Obtain object storage catalogues as a dataframe of file/object identifiers.

The object (file) catalogue of available sources is stored in Parquet format on Pawsey object storage. This function retrieves the curated catalogue.

sooty_files(curated = TRUE)

Arguments

curated

logical TRUE by default, ignored with a warning if FALSE

Value

a data frame, see details

Details

The returned curated data frame has columns 'date', 'source' which are the main useful fields, these describe the date of the data in the file, and its full URI (Uniform Resource Identifier) source on S3 object storage. There are also fields 'Bucket', 'Key', and 'Protocol' from which 'source' is constructed.

The original publisher URI can be reconstructed by replacing the value of 'Protocol' in 'source' with 'https://'.

The public object URI can be reconstructed by replacing the value of 'Protocol' in 'source' with 'https://projects.pawsey.org.au'.

By default sooty maintains a local cache of the catalogue, refreshed once per session when internet is available. Set options("sooty.allow.cache" = FALSE) to suppress all disk I/O and use only the bundled sysdata, or options("sooty.cache.path" = tempdir()) to redirect the cache directory. See sooty_cache_info() for details.

Examples

op <- options("sooty.allow.cache" = FALSE)
sooty_files()
#> # A tibble: 232,798 × 7
#>    date                source                Dataset Bucket Key   Protocol Host 
#>    <dttm>              <chr>                 <chr>   <chr>  <chr> <chr>    <chr>
#>  1 1981-09-01 00:00:00 /vsicurl/https://pro… oisst-… idea-… www.… /vsicurl http…
#>  2 1981-09-02 00:00:00 /vsicurl/https://pro… oisst-… idea-… www.… /vsicurl http…
#>  3 1981-09-03 00:00:00 /vsicurl/https://pro… oisst-… idea-… www.… /vsicurl http…
#>  4 1981-09-04 00:00:00 /vsicurl/https://pro… oisst-… idea-… www.… /vsicurl http…
#>  5 1981-09-05 00:00:00 /vsicurl/https://pro… oisst-… idea-… www.… /vsicurl http…
#>  6 1981-09-06 00:00:00 /vsicurl/https://pro… oisst-… idea-… www.… /vsicurl http…
#>  7 1981-09-07 00:00:00 /vsicurl/https://pro… oisst-… idea-… www.… /vsicurl http…
#>  8 1981-09-08 00:00:00 /vsicurl/https://pro… oisst-… idea-… www.… /vsicurl http…
#>  9 1981-09-09 00:00:00 /vsicurl/https://pro… oisst-… idea-… www.… /vsicurl http…
#> 10 1981-09-10 00:00:00 /vsicurl/https://pro… oisst-… idea-… www.… /vsicurl http…
#> # ℹ 232,788 more rows
options(op)