Skip to main content

Array data read where it lives

Virtual stores for Antarctic and Southern Ocean arrays, ausantarctic on source.coop

Each dataset’s chunk index is read once into a table of byte references: which bytes of which file hold each chunk. Every published form (GDAL multidimensional VRT, kerchunk, Icechunk, STAC) is rendered from that one table and checked against GDAL reading the original files. No data is copied to make an index.

22,771,404byte references in one store: GHRSST MUR, 8,909 COGs, 2002 to 2026
27datasets indexed: 20 drafts, 7 in production
4 formsfrom one table, each verified against GDAL on the sources
3 sitesNectar, NCI Gadi and Pawsey build where the bytes are fastest to read
UPSTREAM PUBLISHERS BYTES WE HOST WHERE HARVEST RUNS PUBLISHED INDEX NOAA public S3 (OISST) NSIDC https (G02202) Uni Bremen (AMSR2 ASI) NCI THREDDS gb6 (BRAN) PO.DAAC (MUR netCDF) oceancolour.org (ESA CCI) PANGAEA, ibcso.org (IBCSO) UK PDC (Bedmap3) BODC / CEDA (GEBCO) PGC bucket (REMA, listed only) source.coop ausantarctic/ ghrsst-mur-v2/ 4.0 TiB ibcso/<release>/cog/ bedmap/bedmap3/cog/ gebco/GEBCO_<year>.tif Pawsey Acacia aad-derived/esacci-oc/ Nectar VM (docker) OISST, sea ice, GHRSST, IBCSO, Bedmap3, GEBCO NCI Gadi (PBS) BRAN, scanned in place on /g/data/gb6 Pawsey Setonix (Slurm) ESA CCI: convert to COG, write Acacia, harvest scratch (unlisted) ausantarctic-scratch/ bluelink/ ghrsst/ oisst/ seaice/ oceancolour/ 20 datasets production ibcso/virtual/ bedmap/virtual/ gebco/virtual/ COG scan in place (theirs) publish aad-filelist listing corpus lists every source weekly, ours included; keeps every run lists file list per build bowerbird-byterefs central refs + stale check collect tables every reference in an index points back at the bytes, wherever they are (decision 0004 remap rules give the public URL) theirs: publisher hosts ours: COGs we host runs today file listing proposed references point back to bytes
Our COGs are made by ghrsst-cogger (MUR), VM scripts (IBCSO, Bedmap3, GEBCO) and Setonix jobs (ESA CCI). raad.virtualize is the engine on all three sites: one checkout per machine, same run.sh.

Datasets

Every dataset, where its bytes are, and the URL of each form.

How it works

Plan, harvest, collate, render, verify; the table in brief.

Readers

Which reader opens which form, by codec.

Status 2026-11-05: drafts in ausantarctic/ausantarctic-scratch (unlisted); IBCSO, Bedmap3 and GEBCO in production. Moving to production with the AAD cloud team.