Skip to main content

GHRSST virtual indexes on source.coop

Layout of the byte-reference indexes (flat refs, kerchunk Parquet, Icechunk, MDIM VRT, STAC) for the GHRSST MUR COGs in the source.coop repository ausantarctic/ghrsst-mur-v2, the drafting procedure in this repository, and the changes to AustralianAntarcticDivision/ghrsst-cogger that would take the procedure over.

Status: draft. Drafts of every index family (GHRSST, bluelink, others) go to the scratch product ausantarctic/ausantarctic-scratch, so that drafting does not touch the production pipeline (decision 2026-11-02). The scratch group ghrsst/ holds one dataset, ghrsst-mur-cog-analysed_sst (full series, products/ghrsst/datasets.tsv); the procedure is under “Full series in the scratch group” below. Credential and product details are from the source.coop documentation (docs.source.coop, read 2026-10-29).

Repository layout

The COGs and the cogger VRT stay where they are. The indexes go under one new top-level prefix, virtual/, one directory per dataset (a dataset is one store and its rendered forms; several can sit side by side).

ausantarctic/ghrsst-mur-v2/
  YYYY/MM/DD/<date>090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1_<var>.tif
  ghrsst-mur-v2.vrt                 cogger VRT (unchanged)
  virtual/
    <DATASET>/
      mdim.vrt                      GDAL multidim VRT (SourceSlab counts)
      kerchunk/                     kerchunk Parquet, fsspec layout (Zarr v2)
        .zmetadata
        <var>/refs.<n>.parq
      icechunk/                     Icechunk repository, branch main
      flat/                         flat refs store
        arrays.json
        sources.parquet
        refs_<var>.parquet
      thumbnail.png                 overview of one field (collection asset)
      stac/
        collection.json             one asset per form above, thumbnail,
                                    items.parquet (geoparquet-items)
        items.ndjson                one item per COG
        items.parquet               stac-geoparquet (rustac)
      plan.vrt                      plan, verbatim source URIs
      store.sqlite                  byterefs/0.1 store

ausantarctic/ausantarctic-scratch/
  catalog.json                      root catalog, one child per group
  ghrsst/
    catalog.json                    group catalog, one child per dataset
    <DATASET>/                      drafts, same layout as above
  bluelink/, oisst/, seaice/        the other groups

Notes on the layout:

  • Every form references the COGs by absolute URL (https://data.source.coop/ausantarctic/ghrsst-mur-v2/YYYY/...), so a dataset directory can be copied between products unchanged. Only the STAC collection holds its own location (STAC_BASE): promotion from the scratch product to ausantarctic/ghrsst-mur-v2/virtual/ is a copy of the directory followed by a re-render of stac/ with the new base.
  • The source listing in config/ghrsst-cog.sh takes only four-digit top-level names, so virtual/ is not mistaken for data. Any listing in ghrsst-cogger that walks the repository root needs the same check (not confirmed; the cogger code was not reviewed for this draft).
  • The scratch product carries a README.md stating that its contents are transient and may be removed without notice. verify-published reads anonymously, so the product must be publicly readable (not checked).
  • The Icechunk repository registers one virtual chunk container, https://data.source.coop/. Readers authorize it with icechunk.credentials.HttpAccess (anonymous).

Dataset names

Current: ghrsst-mur-cog-<var> (one array per store, from config/ghrsst-cog.sh), with a year suffix for subsets.

Target: one dataset holding every array that cogger writes (analysed_sst, analysis_error, mask, sea_ice_fraction, and the others cogger adds), under a name without the variable, such as mur-daily. One Icechunk repository and one kerchunk directory then serve an xarray Dataset with all variables on one time axis. This needs py/plan_cog.py to place several arrays from per-variable files (one --array and pattern per variable), which it does not do yet.

Sizes

From the full analysed_sst series (8909 days, 22,771,404 refs, built on the Nectar VM, 2026-10):

Item Size or time
store.sqlite 1.2 GB
Icechunk 568 MB, 23 commits, 105 s
harvest (GDAL, py/harvest.py) 17 min
harvest (async-tiff via rustycogs, IFD 0 and overviews) 137 s, 30.5M rows
collate 6 min
render (flat, VRT, kerchunk) 5 min

A five-array dataset is roughly five times these sizes, except where arrays are smaller (mask is Byte).

Trial procedure

Scratch product

Drafts go to ausantarctic/ausantarctic-scratch, the default target of config/source-coop.sh, rather than to the production product. The scratch product needs no change to ausantarctic/ghrsst-mur-v2, its credentials cannot write to the production data, and it can be emptied without coordination. The forms reference the COGs in ausantarctic/ghrsst-mur-v2 by absolute URL, so they work from any product. The same product holds bluelink and other drafts, one group prefix per product family (ghrsst/<DATASET>/, bluelink/<DATASET>/, …; the dataset config’s PRODUCT_GROUP, matching products/<group>/), under one root catalog.json.

Below the group prefix the layout is the production one below virtual/ (ghrsst/<DATASET>/... in scratch, virtual/<DATASET>/... in ausantarctic/ghrsst-mur-v2), so promotion is the copy of the dataset directory plus stac/ re-render described above. Another product (a personal one, for example) is a SOURCECOOP_REPO setting away; product ID rules: 3 to 39 characters, letters, digits and single hyphens, not starting or ending with a hyphen.

Credentials

Three ways to get write credentials, all exchanged at the data proxy (https://data.source.coop; any region value is accepted):

Method Use Lifetime
“View Credentials” on the product page one-off runs temporary, expiry shown
source-coop login, then source-coop creds (AWS credential_process) interactive runs one hour, renewed by AWS CLI and SDKs; login every 30 days
service account: API key (sck_...) or GitHub Actions OIDC scheduled jobs (cogger) one hour per exchange; GHA up to 12 h with role-duration-seconds

All three give an AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN. Exported keys are not renewed, so a publish that runs past an hour fails with ExpiredToken. A one-year trial is well under that; a full-series publish goes in parts. Service account keys are exchanged through STS at https://data.source.coop/.sts; GDAL would send them to AWS STS instead, so config/source-coop.sh sets CPL_AWS_WEB_IDENTITY_ENABLE=NO.

A service account is scoped to products its owner owns, with Read or Read and write per product. For cogger, a GitHub Actions trust on the cogger repository and branch avoids storing a secret.

GDAL writes

On 2026-10-29 the proxy accepted writes from Icechunk with the product’s temporary keys but rejected every signed GDAL /vsis3/ request (PUT, multipart initiate, list) with InvalidRequest: missing SignedHeaders. Anonymous GDAL reads work. The likely cause is the proxy’s parsing of the Authorization header that GDAL writes (not confirmed). Until that is resolved, config/source-coop.sh sets PUBLISH_WITH=obstore, and the publish step uploads with py/s3_sync.py (obstore) instead of gdal vsi sync. The image needs pip install obstore.

Write check

py/sourcecoop_check.py writes under <product>/_write-check/<time>/ and removes what it wrote: a small PUT and a 120 MB multipart PUT (obstore), a listing, an anonymous https read, an Icechunk repository created and committed twice directly on the proxy (conditional writes, needed for later in-place commits), a GDAL PUT (informational), and deletes.

# needs the product's AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN exported
python py/sourcecoop_check.py ausantarctic/ausantarctic-scratch

Full series in the scratch group

On a docker host (the Nectar VM), hpc/vm/docker-run.sh runs run.sh and catalog.sh in the gdal-r-python-extras image and installs obstore, imagecodecs and fastparquet into the container. The listing is of the whole bucket, so a rebuild picks up every day cogger has written.

The plan step lists the days with no COG and stops when there are more than GHRSST_MAX_MISSING (default 0): a listing cut short by HTTP 503 from data.source.coop fails there instead of producing a shorter series. The grist check reads a 2048 x 2048 window of each checked field (GRIST_VERIFY_WINDOW) rather than the 1.3 GB field.

Build and local verify (no keys; about 40 minutes, most of it the harvest):

cd ~/Git/raad.virtualize && git pull
nohup bash hpc/vm/docker-run.sh run.sh config/ghrsst-cog.sh config/source-coop.sh \
  deps plan harvest collate render icechunk grist stac verify > ghrsst-build.log 2>&1 < /dev/null &
grep -E '^== |missing|FAIL|DIFFERENT|Error' ghrsst-build.log

The stac and grist steps must run with config/source-coop.sh sourced, so that the collection and the catalog dsn carry the public URL.

Publish (fresh product keys; about 3 GB, store.sqlite 1.2 GB of it). If the token expires part way, export new keys and run again with S3_SYNC_ARGS=--skip-same-size, which leaves out what is already uploaded:

# needs AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN of ausantarctic-scratch exported
cd ~/Git/raad.virtualize
nohup bash hpc/vm/docker-run.sh run.sh config/ghrsst-cog.sh config/source-coop.sh publish \
  > ghrsst-publish.log 2>&1 < /dev/null &
grep -E '^== |s3_sync: [0-9]+ files|Error|Expired' ghrsst-publish.log

Read back anonymously, then the group and root catalogs (keys needed for the two publish steps only):

cd ~/Git/raad.virtualize
bash hpc/vm/docker-run.sh run.sh config/ghrsst-cog.sh config/source-coop.sh verify-published 2>&1 | tee ghrsst-verify.log
bash hpc/vm/docker-run.sh catalog.sh products/ghrsst config/source-coop.sh check publish
bash hpc/vm/docker-run.sh catalog.sh products config/source-coop.sh check publish

verify-published counts the GDAL kerchunk reads as a known limit (imagecodecs_delta); the kerchunk form is checked with fsspec and imagecodecs instead.

One-year trial

config/source-coop.sh is the publish target. SOURCECOOP_REPO names the product (default ausantarctic/ausantarctic-scratch; any other ausantarctic/ product is refused unless SOURCECOOP_PRODUCTION=1), SOURCECOOP_PREFIX defaults to the config’s PRODUCT_GROUP (ghrsst) in the scratch product and to virtual elsewhere, and STAC_BASE is set to the public URL of the dataset directory.

export GHRSST_YEARS=2024
# build and verify locally
bash run.sh config/ghrsst-cog.sh config/source-coop.sh
# publish (product credentials exported)
bash run.sh config/ghrsst-cog.sh config/source-coop.sh publish
# read back anonymously, as a public reader
env -u AWS_ACCESS_KEY_ID -u AWS_SECRET_ACCESS_KEY -u AWS_SESSION_TOKEN AWS_NO_SIGN_REQUEST=YES \
  bash run.sh config/ghrsst-cog.sh config/source-coop.sh verify-published

The stac step must run with config/source-coop.sh sourced, so that the collection carries the public base. Because the forms are rendered before publish, a local build of the same dataset can be published to the AADC transfer bucket and to source.coop without a rebuild, apart from stac/.

Checks before promotion to production

  1. Completeness: the harvested source count equals the listed COG count, and every source has its full tile grid (an earlier full run lost 19 sources to HTTP 503 from data.source.coop without failing). The build stops on a shortfall rather than publishing a shorter series.
  2. Overviews: the contract states whether overview IFDs are indexed. The current stores index IFD 0 only (22.77M refs); the async-tiff harvest returns overviews too (30.5M rows). The scratch stores index every level as <var>_x<f> (GHRSST_OVERVIEWS, default 1, py/plan_cog.py –overviews; resampling unknown, set in cogger) until the team agrees to fold the indexes into production.
  3. Anonymous read-back passes for VRT and Icechunk (kerchunk via GDAL stays a known limit until GDAL Zarr reads the TIFF predictor filter).
  4. The async-tiff harvest is the harvest step for COG sources, with the GDAL harvest kept as the cross-check.

Update cadence

MUR rewrites its newest days for several days after first publication, and a full rescan takes minutes, so each run rebuilds the dataset from a full listing. What changes between runs:

  • flat refs, kerchunk, VRT, STAC: rewritten whole. The kerchunk record files are ordered with time first, so only the last records differ. The publish step syncs by timestamp, so every rewritten file is uploaded; a content (ETag) sync strategy would upload only the changed ones.
  • Icechunk: a full rebuild produces a new repository with no history. Commit continuity (one commit per new or changed day on an existing repository) is a later step and needs --split 1 on the time axis so that a day’s commit rewrites one manifest.
  • store.sqlite (1.2 GB per array): uploading it each day is the largest cost. Open: publish it only on a slower cadence, or not at all and keep the flat refs as the published contract.

ghrsst-cogger update plan

cogger already writes the COGs, holds the source.coop credential and knows when a new day has landed, so it owns the trigger; this repository owns the engine and the store contract.

  1. Pin the engine: cogger calls run.sh from a tagged release of this repository (or its successor) in the gdal-r-python-extras image, with config/ghrsst-cog.sh (multi-array form) and config/source-coop.sh.
  2. Trigger: after cogger’s daily COG write succeeds, a job runs plan, harvest, collate, render, icechunk, stac, verify, then publish and verify-published to virtual/ (SOURCECOOP_PRODUCTION=1).
  3. Runner: a GitHub Actions job in cogger (standard runner: 4 CPU, 16 GB, about 14 GB of disk) fits a full rebuild of one array (about 15 min with the async-tiff harvest); five arrays need the work directory on a larger runner or a per-array split. AWS Lambda does not fit (15 min limit, 10 GB of /tmp). Pawsey (Setonix, Slurm) is the interim runner.
  4. Failure policy: verify or completeness failure stops before publish and leaves the previous virtual/ contents in place.
  5. Provenance: the store meta records the engine commit and config; the STAC collection repeats it (rv:meta).

Open questions

  • Prefix name virtual/ and dataset name mur-daily.
  • Whether store.sqlite is published (see Update cadence).
  • Whether the cogger VRT ghrsst-mur-v2.vrt is replaced by virtual/<DATASET>/mdim.vrt or kept alongside.
  • Who at source.coop needs to know about the new prefix (the operators maintain the repository listing and URL rewrites).