Skip to main content

How it works

A virtual store is a table plus a plan. The plan places every source file in one global array; the table holds one row per chunk of every source. Every form is a projection of the two.

Step Writes Does
plan plan.vrt where each file sits in the global array (gdal mdim mosaic, py/plan_cog.py)
harvest chunk index per file offsets and sizes of every chunk, with GDAL or h5py (py/harvest.py)
collate store.sqlite one byterefs/0.1 table per dataset
render flat/, kerchunk/, icechunk/, mdim.vrt, stac/ each form from the table
verify each form against GDAL reading the sources (py/verify.py)
publish upload, then read every form back anonymously (verify-published)

All steps are run.sh <dataset config> [publish target] [steps].

The table

Three files, readable by any Parquet reader:

arrays.json            dims, shapes, chunks, dtype, fill, codec, remap rules, meta
refs_<array>.parquet   dim_0 ... dim_n, present, path, offset, size, info
sources.parquet        source_id, source, size, mtime, etag

One row of OISST sst:

dim_0..3  0 0 0 0      present 1
path      .../avhrr/198109/oisst-avhrr-v02r01.19810901.nc
offset    1025881      size 683590
info      COMPRESSION=DEFLATE; FILTER=SHUFFLE; ENDIANNESS=LITTLE

Specification: byterefs/spec, spec/byterefs-0.1.md. Stores built before 2026-11 say mdim-refs/0.2, the same layout.

One table, several endpoints

Paths are stored as scanned; a local mirror is the fastest place to scan. Remap rules carried with the table give each reader a path it can open: public for https, gdal for /vsicurl/. A rule asserts the bytes are identical at both ends, and verify checks it (byterefs decision 0004).

target  prefix                                        replacement
public  /rdsi/PUBLIC/raad/data/www.ncei.noaa.gov/...  https://noaa-cdr-...s3.amazonaws.com/data/v2.1/avhrr/
gdal    (same)                                        /vsicurl/https://noaa-cdr-...s3.amazonaws.com/...

What verify checks

Per array: the first chunk, one full row across the last dimension, four random present chunks, one absent chunk, plus scale_factor and fill. Raw values must hash the same as GDAL reading the sources through plan.vrt.

File lists and staleness

A build reads its file list from the aad-filelist listing corpus and keeps a record of every file it referenced (sources.meta.tsv). The stale step compares that record with the corpus now: a file replaced at the same URL, or withdrawn, is flagged before its byte offsets mislead a reader.