Pipeline results and findings
Runs of run.sh between 2026-10-20 and 2026-10-23. “Identical” means the raw (unscaled) values of every checked window hash the same as GDAL reading the sources through the plan VRT. verify checks the first chunk, one full row across the last dimension, four random present chunks and one absent chunk per array, plus scale_factor and fill value.
Runs
| Run | Result |
|---|---|
| OISST, 30 files, 4 arrays (NOAA S3) | VRT, kerchunk and Icechunk identical. scale_factor and fill exact in every form. xarray opens the Icechunk store and decodes time. |
| Synthetic BRAN, 2 files, 168 chunks, 28 never written | VRT and Icechunk identical, including absent chunks. kerchunk identical except absent chunks (finding 3). |
wxy-refs (wxylib core 1d5f349) on out/flat |
wxy warp --remap public identical to GDAL on OISST and synthetic BRAN. |
| Scale test, 1,116,900 fake refs | collate 11 s; render 15 s; Icechunk 8 s. Store 87 MB, flat 6 MB, kerchunk 14 MB, Icechunk 22 MB (about 20 B per ref). |
BRAN2023 temp 2010 over THREDDS |
1,116,900 refs. collate 12 s, render 16 s, Icechunk 5 s. VRT and kerchunk identical; Icechunk identical once read concurrency is lowered (finding 6). Published to the AADC transfer bucket; GDAL opens the VRT from there. |
BRAN2023 salt Jan-Feb 2010 over THREDDS |
180,540 refs. All forms identical, locally and read back from the bucket. |
GHRSST MUR analysed_sst COGs 2024 (source.coop) |
935,496 refs. harvest 47 s (8 workers), collate 14 s, render 12 s, Icechunk 4 s (18.4 MB). Published; VRT and Icechunk identical read back from the bucket. kerchunk not read by GDAL (finding 10). |
GHRSST MUR analysed_sst COGs 2002-2026 |
22,771,404 refs from 8,909 files. harvest 1,045 s, collate 385 s (1.2 GB store). Render first stopped on the arrow string limit (finding 11); after the fix, render 282 s (flat 136 s, kerchunk 139 s), Icechunk 105 s, 568 MB in 23 commits. |
| STAC render, synthetic BRAN (2 netCDF) and synthetic MUR COGs (3 days, one missing) | Collection and items validate against STAC 1.1.0 core, datacube 2.2.0, xarray-assets 1.0.0, file 2.1.0 and projection 2.0.0 schemas. From the collection: the Icechunk asset opened with its xarray:open_kwargs and rv:virtual_chunk_containers, and the VRT asset, both match the item’s source file. GDAL STACIT opens the items; GDAL GTI opens items.parquet (rustac stac-geoparquet) directly; gdalcubes builds a daily cube from the items with the missing day empty. |
| terra on the rendered VRT | terra::rast("mdim.vrt", md = TRUE) reads BRAN-shaped VRTs with time and depth decoded. Before this change the COG VRT had no time units (the plan’s <Unit> never reached the store), so terra, xarray and STAC saw raw day numbers; collate now takes coordinate units from the plan where the harvest has none. terra still guesses the CRS: the store holds no SRS (finding 13). |
| STAC render, 8,909 items (GHRSST-shaped store with synthetic placements) | 44 s in R (JSON serialization is most of it); items.ndjson 7.7 MB, items/ 8,909 files, collection 1.1 MB. |
| Publish to S3 | Tested against a local S3 mock (moto) on OISST and synthetic BRAN, then against the AADC transfer bucket. |
Findings
Editor’s note: here 1. “gdal getrawblockinfo is wrong” the llm was using 3.13 and not 3.14 and I hadn’t primed it up for an assumption of my own MDS. All this is very early first written from interesting experiments. GDAL gained supremely fast access to the HDF5 byte refs extractor recently.
- GDAL
GetRawBlockInfois wrong for netCDF4/HDF5 files with unwritten chunks, and slow. GDAL 3.13 turns block coordinates into a linear index forH5Dget_chunk_info(index), which HDF5 counts over allocated chunks only, so one unwritten chunk shifts every later answer. GDAL master (0896c69ac3) fills all blocks withH5Dchunk_iterwhen block 0 is asked first; the fallback path is unchanged. Reported upstream. The harvest detects sparse files and rescans them with h5py;--engine autouses h5py for local files and GDAL for remote ones. - THREDDS cost of a harvest. One BRAN month over THREDDS: GDAL 3.13 about 70 s, 1.7k GETs, 35 MB; GDAL 3.14dev 39 s, 1.7k GETs, 33.5 MB; h5py with 64 KB fsspec blocks 53 s, 1.8k GETs, 115 MB; with 1 MB blocks over 500 MB. The chunk index itself is about 33 MB per file and the time is round-trip bound (about 23 ms per GET), so parallel files are the lever.
py/probe_harvest.py SOURCEmeasures this for any source. - GDAL
/vsikerchunk_parquet_ref/fails on absent chunks. A row with null path and null raw (fsspec’s “no such chunk”) is opened as a path. verify labels it rather than failing. Does not affect fully allocated datasets such as BRAN. - R JSON writers lose precision. jsonlite’s default
digits = 4in blocklist’s.zmetadatawriter is the cause of BRAN’sscale_factor0.0078 in earlier indexes. The copies here usedigits = I(17); the upstream one-line fixes are in blocklist and byteref. - Packing is not checked by
gdal mdim mosaic. The mosaic checks dtype, nodata and dimensions across sources but takes scale and offset from the first. The harvest records packing per source and collate stops if it differs. With the HDF5 driver (no netCDF driver),scale_factoris not lifted to GetScale and the mosaic VRT keeps it as text at reduced precision; verify reads its reference value from the first source. - THREDDS resets concurrent connections. An Icechunk read across 12 chunks fails with “Connection reset by peer” at zarr’s default
async.concurrencyof 10 and succeeds at 4. GDAL reuses connections and does not trip it. verify steps concurrency down automatically. - Reader gaps. GDAL’s Zarr v3 reader has no
numcodecs.shuffle, so GDAL skips shuffled arrays in the Icechunk store. xarray decodes BRAN to float64 from the Zarr stores because JSON attributes lose the float32 type ofscale_factor(a gap in the store’s attribute record). gdal mdim mosaicrejects a 2-step time axis as irregularly spaced.wxy pointsandwxy zonalhave no--remap, unlikewxy warp.- TIFF predictor has no portable Zarr codec.
PREDICTOR=2is horizontal differencing restarted on every row. numcodecsdeltaruns over the flattened chunk and decodes the first value of every row wrongly, so the forms useimagecodecs_delta(axis -1), as tifffile writes it. Readers of the kerchunk and Icechunk forms need imagecodecs; GDAL’s Zarr driver reads neither form. The VRT reads the TIFFs directly. - Arrow string arrays hold at most 2 GB. The path column of the full GHRSST series is 3.3 GB; the flat store writer now builds columns in chunks of 1M rows.
- GDAL reads COG tiles serially through the VRT. One full-width row of one GHRSST day (71 tiles over https) took 252 s through the plan and 166 s through the rendered VRT; Icechunk read it in 6 s. Probably item 15 rather than the tile fetch: 366 source opens at 0.5-1 s each covers the time. Not yet re-measured with the counted form.
- The store holds no SRS. The plan VRT carries it (
<SRS>on the data array) but collate drops it, so the rendered VRT, kerchunk and Icechunk forms are CRS-less and terra guesses lon/lat. The STAC render reads the SRS from the plan beside the store. Recording it inarrayswould fix every form. - Stores must state what readers would otherwise guess. A browser explorer over the two BRAN2023 kerchunk stores in mdsumner/virtualized found every guess was about meaning, not layout: fill values that disagree (
fill_value-32767 against_FillValue-32768 in the VirtualiZarr build), rounded packing (item 4), bareNaNin JSON, and dimension roles. Collate now stops when_FillValueormissing_valuedisagree with nodata; harvest writes strict JSON; the plan’s dimensiontypeanddirectionare kept inmeta.dimensionsand written back by the VRT render;arrays.jsoncarriesdimensionsand a per-arraysummary(grid chunks, present, absent, referenced bytes). - GDAL’s multidim VRT opens every source whose extent it does not know. A one-chunk read of the full GHRSST plan (8909 daily COGs over /vsicurl/) opened the sources in date order, a HEAD and a 16 KB header GET each, at about 1 s per source: hours per window, though only one source holds the chunk. A
<SourceBand>source with no<SourceSlab>count is opened to learn its shape; with<SourceSlab offset="0,0,0" count="1,ny,nx" step="1,1,1"/>beside the<SourceTranspose>, a read opens only the sources it touches (69 synthetic COGs: 3 opens for a 3-day window, against 91 without).AdviseRead()on the array does not help: it opens every source too. plan_cog.py and the VRT render now write the count; plans and VRTs built earlier lack it, so verify also reads each window through a copy of the VRT holding only the touched sources (arrays with 64 or more sources). - Sources change after harvest. GHRSST MUR rewrites its newest days for several days after first publication: of the last 10 days in the full-series store, 5 had a new size and mtime on source.coop three days after the harvest, and their refs failed to decode (“Zstd decompression error”) while GDAL on the plan read the new files. Nothing in a byte-ref form detects this. The harvest records size and mtime per source;
harvest.py --recheck N(HARVEST_RECHECKin run.sh) stats the last N sources again and rescans those that changed. A published store goes stale the same way, so an index of a live collection needs the recheck on every update, and a reader cannot tell without a validator (an etag or size check before trusting the refs). - fastparquet drops kerchunk paths after a dictionary fallback. arrow writes the
pathcolumn ofrefs.<n>.parqdictionary-encoded and falls back to plain encoding once the dictionary passes 1 MiB (about 8,000 distinct URLs). fastparquet 2026.5.0+ under pandas 3, fsspec’s default engine for kerchunk Parquet, returns null for every path after the fallback, and xarray reads fill values with no error: OISSTsstis all NaN from time index 8192, sea ice G02202 from index 11264. Stores with many chunks per source file (bluelink, GHRSST, ASI) stay under the limit. fastparquet 2026.3.0 reads these files correctly; the regression is its pandas 3 arrow string path. Reported by Michael Sumner from raadtools. The kerchunk writer now setsuse_dictionary = FALSE; verify compares fastparquet with pyarrow on every refs file (py/check_kerchunk_parquet.py). Readers of older stores passengine="pyarrow"toLazyReferenceMapper(Reading published stores).
Legs
| Leg | Lang | Caveats |
|---|---|---|
| plan | GDAL CLI, Python | Sources passed as a response file (@sources.txt). GeoTIFF sources use py/plan_cog.py, since the mosaic does not take GeoTIFF. |
| harvest | Python | No R producer yet; the gdalraster fork getRawBlockRefs() would write the same fragments. |
| collate | R | refs_<array> is a view with no index; materialise it beyond a few years of BRAN. |
| flat | R | Paths verbatim; remap rules travel in arrays.json. Columns built in chunks of 1M rows. |
| MDIM VRT | R | Loses dimension type and direction, which the store does not hold. |
| kerchunk | R | Builds the whole positional grid in memory. |
| Icechunk | Python | One virtual chunk container per host. Manifests split along the first axis (--split, default 365) and the load commits every --commit-every refs (default 1M): the GHRSST MUR series, 22.8M refs, builds in 255 s at 4.7 GB peak (568 MB store). |
| STAC | R | Item time range and bbox from placements and coordinates; CRS and dimension types from the plan VRT (the store holds neither). Media types for the kerchunk and Icechunk assets follow no settled convention (application/vnd+zarr, application/vnd.zarr+icechunk). |
| verify | Python | --published reads the forms from object storage. |