Related
This repository is the engine. The others hold what it reads, what it follows, and what reads it.
| Repository | Why it exists |
|---|---|
| mdsumner/aad-filelist | Listing corpus: every source listed on a schedule, every run kept. Builds read their file lists here; a file replaced at the same URL shows up as a change. |
| mdsumner/bowerbird-byterefs | Will collect every dataset’s table in one place and run the stale check against the corpus. |
| byterefs/spec | The table format, so other readers (wxy-refs, arrir, DuckDB) can use it without this code. |
| AustralianAntarcticDivision/ghrsst-cogger | Writes the GHRSST MUR COGs indexed here (ausantarctic/ghrsst-mur-v2). |
| hypertidy/grist | Reads the published catalogs from R, one row per date and array. |
| hypertidy/vrtstack, hypertidy/blocklist | COG collections as multidim VRT; the kerchunk writer vendored here. |
Elsewhere
Earthmover’s icechunk-datacube-ingestion agent skill is a procedure for building one Icechunk repository from many files with VirtualiZarr, and the GOES-16 ingest behind it is the largest worked example (7.1 billion refs). Its practices hold here too: commit in batches, split manifests past about a million refs (along the append axis about as often as you commit), load only coordinate variables, and keep the write-time cache small. py/render_icechunk.py follows them, rendering Icechunk from the table rather than from the files, and with --relocatable it writes remap roots as Icechunk named containers, so one store reads from THREDDS or in place on /g/data (reading in place). One exception: the skill calls kerchunk deprecated and steers away from it. We keep writing kerchunk (JSON and Parquet), because GDAL and other readers outside Python need it today; Icechunk is one output among several.