4  A virtual store: BRAN on THREDDS

Most of the world’s gridded data is not Zarr and will not be rewritten as Zarr. It is NetCDF and HDF5, in archives, served as files. A virtual Zarr store leaves those files alone and keeps a table of references: for each chunk, which file, which byte offset, how many bytes. Kerchunk and VirtualiZarr write such tables; zaro reads them (kerchunk JSON and Parquet, VirtualiZarr Parquet manifests, byterefs tables) and from then on the store behaves like any other.

BRAN is CSIRO’s Bluelink ocean reanalysis: daily fields at 51 depths on a 0.1 degree global grid, as monthly NetCDF files on NCI’s THREDDS server (BRAN2020 to 2009, BRAN2023 from 2010). Reference stores for it, one per variable and each in several forms, are published on Source Cooperative, built with raad.virtualize. Temperature is one array of 3600 x 1500 x 51 x 12265 values, every day from 1993, in chunks of 300 x 300 x 1 x 1: each chunk is a tile of one depth on one day, a few tens of kilobytes inside a monthly NetCDF file. That is 37.5 million chunks, and the table that locates them is itself read lazily: it is 376 Parquet files of 100,000 references each, and only the files a read touches are fetched. This chapter reads the kerchunk form of the store; Chapter 6 reads the same references in another form.

bran <- zaro("virtualizarr://https://data.source.coop/ausantarctic/ausantarctic-scratch/bluelink/bluelink-daily-temp/kerchunk")
temp <- zaro_array(bran, "temp", cf = TRUE)    # degrees C
dim(temp)
#> [1]  3600  1500    51 12265

No NetCDF library is involved, and nothing is copied. (zaro calls this layout virtualizarr:// because VirtualiZarr writes it too; it is kerchunk’s Parquet format.)

4.1 The reads

A surface map for one day (60 chunks, all in one file); a profile at one cell (51 chunks, one per depth); a north-south section by depth; and a month of surface values at one cell.

# 04-virtual.R -- BRAN ocean temperature through a virtual store.
# ASCII only.
#
# The data are monthly NetCDF files on NCI's THREDDS server. A kerchunk
# Parquet reference store on Source Cooperative (built by raad.virtualize)
# says where each chunk's bytes are; zaro reads those byte ranges in
# place. temp is (xt_ocean 3600, yt_ocean 1500, st_ocean 51, Time 12265:
# daily from 1993), int16, chunks of 300 x 300 x 1 x 1.
# THREDDS accepts about 4 concurrent requests: BOOK_THREDDS_CON (default 4).
here <- function() {
  f <- sub("^--file=", "", grep("^--file=", commandArgs(FALSE), value = TRUE))
  if (length(f)) dirname(f[1]) else "scripts"
}
source(file.path(here(), "_bench.R"))
suppressPackageStartupMessages(library(zaro))
options(zaro.verbose = FALSE,
        zaro.max_con = as.integer(Sys.getenv("BOOK_THREDDS_CON", "4")))
count_http()

bran <- zaro(paste0("virtualizarr://https://data.source.coop/ausantarctic/",
                    "ausantarctic-scratch/bluelink/bluelink-daily-temp/kerchunk"))
ds <- zaro_dataset(bran)
temp <- zaro_array(ds, "temp", cf = TRUE)       # degrees C, land as NA
cat("temp:", paste(dim(temp), collapse = " x "), "\n")

xt <- as.vector(zaro_read(ds, "xt_ocean"))
yt <- as.vector(zaro_read(ds, "yt_ocean"))
i <- which.min(abs(xt - 150))                   # east of Tasmania
j <- which.min(abs(yt - -42))
jj <- which(yt >= -50 & yt <= -40)
nt <- dim(temp)[4]

bench("surface map, one day: temp[, , 1, day]", temp, temp[, , 1, nt])
bench("profile, 51 depths at one cell: temp[i, j, , day]", temp, temp[i, j, , nt])
bench("section, 40S to 50S by depth: temp[i, jj, , day]", temp, temp[i, jj, , nt])
bench("surface series, 30 days at one cell: temp[i, j, 1, days]", temp,
      temp[i, j, 1, (nt - 29):nt])
finish()
2026-11-22 04:38 UTC on a5af98a4c685; R-devel patched r90638, stock r90638
Read Values Chunks MB Fetch calls, stock Fetch calls, patched Requests, stock Requests, patched Seconds, stock Seconds, patched Faster by Same values
surface map, one day: temp[, , 1, day] 5,400,000 60 3.5 60 1 60 2 18.20 8.75 2.1x yes
profile, 51 depths at one cell: temp[i, j, , day] 51 51 1.5 51 1 51 51 11.46 3.67 3.1x yes
section, 40S to 50S by depth: temp[i, jj, , day] 5,100 102 4.4 102 1 102 102 22.96 7.08 3.2x yes
surface series, 30 days at one cell: temp[i, j, 1, days] 30 30 1.2 30 1 30 30 7.96 2.77 2.9x yes

This store shows something the native Zarr store could not. When zaro is given a whole batch, it looks at the byte ranges, and ranges that touch in the same file are merged into one request. The 60 tiles of a surface map sit end to end in the NetCDF file, so a planned read of the map can be a single HTTP request. Asked one chunk at a time, there is nothing to merge: 60 requests to a THREDDS server, in series. (Reads from THREDDS in this book are capped at 4 concurrent requests; the server resets connections not far above that.)

So here the Requests column can differ between builds, while MB stays the same. Fewer, larger requests are kinder to an archive server than many small ones, as well as faster for the reader. The patch is what lets R be the polite client.

Read Element calls, stock Element calls, patched
surface map, one day: temp[, , 1, day] 5,400,000 0
profile, 51 depths at one cell: temp[i, j, , day] 51 0
section, 40S to 50S by depth: temp[i, jj, , day] 5,100 0
surface series, 30 days at one cell: temp[i, j, 1, days] 30 0
Note

If the stock column says “failed”, that is a result too, and for a while it was the only one. On the first renders of this chapter (GitHub runners and Pawsey, 2026-11-21 and 22) the patched build completed every read and the stock build completed none: each attempt ended, after about 20 requests, with the server resetting the connection. The cause was the rate of new connections. On stock R every chunk is its own fetch call, and zaro then opened a new connection for each call, so 60 small chunks were 60 TLS handshakes in quick succession. zaro now keeps its connections open between fetch calls, and with that the stock build gets through, and is faster in every chapter. The comparison here is the fairer for it: what remains between the columns is what asking once is worth, not the cost of a handshake per chunk.

4.2 A track is already fine

One read from this store needs no patch: values along a track, one (x, y, depth, day) per row, is a matrix subscript (Chapter 1). A 30-day track off Tasmania was 30 byte ranges from 30 files, one fetch call, 2 MB in half a second (measured 2026-10-03). The profile and the section above are the same data asked for as rectangles, and that is the only reason they fare differently.