# 06-icechunk.R -- seal dives and the ocean model under them, both from
# virtual stores on Source Cooperative. ASCII only.
#
# meop-ctd-2024: IMOS seal CTD profiles (MEOP quality control), one cube of
# 126,518 profiles by 1000 levels (1 dbar). Each profile is one virtual
# chunk: 4000 bytes of a classic netCDF file in the IMOS public bucket.
# bluelink-daily-temp: BRAN temperature, daily from 1993, (xt_ocean,
# yt_ocean, st_ocean 51, Time) in chunks of 300 x 300 x 1 x 1. Virtual
# chunks: byte ranges of netCDF files on NCI's THREDDS server, so a model
# profile under one dive is 51 chunks. Read here through the flat
# Parquet form of the same references (see the chapter for why).
#
# THREDDS accepts about 4 concurrent requests: BOOK_THREDDS_CON (default 4).
here <- function() {
f <- sub("^--file=", "", grep("^--file=", commandArgs(FALSE), value = TRUE))
if (length(f)) dirname(f[1]) else "scripts"
}
source(file.path(here(), "_bench.R"))
suppressPackageStartupMessages(library(zaro))
options(zaro.verbose = FALSE,
zaro.max_con = as.integer(Sys.getenv("BOOK_THREDDS_CON", "4")))
if (!requireNamespace("tabularberg", quietly = TRUE)) {
cat("tabularberg is not installed in this R; nothing run\n")
quit(status = 0)
}
count_http()
coop <- "https://data.source.coop/ausantarctic/ausantarctic-scratch/"
## the seals: an Icechunk repository, pinned to the snapshot it opens at
ice <- zaro_icechunk(paste0(coop, "imos-tracking/meop-ctd-2024/icechunk"),
authorize = "https://", direct = TRUE)
cat("meop-ctd-2024 snapshot:", ice@snapshot, "\n")
meop <- zaro_dataset(ice)
obs_t <- zaro_array(meop, "TEMP_ADJUSTED", cf = TRUE) # N_LEVELS, N_PROF
obs_p <- zaro_array(meop, "PRES_ADJUSTED", cf = TRUE)
cat("TEMP_ADJUSTED:", paste(dim(obs_t), collapse = " x "), "\n")
juld <- as.vector(zaro_read(meop, "JULD")) # days since 1950-01-01
plat <- as.vector(zaro_read(meop, "LATITUDE"))
plon <- as.vector(zaro_read(meop, "LONGITUDE"))
tag <- as.vector(zaro_read(meop, "tag"))
ok <- is.finite(juld) & is.finite(plat) & is.finite(plon) &
juld > 25202 & juld < 25567 & # the year 2019
plat > -60 & plat < -45 & plon > 60 & plon < 90 # Kerguelen plateau
stopifnot(any(ok))
one <- which(ok & tag == tag[which(ok)[1]]) # one animal
sel <- one[unique(round(seq(1, length(one), length.out = 5)))]
dives <- data.frame(profile = sel, lon = plon[sel], lat = plat[sel],
date = as.Date("1950-01-01") + floor(juld[sel]))
print(dives)
## the model
bran <- zaro_dataset(zaro(paste0("flatrefs://", coop,
"bluelink/bluelink-daily-temp/flat"),
target = "public"))
temp <- zaro_array(bran, "temp", cf = TRUE) # degrees C, land as NA
cat("temp:", paste(dim(temp), collapse = " x "), "\n")
xt <- as.vector(zaro_read(bran, "xt_ocean"))
yt <- as.vector(zaro_read(bran, "yt_ocean"))
zt <- as.vector(zaro_read(bran, "st_ocean"))
tunits <- bran$arrays$Time@attributes$units
days <- as.Date(sub("^days since ([0-9-]+).*", "\\1", tunits)) +
as.vector(zaro_read(bran, "Time"))
wrap <- function(v) ((v + 180) %% 360) - 180
i <- vapply(dives$lon, function(v) which.min(abs(wrap(xt - v))), 1L)
j <- vapply(dives$lat, function(v) which.min(abs(yt - v)), 1L)
k <- vapply(as.numeric(dives$date), function(v) which.min(abs(as.numeric(days) - v)), 1L)
nz <- dim(temp)[3]
nd <- length(sel)
seal <- bench("seal: 5 dives of one animal, TEMP_ADJUSTED[, dives]", obs_t, obs_t[, sel])
a <- bench("model: the profile under each dive, a loop of temp[i, j, , t]", temp,
vapply(seq_len(nd), function(p) temp[i[p], j[p], , k[p]], numeric(nz)))
idx <- cbind(rep(i, each = nz), rep(j, each = nz), rep(seq_len(nz), nd), rep(k, each = nz))
b <- bench("model: the same values as one matrix subscript, temp[idx]", temp, temp[idx])
if (!is.null(a) && !is.null(b)) {
stopifnot(isTRUE(all.equal(as.vector(a), b)), any(is.finite(a)))
}
## both, for the figure in the chapter
if (dir.exists("results") && !is.null(seal) && !is.null(a)) {
pres <- obs_p[, sel]
out <- rbind(
data.frame(dive = rep(seq_len(nd), each = nrow(seal)), source = "seal",
depth = as.vector(pres), temp = as.vector(seal)),
data.frame(dive = rep(seq_len(nd), each = nz), source = "model",
depth = rep(zt, nd), temp = as.vector(a)))
out <- out[is.finite(out$depth) & is.finite(out$temp), ]
out$label <- sprintf("%s %.1fE %.1fS", dives$date, dives$lon, -dives$lat)[out$dive]
utils::write.csv(out, "results/06-dives.csv", row.names = FALSE)
}
finish()6 Icechunk: a versioned store
Icechunk puts a transaction log in front of Zarr: snapshots, branches and tags, so a reader can pin a version and keep reading it while writers move on. Chunks are found through the snapshot’s manifest, not by key, and a chunk can be native (stored in the repository), inline, or virtual (a byte range of a file somewhere else).
In R, tabularberg wraps the Icechunk Rust library, and zaro opens a repository as a store.
6.1 A seal, and the ocean model under it
Two virtual stores on Source Cooperative, both built with raad.virtualize and published as Icechunk repositories among other forms. Neither holds a copy of the data.
- IMOS seal CTD profiles (
imos-tracking/meop-ctd-2024). Elephant seals, Weddell seals and fur seals carry satellite-relay CTD tags, and each dive is a temperature and salinity profile at one place and time. The 2024 release is 126,518 profiles of 1000 levels, in 265 classic netCDF files in the IMOS public bucket. The repository presents them as one cube, each profile a chunk: 4000 bytes of one of those files. - Bluelink (BRAN) temperature (
bluelink/bluelink-daily-temp). The reanalysis of Chapter 4, daily from 1993, every chunk a byte range of a netCDF file on NCI’s THREDDS server.
coop <- "https://data.source.coop/ausantarctic/ausantarctic-scratch/"
meop <- zaro_icechunk(paste0(coop, "imos-tracking/meop-ctd-2024/icechunk"),
authorize = "https://", direct = TRUE)
meop@snapshot # the version this store is pinned to
obs <- zaro_array(meop, "TEMP_ADJUSTED", cf = TRUE) # level, profile
bran <- zaro(paste0("flatrefs://", coop, "bluelink/bluelink-daily-temp/flat"),
target = "public")
temp <- zaro_array(bran, "temp", cf = TRUE) # x, y, depth, dayThe model is not read through Icechunk here yet. zaro’s Icechunk store loads the whole reference table of an array before it looks up a chunk. For the seal cube that is 126,518 rows. For Bluelink temperature it is 37.5 million, and the first render of this chapter ran out of memory there, in both builds. Each Bluelink dataset is published in several forms from the same references, so this chapter reads the flat Parquet form, which zaro reads one row group at a time. The Icechunk read needs zaro to ask for only the manifest rows a request covers.
The job: take five dives of one seal on the Kerguelen plateau in 2019, and for each get what the seal measured and what the model has at that cell on that day. In R that is two lines. What the seal measured is a block of a matrix, obs[, dives]. What the model has is a profile, temp[i, j, , t], for each dive: 51 chunks, one per depth level.
| Read | Values | Chunks | MB | Fetch calls, stock | Fetch calls, patched | Requests, stock | Requests, patched | Seconds, stock | Seconds, patched | Faster by | Same values |
|---|---|---|---|---|---|---|---|---|---|---|---|
| seal: 5 dives of one animal, TEMP_ADJUSTED[, dives] | 5,000 | 5 | 0.0 | 5 | 1 | 5 | 5 | 1.71 | 1.07 | 1.6x | yes |
| model: the profile under each dive, a loop of temp[i, j, , t] | 255 | 255 | 15.2 | 255 | 5 | 256 | 256 | 75.02 | 26.09 | 2.9x | yes |
| model: the same values as one matrix subscript, temp[idx] | 255 | 255 | 15.2 | 1 | 1 | 255 | 255 | 19.25 | 16.15 | 1.2x | yes |
Three hosts are involved in these reads: Source Cooperative for the references, the IMOS bucket in Sydney for the seal profiles, NCI in Canberra for the model. On stock R every chunk from either archive is its own round trip, in series. With the method, each [ is one batch.
The third row is the workaround a careful user can write today: build the index matrix for every level of every dive by hand, and R passes it through whole in both builds. It is the fastest of the three, and it is the code nobody writes first.
| Read | Element calls, stock | Element calls, patched |
|---|---|---|
| seal: 5 dives of one animal, TEMP_ADJUSTED[, dives] | 5,000 | 0 |
| model: the profile under each dive, a loop of temp[i, j, , t] | 255 | 0 |
| model: the same values as one matrix subscript, temp[idx] | 0 | 0 |
tabularberg needs a Rust toolchain to build; the book image includes it unless built with WITH_ICECHUNK=false (Appendix A), and without it this chapter has no table. Requests to THREDDS are capped at 4 at a time. The Source Cooperative location is a scratch area: datasets are added there over time.
6.2 Why the batch matters more here
With a plain Zarr store a chunk’s address is its key. With Icechunk each chunk is looked up in a manifest, and a virtual chunk then points at another host. A reader given the whole request resolves all the references in one pass over the manifest and can group the fetches by host. A reader given one element at a time does that lookup once per chunk. The deeper the indirection, the more a store gains from being asked once.