3  A cloud-native store: Copernicus Marine

Copernicus Marine publishes its gridded products as Zarr V2 on public object storage, and publishes each one twice. The timeChunked copy has chunks shaped like maps: one day of global sea level is nine chunks of 1024 x 512 cells. The geoChunked copy has chunks shaped like time series: 33 years at one cell is six chunks of 16 x 8 cells by 2264 days. The publisher has already done the work of making both kinds of read cheap.

With zaro, either copy opens as a lazy R array, unpacked to metres with missing values as NA:

ds  <- zaro_dataset(zaro("https://s3.waw3-1.cloudferro.com/.../timeChunked.zarr"))
sla <- zaro_array(ds, "sla", cf = TRUE)
dim(sla)
#> [1]  2880  1440 12188
map <- sla[, , 12188]          # the latest day

That last line is the natural thing to write, and it is the read the store was chunked for. On stock R it is 4.1 million element calls and nine separate requests, one after another. The same goes for the series sla[i, j, ] on the other copy.

3.1 The reads

Five reads: a global map, a regional block of two weeks, and a full series at one cell from the sea level product; then, from the global waves product (3-hourly since 1980, chunks of 4 x 8 cells by 17,720 steps), 46 years at one cell and a year in a small box.

# 03-cmems.R -- Copernicus Marine Zarr V2 stores, read in place. ASCII only.
#
# Global daily sea level (1/8 degree, 1993 on) is published twice: chunked
# for maps (timeChunked) and for time series (geoChunked). Global 3-hourly
# waves (0.2 degree, 1980 on) is read from its geoChunked copy. Public, no
# credentials. BOOK_MAX_CON (default 8) caps concurrent connections.
here <- function() {
  f <- sub("^--file=", "", grep("^--file=", commandArgs(FALSE), value = TRUE))
  if (length(f)) dirname(f[1]) else "scripts"
}
source(file.path(here(), "_bench.R"))
suppressPackageStartupMessages(library(zaro))
options(zaro.verbose = FALSE,
        zaro.max_con = as.integer(Sys.getenv("BOOK_MAX_CON", "8")))
count_http()

host <- "https://s3.waw3-1.cloudferro.com/"
sl <- paste0("SEALEVEL_GLO_PHY_L4_MY_008_047/",
             "cmems_obs-sl_glo_phy-ssh_my_allsat-l4-duacs-0.125deg_P1D_202411/")
ds_t <- zaro_dataset(zaro(paste0(host, "mdl-arco-time-045/arco/", sl, "timeChunked.zarr")))
ds_g <- zaro_dataset(zaro(paste0(host, "mdl-arco-geo-045/arco/", sl, "geoChunked.zarr")))
sla_t <- zaro_array(ds_t, "sla", cf = TRUE)     # longitude, latitude, time
sla_g <- zaro_array(ds_g, "sla", cf = TRUE)
cat("sea level:", paste(dim(sla_t), collapse = " x "), "\n")

lon <- as.vector(zaro_read(ds_t, "longitude"))
lat <- as.vector(zaro_read(ds_t, "latitude"))
i <- which.min(abs(lon - 155))                  # Tasman Sea
j <- which.min(abs(lat - -35))
ix <- which(lon >= 145 & lon <= 165)
iy <- which(lat >= -45 & lat <= -25)
nt <- dim(sla_t)[3]

bench("sea level, one day's map: sla[, , day]", sla_t, sla_t[, , nt])
bench("sea level, Tasman Sea for 14 days: sla[ix, iy, days]", sla_t,
      sla_t[ix, iy, (nt - 13):nt])
bench("sea level, every day at one cell: sla[i, j, ]", sla_g, sla_g[i, j, ])

wav <- zaro(paste0(host, "mdl-arco-geo-032/arco/GLOBAL_MULTIYEAR_WAV_001_032/",
                   "cmems_mod_glo_wav_my_0.2deg_PT3H-i_202411/geoChunked.zarr"))
hs <- zaro_array(wav, "VHM0", cf = TRUE)        # longitude, latitude, time
cat("waves:", paste(dim(hs), collapse = " x "), "\n")
wlon <- as.numeric(unlist(zaro_read(wav, "longitude")))
wlat <- as.numeric(unlist(zaro_read(wav, "latitude")))
wi <- which.min(abs(wlon - 140))                # Southern Ocean, south of Australia
wj <- which.min(abs(wlat - -50))
wt <- dim(hs)[3]

bench("waves, every 3 hours since 1980 at one cell: hs[i, j, ]", hs, hs[wi, wj, ])
bench("waves, a year in a 2 degree box: hs[i:(i+9), j:(j+9), year]", hs,
      hs[wi:(wi + 9), wj:(wj + 9), (wt - 2919):wt])
finish()
2026-11-22 04:36 UTC on a5af98a4c685; R-devel patched r90638, stock r90638
Read Values Chunks MB Fetch calls, stock Fetch calls, patched Requests, stock Requests, patched Seconds, stock Seconds, patched Faster by Same values
sea level, one day’s map: sla[, , day] 4,147,200 9 4.2 9 1 9 9 2.96 1.50 2.0x yes
sea level, Tasman Sea for 14 days: sla[ix, iy, days] 358,400 28 13.7 28 1 28 28 6.30 1.58 4.0x yes
sea level, every day at one cell: sla[i, j, ] 12,188 6 3.2 6 1 6 6 1.37 0.35 4.0x yes
waves, every 3 hours since 1980 at one cell: hs[i, j, ] 136,360 8 4.6 8 1 8 8 1.84 1.26 1.5x yes
waves, a year in a 2 degree box: hs[i:(i+9), j:(j+9), year] 292,000 9 3.7 9 1 9 9 1.86 1.21 1.5x yes

What to look for:

  • Requests are equal in the two builds, or close to it, and so are the megabytes. Nobody is reading more or less data.
  • Fetch calls drop to one. That one call carries every chunk, and zaro sends them through a single pool of 8 connections.
  • The gain grows with the number of chunks in the read, up to the connection cap. Nine chunks cannot be more than nine times faster. The regional block, with a few dozen chunks, is where the batch earns the most.
  • Decoding still happens in R, chunk by chunk, in both builds. That part of the time does not move.
Read Element calls, stock Element calls, patched
sea level, one day’s map: sla[, , day] 4,147,200 0
sea level, Tasman Sea for 14 days: sla[ix, iy, days] 358,400 0
sea level, every day at one cell: sla[i, j, ] 12,188 0
waves, every 3 hours since 1980 at one cell: hs[i, j, ] 136,360 0
waves, a year in a 2 degree box: hs[i:(i+9), j:(j+9), year] 292,000 0

3.2 The layout is the publisher’s; the plan should be R’s

The two copies exist because no single chunk shape serves maps and series both. Choosing the copy is the user’s one decision. After that, which chunks to fetch and how to fetch them together is arithmetic on the subscripts, and R has the subscripts. On stock R that arithmetic is thrown away at the last step.

The connection cap matters as much as the plan once the plan exists. Against this endpoint, 16 connections were about 1.5 times faster than 8 for a 268-chunk read, and 32 about 2 times (measured 2026-11-16). Set BOOK_MAX_CON to try it; the default stays at a polite 8.