5  Sharded Zarr V3: Sentinel-2

Zarr V3 added sharding: many small chunks packed into one object, with an index at one end saying where each chunk’s bytes are. Small chunks make small reads possible; shards keep the object count sane. ESA’s EOPF sample service publishes Sentinel-2 Level-2A products this way, each band a matrix at 10, 20, 60 m and coarser.

To a reader, a shard is one more level of byte references: a ranged read of the index, then each chunk is a byte range of the shard file. That is the same shape as a virtual store, and zaro plans it the same way.

This chapter is here for a second reason. A band is a matrix, and m[i, j] takes a different path through R than x[i, j, k]: MatrixSubset(), not ArraySubset(). R’s source calls the split “purely an optimization”, but it means the element loop exists twice, and the patch offers the method from both places. These reads exercise the matrix one.

# 05-sharded.R -- a sharded Zarr V3 product: Sentinel-2 L2A from the EOPF
# sample service. ASCII only.
#
# Bands are matrices, so every read here is m[i, j]: the MatrixSubset()
# path in R, separate from the array path. Each band is one shard file
# holding many inner chunks; a chunk is a byte range of the shard.
here <- function() {
  f <- sub("^--file=", "", grep("^--file=", commandArgs(FALSE), value = TRUE))
  if (length(f)) dirname(f[1]) else "scripts"
}
source(file.path(here(), "_bench.R"))
suppressPackageStartupMessages(library(zaro))
options(zaro.verbose = FALSE,
        zaro.max_con = as.integer(Sys.getenv("BOOK_MAX_CON", "8")))
count_http()

eopf <- zaro(paste0("https://s3.explorer.eopf.copernicus.eu/esa-zarr-sentinel-explorer-fra/",
                    "tests-output/sentinel-2-l2a/",
                    "S2C_MSIL2A_20260120T084301_N0511_R064_T36UXA_20260120T122910.zarr"))
r120 <- zaro_dataset(eopf, group = "measurements/reflectance/r120m")
r10 <- zaro_dataset(eopf, group = "measurements/reflectance/r10m")
b120 <- zaro_array(r120, "b04")                 # x, y
b10 <- zaro_array(r10, "b04")
cat("b04 at 120 m:", paste(dim(b120), collapse = " x "),
    " at 10 m:", paste(dim(b10), collapse = " x "), "\n")

bench("whole band at 120 m: b[, ]", b120, b120[, ])
bench("1000 x 1000 window at 10 m: b[5001:6000, 5001:6000]", b10,
      b10[5001:6000, 5001:6000])
bench("every 10th pixel of a 4000 x 4000 window at 10 m", b10,
      b10[seq(3001, 7000, 10), seq(3001, 7000, 10)])
finish()
2026-11-22 04:41 UTC on a5af98a4c685; R-devel patched r90638, stock r90638
Read Values Chunks MB Fetch calls, stock Fetch calls, patched Requests, stock Requests, patched Seconds, stock Seconds, patched Faster by Same values
whole band at 120 m: b[, ] 837,225 25 2.6 25 1 25 1 7.32 1.57 4.7x yes
1000 x 1000 window at 10 m: b[5001:6000, 5001:6000] 1,000,000 25 4.5 25 1 25 8 6.59 1.14 5.8x yes
every 10th pixel of a 4000 x 4000 window at 10 m 160,000 289 51.4 289 1 289 30 77.80 3.45 22.6x yes

The third read is a strided one: every tenth pixel of a 4000 x 4000 window, a quick-look. On a lazy array that is still every chunk under the window, because a compressed chunk is all or nothing. What the plan saves is, again, the round trips.

Read Element calls, stock Element calls, patched
whole band at 120 m: b[, ] 837,225 0
1000 x 1000 window at 10 m: b[5001:6000, 5001:6000] 1,000,000 0
every 10th pixel of a 4000 x 4000 window at 10 m 160,000 0
Note

The product URL is under a tests-output path on the sample service and may move. If this chapter shows no results, the script needs a current product; any Sentinel-2 L2A Zarr from the service will do.