# 05-sharded.R -- a sharded Zarr V3 product: Sentinel-2 L2A from the EOPF
# sample service. ASCII only.
#
# Bands are matrices, so every read here is m[i, j]: the MatrixSubset()
# path in R, separate from the array path. Each band is one shard file
# holding many inner chunks; a chunk is a byte range of the shard.
here <- function() {
f <- sub("^--file=", "", grep("^--file=", commandArgs(FALSE), value = TRUE))
if (length(f)) dirname(f[1]) else "scripts"
}
source(file.path(here(), "_bench.R"))
suppressPackageStartupMessages(library(zaro))
options(zaro.verbose = FALSE,
zaro.max_con = as.integer(Sys.getenv("BOOK_MAX_CON", "8")))
count_http()
eopf <- zaro(paste0("https://s3.explorer.eopf.copernicus.eu/esa-zarr-sentinel-explorer-fra/",
"tests-output/sentinel-2-l2a/",
"S2C_MSIL2A_20260120T084301_N0511_R064_T36UXA_20260120T122910.zarr"))
r120 <- zaro_dataset(eopf, group = "measurements/reflectance/r120m")
r10 <- zaro_dataset(eopf, group = "measurements/reflectance/r10m")
b120 <- zaro_array(r120, "b04") # x, y
b10 <- zaro_array(r10, "b04")
cat("b04 at 120 m:", paste(dim(b120), collapse = " x "),
" at 10 m:", paste(dim(b10), collapse = " x "), "\n")
bench("whole band at 120 m: b[, ]", b120, b120[, ])
bench("1000 x 1000 window at 10 m: b[5001:6000, 5001:6000]", b10,
b10[5001:6000, 5001:6000])
bench("every 10th pixel of a 4000 x 4000 window at 10 m", b10,
b10[seq(3001, 7000, 10), seq(3001, 7000, 10)])
finish()5 Sharded Zarr V3: Sentinel-2
Zarr V3 added sharding: many small chunks packed into one object, with an index at one end saying where each chunk’s bytes are. Small chunks make small reads possible; shards keep the object count sane. ESA’s EOPF sample service publishes Sentinel-2 Level-2A products this way, each band a matrix at 10, 20, 60 m and coarser.
To a reader, a shard is one more level of byte references: a ranged read of the index, then each chunk is a byte range of the shard file. That is the same shape as a virtual store, and zaro plans it the same way.
This chapter is here for a second reason. A band is a matrix, and m[i, j] takes a different path through R than x[i, j, k]: MatrixSubset(), not ArraySubset(). R’s source calls the split “purely an optimization”, but it means the element loop exists twice, and the patch offers the method from both places. These reads exercise the matrix one.
| Read | Values | Chunks | MB | Fetch calls, stock | Fetch calls, patched | Requests, stock | Requests, patched | Seconds, stock | Seconds, patched | Faster by | Same values |
|---|---|---|---|---|---|---|---|---|---|---|---|
| whole band at 120 m: b[, ] | 837,225 | 25 | 2.6 | 25 | 1 | 25 | 1 | 7.32 | 1.57 | 4.7x | yes |
| 1000 x 1000 window at 10 m: b[5001:6000, 5001:6000] | 1,000,000 | 25 | 4.5 | 25 | 1 | 25 | 8 | 6.59 | 1.14 | 5.8x | yes |
| every 10th pixel of a 4000 x 4000 window at 10 m | 160,000 | 289 | 51.4 | 289 | 1 | 289 | 30 | 77.80 | 3.45 | 22.6x | yes |
The third read is a strided one: every tenth pixel of a 4000 x 4000 window, a quick-look. On a lazy array that is still every chunk under the window, because a compressed chunk is all or nothing. What the plan saves is, again, the round trips.
| Read | Element calls, stock | Element calls, patched |
|---|---|---|
| whole band at 120 m: b[, ] | 837,225 | 0 |
| 1000 x 1000 window at 10 m: b[5001:6000, 5001:6000] | 1,000,000 | 0 |
| every 10th pixel of a 4000 x 4000 window at 10 m | 160,000 | 0 |
The product URL is under a tests-output path on the sample service and may move. If this chapter shows no results, the script needs a current product; any Sentinel-2 L2A Zarr from the service will do.