Skip to contents

Array stores are read in chunks: a GeoTIFF in blocks, a zarr or netCDF array in its chunks. A read touches every chunk its selection overlaps, and pays for each one whole, so the chunk layout decides what a query costs. xr_chunks() maps it. It returns a table with one row per chunk that a statement touches: each chunk’s index range and coordinate extent along every dimension, how many of its cells the statement selects, and a rectangle over the last two dimensions. Only coordinates and metadata are read, never the data.

The blocks of a COG

A Sentinel-2 true-colour image, one 10980 by 10980 pixel UTM tile stored as a Cloud-Optimized GeoTIFF in 1024 by 1024 blocks, read through the gdalxarray engine (it needs the GDAL Python bindings):

tci <- "/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/36/Q/WD/2020/7/S2A_36QWD_20200701_0_L2A/TCI.tif"
cog <- dbConnect(xarray(), tci, engine = "gdalxarray", multidim = FALSE)
blocks <- xr_chunks(cog, "ds.band_data.isel(band=0)")
attr(blocks, "chunking")
#> [1] "storage"
nrow(blocks)
#> [1] 121
head(blocks[, c("y_chunk", "x_chunk", "y_start", "y_stop", "x_start",
                "x_stop", "cells")], 3)
#>   y_chunk x_chunk y_start y_stop x_start x_stop   cells
#> 1       0       0       0   1024       0   1024 1048576
#> 2       0       1       0   1024    1024   2048 1048576
#> 3       0       2       0   1024    2048   3072 1048576
table(blocks$cells)
#> 
#>  547600  757760 1048576 
#>       1      20     100

The blocks along the right and bottom edges are partial, which the cells column shows. Each dimension gets _chunk (the chunk’s index along it), _start and _stop (its 0-based index range, stop exclusive, as in isel(d=slice(start, stop))) and, when the dimension has a coordinate, _min and _max (its extent, to the cell edges):

blocks[1, c("y_min", "y_max", "x_min", "x_max")]
#>     y_min   y_max  x_min  x_max
#> 1 1889780 1900020 499980 510220
class(blocks$geometry)
#> [1] "wk_rct"  "wk_rcrd"
sf::st_crs(wk::wk_crs(blocks$geometry))$Name
#> [1] "WGS 84 / UTM zone 36N"

geometry is a wk::rct() carrying the CRS, so it can be drawn, intersected, or converted with sf::st_as_sfc().

What a selection touches

On a selection, the table lists only the chunks the selection touches, and selected counts the cells it reads from each:

win <- xr_chunks(cog,
  "ds.band_data.isel(band=0, y=slice(3000, 5500), x=slice(2500, 6000))")
nrow(win)
#> [1] 16
win[, c("y_chunk", "x_chunk", "cells", "selected")] |> head()
#>   y_chunk x_chunk   cells selected
#> 1       2       2 1048576    41184
#> 2       2       3 1048576    73728
#> 3       2       4 1048576    73728
#> 4       2       5 1048576    63360
#> 5       3       2 1048576   585728
#> 6       3       3 1048576  1048576
## the share of the cells in the touched blocks that the window uses
sum(win$selected) / sum(win$cells)
#> [1] 0.5215406
used <- win$selected / win$cells
plot(blocks$geometry, border = "grey60")
plot(win$geometry, add = TRUE, border = "firebrick",
     col = rgb(178, 34, 34, alpha = 255 * (0.15 + 0.6 * used),
               maxColorValue = 255))

The 11 by 11 grid of storage blocks of a Sentinel-2 tile, with the blocks that a window touches shaded by the share of their cells the window reads

The interior blocks are read whole and the edge blocks only partly, but all of them are fetched.

Chunks in time

Chunking is not only spatial. The ARCO ERA5 store on Google Cloud keeps one hour per chunk, each a whole global field:

reticulate::py_require(c("zarr", "gcsfs"))
era <- dbConnect(xarray(),
  "gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3",
  engine = "zarr", storage_options = reticulate::dict(token = "anon"))

## a point time series for January 1980
pt <- xr_chunks(era,
  "ds['2m_temperature'].sel(time=slice('1980-01-01', '1980-01-31'),
                             latitude=-42.75, longitude=147.25)")
nrow(pt)
#> [1] 744
sum(pt$selected)
#> [1] 744
sum(pt$cells)
#> [1] 772450560

A month of hourly values at one point touches 744 chunks, and reads a million cells for every value it uses. This store’s layout favours global maps at an instant, not long series at a point.

The whole variable has more chunks than the default limit of a million rows (max_chunks, or getOption("xrdbi.max_chunks")), so mapping it needs force = TRUE:

try(xr_chunks(era, "ds['2m_temperature']"))
#> Error : the selection touches 1,323,648 chunks, above the guard (1,000,000); select less, raise options(xrdbi.max_chunks=), or use force = TRUE

Several variables

A Dataset statement maps every data variable in it under the one selection, and array says whose chunk each row is. Here are two surface variables over six hours:

two <- xr_chunks(era, paste0(
  "ds[['2m_temperature', '10m_u_component_of_wind']]",
  ".sel(time=slice('1980-01-01T00', '1980-01-01T05'))"))
table(two$array)
#> 
#> 10m_u_component_of_wind          2m_temperature 
#>                       6                       6
two[c(1, 7), c("array", "time_chunk", "time_min", "cells")]
#>                     array time_chunk   time_min   cells
#> 1          2m_temperature     701256 1980-01-01 1038240
#> 7 10m_u_component_of_wind     701256 1980-01-01 1038240

The variables need not share dimensions. A variable has missing values in the columns of any dimension it lacks: here that is level for the surface variable, beside geopotential on 37 pressure levels:

mix <- xr_chunks(era, paste0(
  "ds[['2m_temperature', 'geopotential']]",
  ".sel(time='1980-01-01T00', latitude=slice(-40, -45), longitude=slice(145, 150))"))
mix[, c("array", "level_chunk", "latitude_chunk", "cells", "selected")]
#>            array level_chunk latitude_chunk    cells selected
#> 1 2m_temperature          NA              0  1038240      441
#> 2   geopotential           0              0 38414880    16317
dbDisconnect(era)

Shards

A zarr v3 array can be sharded: its chunks are grouped into larger stored objects, the shards, so a store holds fewer, bigger files while reads can still fetch single chunks. level = "shards" maps the shards, and at the chunk level each chunk gets a _shard column per dimension, the shard it lives in. A small sharded store:

path <- file.path(tempfile(), "sharded.zarr")
dir.create(dirname(path))
reticulate::py_run_string(sprintf("
import xarray as xr, numpy as np, pandas as pd
xr.Dataset({'v': (('time', 'y', 'x'), np.zeros((4, 120, 200), 'float32'))},
  coords={'time': pd.date_range('2026-01-01', periods=4),
          'y': np.arange(120) + 0.5, 'x': np.arange(200) + 0.5}).to_zarr(
  '%s', mode='w', zarr_format=3, consolidated=False,
  encoding={'v': {'chunks': (1, 20, 25), 'shards': (2, 60, 100)}})
", path))
z <- dbConnect(xarray(), path, engine = "zarr", consolidated = FALSE)

shards <- xr_chunks(z, "ds.v", level = "shards")
nrow(shards)
#> [1] 8
unique(shards$cells)
#> [1] 12000
chunks <- xr_chunks(z, "ds.v")
nrow(chunks)
#> [1] 192
table(paste(chunks$y_shard, chunks$x_shard))
#> 
#> 0 0 0 1 1 0 1 1 
#>  48  48  48  48
win <- xr_chunks(z, "ds.v.isel(time=1, y=slice(50, 70), x=slice(90, 110))")
win[, c("y_chunk", "x_chunk", "y_shard", "x_shard", "selected")]
#>   y_chunk x_chunk y_shard x_shard selected
#> 1       2       3       0       0      100
#> 2       2       4       0       1      100
#> 3       3       3       1       0      100
#> 4       3       4       1       1      100
one <- chunks$time_chunk == 0
plot(chunks$geometry[one], border = "grey70")
plot(shards$geometry[shards$time_chunk == 0], add = TRUE, lwd = 2)
plot(win$geometry, add = TRUE, col = rgb(178, 34, 34, 100, maxColorValue = 255),
     border = "firebrick")
rect(90, 50, 110, 70, lty = 2)   # the window itself

One time step of a sharded array: 48 chunks in 4 shards, and a small window crossing the corner where four chunks and four shards meet

The window (dashed) is 20 by 20 cells, but it crosses a corner, so it touches four chunks (shaded) in four shards (heavy lines).

Keys for byte references

A map of storage chunks starts with the chunk key of a byterefs table: array, and dim_0, dim_1, …, the chunk’s 0-based position along each stored dimension, in the variable’s dimension order. That is the same as the _chunk columns, under the names other readers use:

chunks[1:3, c("array", "dim_0", "dim_1", "dim_2", "time_chunk", "y_chunk",
              "x_chunk")]
#>   array dim_0 dim_1 dim_2 time_chunk y_chunk x_chunk
#> 1     v     0     0     0          0       0       0
#> 2     v     0     0     1          0       0       1
#> 3     v     0     0     2          0       0       2

byterefs views (refs_<array>) and zaro’s plans (zaro_plan()) are keyed the same way, one array at a time. xr_chunks() reads no bytes, so for each chunk’s presence, path and byte range, join its map to one of those on the keys. In a sharded array the keys index the inner chunks, as byterefs rows do, so a shard map has no keys. With zaro, each chunk’s shard object and its byte range within it come from the shard index:

keys <- c("dim_0", "dim_1", "dim_2")
p <- zaro::zaro_plan(zaro::zaro(path), "v", chunks = as.matrix(chunks[keys]))
merge(chunks, as.data.frame(p)[c(keys, "present", "path", "offset", "size")],
      by = keys)

Where chunk shapes come from

Chunk shapes come from the variable’s encoding["preferred_chunks"], which xarray’s backends set from the storage layout: zarr, netCDF through netCDF4 or h5netcdf, and GDAL through gdalxarray, classic and multidimensional. The chunking attribute says which source was used:

  • "storage": the storage layout, as above
  • "shards": the shards, with level = "shards"
  • "dask": the dask chunks of a dask-backed variable that records no storage layout
  • "none": no chunking known; the variable is one chunk

The selection is placed on the stored grid by its coordinate values, so it must be an orthogonal selection of stored variables, by sel()/isel() with labels, slices or lists. A pointwise selection onto a new dimension cannot be placed. Shards, dask chunks and whole variables are not storage chunks, so their maps have no dim_0 .. keys; for a Dataset whose variables differ, chunking has one value per variable.

On a curvilinear grid, each chunk also gets the envelope of its block of 2-D longitude and latitude: the curvilinear grids article shows one.

Where next