Skip to contents

The block structure of a stored variable as a table: one row per storage chunk that the selection touches, with each chunk's index range and coordinate extent along every dimension, and a rectangle over the last two dimensions (y and x, as for xr_raster()). Run it on the whole variable ("ds.sst") to see its chunk grid, or on a selection ("ds.sst.sel(lat=slice(-60, -50))") to see which chunks reading it would touch. A Dataset statement ("ds[['sst', 'ice']].sel(lat=slice(-60, -50))") maps every data variable in it under the one selection, with an array column to say whose chunk each row is.

Usage

xr_chunks(
  x,
  statement = NULL,
  params = NULL,
  level = c("chunks", "shards"),
  max_chunks = getOption("xrdbi.max_chunks", 1e+06),
  force = FALSE
)

Arguments

x

an XarrayResult, an XarrayConnection (with statement), or a lazy tbl(), whose map is of what collecting it would read (for a grid table, every member it reads)

statement

a Python expression selecting a stored variable, or a Dataset of them, when x is a connection

params

named list of values the statement refers to, when x is a connection (see XarrayResult)

level

"chunks" (the default) maps storage chunks; "shards" maps the shards of a sharded zarr v3 array, each a stored object holding a grid of inner chunks. At the chunk level a sharded array also gets a d_shard column per dimension, the shard each chunk lives in.

max_chunks

guard on the number of rows, default getOption("xrdbi.max_chunks", 1e6)

force

bypass the guard

Value

a data frame, one row per chunk touched, chunks in C order (the last dimension fastest), variable by variable for a Dataset. First array, the stored variable's name, and, when the shapes are storage chunks (not shards, dask chunks or one whole variable), the byterefs chunk key dim_0 .. dim_{n-1} (integer). In a Dataset, a variable has missing values in the columns of dimensions it does not have, and in the keys beyond its rank. For each dimension d: d_chunk, the chunk's index along d; d_start and d_stop, its 0-based index range, stop exclusive (as in isel(d=slice(start, stop))); and, when d has a coordinate, d_min and d_max, its extent: cell edges for a numeric coordinate, the first and last values otherwise. Then cells, the chunk's size in cells, and selected, how many of them the selection reads. When the last two dimensions have numeric coordinates and the wk package is installed, geometry is a wk::rct() over them, with the CRS when one is recorded (as for xr_raster()). On a curvilinear grid, where 2-D longitude/latitude coordinates (recognised by CF standard_name, units, or common names) span the last two dimensions, each chunk also gets the envelope of its block of those values (lon_min, lon_max, lat_min, lat_max, named after the coordinates; cell centres, so half a cell inside the edges, and not split at the antimeridian), and geometry is that envelope in OGC:CRS84 when the dimensions have no 1-D coordinates of their own. Attribute chunking says where the shapes came from: "storage", "shards", "dask" or "none"; for a Dataset whose variables differ, one value per variable, named by variable.

Details

Chunk shapes come from the variable's encoding["preferred_chunks"], which xarray's backends (zarr, netCDF via netCDF4 or h5netcdf, GDAL via gdalxarray, classic and multidimensional) set from the storage layout. Without it, a dask-backed selection maps its own dask chunks; otherwise the variable is one chunk. The selection is placed on the source grid by its coordinate values, so it must be an orthogonal selection of stored variables (sel()/isel() by labels, slices, or lists); pointwise selections onto a new dimension are not supported. Nothing but coordinates is read.

Storage chunk maps carry the chunk key of the byterefs table, dim_0, dim_1, ...: the chunk's 0-based position along each stored dimension, in the variable's dimension order. A byterefs view refs_<array> and the plans of other readers (such as zaro's zaro_plan()) are keyed the same way, one array at a time, so a chunk map joins to them on dim_0 .. for what they record of each chunk (presence, path, byte range), which xr_chunks() does not read.

See also

xr_raster() to read the cells; XarrayResult for statements and params; xrdbi-package for an overview.

Examples

if (FALSE) { # \dontrun{
## needs a Python environment with xarray (and here zarr, gcsfs)
con <- dbConnect(xarray(),
  "gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3",
  engine = "zarr", storage_options = reticulate::dict(token = "anon"))
## one chunk per hour: a month of one variable reads 744 chunks
ch <- xr_chunks(con,
  "ds['2m_temperature'].sel(time=slice('1980-01-01', '1980-01-31'))")
nrow(ch)

## two variables of one grid under one selection: a row per variable
## and chunk, keyed by array and dim_0 .. as byterefs tables are
ch <- xr_chunks(con, paste0(
  "ds[['2m_temperature', '10m_u_component_of_wind']]",
  ".sel(time=slice('1980-01-01T00', '1980-01-01T05'))"))
ch[, c("array", "dim_0", "dim_1", "dim_2", "time_min", "cells")]

## a GeoTIFF's block grid (gdalxarray needs the GDAL Python bindings;
## multidim = FALSE opens it as a classic raster: one variable,
## band_data, over (band, y, x))
tif <- system.file("extdata/autotest/rgbsmall.tif", package = "xrdbi")
con <- dbConnect(xarray(), tif, engine = "gdalxarray", multidim = FALSE)
ch <- xr_chunks(con, "ds.band_data")
ch[, c("band_chunk", "y_start", "y_stop", "x_start", "x_stop", "cells")]
plot(ch$geometry)
} # }