The block structure of a stored variable as a table: one row per
storage chunk that the selection touches, with each chunk's index
range and coordinate extent along every dimension, and a rectangle
over the last two dimensions (y and x, as for xr_raster()). Run it
on the whole variable ("ds.sst") to see its chunk grid, or on a
selection ("ds.sst.sel(lat=slice(-60, -50))") to see which chunks
reading it would touch. A Dataset statement
("ds[['sst', 'ice']].sel(lat=slice(-60, -50))") maps every data
variable in it under the one selection, with an array column to say
whose chunk each row is.
Arguments
- x
an XarrayResult, an XarrayConnection (with
statement), or a lazy tbl(), whose map is of what collecting it would read (for a grid table, every member it reads)- statement
a Python expression selecting a stored variable, or a Dataset of them, when
xis a connection- params
named list of values the statement refers to, when
xis a connection (see XarrayResult)- level
"chunks"(the default) maps storage chunks;"shards"maps the shards of a sharded zarr v3 array, each a stored object holding a grid of inner chunks. At the chunk level a sharded array also gets ad_shardcolumn per dimension, the shard each chunk lives in.- max_chunks
guard on the number of rows, default
getOption("xrdbi.max_chunks", 1e6)- force
bypass the guard
Value
a data frame, one row per chunk touched, chunks in C order
(the last dimension fastest), variable by variable for a Dataset.
First array, the stored variable's name, and, when the shapes are
storage chunks (not shards, dask chunks or one whole variable), the
byterefs chunk key dim_0 .. dim_{n-1} (integer). In a Dataset, a
variable has missing values in the columns of dimensions it does
not have, and in the keys beyond its rank. For each dimension d:
d_chunk, the
chunk's index along d; d_start and d_stop, its 0-based index
range, stop exclusive (as in isel(d=slice(start, stop))); and,
when d has a coordinate, d_min and d_max, its extent: cell
edges for a numeric coordinate, the first and last values
otherwise. Then cells, the chunk's size in cells, and selected,
how many of them the selection reads. When the last two dimensions
have numeric coordinates and the wk package is installed,
geometry is a wk::rct() over them, with the CRS when one is
recorded (as for xr_raster()). On a curvilinear grid, where 2-D
longitude/latitude coordinates (recognised by CF standard_name,
units, or common names) span the last two dimensions, each chunk
also gets the envelope of its block of those values (lon_min,
lon_max, lat_min, lat_max, named after the coordinates; cell
centres, so half a cell inside the edges, and not split at the
antimeridian), and geometry is that envelope in OGC:CRS84 when
the dimensions have no 1-D coordinates of their own. Attribute
chunking says where the shapes came from: "storage",
"shards", "dask" or "none"; for a Dataset whose variables
differ, one value per variable, named by variable.
Details
Chunk shapes come from the variable's encoding["preferred_chunks"],
which xarray's backends (zarr, netCDF via netCDF4 or h5netcdf, GDAL
via gdalxarray, classic and multidimensional) set from the storage
layout. Without it, a dask-backed selection maps its own dask chunks;
otherwise the variable is one chunk. The selection is placed on the
source grid by its coordinate values, so it must be an orthogonal
selection of stored variables (sel()/isel() by labels, slices,
or lists); pointwise selections onto a new dimension are not
supported. Nothing but coordinates is read.
Storage chunk maps carry the chunk key of the
byterefs table, dim_0,
dim_1, ...: the chunk's 0-based position along each stored
dimension, in the variable's dimension order. A byterefs view
refs_<array> and the plans of other readers (such as zaro's
zaro_plan()) are keyed the same way, one array at a time, so a chunk
map joins to them on dim_0 .. for what they record of each chunk
(presence, path, byte range), which xr_chunks() does not read.
See also
xr_raster() to read the cells; XarrayResult for
statements and params; xrdbi-package for an overview.
Examples
if (FALSE) { # \dontrun{
## needs a Python environment with xarray (and here zarr, gcsfs)
con <- dbConnect(xarray(),
"gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3",
engine = "zarr", storage_options = reticulate::dict(token = "anon"))
## one chunk per hour: a month of one variable reads 744 chunks
ch <- xr_chunks(con,
"ds['2m_temperature'].sel(time=slice('1980-01-01', '1980-01-31'))")
nrow(ch)
## two variables of one grid under one selection: a row per variable
## and chunk, keyed by array and dim_0 .. as byterefs tables are
ch <- xr_chunks(con, paste0(
"ds[['2m_temperature', '10m_u_component_of_wind']]",
".sel(time=slice('1980-01-01T00', '1980-01-01T05'))"))
ch[, c("array", "dim_0", "dim_1", "dim_2", "time_min", "cells")]
## a GeoTIFF's block grid (gdalxarray needs the GDAL Python bindings;
## multidim = FALSE opens it as a classic raster: one variable,
## band_data, over (band, y, x))
tif <- system.file("extdata/autotest/rgbsmall.tif", package = "xrdbi")
con <- dbConnect(xarray(), tif, engine = "gdalxarray", multidim = FALSE)
ch <- xr_chunks(con, "ds.band_data")
ch[, c("band_chunk", "y_start", "y_stop", "x_start", "x_stop", "cells")]
plot(ch$geometry)
} # }