tbl(con, from) returns a lazy table over one variable, or over a
grid of variables: dplyr verbs
accumulate, and no data are read until dplyr::collect() (or a
print, which shows a small preview). The verbs compile to a single
Python statement, shown by dplyr::show_query() and run through
DBI::dbGetQuery() like any other statement. Building it checks
field names and probes coordinates, which needs Python, but reads no
data.
Usage
# S3 method for class 'XarrayConnection'
tbl(src, from, ...)Arguments
- src
an XarrayConnection
- from
a variable name (a data variable, or a coordinate); several data variables on one grid; or the name of a grid, as
xr_grids()lists it (either view)- ...
unused
Value
a lazy tbl_xarray; dplyr::collect() returns a tibble
Details
A grid is the set of variables on one set of dimensions (see
xr_grids()), and a grid table has a column per member, as tidync's
hyper_tibble() does: tbl(con, c("sst", "ice")) for some members,
or tbl(con, "time,zlev,lat,lon") for a grid by its name. It compiles
to a Dataset statement, ds[['sst', 'ice']], and the verbs apply as
they do to one variable, with these differences:
select()decides which members are read: the statement names the selected members, plus any a filter needs (which are dropped after it). It must name at least one member.the guard counts values, cells times members.
printing previews at most
getOption("xrdbi.preview_members", 4)members, since each is at least a chunk read.variables on different grids are refused, because a statement over both would repeat each value along the dimensions its variable lacks (a surface field across 37 pressure levels).
xr_chunks() takes a lazy table too, and maps what collecting it
would read: for a grid table, a row per member read and chunk.
Supported verbs: filter(), select() (plain column names),
head(), collect(), show_query(), and as.data.frame().
filter() takes ==, %in%, <, <=, >, >=, between() and
& (or several arguments), each comparing a field with a value;
there is no |, ! or !=. collect() is guarded like
DBI::dbFetch(): above max_cells cells read (default
getOption("xrdbi.max_cells", 1e7)) it errors unless
collect(force = TRUE); as.data.frame() takes the same arguments
and gives a plain data frame. Printing shows the
first getOption("xrdbi.print_n", 6) rows, reading at most one chunk;
options(xrdbi.print_preview = FALSE) turns the preview off.
Filters are pushed into xarray where they can be, and the rest applied after (the "residual"):
a filter on a dimension (
==,%in%, or a range) compiles to an inclusive.sel(), so only the part of the array it selects (its "hull") is read;%in%is an exact.sel([...])strict inequalities on a dimension use the inclusive hull, then a boolean
.isel()on that dimension's index trims the boundary, so the result is still an xarray objectpredicates on data variables, and mixed forms on one dimension, are kept as a pandas
.query()applied Python-side, before conversion to Rpredicates on non-dimension coordinates (2-D
lat/lonof a curvilinear grid, a 1-D coordinate alongtime) stay in that query for exactness, but are checked against the coordinates at collect time and the read narrowed to the index block holding every passing cell (.isel()with literal slices, visible inshow_query()). Dimensions without coordinates are pinned to their positions first (.assign_coords(y=range(n))), so they render as they would unfilteredwithout such a query,
head()andselect()apply in R to the fetched rows, andhead(n)first narrows the selection to the leading block that holds itsnrows, so a peek at a huge variable reads one corner;dplyr::show_query()lists these R stepsat collect time each hull dimension is asked for its direction, and the slice is oriented to match:
filter(latitude > 18, latitude < 19)works identically on ascending and DESCENDING coordinates
See also
XarrayResult for statements written in Python (and
values passed with params); xr_query(); xr_grids() for the
grids; the guard and preview options in xrdbi-package.
Examples
if (FALSE) { # \dontrun{
## needs a Python environment with xarray (and here zarr, gcsfs)
library(dplyr)
con <- dbConnect(xarray(),
"gs://gcp-public-data-arco-era5/ar/full_37-1h-0p25deg-chunk-1.zarr-v3",
engine = "zarr", storage_options = reticulate::dict(token = "anon"))
u <- tbl(con, "100m_u_component_of_wind") |>
filter(latitude > 18, latitude < 19, longitude >= 73, longitude <= 74,
time >= as.POSIXct("1980-01-01", tz = "UTC"),
time < as.POSIXct("1980-01-02", tz = "UTC"))
show_query(u)
collect(u)
## the store's time axis starts in 1900, its data in 1940
tbl(con, "2m_temperature") |>
filter(time >= as.POSIXct("1980-01-01", tz = "UTC")) |>
head(3) |> collect()
## the surface grid: 262 members, of which select() reads three
xr_grids(con)
wind <- tbl(con, "time,latitude,longitude") |>
filter(time >= as.POSIXct("2020-01-01 00:00", tz = "UTC"),
time <= as.POSIXct("2020-01-01 05:00", tz = "UTC"),
latitude >= -43, latitude <= -42,
longitude >= 147, longitude <= 148) |>
select(time, latitude, longitude, `2m_temperature`,
`10m_u_component_of_wind`, `10m_v_component_of_wind`)
show_query(wind)
collect(wind)
xr_chunks(wind) # 18 chunks: 6 hours, 3 members
} # }