xrdbi: 'DBI' Backend for 'xarray' via a Live 'reticulate' Session
Source:R/xrdbi-package.R
xrdbi-package.RdTreats a live Python session holding an 'xarray' https://xarray.dev object as a database connection. Tables are variables, fields are dimensions and coordinates, and queries are Python expressions (or compiled from a small structured spec or from 'dplyr' verbs) evaluated lazily against the dataset. Fetching renders the tidy long form, one row per cell, built in R from the values; results can also render as an in-memory raster.
Articles
The package website has a Get started guide and articles with live examples: querying with dplyr, parameters, the shape and types of results, data sources, rasters, chunk maps, curvilinear grids and vector cubes, Python environments, and design notes.
How xarray maps to DBI
A connection holds one xarray Dataset, open in the Python session.
Tables are its variables, data variables and coordinates alike:
DBI::dbListTables()lists them all, whiledbGetInfo()$tableslists only the data variables.The fields of a table are its dimensions, then its non-dimension coordinates, then the variable itself: the columns its rows will have (
DBI::dbListFields()).A statement is a Python expression over
ds(the Dataset) andxr(the module), such as"ds.sst.isel(time=0)". It runs when sent; a selection reads nothing there, a computation runs in full (see XarrayResult).Fetching renders the result in long form, as xarray's
to_dataframe().reset_index()would: one row per cell, a column per coordinate. Missing floats stayNaN, times becomePOSIXctin UTC (as xarray stores them), and the geometry of a vector data cube becomes awk::wkb()vector with its CRS when wk is installed (WKT strings otherwise).
Functions by task
Connect:
xarray()is the driver; dbConnect() opens a path or URL, runs a builder function, or adopts a Python object;dbDisconnect()closes it.Look around: print the connection, or use
dbListTables(),dbListFields()anddbGetInfo()(XarrayConnection).Query in Python:
dbGetQuery(con, statement), ordbSendQuery()thendbFetch()in pages; pass values from R withparamsand rebind withdbBind()(XarrayResult).xr_query()writes a statement from a structured selection.Query with dplyr: tbl() gives a lazy table for
filter(),select(),head()andcollect(), compiled to a statement thatdplyr::show_query()prints. A table is one variable, or a grid of them (the variables on one set of dimensions, a column each), whichxr_grids()lists.Read a whole variable: dbReadTable().
Render other ways:
xr_raster()gives a 2-D or 3-D slab in gdalraster's in-memory raster form;xr_chunks()maps the storage chunks a selection touches.Use Python directly:
xr_dataset()andxr_module()return the Dataset and the xarray module, unconverted.
Guards and options
Reads that materialize data are guarded: above a size they error,
naming the size, unless the call is given force = TRUE. Each guarded
function also takes the threshold as an argument; the options set the
defaults.
xrdbi.max_cells(default1e7): the most cells a read renders: the firstdbFetch()of a result (sodbGetQuery()),dbReadTable(),collect()of a tbl_xarray, andxr_raster(). It sizes the result of a statement, not its computation:.mean('time')runs atdbSendQuery(), before the guard sees the result.xrdbi.max_chunks(default1e6): the most rowsxr_chunks()returns.xrdbi.print_n(default6): the rows a printed tbl_xarray previews, reading at most one chunk.xrdbi.print_preview(defaultTRUE):FALSEprints a tbl_xarray's compiled statement without the preview's data read (compiling still reads coordinates).xrdbi.preview_members(default4): the most members of a grid tbl_xarray a print previews, each at most a chunk read.
R packages used when installed
wk: geometry. Vector cube geometry arrives as
wk::wkb()(WKT strings without it), geometryparamsare sent through it, andxr_chunks()returns each chunk's rectangle as awk::rct().sf: not needed, but
sf::st_as_sfc()converts any of those.gdalraster: not needed by
xr_raster(), but its output is in the form gdalraster'sread_ds()returns, soplot_raster()takes it.
Python requirements
Loading xrdbi declares its core Python requirements with
reticulate::py_require(): 'xarray' (which brings 'pandas' and
'numpy') and 'netCDF4', the netCDF engine for local netCDF-3 and
netCDF-4 files. When no Python is configured, reticulate provisions
them in a managed environment on first use. Declare the IO engines
your data needs the same way, before connecting (or after: reticulate
then switches to an environment that adds them):
zarr stores:
reticulate::py_require("zarr"), with"gcsfs"or"s3fs"for cloud bucketsnetCDF/HDF5 over HTTPS via h5netcdf:
reticulate::py_require(c("h5netcdf", "h5py", "fsspec", "aiohttp"))many files as one dataset (
open_mfdataset):reticulate::py_require("dask")vector data cubes:
reticulate::py_require(c("xvec", "geopandas"))
The 'gdalxarray' engine needs the GDAL Python bindings, which pip
cannot install: use a conda environment (conda-forge gdal) or an
image with GDAL, such as ghcr.io/hypertidy/gdal-r-python. With a
Python you configure yourself (RETICULATE_PYTHON, a conda
environment), install the packages there; reticulate does not
provision into it.
Author
Maintainer: Michael Sumner mdsumner@gmail.com
Authors:
Michael Sumner mdsumner@gmail.com
Other contributors:
The GDAL/OGR contributors (test fixtures in inst/extdata/autotest, see inst/COPYRIGHTS) [copyright holder]
Examples
if (FALSE) { # \dontrun{
## needs a Python environment with xarray (see Python requirements)
library(DBI)
f <- system.file("extdata/autotest/trmm.nc", package = "xrdbi")
con <- dbConnect(xarray(), f)
con
dbGetQuery(con, "ds.pcp.isel(time=0, latitude=0, longitude=slice(0, 3))")
library(dplyr)
tbl(con, "pcp") |> filter(latitude > -14, latitude < -13) |> collect()
dbDisconnect(con)
} # }