== patched ==
variant: variant
r_svn_commit: e1b41a6758b02ea43008cdc7199abda4c9c21ef3
svn_revision: 90638
svn_date: 2026-10-03
make_check: true
patches:
- 0001-ALTREP-Extract_array_subset.patch sha256:e61595222a6a261906730831fbebf58d4db30545255e3d5831d2a37617edbcd7
R Under development (unstable) (2026-10-03 r90638)
altarr 0.1.0.9000
zaro 0.0.1.9048
tabularberg 0.0.0.9001
blosc 0.1.2
array-subset method compiled into altarr: TRUE
== stock ==
variant: baseline
r_svn_commit: e1b41a6758b02ea43008cdc7199abda4c9c21ef3
svn_revision: 90638
svn_date: 2026-10-03
make_check: false
patches:
WARNING: ignoring environment value of R_HOME
R Under development (unstable) (2026-10-03 r90638)
altarr 0.1.0.9000
zaro 0.0.1.9048
tabularberg 0.0.0.9001
blosc 0.1.2
array-subset method compiled into altarr: FALSE
rendered: 2026-11-22 04:34 UTC on a5af98a4c685
mode: live
Lazy arrays in base R
Worked examples for the proposed ALTREP Extract_array_subset method
Preface
R’s arrays are a vector with a dim attribute, and R’s [ is one of the richest indexing languages there is. ALTREP lets a package supply the vector: the values can live in a Zarr store on object storage, in NetCDF files behind a reference table, or in an Icechunk repository, and be read only when R asks for them. The object has no class. To R, and to every function ever written for arrays, it is an array.
That works today, with one gap. When you write x[i, j, k], R asks the object for the result one element at a time. A store that could have fetched the 60 chunks it needs in one concurrent batch instead sees millions of single-element questions and answers them chunk by chunk, in series.
The proposed change to R is one optional ALTREP method, Extract_array_subset: about 100 lines that let R hand the whole request over. This book is the case for it, made with reads from real stores.
The short version:
x[cbind(i, j, k)]andx[i]already reach an ALTREP array whole, on any released R. Planned point extraction works now (1 What planning buys: points).x[i, j, k]andm[i, j]do not. Same chunks, same bytes, but one round trip per chunk instead of one batch (2 The gap: rectangles).- On real stores that is the difference between a map arriving in one merged request and in 60 (3 A cloud-native store: Copernicus Marine, 4 A virtual store: BRAN on THREDDS, 5 Sharded Zarr V3: Sentinel-2, 6 Icechunk: a versioned store).
- With the method, code that knows nothing about lazy arrays gets planned reads:
apply(), loops, negative and logical and character subscripts (7 Code that was not written for lazy arrays). - The patch changes nothing for any existing class, and the package ABI does not move (8 The patch).
How to read the tables
Every table in this book is produced when the book is rendered. Each example is a script in book/scripts that is run twice: once with stock R-devel and once with R-devel plus the patch, both in the same container, with the same altarr and zaro source. The columns:
- Chunks: distinct chunks the read needs. The same in both builds.
- Fetch calls: how many times R’s requests reached the store as a batch. One is the goal.
- Requests and MB: HTTP requests sent and bytes received, for stores read over the network.
- Seconds: wall time. The patched build runs first, so a warm cache anywhere along the way counts against the patch, not for it.
- Same values: the two builds returned the same numbers.
A read that fails, usually a server dropping a connection, is retried after a pause, and the time shown is the attempt that succeeded. A cell that says “failed” is a read that did not complete in three attempts. Timings from a live store are one run on one day; read them as ratios, not benchmarks. Where a table’s caption says “Earlier run”, this render did not reach that store and the last saved result is shown instead.
This render
To run it again, on Pawsey or anywhere with a container runtime, see Appendix A — Running the book.