# 07-ordinary.R -- R code that was not written for lazy arrays. ASCII only.
#
# The same simulated store as 02-rectangles.R, with names on the time
# dimension. Every read below is plain base R.
here <- function() {
f <- sub("^--file=", "", grep("^--file=", commandArgs(FALSE), value = TRUE))
if (length(f)) dirname(f[1]) else "scripts"
}
source(file.path(here(), "_bench.R"))
x <- synthetic(c(1000, 800, 60), c(100, 100, 10),
dimnames = list(NULL, NULL, paste0("t", 1:60)))
# negative, logical and character subscripts: R resolves them all
keep <- rep(c(TRUE, FALSE), 400)
bench("x[-(1:900), keep, \"t30\"]", x, x[-(1:900), keep, "t30"])
# a function written for ordinary arrays
time_mean <- function(a, i, j) apply(a[i, j, , drop = FALSE], c(1, 2), mean)
bench("apply() over x[101:150, 201:260, , drop = FALSE]", x,
time_mean(x, 101:150, 201:260))
# a loop over time steps
bench("for each of 20 steps: sum(x[201:400, 301:500, k])", x,
vapply(1:20, function(k) sum(x[201:400, 301:500, k]), numeric(1)))
# R's own wrapper object, made by changing attributes on a shared array
y <- x
dimnames(y) <- list(lon = NULL, lat = NULL, time = paste0("t", 1:60))
bench("after dimnames<- on a copy: y[1:250, 1:250, 1:15]", x, y[1:250, 1:250, 1:15])
finish()7 Code that was not written for lazy arrays
The examples so far could all be rescued today by calling altarr_extract() in place of [. This chapter is about why that is not an answer.
The promise of a classless ALTREP array is that it is an array. Nobody has to write methods for it and nobody has to know it is lazy. Thirty years of R code takes arrays and indexes them with [. Every one of those functions works on a lazy array on stock R, correctly, and reads its data one element at a time. A special extract function helps only the code that calls it.
With the method in R, the planning happens underneath [, so it happens for all of them.
| Read | Values | Chunks | Fetch calls, stock | Fetch calls, patched | Seconds, stock | Seconds, patched | Faster by | Same values |
|---|---|---|---|---|---|---|---|---|
| x[-(1:900), keep, “t30”] | 40,000 | 8 | 8 | 1 | 0.42 | 0.06 | 6.6x | yes |
| apply() over x[101:150, 201:260, , drop = FALSE] | 3,000 | 6 | 6 | 1 | 0.34 | 0.08 | 4.4x | yes |
| for each of 20 steps: sum(x[201:400, 301:500, k]) | 20 | 8 | 8 | 2 | 0.47 | 0.13 | 3.7x | yes |
| after dimnames<- on a copy: y[1:250, 1:250, 1:15] | 937,500 | 18 | 18 | 1 | 1.00 | 0.09 | 10.8x | yes |
Read by read:
- Negative, logical and character subscripts. R resolves all of them to positions before the method is called, with its own rules for recycling, name matching and
NA. A lazy array class gets R’s full indexing language without implementing any of it, and cannot get it subtly wrong. apply()over a block. The function takes an array and indexes it. It has never heard of altarr.- A loop over time steps. Each step is one planned read, and steps that fall in chunks already read cost nothing, so the loop makes one fetch call per layer of chunks it enters.
- After
dimnames<-on a copy. R does not copy the data; it wraps the object in its own ALTREP wrapper class. The patch makes the wrapper forward the method, so a renamed array is still planned.
| Read | Element calls, stock | Element calls, patched |
|---|---|---|
| x[-(1:900), keep, “t30”] | 40,000 | 0 |
| apply() over x[101:150, 201:260, , drop = FALSE] | 180,000 | 0 |
| for each of 20 steps: sum(x[201:400, 301:500, k]) | 800,000 | 0 |
| after dimnames<- on a copy: y[1:250, 1:250, 1:15] | 937,500 | 0 |
7.1 Where this reaches
The same thing holds further up the stack, in packages that sit on lazy arrays:
- ndr (labelled n-dimensional arrays, in the manner of xarray) selects with
do.call("[", ...). On stock R it has to detect a lazy array and callaltarr_extract(). With the method it does not need to know. - GDAL7 presents GDAL multidimensional arrays and raster bands with
as_altarr(). Given the whole request, a band indexed asb[rows, cols]can be one GDAL read of the window, with oneAdviseReadahead of it. - Any reader could do the same. The method is not specific to altarr: arrow, a NetCDF or HDF5 package, a database-backed matrix, a memory-mapped file class could each implement it.
7.2 Still open
- Wrapper classes in packages. R’s own wrapper forwards the method. A package that defines its own ALTREP view over another object (rray 0.2.0 has one) cannot forward it, because the dispatch function is internal to R. Whether to export it is a question for R core. Not yet tested here.
- Subassignment.
x[i, j, k] <- valueon a lazy array materializes it. A write-side method would be the companion to this one; it is not part of this proposal. is.na()has no ALTREP hook, somean(x, na.rm = TRUE)still walks elements. Subsetting first, then computing on the ordinary array that comes back, is the pattern that works, and it is the one this patch makes fast.