# SparseArray objects

Hervé Pagès1

1Fred Hutchinson Cancer Research Center, Seattle, WA

#### Compiled 7 November 2023; Modified 24 July 2023

#### Package

SparseArray 1.2.2

# Contents

# 1 Introduction

SparseArray is an infrastructure package that provides an array-like container for efficient in-memory representation of multidimensional sparse data in R.

The package defines the SparseArray virtual class and two concrete subclasses: COO_SparseArray and SVT_SparseArray. Each subclass uses its own internal representation of the nonzero multidimensional data, the “COO layout” and the “SVT layout”, respectively.

Note that the SparseArray virtual class could easily be extended by other S4 classes that intent to implement alternative internal representations of the nonzero multidimensional data.

This vignette focuses on the SVT_SparseArray container.

# 2 Install and load the package

    if (!requireNamespace("BiocManager", quietly=TRUE))
        install.packages("BiocManager")
    BiocManager::install("SparseArray")

    library(SparseArray)

# 3 SVT_SparseArray objects

The SVT_SparseArray container provides an efficient representation of the nonzero multidimensional data via a novel layout called the “SVT layout”.

Note that SVT_SparseArray objects mimic as much as possible the behavior of ordinary matrix and array objects in base R. In particular, they suppport most of the “standard matrix and array API” defined in base R and in the matrixStats package from CRAN.

## 3.1 Construction

SVT_SparseArray objects can be constructed in many ways. A common way is to coerce an ordinary matrix or array to SVT_SparseArray:

    m <- matrix(0L, nrow=6, ncol=4)
    m[c(1:2, 8, 10, 15:17, 24)] <- (1:8)*10L
    svt1 <- as(m, "SVT_SparseArray")
    svt1

    ## <6 x 4 SVT_SparseMatrix> of type "integer" (nzcount=8):
    ##      [,1] [,2] [,3] [,4]
    ## [1,]   10    0    0    0
    ## [2,]   20   30    0    0
    ## [3,]    0    0   50    0
    ## [4,]    0   40   60    0
    ## [5,]    0    0   70    0
    ## [6,]    0    0    0   80

    a <- array(0L, 5:3)
    a[c(1:2, 8, 10, 15:17, 20, 24, 40, 56:60)] <- (1:15)*10L
    svt2 <- as(a, "SVT_SparseArray")
    svt2

    ## <5 x 4 x 3 SVT_SparseArray> of type "integer" (nzcount=15):
    ## ,,1
    ##      [,1] [,2] [,3] [,4]
    ## [1,]   10    0    0   60
    ## [2,]   20    0    0   70
    ## [3,]    0   30    0    0
    ## [4,]    0    0    0    0
    ## [5,]    0   40   50   80
    ##
    ## ,,2
    ##      [,1] [,2] [,3] [,4]
    ## [1,]    0    0    0    0
    ## [2,]    0    0    0    0
    ## [3,]    0    0    0    0
    ## [4,]   90    0    0    0
    ## [5,]    0    0    0  100
    ##
    ## ,,3
    ##      [,1] [,2] [,3] [,4]
    ## [1,]    0    0    0  110
    ## [2,]    0    0    0  120
    ## [3,]    0    0    0  130
    ## [4,]    0    0    0  140
    ## [5,]    0    0    0  150

Alternatively, the ordinary matrix or array can be passed to the `SVT_SparseArray` constructor function:

    svt1 <- SVT_SparseArray(m)
    svt2 <- SVT_SparseArray(a)

Note that coercing an ordinary matrix or array to SparseArray is also supported and will produce the same results:

    svt1 <- as(m, "SparseArray")
    svt2 <- as(a, "SparseArray")

This is because, for most use cases, the SVT_SparseArray representation is more efficient than the COO_SparseArray representation, so the former is usually preferred over the latter.

For the same reason, the `SparseArray` constructor function will also give the preference to the SVT_SparseArray representation:

    svt1 <- SparseArray(m)
    svt2 <- SparseArray(a)

This is actually the most convenient way to turn an ordinary matrix or array into an SVT_SparseArray object.

Coercion back to ordinary matrix or array is supported:

    as.array(svt1)  # same as as.matrix(svt1)

    ##      [,1] [,2] [,3] [,4]
    ## [1,]   10    0    0    0
    ## [2,]   20   30    0    0
    ## [3,]    0    0   50    0
    ## [4,]    0   40   60    0
    ## [5,]    0    0   70    0
    ## [6,]    0    0    0   80

    as.array(svt2)

    ## , , 1
    ##
    ##      [,1] [,2] [,3] [,4]
    ## [1,]   10    0    0   60
    ## [2,]   20    0    0   70
    ## [3,]    0   30    0    0
    ## [4,]    0    0    0    0
    ## [5,]    0   40   50   80
    ##
    ## , , 2
    ##
    ##      [,1] [,2] [,3] [,4]
    ## [1,]    0    0    0    0
    ## [2,]    0    0    0    0
    ## [3,]    0    0    0    0
    ## [4,]   90    0    0    0
    ## [5,]    0    0    0  100
    ##
    ## , , 3
    ##
    ##      [,1] [,2] [,3] [,4]
    ## [1,]    0    0    0  110
    ## [2,]    0    0    0  120
    ## [3,]    0    0    0  130
    ## [4,]    0    0    0  140
    ## [5,]    0    0    0  150

## 3.2 Accessors

The standard array accessors are supported:

    dim(svt2)

    ## [1] 5 4 3

    length(svt2)

    ## [1] 60

    dimnames(svt2) <- list(NULL, letters[1:4], LETTERS[1:3])
    svt2

    ## <5 x 4 x 3 SVT_SparseArray> of type "integer" (nzcount=15):
    ## ,,A
    ##       a  b  c  d
    ## [1,] 10  0  0 60
    ## [2,] 20  0  0 70
    ## [3,]  0 30  0  0
    ## [4,]  0  0  0  0
    ## [5,]  0 40 50 80
    ##
    ## ,,B
    ##        a   b   c   d
    ## [1,]   0   0   0   0
    ## [2,]   0   0   0   0
    ## [3,]   0   0   0   0
    ## [4,]  90   0   0   0
    ## [5,]   0   0   0 100
    ##
    ## ,,C
    ##        a   b   c   d
    ## [1,]   0   0   0 110
    ## [2,]   0   0   0 120
    ## [3,]   0   0   0 130
    ## [4,]   0   0   0 140
    ## [5,]   0   0   0 150

Some additional accessors defined in the S4Arrays / SparseArray framework:

    type(svt1)

    ## [1] "integer"

    type(svt1) <- "double"
    svt1

    ## <6 x 4 SVT_SparseMatrix> of type "double" (nzcount=8):
    ##      [,1] [,2] [,3] [,4]
    ## [1,]   10    0    0    0
    ## [2,]   20   30    0    0
    ## [3,]    0    0   50    0
    ## [4,]    0   40   60    0
    ## [5,]    0    0   70    0
    ## [6,]    0    0    0   80

    is_sparse(svt1)

    ## [1] TRUE

Other accessors/extractors specific to sparse arrays:

    ## Get the number of nonzero array elements in 'svt1':
    nzcount(svt1)

    ## [1] 8

    ## Extract the "linear indices" of the nonzero array elements in 'svt1':
    nzwhich(svt1)

    ## [1]  1  2  8 10 15 16 17 24

    ## Extract the "array indices" (a.k.a. "array coordinates") of the
    ## nonzero array elements in 'svt1':
    nzwhich(svt1, arr.ind=TRUE)

    ##      [,1] [,2]
    ## [1,]    1    1
    ## [2,]    2    1
    ## [3,]    2    2
    ## [4,]    4    2
    ## [5,]    3    3
    ## [6,]    4    3
    ## [7,]    5    3
    ## [8,]    6    4

    ## Extract the values of the nonzero array elements in 'svt1' and return
    ## them in a vector "parallel" to 'nzwhich(svt1)':
    #nzvals(svt1)  # NOT READY YET!

    sparsity(svt1)

    ## [1] 0.6666667

See `?SparseArray` for more information and additional examples.

## 3.3 Subsetting and subassignment

    svt2[5:3, , "C"]

    ## <3 x 4 SVT_SparseMatrix> of type "integer" (nzcount=3):
    ##        a   b   c   d
    ## [1,]   0   0   0 150
    ## [2,]   0   0   0 140
    ## [3,]   0   0   0 130

Like with ordinary arrays in base R, assigning values of type `"double"` to an SVT_SparseArray object of type `"integer"` will automatically change the type of the object to `"double"`:

    type(svt2)

    ## [1] "integer"

    svt2[5, 1, 3] <- NaN
    type(svt2)

    ## [1] "double"

See `?SparseArray_subsetting` for more information and additional examples.

## 3.4 Summarization methods (whole array)

The following summarization methods are provided at the moment: `anyNA()`, `any`, `all`, `min`, `max`, `range`, `sum`, `prod`, `mean`, `var`, `sd`.

    anyNA(svt2)

    ## [1] TRUE

    range(svt2, na.rm=TRUE)

    ## [1]   0 150

    mean(svt2, na.rm=TRUE)

    ## [1] 20.33898

    var(svt2, na.rm=TRUE)

    ## [1] 1717.124

See `?SparseArray_summarization` for more information and additional examples.

## 3.5 Operations from the ‘Ops’, ‘Math’, ‘Math2’, and ‘Complex’ groups

SVT_SparseArray objects support operations from the ‘Ops’, ‘Math’, `Math2`, and ‘Complex’ groups, with some restrictions. See `?S4groupGeneric` in the methods package for more information about these group generics.

    signif((svt1^1.5 + svt1) %% 100 - 0.6 * svt1, digits=2)

    ## <6 x 4 SVT_SparseMatrix> of type "double" (nzcount=8):
    ##       [,1]  [,2]  [,3]  [,4]
    ## [1,]  36.0   0.0   0.0   0.0
    ## [2,]  -2.6  76.0   0.0   0.0
    ## [3,]   0.0   0.0 -26.0   0.0
    ## [4,]   0.0  69.0 -11.0   0.0
    ## [5,]   0.0   0.0  14.0   0.0
    ## [6,]   0.0   0.0   0.0  48.0

See `?SparseArray_Ops`, `?SparseArray_Math`, and `?SparseArray_Complex`, for more information and additional examples.

## 3.6 Other operations on SVT_SparseArray objects

More operations will be added in the future e.g. `is.na()`, `is.infinite()`, `is.nan()`, etc…

## 3.7 Generate a random SVT_SparseArray object

Two convenience functions are provided for this:

    randomSparseArray(c(5, 6, 2), density=0.5)

    ## <5 x 6 x 2 SVT_SparseArray> of type "double" (nzcount=30):
    ## ,,1
    ##        [,1]   [,2]   [,3]   [,4]   [,5]   [,6]
    ## [1,]  0.000  0.000  0.000  0.000  3.500  0.000
    ## [2,]  0.000  0.000 -1.300 -0.760 -0.017  0.000
    ## [3,]  0.420  0.000  0.500  0.000  0.000  0.000
    ## [4,] -0.420  0.000  0.470  0.000  0.000  0.000
    ## [5,] -1.300 -0.640 -0.440 -0.110  0.000 -0.820
    ##
    ## ,,2
    ##        [,1]   [,2]   [,3]   [,4]   [,5]   [,6]
    ## [1,]  0.058  0.000 -0.073  0.000  1.300 -0.480
    ## [2,]  0.000 -1.100 -1.800 -1.800  0.000  0.000
    ## [3,] -1.800 -0.880  0.000 -0.810  0.950  0.000
    ## [4,]  0.000  0.000  1.700 -0.270  1.400  0.000
    ## [5,] -0.230  1.600  0.870  0.000  0.000  0.000

    poissonSparseArray(c(5, 6, 2), density=0.5)

    ## <5 x 6 x 2 SVT_SparseArray> of type "integer" (nzcount=28):
    ## ,,1
    ##      [,1] [,2] [,3] [,4] [,5] [,6]
    ## [1,]    0    0    1    0    3    2
    ## [2,]    0    0    0    1    0    0
    ## [3,]    1    3    0    0    1    0
    ## [4,]    1    0    0    1    0    3
    ## [5,]    1    2    0    0    1    0
    ##
    ## ,,2
    ##      [,1] [,2] [,3] [,4] [,5] [,6]
    ## [1,]    0    1    0    2    0    0
    ## [2,]    3    0    1    0    0    1
    ## [3,]    0    0    2    1    1    1
    ## [4,]    2    1    1    0    2    1
    ## [5,]    0    0    0    0    0    1

See `?randomSparseArray` for more information and additional examples.

# 4 SVT_SparseMatrix objects

## 4.1 Transposition

    t(svt1)

    ## <4 x 6 SVT_SparseMatrix> of type "double" (nzcount=8):
    ##      [,1] [,2] [,3] [,4] [,5] [,6]
    ## [1,]   10   20    0    0    0    0
    ## [2,]    0   30    0   40    0    0
    ## [3,]    0    0   50   60   70    0
    ## [4,]    0    0    0    0    0   80

Note that multidimensional transposition is supported via `aperm()`:

    aperm(svt2)

    ## <3 x 4 x 5 SVT_SparseArray> of type "double" (nzcount=16):
    ## ,,1
    ##     a   b   c   d
    ## A  10   0   0  60
    ## B   0   0   0   0
    ## C   0   0   0 110
    ##
    ## ,,2
    ##     a   b   c   d
    ## A  20   0   0  70
    ## B   0   0   0   0
    ## C   0   0   0 120
    ##
    ## ,,3
    ##     a   b   c   d
    ## A   0  30   0   0
    ## B   0   0   0   0
    ## C   0   0   0 130
    ##
    ## ,,4
    ##     a   b   c   d
    ## A   0   0   0   0
    ## B  90   0   0   0
    ## C   0   0   0 140
    ##
    ## ,,5
    ##     a   b   c   d
    ## A   0  40  50  80
    ## B   0   0   0 100
    ## C NaN   0   0 150

See `?SparseArray_aperm` for more information and additional examples.

## 4.2 Combine multidimensional objects along a given dimension

Like ordinary matrices in base R, SVT_SparseMatrix objects can be combined by rows or columns, with `rbind()` or `cbind()`:

    svt3 <- poissonSparseMatrix(6, 2, density=0.5)

    cbind(svt1, svt3)

    ## <6 x 6 SVT_SparseMatrix> of type "double" (nzcount=13):
    ##      [,1] [,2] [,3] [,4] [,5] [,6]
    ## [1,]   10    0    0    0    0    0
    ## [2,]   20   30    0    0    1    0
    ## [3,]    0    0   50    0    1    0
    ## [4,]    0   40   60    0    1    1
    ## [5,]    0    0   70    0    0    0
    ## [6,]    0    0    0   80    0    1

Note that multidimensional objects can be combined along any dimension with `abind()`:

    svt4a <- poissonSparseArray(c(5, 6, 2), density=0.4)
    svt4b <- poissonSparseArray(c(5, 6, 5), density=0.2)
    svt4c <- poissonSparseArray(c(5, 6, 4), density=0.2)
    abind(svt4a, svt4b, svt4c)

    ## <5 x 6 x 11 SVT_SparseArray> of type "integer" (nzcount=74):
    ## ,,1
    ##      [,1] [,2] [,3] [,4] [,5] [,6]
    ## [1,]    0    1    0    1    0    3
    ## [2,]    0    0    0    0    0    1
    ## [3,]    1    0    2    2    1    0
    ## [4,]    0    0    2    1    1    2
    ## [5,]    0    1    1    0    0    1
    ##
    ## ...
    ##
    ## ,,11
    ##      [,1] [,2] [,3] [,4] [,5] [,6]
    ## [1,]    0    0    0    2    0    0
    ## [2,]    1    0    0    0    0    0
    ## [3,]    0    0    1    1    0    0
    ## [4,]    2    0    0    0    0    3
    ## [5,]    0    0    0    0    0    0

    svt5a <- aperm(svt4a, c(1, 3:2))
    svt5b <- aperm(svt4b, c(1, 3:2))
    svt5c <- aperm(svt4c, c(1, 3:2))
    abind(svt5a, svt5b, svt5c, along=2)

    ## <5 x 11 x 6 SVT_SparseArray> of type "integer" (nzcount=74):
    ## ,,1
    ##       [,1]  [,2]  [,3]  [,4] ...  [,8]  [,9] [,10] [,11]
    ## [1,]     0     0     0     0   .     0     0     0     0
    ## [2,]     0     0     0     0   .     0     0     0     0
    ## [3,]     1     1     2     0   .     1     1     0     2
    ## [4,]     0     0     0     0   .     0     0     0     0
    ## [5,]     0     0     0     0   .     0     0     0     0
    ##
    ## ...
    ##
    ## ,,6
    ##       [,1]  [,2]  [,3]  [,4] ...  [,8]  [,9] [,10] [,11]
    ## [1,]     0     0     0     0   .     0     0     0     0
    ## [2,]     0     0     0     0   .     0     0     0     0
    ## [3,]     1     0     0     1   .     0     2     1     3
    ## [4,]     0     0     0     0   .     0     0     0     0
    ## [5,]     0     0     0     0   .     0     0     0     0

See `?SparseArray_abind` for more information and additional examples.

## 4.3 Matrix multiplication and cross-product

Like ordinary matrices in base R, SVT_SparseMatrix objects can be multiplied with the `%*%` operator:

    m6 <- matrix(0L, nrow=5, ncol=6, dimnames=list(letters[1:5], LETTERS[1:6]))
    m6[c(2, 6, 12:17, 22:30)] <- 101:117
    svt6 <- SparseArray(m6)

    svt6 %*% svt3

    ##   [,1] [,2]
    ## a  209  220
    ## b  211  222
    ## c  104  115
    ## d  105  116
    ## e  106  117

They also support `crossprod()` and `tcrossprod()`:

    crossprod(svt3)

    ##      [,1] [,2]
    ## [1,]    3    1
    ## [2,]    1    2

See `?SparseMatrix_mult` for more information and additional examples.

## 4.4 matrixStats methods

The SparseArray package provides memory-efficient col/row summarization methods for SVT_SparseMatrix objects:

    colVars(svt6)

    ##      A      B      C      D      E      F
    ## 2040.2 2080.8 2185.3 3467.0 2443.3    2.5

Note that multidimensional objects are supported:

    colVars(svt2)

    ##      A    B   C
    ## a   80 1620 NaN
    ## b  380    0   0
    ## c  500    0   0
    ## d 1520 2000 250

    colVars(svt2, dims=2)

    ##        A        B        C
    ## 732.6316 857.6316      NaN

    colAnyNAs(svt2)

    ##       A     B     C
    ## a FALSE FALSE  TRUE
    ## b FALSE FALSE FALSE
    ## c FALSE FALSE FALSE
    ## d FALSE FALSE FALSE

    colAnyNAs(svt2, dims=2)

    ##     A     B     C
    ## FALSE FALSE  TRUE

See `?matrixStats_methods` for more information and additional examples.

## 4.5 `rowsum()` method

A `rowsum()` method is provided:

    rowsum(svt6, group=c(1:3, 2:1))

    ##     A   B   C   D   E   F
    ## 1   0 102 106 107 112 230
    ## 2 101   0 208 108 220 230
    ## 3   0   0 104   0 110 115

See `?rowsum_methods` for more information and additional examples.

## 4.6 Read/write a sparse matrix from/to a CSV file

Use `writeSparseCSV()` to write a sparse matrix to a CSV file:

    csv_file <- tempfile()
    writeSparseCSV(m6, csv_file)

Use `readSparseCSV()` to read the file. This will import the data as an SVT_SparseMatrix object:

    readSparseCSV(csv_file)

    ## <5 x 6 SVT_SparseMatrix> of type "integer" (nzcount=17):
    ##     A   B   C   D   E   F
    ## a   0 102   0 107   0 113
    ## b 101   0 103 108 109 114
    ## c   0   0 104   0 110 115
    ## d   0   0 105   0 111 116
    ## e   0   0 106   0 112 117

See `?readSparseCSV` for more information and additional examples.

# 5 Comparison with dgCMatrix objects

The nonzero data of a SVT_SparseArray object is stored in a Sparse Vector Tree. This internal data representation is referred to as the “SVT layout”. It is similar to the “CSC layout” (compressed, sparse, column-oriented format) used by CsparseMatrix derivatives from the Matrix package, like dgCMatrix or lgCMatrix objects, but with the following improvements:

- The “SVT layout” supports sparse arrays of arbitrary dimensions.

- With the “SVT layout”, the sparse data can be of any type. Whereas CsparseMatrix derivatives only support sparse data of type `"double"` or `"logical"` at the moment.

- The “SVT layout” imposes no limit on the number of nonzero array elements that can be stored. With dgCMatrix/lgCMatrix objects, this number must be < 2^31.

- Overall, the “SVT layout” allows more efficient operations on SVT_SparseArray objects.

See `?SVT_SparseArray` for more information.

# 6 Learn more

Please consult the individual man pages in the SparseArray package to learn more about SVT_SparseArray objects and about the package. A good starting point is the man page for SparseArray objects: `?SparseArray`

# 7 Session information

    sessionInfo()

    ## R version 4.3.2 (2023-10-31)
    ## Platform: x86_64-pc-linux-gnu (64-bit)
    ## Running under: Ubuntu 22.04.3 LTS
    ##
    ## Matrix products: default
    ## BLAS:   /home/biocbuild/bbs-3.18-bioc/R/lib/libRblas.so
    ## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
    ##
    ## locale:
    ##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
    ##  [3] LC_TIME=en_GB              LC_COLLATE=C
    ##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
    ##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
    ##  [9] LC_ADDRESS=C               LC_TELEPHONE=C
    ## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
    ##
    ## time zone: America/New_York
    ## tzcode source: system (glibc)
    ##
    ## attached base packages:
    ## [1] stats4    stats     graphics  grDevices utils     datasets  methods
    ## [8] base
    ##
    ## other attached packages:
    ##  [1] SparseArray_1.2.2     S4Arrays_1.2.0        IRanges_2.36.0
    ##  [4] abind_1.4-5           S4Vectors_0.40.1      MatrixGenerics_1.14.0
    ##  [7] matrixStats_1.1.0     BiocGenerics_0.48.1   Matrix_1.6-1.1
    ## [10] BiocStyle_2.30.0
    ##
    ## loaded via a namespace (and not attached):
    ##  [1] crayon_1.5.2        cli_3.6.1           knitr_1.45
    ##  [4] rlang_1.1.2         xfun_0.41           jsonlite_1.8.7
    ##  [7] htmltools_0.5.7     sass_0.4.7          rmarkdown_2.25
    ## [10] grid_4.3.2          evaluate_0.23       jquerylib_0.1.4
    ## [13] fastmap_1.1.1       yaml_2.3.7          bookdown_0.36
    ## [16] BiocManager_1.30.22 compiler_4.3.2      XVector_0.42.0
    ## [19] lattice_0.22-5      digest_0.6.33       R6_2.5.1
    ## [22] bslib_0.5.1         tools_4.3.2         zlibbioc_1.48.0
    ## [25] cachem_1.0.8