# Supporting arbitrary matrix classes

Aaron Lun

#### 30 October 2018

#### Package

beachmat 1.4.0

# 1 Background

Ordinarily, direct support for an input matrix class would require the appropriate methods to be defined in beachmat at compile time. This is the case for the most widely used matrix classes but is somewhat restrictive for other community-contributed matrix representations. Fortunately, R provides a mechanism to link across shared libraries from different packages. This means that package developers who define a R matrix type can also define C++ methods for direct input support in beachmat-dependent code. By doing so, we can improve efficiency of access to these new classes by avoiding the need for block processing via R.

A functioning demonstration of this approach is available in the `extensions` test package. This vignette will provide an explanation of the code in `extensions`, and we suggest examining the source code at the same time:

    system.file("extensions", package="beachmat")

    ## [1] "/tmp/RtmpS3CKSB/Rinst3ae91d9a18df/beachmat/extensions"

# 2 Setting up in R

Assume that we have already defined a new matrix-like S4 class (here, `AaronMatrix`). To notify the beachmat API that direct access support is available, we need to:

- define a method for the `supportCppAccess()` generic (from beachmat) for this class. This should return `TRUE` if direct support is available (obviously).
- define a method for the `type()` generic from the DelayedArray package. This should return the type of the matrix, i.e., integer, logical, numeric or character.

It is possible to only have direct support for particular data types of the given matrix representation. The example in `extensions` only directly supports integer and character `AaronMatrix` objects11 Because I was too lazy to add all of them. and will only return `TRUE` for such types.

# 3 Setting up in C++

We will use integer matrices for demonstration, though it is simple to generalize this to all types by replacing `_integer` with, e.g., `_character` 22 Some understanding of C++ templates will greatly simplify the definition of the same methods for different types.. First, we define a `create()` function that takes a `SEXP` object and returns a `void` pointer. This should presumably point to some C++ class that can contain intermediate data structures for efficient access.

    void * ptr = create_integer(in /* SEXP */);

We define a `clone()` function that performs a deep copy of the aforementioned pointer.

    void * ptr_copy = clone_integer(ptr /* void* */);

We define a `destroy()` function the frees the memory pointed to by `ptr`.

    destroy_integer(ptr /* void* */);

We define a `get_dim()` function that records the number of rows and columns in the object pointed to by `ptr`. Note the references on the `size_t&` arguments.

    get_dim_integer(
        ptr, /* void* */
        nrow, /* size_t& */
        ncol /* size_t& */
    );

# 4 Defining getter methods

## 4.1 For all types

In general, the getter methods follow the same structure as that described for the input API. We expect a `load()` method to obtain a specified entry of the matrix:

    int val = load_integer(
        ptr, /* void* */
        r, /* size_t */
        c /* size_t */
    );

The returned `val` should reflect the matrix type. For example, `val` should be a `Rcpp::String` for character matrices, a `double` for numeric matrices, and an `int` for logical matrices.

Developers can assume that `r` and `c` are valid, i.e., within `[0, nrow)` and `[0, ncol)` respectively. These checks are performed by beachmat and do not have to be repeated within developer-defined functions33 Obviously, the dimensions of the matrix pointed to by `ptr` should not change!.

## 4.2 For non-numeric types

Here, we will use character matrices44 Character matrices tend to require some special attention, as character arrays need to be coerced to `Rcpp::String` objects to be returned in `in`. as an example. We expect a `load_col()` method to obtain a column of the matrix:

    load_col_character(
        ptr, /* void* */
        c, /* size_t */
        in, /* Rcpp::StringVector::iterator */
        first, /* size_t */
        last /* size_t */
    );

… and a `load_row()` method to obtain a row of the matrix:

    load_row_character(
        ptr, /* void* */
        r, /* size_t */
        in, /* Rcpp::StringVector::iterator */
        first, /* size_t */
        last /* size_t */
    );

We expect a `load_cols()` method to obtain multiple columns:

    load_cols_character(
        ptr, /* void* */
        indices, /* Rcpp::IntegerVector::iterator */
        n, /* size_t */
        in, /* Rcpp::StringVector::iterator */
        first, /* size_t */
        last /* size_t */
    );

… and a `load_rows()` method to obtain multiple rows:

    load_cols_character(
        ptr, /* void* */
        indices, /* Rcpp::IntegerVector::iterator */
        n, /* size_t */
        in, /* Rcpp::StringVector::iterator */
        first, /* size_t */
        last /* size_t */
    );

In all cases, `first` and `last` can be assumed to be valid, i.e., `first <= last` and both in `[0, nrow)` or `[0, ncol)` (for column and row access, respectively). Indices in `indices` can also be assumed to be valid, i.e., within matrix dimensions and strictly increasing.

## 4.3 Numeric types

For integer, logical or numeric matrices, we need to account for type conversions. This is done by defining the following functions (using integer matrices as an example):

- `load_col2int_integer`
- `load_col2dbl_integer`
- `load_row2int_integer`
- `load_row2dbl_integer`
- `load_cols2int_integer`
- `load_cols2dbl_integer`
- `load_rows2int_integer`
- `load_rows2dbl_integer`

Taking the single-column getter as an example:

    load_col2int_character(
        ptr, /* void* */
        c, /* size_t */
        in, /* Rcpp::IntegerVector::iterator */
        first, /* size_t */
        last /* size_t */
    );

    load_col2dbl_character(
        ptr, /* void* */
        c, /* size_t */
        in, /* Rcpp::NumericVector::iterator */
        first, /* size_t */
        last /* size_t */
    );

We explicitly define conversions here as the cross-library linking framework does not support templating of `in`. If we only defined a `load_col()` function of the same type, we would need to perform two copies: once to copy to an integer vector, and then another to copy to the output double-precision vector.

# 5 Defining special getters

## 5.1 Constant column access

We define a `load_const_col()` function to obtain an iterator to a contiguous stretch of memory defining a column of the matrix. Again, `ptr` is a pointer to the location in memory containing the matrix object. All other arguments are as described in the previous workflow.

    Rcpp::IntegerVector::iterator out = load_const_col_integer(
        ptr, /* void* */
        c, /* size_t */
        in, /* Rcpp::IntegerVector::iterator */
        first, /* size_t */
        last /* size_t */
    );

This function should only be special for representations where entire columns are stored contiguously. All other representations should simply copy data into `in`. It is unwise to try to be too smart, as the returned iterator `out` must be valid throughout the lifetime of the matrix. This means that, if all columns were accessed, the entire matrix would need to be stored in memory to ensure that all iterators were valid. Such a strategy means that any matrix representation will effectively become a dense array.

Obviously, `out` and `in` should reflect the matrix type. For example, for `load_const_col_character()`, both of them should be `Rcpp::StringVector::iterator` objects.

## 5.2 Indexed column access

We define a `load_const_col_indexed()` function to obtain iterators to “non-zero” elements of the matrix55 See the previous workflow to clarify the definition of non-zero.. Again, `ptr` is a pointer to the location in memory containing the matrix object. The number of indexed elements should be returned in `n`.

    size_t n = load_const_col_integer(
        ptr, /* void* */
        c, /* size_t */
        index, /* Rcpp::IntegerVector::iterator& */
        values, /* Rcpp::IntegerVector::iterator& */
        first, /* size_t */
        last /* size_t */
    );

Note that both `index` and `values` are references to iterator objects. They should be modified to point to internal data structures in `ptr`. The modified values will then be returned as part of the output of `get_const_col_indexed()` in the input API.

This function should only be special for representations where non-zero values and their indices are stored by column (i.e., variants of column-sparse compressed matrices). All other representations should simply copy data into `values`, and set `index` to a sequence of integers from `[first, last)`. It is possible to streamline the setting of `index` by creating an increasing sequence once and storing it in `ptr` - see the example in `extensions` for more details.

Obviously, `values` should reflect the matrix type. For example, for `load_const_col_character()`, it should be a `Rcpp::StringVector::iterator&` reference.

# 6 Ensuring discoverability

We use the `R_RegisterCCallable()` function from the R API to register the above functions (see here for an explanation). This ensures that they can be found by beachmat when an `AaronMatrix` instance is encountered. Note that the functions must be defined with C-style linkage in order for this procedure to work properly, hence the use of `extern "C"`.

Needless to say, the `NAMESPACE` should contain an appropriate `useDynLib` command. This means that shared library will be loaded along with the package, allowing beachmat to access the registered routines within.