# Using HDF5-backed matrices with beachmat

Aaron Lun

#### 24 October 2023

#### Package

beachmat.hdf5 1.0.0

# 1 Overview

beachmat.hdf5 provides a C++ API to extract numeric data from HDF5-backed matrices from the HDF5Array package. This extends the beachmat package to the matrix representations in the tatami_hdf5 library. By including this package, users and developers can enable tatami-compatible C++ code to operate natively on file-backed data via the HDF5 C library.

# 2 For users

Users can simply load the package in their R session:

    library(beachmat.hdf5)

This will automatically extend beachmat’s functionality to HDF5Array matrices. Any package code based on beachmat will now be able to access HDF5 data natively without any further work.

# 3 For developers

Developers should read the beachmat developer guide if they have not done so already.

Developers can import beachmat.hdf5 in their packages to guarantee native support for HDF5Array classes. This registers more `initializeCpp()` methods that initializes the appropriate C++ representations for these classes. Of course, this adds some more dependencies to the package, which may or may not be acceptable; some developers may prefer to leave this choice to the user or hide it behind an optional parameter to reduce the installation burden (e.g., if HDF5-backed matrices are not expected to be a common input in the package workflow).

It’s worth noting that beachmat by itself will already work with `HDF5Matrix`, `H5SparseMatrix`, etc. objects even without loading beachmat.hdf5. However, this is not as efficient as any package C++ code needs to go back into R to extract the matrix data via `DelayedArray::extract_array()` and friends. Importing beachmat.hdf5 provides native support without the need for calls to R functions.

# 4 In-memory caching

The `initializeCpp()` methods for the HDF5Array classes have an optional `memorize=` parameter. If this is `TRUE`, the entire matrix is loaded from the HDF5 file into memory and stored in a global cache on first use. Any subsequent calls to `initializeCpp()` on the same matrix instance will re-use the cached value.

In-memory caching is intended for functions or workflows that need to iterate through the matrix multiple times. By setting `memorize=TRUE`, developers can pay an up-front loading cost to avoid the repeated penalty of disk access on subsequent iterations. Obviously, this assumes that the matrix is still small enough that an in-memory store is feasible.

For long-running analyses, users may call `beachmat::flushMemoryCache()` to clear the cache.

# 5 Other comments

beachmat.hdf5 vendors the tatami_hdf5 libraries, which can be made available to package C++ code by including `beachmat.hdf5` in the package `LinkingTo`. However, if this is done, developers should `#include "Rtatami_hdf5.hpp"` rather than including the tatami_hdf5 headers directly. The former will define the appropriate macros for thread-safe access to the HDF5 file.

# Session information

    sessionInfo()

    ## R version 4.3.1 (2023-06-16)
    ## Platform: x86_64-pc-linux-gnu (64-bit)
    ## Running under: Ubuntu 22.04.3 LTS
    ##
    ## Matrix products: default
    ## BLAS:   /home/biocbuild/bbs-3.18-bioc/R/lib/libRblas.so
    ## LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.10.0
    ##
    ## locale:
    ##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
    ##  [3] LC_TIME=en_GB              LC_COLLATE=C
    ##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
    ##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
    ##  [9] LC_ADDRESS=C               LC_TELEPHONE=C
    ## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
    ##
    ## time zone: America/New_York
    ## tzcode source: system (glibc)
    ##
    ## attached base packages:
    ## [1] stats     graphics  grDevices utils     datasets  methods   base
    ##
    ## other attached packages:
    ## [1] beachmat.hdf5_1.0.0 knitr_1.44          BiocStyle_2.30.0
    ##
    ## loaded via a namespace (and not attached):
    ##  [1] crayon_1.5.2          cli_3.6.1             rlang_1.1.1
    ##  [4] xfun_0.40             jsonlite_1.8.7        DelayedArray_0.28.0
    ##  [7] S4Vectors_0.40.0      htmltools_0.5.6.1     sass_0.4.7
    ## [10] stats4_4.3.1          MatrixGenerics_1.14.0 rmarkdown_2.25
    ## [13] grid_4.3.1            abind_1.4-5           evaluate_0.22
    ## [16] jquerylib_0.1.4       fastmap_1.1.1         Rhdf5lib_1.24.0
    ## [19] IRanges_2.36.0        yaml_2.3.7            bookdown_0.36
    ## [22] BiocManager_1.30.22   compiler_4.3.1        Rcpp_1.0.11
    ## [25] rhdf5filters_1.14.0   XVector_0.42.0        rhdf5_2.46.0
    ## [28] lattice_0.22-5        digest_0.6.33         R6_2.5.1
    ## [31] SparseArray_1.2.0     HDF5Array_1.30.0      bslib_0.5.1
    ## [34] Matrix_1.6-1.1        tools_4.3.1           beachmat_2.18.0
    ## [37] matrixStats_1.0.0     zlibbioc_1.48.0       S4Arrays_1.2.0
    ## [40] BiocGenerics_0.48.0   cachem_1.0.8