# Developing around the SingleCellExperiment class

Davide Risso and Aaron Lun

#### 30 October 2018

#### Package

SingleCellExperiment 1.4.0

# 1 Introduction

By design, the scope of this package is limited to defining the `SingleCellExperiment` class and some minimal getter and setter methods. For this reason, we leave it to developers of specialized packages to provide more advanced methods for the `SingleCellExperiment` class. For example, scater defines a number of specific methods such as `normalize` and dplyr-like verbs. We also leave it to the developers of existing packages to provide coercion methods from their classes to `SingleCellExperiment`.

The use of `SingleCellExperiment` objects inside package functions is mostly the same as the use of instances of the base `SummarizedExperiment` class. The only exceptions involve direct access to the internal fields of the `SingleCellExperiment` definition. Manipulation of these internal fields in other packages is possible but requires some caution, as we shall discuss below.

# 2 Using the internal fields

## 2.1 Rationale

We use an internal storage mechanism to protect the spike-in and size factor fields from direct manipulation by the user. This ensures that, e.g., only a call to `sizeFactors<-` can change the size factors. The same effect could be achieved by reserving a subset of columns (or column names) as “private” in `colData()` and `rowData()`, though this is not easily implemented.

The internal storage avoids situations where users or functions can silently overwrite these important metadata fields during manipulations of `rowData` or `colData`. This can result in bugs that are difficult to track down, particularly in long workflows involving many functions. It also allows us to add new methods and metadata types to `SingleCellExperiment` without worrying about overwriting user-supplied metadata in existing objects.

Methods to get or set the internal fields are exported for use by developers of packages that depend on SingleCellExperiment. This allows dependent packages to store their own custom fields that are not meant to be directly accessible by the user. However, this requires some care to avoid conflicts between packages.

## 2.2 Conflicts between packages

The concern is that package A and B both define methods that get/set an internal field `X` in a `SingleCellExperiment` instance. Consider the following example object:

    library(SingleCellExperiment)
    counts <- matrix(rpois(100, lambda = 10), ncol=10, nrow=10)
    sce <- SingleCellExperiment(assays = list(counts = counts))
    sce

    ## class: SingleCellExperiment
    ## dim: 10 10
    ## metadata(0):
    ## assays(1): counts
    ## rownames: NULL
    ## rowData names(0):
    ## colnames: NULL
    ## colData names(0):
    ## reducedDimNames(0):
    ## spikeNames(0):

Assume that we have functions that set an internal field `X` in packages A and B.

    # Function in package A:
    AsetX <- function(sce) {
        int_colData(sce)$X <- runif(ncol(sce))
        sce
    }

    # Function in package B:
    BsetX <- function(sce) {
        int_colData(sce)$X <- sample(LETTERS, ncol(sce), replace=TRUE)
        sce
    }

If both of these functions are called, one will clobber the output of the other. This may lead to nonsensical results in downstream procedures.

    sce2 <- AsetX(sce)
    int_colData(sce2)$X

    ##  [1] 0.3774103 0.3056382 0.8796171 0.9862273 0.7325031 0.3756312 0.1708986
    ##  [8] 0.2595704 0.6406194 0.9399857

    sce2 <- BsetX(sce2)
    int_colData(sce2)$X

    ##  [1] "C" "O" "D" "D" "G" "Y" "J" "X" "I" "I"

## 2.3 Using “Inception-style” nesting

We recommend using nested `DataFrame`s to store internal fields in the column-level metadata. The name of the nested element should be set to the package name, thus avoiding clashes between fields with the same name from different packages.

    AsetX_better <- function(sce) {
        int_colData(sce)$A <- DataFrame(X=runif(ncol(sce)))
        sce
    }

    BsetX_better <- function(sce) {
        choice <- sample(LETTERS, ncol(sce), replace=TRUE)
        int_colData(sce)$B <- DataFrame(X=choice)
        sce
    }

    sce2 <- AsetX_better(sce)
    sce2 <- BsetX_better(sce2)
    int_colData(sce2)$A$X

    ##  [1] 0.26005411 0.05588541 0.87922192 0.58440349 0.13547021 0.05622731
    ##  [7] 0.33658376 0.18886351 0.73448241 0.32462765

    int_colData(sce2)$B$X

    ##  [1] "C" "V" "E" "P" "H" "O" "W" "R" "B" "T"

The same approach can be applied to the row-level metadata, e.g., for some per-row field `Y`.

    AsetY_better <- function(sce) {
        int_elementMetadata(sce)$A <- DataFrame(Y=runif(nrow(sce)))
        sce
    }

    BsetY_better <- function(sce) {
        choice <- sample(LETTERS, nrow(sce), replace=TRUE)
        int_elementMetadata(sce)$B <- DataFrame(Y=choice)
        sce
    }

    sce2 <- AsetY_better(sce)
    sce2 <- BsetY_better(sce2)
    int_elementMetadata(sce2)$A$Y

    ##  [1] 0.7141494 0.1771398 0.9900464 0.8537075 0.3864944 0.4062250 0.5435682
    ##  [8] 0.1337825 0.5297460 0.9118756

    int_elementMetadata(sce2)$B$Y

    ##  [1] "O" "D" "V" "U" "R" "D" "A" "U" "Y" "V"

For the object-wide metadata, a nested list is usually sufficient.

    AsetZ_better <- function(sce) {
        int_metadata(sce)$A <- list(Z = "Aaron")
        sce
    }

    BsetZ_better <- function(sce) {
        int_metadata(sce)$B <- list(Z = "Davide")
        sce
    }

    sce2 <- AsetZ_better(sce)
    sce2 <- BsetZ_better(sce2)
    int_metadata(sce2)$A$Z

    ## [1] "Aaron"

    int_metadata(sce2)$B$Z

    ## [1] "Davide"

In this manner, both A and B can set their internal `X`, `Y` and `Z` without interfering with each other. Of course, this strategy assumes that packages do not have the same names as some of the in-built internal fields (which would be very unfortunate).

# 3 Contacting us

If your package accesses the internal fields of the `SingleCellExperiment` class, we suggest you get into contact with us on GitHub. This will help us in planning changes to the internal organization of the class. It will also allow us to contact you with respect to changes or to get feedback.

We are particularly interested in scenarios where multiple packages are defining internal fields with the same scientific meaning. In such cases, it may be valuable to provide getters and setters for this field in SingleCellExperiment directly. This reduces redundancy in the definitions across packages and promotes interoperability. For example, methods from one package can set the field, which can then be used by methods of another package.

# 4 Other design decisions

## 4.1 What’s up with `reducedDims`?

We use a `SimpleList` as the `reducedDims` slot to allow for multiple dimensionality reduction results. One can imagine that different dimensionality reduction techniques will be useful for different aspects of the analysis, e.g., t-SNE for visualization, PCA for pseudo-time inference. We see `reducedDims` as a similar slot to `assays()` in that multiple matrices can be stored, though the dimensionality reduction results need not have the same number of dimensions.

## 4.2 Why derive from a `RangedSummarizedExperiment`?

We decided to extend `RangedSummarizedExperiment` rather than `SummarizedExperiment` because for certain assays it will be essential to have `rowRanges()`. Even for RNA-seq, it is sometimes useful to have `rowRanges()` and other classes to define the genomic coordinates, e.g., `DESeqDataSet` in the DESeq2 package. An alternative would have been to have two classes, `SingleCellExperiment` and `RangedSingleCellExperiment`. However, this seems like an unnecessary duplication as having a class with default empty `rowRanges` seems good enough when one does not need `rowRanges`.

# 5 Session information

    sessionInfo()

    ## R version 3.5.1 Patched (2018-07-12 r74967)
    ## Platform: x86_64-pc-linux-gnu (64-bit)
    ## Running under: Ubuntu 16.04.5 LTS
    ##
    ## Matrix products: default
    ## BLAS: /home/biocbuild/bbs-3.8-bioc/R/lib/libRblas.so
    ## LAPACK: /home/biocbuild/bbs-3.8-bioc/R/lib/libRlapack.so
    ##
    ## locale:
    ##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
    ##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=C
    ##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8
    ##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
    ##  [9] LC_ADDRESS=C               LC_TELEPHONE=C
    ## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
    ##
    ## attached base packages:
    ## [1] parallel  stats4    stats     graphics  grDevices utils     datasets
    ## [8] methods   base
    ##
    ## other attached packages:
    ##  [1] SingleCellExperiment_1.4.0  SummarizedExperiment_1.12.0
    ##  [3] DelayedArray_0.8.0          BiocParallel_1.16.0
    ##  [5] matrixStats_0.54.0          Biobase_2.42.0
    ##  [7] GenomicRanges_1.34.0        GenomeInfoDb_1.18.0
    ##  [9] IRanges_2.16.0              S4Vectors_0.20.0
    ## [11] BiocGenerics_0.28.0         BiocStyle_2.10.0
    ##
    ## loaded via a namespace (and not attached):
    ##  [1] Rcpp_0.12.19           knitr_1.20             XVector_0.22.0
    ##  [4] magrittr_1.5           zlibbioc_1.28.0        lattice_0.20-35
    ##  [7] stringr_1.3.1          tools_3.5.1            grid_3.5.1
    ## [10] xfun_0.4               htmltools_0.3.6        yaml_2.2.0
    ## [13] rprojroot_1.3-2        digest_0.6.18          bookdown_0.7
    ## [16] Matrix_1.2-14          GenomeInfoDbData_1.2.0 BiocManager_1.30.3
    ## [19] bitops_1.0-6           RCurl_1.95-4.11        evaluate_0.12
    ## [22] rmarkdown_1.10         stringi_1.2.4          compiler_3.5.1
    ## [25] backports_1.1.2