---
title: "multiHiCcompare Vignette"
author: "John C. Stansfield & Mikhail G. Dozmorov"
package: multiHiCcompare
output: 
  BiocStyle::html_document
vignette: >
  %\VignetteIndexEntry{multiHiCcompare Vignette}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}  
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

```{r, echo=FALSE, message=FALSE}
library(multiHiCcompare)
```


# Introduction

`multiHiCcompare` is an extension of the original `HiCcompare` package. It provides functions for the joint normalization and detection of differential chromatin interactions between multiple Hi-C datasets. `multiHiCcompare` operates on processed Hi-C data in the form of chromosome-specific chromatin interaction matrices. It accepts four-column tab-separated text files storing chromatin interaction matrices in a sparse matrix format (see [Creating the hicexp object](#hicexp)). Functions to convert popular Hi-C data formats (`.hic`, `.cool`) to sparse format are available (see ?cooleHCT116_r2sparse). `multiHiCcompare` differs from other packages that attempt to compare Hi-C data in that it works on processed data in chromatin interaction matrix format instead of raw sequencing data. In addition, `multiHiCcompare` provides a non-parametric method for the joint normalization and removal of biases between multiple Hi-C datasets for the purpose of comparative analysis. `multiHiCcompare` also provides a general linear model (GLM) based framework for detecting differences in Hi-C data.

# How to use `multiHiCcompare`

## Install `multiHiCcompare` from Bioconductor

```{r, eval = FALSE}
BiocManager::install("multiHiCcompare")
library(multiHiCcompare)
```

## Getting Hi-C Data

You will need processed Hi-C data in the form of sparse upper triangular matrices or BEDPE files in order to use `multiHiCcompare`. Data is available from several sources and two examples for downloading and extracting data are listed below. If you have full Hi-C contact matrices you can convert them to sparse upper triangular format using the full `full2sparse` function as shown in [additional functions](#addfunc)

### Extracting data from `.hic` files

Hi-C data is available from several sources and in many formats. `multiHiCcompare` is built to work with the sparse upper triangular matrix format popularized by the lab of Erez Lieberman-Aiden [http://aidenlab.org/data.html](http://aidenlab.org/data.html).  If you already have Hi-C data either in the form of a sparse upper triangular matrix or a full contact matrix you can skip to the creating the `hicexp` object section. If you obtain data from the Aiden Lab in the `.hic` format you will need to first extract the matrices that you wish to compare. 

  1. Download the `straw` software from [https://github.com/theaidenlab/straw/wiki](https://github.com/theaidenlab/straw/wiki) and install it.
  2. Use `straw` to extract a Hi-C sparse upper triangular matrix. An example is below:
  
Say we downloaded the `GSE63525_K562_combined_30.hic ` file from GEO [https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE63525](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE63525)  

To extract the raw matrix corresponding to chromosome 22 at 500kb resolution we would use the following command within the terminal

  `./straw NONE GSE63525_K562_combined_30.hic  22 22 BP 500000 > K562.chHCT116_r22.500kb.txt`

This will extract the matrix from the `.hic` file and save it to the `K562.chHCT116_r22.500kb.txt` text file, in the sparse upper triangular matrix format. See more examples on on how to use `straw` at [https://github.com/theaidenlab/straw/wiki/CPP#running](https://github.com/theaidenlab/straw/wiki/CPP#running). Straw requires several inputs for the extraction of data from a `.hic` file.

`<NONE/VC/VC_SQRT/KR> <hicFile(s)> <chr1>[:x1:x2] <chr2>[:y1:y2] <BP/FRAG> <binsize>`

The first argument is the normalization method. For use in `multiHiCcompare` you want the raw data so you should selected `NONE`. The second argument is the `.hic` file name. Next is the chromosome numbers of the matrix you want. For an intrachromosomal contact map both should be the same as in the above example. If you want a matrix of interchromosomal interactions you can use different chromosomes i.e. interactions between chromosome 1 and chromosome 2 (Note that `HiCcompare` is only meant to be used on intrachromosomal interactions at this point in development). The next argument is whether you want basepair or fragment files. For `multiHiCcompare` use `BP`. The final argument is the binsize of the matrix (the resolution). To extract a matrix at a resolution of 1MB enter `10000000`. Typical bin sizes are 1MB, 500KB, 100KB, 50KB, 5KB, 1KB. Note that most matrices with resolutions higher than 100KB (i.e. matrices with resolutions of 1KB - 50KB) are typically too sparse (due to insufficient sequencing coverage) for analysis in `multiHiCcompare`. 

From here we can just import the matrix into R as you would normally for any tab-delimited file.

  3. Import the data into R `K562.chr22 <- read.table('K562.chr22.500kb.txt', header=FALSE)`
  4. Repeat these steps for any other Hi-C dataset that you wish to compare to the first dataset using  `multiHiCcompare`.
 
### Extracting data from `.cool` files

The `cooler` software, [http://cooler.readthedocs.io/en/latest/index.html](http://cooler.readthedocs.io/en/latest/index.html), allows access to a large collection of Hi-C data. The cooler index [ftp://cooler.csail.mit.edu/coolers](ftp://cooler.csail.mit.edu/coolers) contains Hi-C data for `hg19` and `mm9` from many different sources. To use data in the `.cool` format in `HiCcompare` follow these steps:

  1. Download and install `cooler` from [http://cooler.readthedocs.io/en/latest/index.html](http://cooler.readthedocs.io/en/latest/index.html)
  2. Download a `.cool` file from the cooler index [ftp://cooler.csail.mit.edu/coolers](ftp://cooler.csail.mit.edu/coolers).
  3. Say we downloaded the `Dixon2012-H1hESC-HindIII-allreps-filtered.1000kb.cool` file. See `cooler dump --help` for data extraction options. To extract the contact matrix we use the following commands in the terminal:  
  `cooler dump --join Dixon2012-H1hESC-HindIII-allreps-filtered.1000kb.cool > dixon.hESC.1000kb.txt`
  4. Read in the text file as you would any tab-delimited file in R  
  `hesc1000kb <- read.table("dixon.hESC.1000kb.txt", header = FALSE)`
  5. Convert to a sparse upper triangular matrix using the `HiCcompare::cooler2sparse` function.  
  `sparse <- cooler2sparse(hesc1000kb) `
  6. Repeat the steps for another Hi-C dataset that you wish to compare to the first dataset.


### Using data from HiC-Pro

HiC-Pro is another tool for processing raw Hi-C data into usable matrix files. HiC-Pro will produce a `.matrix` file and a `.bed` file for the data. These `.matrix` files are in sparse upper triangular format similar to the results of Juicer and the dumped contents of a `.hic` file, however instead of using the genomic start coordinates for the first two columns of the sparse matrix they use an ID number. The `.bed` file contains the mappings for each of these IDs to their genomic coordinates. The original `HiCcompare` includes a function to convert the results of HiC-Pro into a usable format for analysis in `multiHiCcompare`. When using data from HiC-Pro it is important to use the raw `.matrix` files and NOT the iced `.matrix` files. The iced `.matrix` files have already had ICE normalization applied to them and are not suitable for entry into `multiHiCcompare`. Here we convert HiC-Pro data for input into `multiHiCcompare`:

```{r, eval=FALSE}
# read in files
mat <- read.table("hic_1000000.matrix")
bed <- read.table("hic_1000000_abs.bed")
# convert to BEDPE
dat <- HiCcompare::hicpro2bedpe(mat, bed)
# NOTE: hicpro2bedpe returns a list of lists. 
#   The first list, dat$cis, contains the intrachromosomal contact matrices
# NOTE: dat$trans contains the interchromosomal 
#   contact matrix which is not used in multiHiCcompare.
```


See the help using `?HiCcompare::hicpro2bedpe` for more details.


## Parallel Processing

Hi-C data is large, especially at high resolutions, and loess normalization is computationally intensive. `multiHiCcompare` was built with parallelization in mind and the best performance when working with large Hi-C experiments (many samples or high resolution) will be achieved when using a computing cluster. Parallel processing can be used for all normalization and comparison functions by setting `parallel = TRUE` in the function options. `multiHiCcompare` uses the Bioconductor `BiocParallel` package for parallel processing. You can set the number of processors to use on linux with the following command:

```{r, eval = FALSE}
library(BiocParallel)
numCores <- 20
register(MulticoreParam(workers = numCores), default = TRUE)
```

Or on Windows with:

```{r, eval = FALSE}
library(BiocParallel)
numCores <- 20
register(SnowParam(workers = numCores), default = TRUE)
```

where `numCores` is the user-set number of processing cores to be used. For parallel processing in `multiHiCcompare`, jobs are split by chromosome and sometimes distance thus the more processors used, the quicker the function will run. For maximum speed it is recommended to set `numCores` to the maximum number of processors available. 

## Creating the `hicexp` object {#hicexp}

### Sparse upper triangular format

A sparse matrix format represents a relatively compact and human-readable way to store pair-wise interactions. It is a tab-delimited text format containing three columns: "region1" - a start coordinate (in bp) of the first region, "region2" a start coordinate of the second region, and "IF" - the interaction frequency between them (IFs). Zero IFs are dropped (hence, the _sparse_ format). Since the full matrix of chromatin interactions is symmetric, only the upper triangular portion, including the diagonal, is stored. Typically matrices in this format are stored in their own separate text files for each chromosome. For use in `multiHiCcompare` you will need to add an additional column for the chromosome number. The chromosome number should be entered as just the number. Chromosomes such as X, Y, etc. should be entered as 23, 24, etc. If you are planning to analyze data for more than a single chromosome you will need to concatenate these matrices together. A sparse Hi-C matrix ready to be input into `multiHiCcompare` should look like the following:

```{r}
data("HCT116_r1") # load example sparse matrix
head(HCT116_r1)
colnames(HCT116_r1) <- c('chr', 'region1', 'region2', 'IF') # rename columns
head(HCT116_r1) # matrix ready to be input into multiHiCcompare
```


If you have full Hi-C contact matrices you can convert them to sparse upper triangular matrices using the `HiCcompare::full2sparse` function and then add an additional column indicating the chromosome.


Say we have data from 2 experimental conditions with 2 samples each. We can make a `hicexp` object by doing the following.

```{r}
data("HCT116_r1", "HCT116_r2", "HCT116_r3", "HCT116_r4")
hicexp1 <- make_hicexp(HCT116_r1, HCT116_r2, HCT116_r3, HCT116_r4, 
                       groups = c(0, 0, 1, 1), 
                       zero.p = 0.8, A.min = 5, filter = TRUE,
                       remove.regions = hg19_cyto)
hicexp1
```

The `groups` option specifies the experimental groups. You must enter a vector the length of the number of Hi-C matrices with indicators for which group each matrix belongs to. An optional covariate `data.frame` with rows corresponding the Hi-C matrices and columns for each additional covariate can be provided with the `covariates` option. 

Filtering can be performed when creating a `hicexp` object using the `zero.p` and `A.min` options in the `make_hicexp` function. The `zero.p` option allows for filtering by the proportion of zero IFs for an interaction. The `A.min` allows for filtering by a minimum average IF value. These options can be used together or individually to filter your data. Filtering is important to remove interactions with lots of 0 IFs and low average expression. These interactions tend to not be very interesting and can easily become a false positive during difference detection. Additionally, removing these interactions will increase the computational speed of `multiHiCcompare`. If for some reason you do not want to filter the data simply set `filter = FALSE`.

Additionally, you can filter out specific genomic regions such as centromeres or black listed regions. `multiHiCcompare` comes with built in regions to be filtered for hg19 and hg38 which can be accessed like so.

```{r}
data("hg19_cyto")
data("hg38_cyto")

hg19_cyto
```

By default the `make_hicexp` object will have the `remove.regions` option set to use the `hg19_cyto` object. If your data was not aligned to hg19 or you want other regions to be removed you can create a `GenomicRanges` object containing the ranges to be removed and the `remove.regions` option to this object. 



### The `hicexp` object

The `hicexp` S4 class has several slots which can be accessed with the accessor functions `hic_table()`, `results()`, and `meta()`. The `hic_table` slot contains the Hi-C matrix in sparse format. The first four columns are the chromosome, region1 start location, region2 start location, and unit distance. All following chromosomes represent the IFs for that interacting pair from each sample. The `comparison` slot is empty at creation, but will be filled following use of one of the comparison functions. It contains the same first four columns as the `hic_table` slot, but also has the logFC - log fold change between conditions, logCPM - log counts per million, p.value, and p.adj - multiple testing corrected p-value columns which indicate the significance of the difference for each interacting pair of regions between the conditions. Access the `comparison` slot using `results()`. The `metadata` slot contains the `data.frame` of covariates for the experiment. Access the `metadata` slot by using `meta()`. The other slots are mainly for internal use and the typical user does not need to be concerned with them. 



## Normalization

`multiHiCcompare` comes with a few methods for normalizing your Hi-C data. Our joint normalization methods are again based on the MD plot as in the original `HiCcompare`. The MD plot is similar to the MA plot or the Bland-Altman plot. $M$ is the $log2$ difference between the interaction frequencies from the two datasets. $D$ is the unit distance between the two interacting regions. Loess is performed on the data after it is represented in the MD coordinate system. 

### Library scaling

The simplest form of normalization for the purpose of comparing Hi-C data is library scaling. `multiHiCcompare` provides the `hic_scale()` function to scale the Hi-C libraries from each sample to the size of the smallest library. If you believe that any trends present in your data are important differences and not due to bias then you can use library scaling for normalizing your data. Otherwise it is recommended to use either cyclic loess or fast loess. You can perform library scaling as follows. Note that you need to pick a single normalization method. If you use `hic_scale` you will not be able to use cyclic loess normalization. 

```{r, eval=FALSE}
data("hicexp2")
hicexp2 <- hic_scale(hicexp2)
```


### Cyclic Loess Normalization

`multiHiCcompare` provides a cyclic loess method for the joint normalization of multiple Hi-C datasets. The method is based on representing the data on an MD plot. The MD plot is similar to the MA plot (Bland-Altman plot) which is commonly used for the visualization of gene expression differences. $M$ is defined as the log difference between the two data sets $M = log_2(IF_2/IF_1)$, where $IF_1$ and $IF_2$ are interaction frequencies of the first and the second Hi-C datasets, respectively. $D$ is defined as the distance between two interacting regions, expressed in unit-length of the $X$ resolution of the Hi-C data. A loess regression curve is fit through the MD plot and used to remove global biases by centering the $M$ differences around $M=0$ baseline.

The cyclic loess algorithm proceeds through the following steps. 

1. Choose two out of the $N$ total samples then generate an MD plot. 
2. Fit a loess curve $f(d)$ to the MD plot. 
3. Subtract $f(d)/2$ from the first dataset and add $f(d)/2$ to the second. 
4. Repeat until all unique pairs have been compared. 
5. Repeat until convergence. 

To perform cyclic loess on your Hi-C data you will need to use the `cyclic_loess()` function as shown below:

```{r}
hicexp1 <- cyclic_loess(hicexp1, verbose = FALSE, 
                        parallel = FALSE, span = 0.2)
# make MD plot
MD_hicexp(hicexp1)
```

As can be seen in the above MD plots, the data for each sample has been jointly normalized with all other samples. Note that the span option can be set by the user. A user set span will run quicker than the default option of automatically calculating the span. It is best to use the automatic span calculation if you have not worked with the data before, but if you are familiar with it then setting the span is a way to speed up processing. The `hic_table` slot in the `hicexp` object has also been updated with the normalized IFs.

```{r}
hic_table(hicexp1)
```

The run time of cyclic loess can be decreased when multiple processors are available by setting the `parallel` option to `TRUE`. This option splits up the data by chromosome and sends each chromosome's data to a parallel processor. 

### Fast Loess Normalization (Fastlo)

In addition to the standard cyclic loess method, `multiHiCcompare` also implements the Fast Loess (Fastlo) joint normalization algorithm. Our implementation of fastlo is adapted to Hi-C data on a per-distance basis. To perform "fastlo" on Hi-C data we first split the data into $p$ pooled matrices. The "progressive pooling" is used to split up the Hi-C matrix by unit distance such that distance 0 is its own pool, distances 1 and 2 are pooled, distance 3, 4, 5 are pooled, and so on until all unit distances belong to one of $p$ pools. Each matrix will have an $IF_{gj}$ value with $g$ interacting pairs for each of the $j$ samples. These $p$ matrices can then be input into the "fastlo" algorithm using the following steps. 

1. Create the vector $\hat{IF}_{pgj}$, the row means of the $p^{th}$ matrix. This is the equivalent of creating an average IF at distance pool $p$. 
2. Plot $\hat{IF}_{p}$ versus $(IF_{pg} - \hat{IF_p})$ for each sample $j$. This is equivalent to an MA plot at a genomic distance pool $p$. 
3. Fit a loess curve $f(x)$ to the plot. 
4. Subtract $f(x)$ from sample $j$. 
5. Repeat for all remaining replicates. 
6. Repeat until algorithm converges. 

You can perform fastlo normalization on your data as follows:


```{r}
data("hicexp2")
# perform fastlo normalization
hicexp2 <- fastlo(hicexp2, verbose = FALSE, parallel = FALSE)
# make MD plot
MD_hicexp(hicexp2)
```


Again, the above MD plots show the normalized data. `fastlo()` can also make use of parallelization to speed up computation speeds by setting the `parallel` option to `TRUE`. The results of `fastlo()` and `cyclic_loess()` may be slightly different, but both should result in the removal of biases between Hi-C datasets. `fastlo()` will have quicker run times compared to `cyclic_loess()`, but `cyclic_loess()` will likely give a slightly better normalization.  


## Difference Detection

`multiHiCcompare` provides two main ways to perform a differential comparison between the groups or conditions of your Hi-C experiment. For simple experiments where only a comparison between two groups is being made, the `hic_exactTest()` function can be used. For more complex experiments with covariates or multiple groups, the `hic_glm()` function should be used. Both of these functions make use of the `edgeR` package for fitting negative binomial models to the Hi-C data. For the difference detection steps, `multiHiCcompare` first splits the data up by distance using the progressive pooling described in the fastlo section. Each distance pool is then treated similar to an independent RNA-seq data matrix on which `edgeR`'s functions are applied to fit the specified model. This process is illustrated in Figure 1 below.

![](multiHiCcompare_glm.png)

**Figure 1.** The off-diagonal analysis of multiple Hi-C replicates.Dashed lines represent the off-diagonal vectors of interaction frequencies at a given distance between interacting regions. Right: Converted into a matrix format similar to RNA-seq data, IFs can be loess normalized, variance across replicates can be estimated using an empirical Bayes approach and differences can be detected using log-linear GLMs. 


### Exact Test

For simple Hi-C experiments the `hic_exactTest()` function can be used as shown below:

```{r}
hicexp1 <- hic_exactTest(hicexp1, p.method = 'fdr', 
                         parallel = FALSE)

# plot results
MD_composite(hicexp1)
```

The above composite MD plot displays where any significant differences are detected between the two groups. This function can also be sped up by using the `parallel` option. The results of the comparison are then saved in the `comparison` slot of the `hicexp` object.

```{r}
results(hicexp1)
```

In this `data.table` the first 3 columns represent the identity of the interaction, then followed by the unit genomic distance (`D`), the log fold change difference between the groups (`logFC`), the log counts per million for the interaction (`logCPM`), the p-value and finally the multiple testing correction p-value (`p.adj`). The type of multiple testing applied can be changed using the `p.method` option. To view what other adjustment methods are available look at the help for `?p.adjust`. 

### GLM Methods

For more complex Hi-C experiments the `hic_glm()` function must be used. In order to use this function, a design matrix must first be created. Here use the `hicexp2` object with some covariates and create the design matrix.

```{r}
batch <- c(1,2,1,2)
# produce design matrix
d <- model.matrix(~factor(meta(hicexp2)$group) + factor(batch))
```

The design matrix should contain the covariates of interest. Any categorical variables should be entered as factors. Next, the comparison of interest will need to be specified using either the `contrast` or the `coef` option. For this example we are interested in the group difference thus we can set `coef = 2` to test if the group effect is equal to 0. For more information on using `contrast` and `coef` please see the `edgeR` user manual. Now we are ready to perform the comparison.

```{r, eval=FALSE}
hicexp2 <- hic_glm(hicexp2, design = d, coef = 2, method = "QLFTest", p.method = "fdr", parallel = FALSE)
```


There are 3 methods by which `hic_glm()` can be performed. The default method is to the use the `QLFTest` which makes use of the quasi-likelihood model. Additionally there is the `LRTest` which conducts a likelihood ratio test. The final method is the `Treat` method which conducts a test relative to a specified fold change threshold. For this option the `M` option will need to be used to specify the log2 fold change threshold.

```{r, eval=FALSE}
# use Treat option
hicexp2 <- hic_glm(hicexp2, design = d, coef = 2, method = "Treat", 
                  M = 0.5, p.method = "fdr", parallel = FALSE)
```

## Downstream analysis

For downstream analysis of the results of `multiHiCcompare` you may want to filter the results. This can be accomplished by using the `topDirs` function. Setting the `return_df = 'pairedbed'` will return a table of of the interacting pairs filtered by your specifications.

```{r}
td <- topDirs(hicexp1, logfc_cutoff = 0.5, logcpm_cutoff = 0.5,
        p.adj_cutoff = 0.2, return_df = 'pairedbed')
head(td)
```

Additionally you can summarize the regions that were detected as significant at least once by setting `return_df = 'bed'`.

```{r}
counts <- topDirs(hicexp1, logfc_cutoff = 0.5, logcpm_cutoff = 0.5, p.adj_cutoff = 0.2, alpha = 0.1, return_df = 'bed')
head(counts)
```

The resulting table when `return_df = 'bed'` is used can be input into the following plot function to visualize the p-values of the top differentially interacting regions (DIRs).

```{r}
plot_pvals(counts)
```

Or to visualize the counts of the top DIRs.

```{r}
plot_counts(counts)
```



## Other functions

There are several other functions included in `multiHiCcompare`. `manhattan_hicexp()` produces a Manhattan plot for the results of a comparison to identify "hotspot" regions of the genome where large numbers of significant differences are found.

```{r}
manhattan_hicexp(hicexp1, method = "standard")
```

There are also the MD plotting functions `MD_hicexp()` which will plot MD plots for each unique pair of samples in the `hicexp` object.

```{r}
MD_hicexp(hicexp1, prow = 2, pcol = 3)
```

The `MD_composite()` function will plot a composite MD plot of the results of a comparison where the significant differences are highlighted.

```{r}
MD_composite(hicexp1)
```


# Session Info

```{r, echo=FALSE}
sessionInfo()
```



