%\VignetteEngine{knitr::knitr}
%\VignetteIndexEntry{oligo User's Guide}
%\VignetteDepends{oligoData, pd.hg18.60mer.expr, maqcExpression4plex, genefilter, limma, pd.hg.u95av2, RSQLite, biomaRt, AnnotationDbi, Biostrings,BSgenome.Hsapiens.UCSC.hg18,GenomeGraphs,RCurl,ACME,pd.huex.1.0.st.v2,pd.mapping50k.xba240}
%\VignetteKeywords{preprocessing, microarray, normalization, summarization}
%\VignettePackage{oligo}

%% compiling this:
%% Rscript -e 'library(knitr); knit("oug.Rnw"); system("pdflatex oug"); system("pdflatex oug"); system("bibtex oug"); system("bibtex oug"); system("pdflatex oug"); system("pdflatex oug")'


\documentclass[12pt, letterpaper]{book}
\usepackage{subfigure}
%\usepackage{graphicx, setspace, amsfonts, amsmath, multirow, hyperref, amssymb, rotating}

%% Preamble
<<style-knitr, eval=TRUE, echo=FALSE, results="asis">>=
BiocStyle::latex()
@


\title{\Biocpkg{oligo} User's Guide}
\author{Benilton S. Carvalho}

\begin{document}

\maketitle

\tableofcontents

<<loadingBack, message=FALSE, warning=FALSE, echo=FALSE>>=
suppressPackageStartupMessages(library(oligo))
suppressPackageStartupMessages(library(oligoData))
suppressPackageStartupMessages(library(pd.hg.u95av2))
suppressPackageStartupMessages(library(RSQLite))
@

\chapter{Introduction}
\label{chap:intro}

\Biocpkg{Oligo} is a \Bioconductor{} package for preprocessing
oligonucleotide microarrays. It currently supports chips produced by
Affymetrix and NimbleGen and uses files provided by these manufacturers
in their native format. The package provides a unified framework for
preprocessing and uses the data representation established by the
\Bioconductor{} project, which simplifies the interface with other packages
in the project.

The \Biocpkg{oligo} package allows users to preprocess their
microarray data using \R{}. This is a convenient approach as analysts
can combine the preprocessed data with a number of tools already
implemented in \R{}, like downstream analyses and visualization.

The software is designed to support large datasets and also provides
parallel execution of common tasks like background subtraction,
normalization and summarization.

This guide describes \Biocpkg{oligo} and its features as available on
\R{} Version 3.2.0 with BioConductor Version 3.1.

\chapter{Preamble}
\label{chap:preamble}

\section{Citing \Biocpkg{oligo}}
\label{sec:citing}

The \Biocpkg{oligo} package is comprised of a collection of tools
developed by different authors. Please cite their work appropriately.

If you use \Biocpkg{oligo}, please cite:
\begin{description}
\item Carvalho and Irizarry. A Framework for Oligonucleotide Microarray
  Preprocessing. Bioinformatics (2010) vol. 16 (19) pp. 2363-2367.
\end{description}

If you use the SNPRMA and/or CRLMM algorithm implemented in \Biocpkg{oligo}, please
also cite:
\begin{description}
\item Carvalho et al. Exploration, normalization, and genotype calls of
  high-density oligonucleotide SNP array data. Biostatistics (2007)
  vol. 8 (2) pp. 485-99.
\end{description}

If you use the RMA algorithm, please cite:
\begin{description}
\item Bolstad, B.M., Irizarry R. A., Astrand M., and Speed, T.P. (2003),
  A Comparison of Normalization Methods for High Density Oligonucleotide
  Array Data Based on Bias and Variance. Bioinformatics 19(2):185-193;
\item Rafael. A. Irizarry, Benjamin M. Bolstad, Francois Collin, Leslie
  M. Cope, Bridget Hobbs and Terence P. Speed (2003), Summaries of
  Affymetrix GeneChip probe level data Nucleic Acids Research 31(4):e15;
\item Irizarry, RA, Hobbs, B, Collin, F, Beazer-Barclay, YD, Antonellis,
  KJ, Scherf, U, Speed, TP (2003) Exploration, Normalization, and
  Summaries of High Density Oligonucleotide Array Probe Level
  Data. Biostatistics .Vol. 4, Number 2: 249-264.
\end{description}

If you use the PLM algorithm, please cite:
\begin{description}
\item Bolstad, BM (2004). Low Level Analysis of High-density
  Oligonucleotide Array Data: Background, Normalization and
  Summarization. PhD Dissertation. University of California, Berkeley.
\end{description}

\fixme{If you use the MAS5 Present/Absent calls... }

\fixme{If you use the DABG Present/Absent calls...}

\section{Installation}
\label{sec:install}

The \Biocpkg{oligo} package is available for download through the
BioConductor project for all platforms. We recommend the installation of
the latest R in order to get the latest features
available in the package, which can be installed using:

<<install, eval=FALSE>>=
if (!requireNamespace("BiocManager", quietly=TRUE))
    install.packages("BiocManager")
BiocManager::install('oligo')
@ 

\Biocpkg{oligo} is in constant development and the users can obtain a
summary of the changes by using the \Rcode{news} command:

<<news, eval=FALSE>>=
## All documented changes
news(package='oligo')
@ 

\section{Requirements}
\label{sec:requirements}

\Biocpkg{oligo} depends on a few packages that will be automatically
installed if the instructions on Section~\ref{sec:install} are
used. These dependencies are available for all platforms and do not
require any intervention for their successful installation.

Once \Biocpkg{oligo} is installed, the users will need to install
the annotation packages associated to the data they want to import. The
annotation packages are built using the \Biocpkg{pdInfoBuilder}
package, but several of them are available for download on the
BioConductor website.

If a user tries to import a dataset for which an annotation package is
not installed on the user's system, \Biocpkg{oligo} will search for it
on the BioConductor website. If the annotation package is found, then
\Biocpkg{oligo} will download and install it automatically. If the
annotation package is not found, \Biocpkg{oligo} will return an error
and the user is expected to build the one using the
\Biocpkg{pdInfoBuilder} package. After the package is built, the user
must install it before attempting to import the data.

\chapter{Getting Started}
\label{chap:getstart}

To get started with \Biocpkg{oligo}, one must load the package, which
can be achieved with the \Rcode{library} command:

<<loadOligo, quiet=TRUE, message=FALSE>>=
library(oligo)
@ 

\Biocpkg{oligo} can appropriately handle data files for Affymetrix and
NimbleGen designs. The supported formats are CEL (Affymetrix) and XYS
(NimbleGen).

\section{Importing Data}
\label{sec:importing}

\subsection{Affymetrix Data}
\label{sec:affydata}

Affymetrix distributes data using CEL files, to simplify the access to
these files, \Biocpkg{oligo} provides the \Rcode{list.celfiles} tool,
which is a wrapper around \Rcode{list.files} (consult the documentation
for \Rcode{list.files} to get detailed information on advanced
usage). The \Rcode{list.celfiles} command should be used to obtain the
list of CEL files at a given directory. We strongly recommend the use of
fully qualified names (i.e., including the whole path) for CEL files, to
minimize the chance of problems. The snippet below shows the syntax to
list CEL files in the hypothetical directory \Robject{myCELs}:

<<listCEL, eval=FALSE>>=
celFiles <- list.celfiles('myCELs', full.names=TRUE)
@ 

The CEL files can be in either binary or text formats. Regardless the
internal structure of the files, \Biocpkg{oligo} can import them
transparently via the command \Rcode{read.celfiles} as shown below:

<<readCEL, eval=FALSE>>=
rawData <- read.celfiles(celFiles)
@ 

\subsection{NimbleGen Data}
\label{sec:nimbledata}

The NimbleGen data supported by oligo is provided as XYS files. They are
produced through the NimbleScan software from the TIFF image and NDF
specification file. The \Rcode{list.xysfiles} function can be used to
simplify the access to the XYS files. If a hypothetical directory
\Robject{myXYSs} contains the XYS files for a given dataset, the suggested
approach to point to these files is as follows:

<<listXYS, eval=FALSE>>=
xysFiles <- list.celfiles('myXYSs', full.names=TRUE)
@ 

The files listed in \Robject{xysFiles} can then be imported using the
\Rcode{read.xysfiles} command:

<<importXYS, eval=FALSE>>=
rawData <- read.xysfiles(xysFiles)
@ 

\section{Containers for Raw Data}
\label{sec:rawdata}

\Biocpkg{oligo} uses different containers to store data at the
feature-level, i.e. data imported from CEL and/or XYS files, as
Table~\ref{tab:classes} shows. This approach improves the flexibility of
the package as it allows any method to behave differently depending on
the type of array from which the data were obtained. As a consequence,
the user benefits from the simplicity of the software, as algorithms
should be able to handle data appropriately independent of their
origin.

\begin{table}[!h]
  \centering
  \begin{tabular}{|c|c|} \hline
    \textbf{Type} & \textbf{Array} \\ \hline
    \Rclass{ExonFeatureSet} & Exon ST \\
    \Rclass{ExpressionFeature} & Expression \\
    \Rclass{GeneFeatureSet} & Gene ST \\
    \Rclass{TilingFeatureSet} & Tiling \\
    \Rclass{SnpFeatureSet} & SNP \\ \hline
  \end{tabular}
  \caption{Types of containers for feature-level data used in oligo.}
  \label{tab:classes}
\end{table}

One simple example is the RMA algorithm. When it is applied to
expression data, the software software uses the usual definition of sets
of features (often referred to as \textit{probesets}) to group features
together for summarization. If the same method is applied to Affymetrix
exon arrays, the software is able to identify that and use the
definition of \textit{meta-probesets} given by Affymetrix to provide
summaries at the transcript level, if such behavior is requested.

\chapter{Visualization and QC Tools}
\label{chap:visqc}

On this chapter, we will demonstrate how \Biocpkg{oligo} can be used
for visualization of data at the feature-level.

To demonstrate the capabilities of the software, the
\Robject{affyExpressionFS} dataset from the \Biocpkg{oligoData}
package will be used.

<<exData>>=
library(oligoData)
data(affyExpressionFS)
@ 

This dataset is comprised of 59 samples on expression arrays provided by
Affymetrix. This dataset is the \textit{Human Genome U95 Data Set}, used
to validade preprocessing algorithms, as it contains genes that were
spiked-in in known concentrations. Below we create a table containing
sample information, using descriptors found on the Affymetrix website. 

The user must pay attention to the fact that the objects handled by \Biocpkg{oligo} always carry information about channels. This information must be reported on a metadata object, which is represented below by the \Robject{metadata} data.frame. Because Affymetrix expression arrays are one-color devices and the information we provide is valid for this channel, we fill the \Robject{channel} column with the value \Robject{ALL}.

<<setData>>=
affyExpressionFS
sns <- sampleNames(affyExpressionFS)
## all 1521 were meant to be 1251
sns <- gsub('1521', '1251', sns)
## removing the 'r' (repeat) flag from the name
sns <- gsub('r\\.CEL$', '\\.CEL', sns)
wafer <- substr(sns, 1, 4)
experiment <- substr(sns, 5, 5)
tmp <- substr(sns, 6, 7)
complex <- rep('+', length(tmp))
complex[tmp == '00'] <- '-'
info <- data.frame(wafer=wafer, experiment=experiment, complex=complex)
rownames(info) <- sns
metadata <- data.frame(labelDescription=c('wafer', 'experiment', 'complex'), channel=factor('_ALL_'))
sampleNames(affyExpressionFS) <- sns
pd <- new('AnnotatedDataFrame', data=info, varMetadata=metadata)
phenoData(affyExpressionFS) <- pd
rm(tmp, wafer, experiment, complex, pd, metadata)
@ 

\section{Pseudo-image Plots}
\label{sec:pseudoimg}

Pseudo-image plots are used to assess the spatial distribution of the
data on the chips. Due to the magnitude of the readings, pseudo-images
using data on the original scale often mask spatial features that may
be present on the arrays. This is why we recommend the use of the
default $\log_2$-scale of the \Rfunction{image} method. One useful
alternative for the $\log_2$-scale pseudo-image is the use of the
ranks of the observations. This can be achieved by setting the
\Robject{transfo} argument on the \Rfunction{image} method.


<<pseudoimg, include=FALSE>>=
image(affyExpressionFS, which=55)
image(affyExpressionFS, which=55, transfo=rank)
@

\begin{figure}[!htp]
  \centering
  \subfigure[$\log_2$-intensities]{\label{fig:pseudoimglog2}\includegraphics[width=.4\textwidth]{figure/pseudoimg-1}}
  \subfigure[Rank]{\label{fig:pseudoimgrank}\includegraphics[width=.4\textwidth]{figure/pseudoimg-2}}
  \caption{Pseudo-images}
  \label{fig:pseudoimg}
\end{figure}


\section{MA Plots}
\label{sec:maplot}

Plotting log-ratios, $M$, \textit{versus} average log-intensities, $A$,
is a strategy to visualize the relationship between these two
variables. Both $M$ and $A$ are computed as a function of a
reference. To illustrate this, the definitions of log-ratios and average
log-intensities of a generic sample, indexed by $i$, and a given
reference, $R$, are given below:

\begin{eqnarray}
  \label{eq:ma}
  M_{i,R} & = & \log I_i - \log I_R \\
  A_{i,R} & = & \frac{1}{2} \left[ \log\left(I_i\right) + \log\left(I_R\right) \right]
\end{eqnarray}

For one color arrays, one common approach is to create MA plots for
every combination of two samples on the (sub-)dataset of interest. On
the snippet below, we use the \Robject{pairs} argument to generate MA
plots for pairs of samples (restricted to the first three samples, which
belong to the same group).

<<maplotpairssmooth, fig.cap='MA Plot using smoothScatter', fig.align='center', fig.width=4, fig.height=4, warning=FALSE>>=
xl <- c(2.8, 4)
yl <- c(-1, 1)
MAplot(affyExpressionFS[, 1:3], pairs=TRUE, ylim=yl, xlim=xl)
@ 


The standard approach to plot data used by \Rfunction{MAplot} is to use
\Rfunction{smoothScatter}, which provides better visualization through
the use of 2-D smoothed densities. This behavior can be changed by
setting the \Robject{plotFun} argument, as shown below. Valid values for
this argument are functions that preferentially take the same arguments
as \Rfunction{smoothScatter}, like the \Rfunction{plot} function.

<<maplotpairspoints, fig.cap='MA Plot using points', fig.align='center', fig.width=4, fig.height=4, warning=FALSE>>=
MAplot(affyExpressionFS[, 1:3], pairs=TRUE, ylim=yl, xlim=xl, plotFun=plot)
@ 

The \Rfunction{MAplot} method also allows the combination of data into
groups. With the code below, the investigator obtains the MA plot for
the first levels of \Robject{wafer}, \textit{'1251'}, comparing the
results against the reference group, \textit{2353}. Note that
\Robject{wafer} is a factor and that the definition of a reference group
was arbitrary and used here just to illustrate the software capabilities.

<<maplotgrpprep>>=
wafer <- affyExpressionFS$wafer
levels(wafer)
@ 

<<maplotgrp, fig.cap='MA Plot comparing groups', fig.align='center', fig.width=4, fig.height=4>>=
MAplot(affyExpressionFS, groups=wafer, which=1, refSamples=3) 
@ 

When the \Robject{groups} argument is not set and the \Robject{pairs}
argument is set to \Rcode{FALSE}, the \Rfunction{MAplot}
method estimates a pseudo-reference sample from the whole dataset passed
to the function. The pseudo-reference sample and the group summaries (if
\Robject{groups} is defined) are estimated using the
\Robject{summaryFun} argument, which must be a function that takes an
$N \times C$ matrix and returns a vector of length $N$. The default
value for \Robject{summaryFun} is \Rfunction{rowMedians}.

\section{Boxplots}
\label{sec:bxp}

Boxplots are used to visualize key components on the distribution of the
data and simplify the comparison of such statistics across samples.

<<boxplotlog2, include=FALSE>>=
boxplot(affyExpressionFS, main='Sample NimbleGen Dataset', ylab='log2-intensity')
@

The call above produces a boxplot for the PM features. If the array
contains other features types, the \Rfunction{boxplot} method can be used
to generate figures for specific probe types by using the
\Robject{which} argument, which take values \Rcode{'pm'}, \Rcode{'mm'},
\Rcode{'bg'}, \Rcode{'both'} and \Rcode{'all'}.

Data transformation can also be applied. The default is to log-transform
(base 2) the data, but other functions can be used, as long as they are
passed through the \Robject{transfo} argument. The example below
presents the boxplot using the original scale.

<<boxplotorig, include=FALSE>>=
boxplot(affyExpressionFS, main='Data on Original Scale',
        ylab='intensity', transfo=identity)
@ 


\begin{figure}[!htp]
  \centering
  \subfigure[log-scale]{\label{fig:bxp1}\includegraphics[width=.4\textwidth]{figure/boxplotlog2-1}}
  \subfigure[original scale]{\label{fig:bxp2}\includegraphics[width=.4\textwidth]{figure/boxplotorig-1}}
  \caption{Boxplots for intensity data: the visualization of the data is simplified if the logarithmic-scale is used}
  \label{fig:bxps}
\end{figure}

The \Rfunction{boxplot} method for \Rclass{FeatureSet} and
\Rclass{ExpressionSet} objects uses a sample of the data (of size
\Robject{nsample}) to produce the plot. Therefore small differences
between consecutive calls to the method are expected. Users interested
in getting the exact same plot should specify a fixed seed through
\Rfunction{set.seed} prior to calling \Rfunction{boxplot}.

\section{Density Plots}
\label{sec:densplots}

Smoothed histograms are also used to assess the distribution of the data
under analysis. They allow the immediate visualization (possibly
non-unique) modes, which can not be reliably detected through the
investigation of boxplots and other graphical tools.

<<densplotlog2, include=FALSE>>=
hist(affyExpressionFS, main='Density Estimate for log-intensities')
@ 

Similar to the \Rfunction{boxplot} method described on
Section~\ref{sec:bxp}, \Rfunction{hist}:

\begin{itemize}
  \item allows subsetting by feature type, if such probes are available on the
    chip, through the \Robject{which} argument;
  \item uses a random sample of the data to generate the plot, requiring
    the use of \Rcode{set.seed} to create reproducible charts. The size
    of the sample is determined by the \Robject{nsample} argument;
  \item permits the use of functions other than $\log_2$ to transform
    the data prior to plotting. The argument \Robject{transfo} is the
    one that handles the transformation function, which should return an
    object with the same attributes as the input.
\end{itemize}

<<densplotorig, include=FALSE>>=
hist(affyExpressionFS, main='Density Estimate for intensities',
     transfo=identity, nsample=15000, xlab='intensity')
@ 

\begin{figure}[!htp]
  \centering
  \subfigure[log-scale]{\label{fig:bxp1}\includegraphics[width=.4\textwidth]{figure/densplotlog2-1}}
  \subfigure[original scale]{\label{fig:bxp2}\includegraphics[width=.4\textwidth]{figure/densplotorig-1}}
  \caption{Density plots for intensity data: the visualization of the data is simplified if the logarithmic-scale is used}
  \label{fig:densps}
\end{figure}

\section{Accessing Probe Sequences}
\label{sec:access-probeseq}

The annotation packages used by \Biocpkg{oligo} store feature
sequences. This is done through instances of \Rclass{DNAStringSet}
objects implemented in the \Biocpkg{Biostrings} package. The sequences
for PM probes can be easily accessed via the \Rfunction{pmSequence}
function, as shown below.

<<obtainSequencesAffyExpression>>=
pmSeq <- pmSequence(affyExpressionFS)
pmSeq[1:5]
@

When importing the data, \Biocpkg{oligo} does not impose any
transformation, so one needs to manually apply, for example, the
$\log_2$ transform to the intensities of PM probes, which can be
accessed with the \Rfunction{pm} function, as needed. Below, we present
how to centralize the $\log_2$-PM intensities for each sample in
\Robject{affyExpressionFS}.
<<obtainPMs, results='hide'>>=
pmsLog2 <- log2(pm(affyExpressionFS))
@

The dependence of intensity on probe sequence is a well established fact
on the microarray literature. The use of the \Biocpkg{oligo} package
simplifies significantly the observation of this event, as it provides
simple access to both observed intensities and annotation. Below, we
estimate the affinity splines coefficients \cite{Wu:2004p1634}.
<<affinityCoefficients>>=
coefs <- getAffinitySplineCoefficients(pmsLog2, pmSeq)
@

On Figure~\ref{fig:seqEffectAffyExpression}, we show how the results
above can be used to estimate the base-position effects on the
$\log_2$-intensities observed for the first sample in the dataset. The
\Rfunction{getBaseProfile} function provides a simple way of using the
affinity coefficients to estimate the effects of interest. It accepts a
\Rcode{plot} argument, which takes logical values, to make the plot
and returns, invisibly, the estimated effects. All the arguments that
can be passed to the \Rfunction{matplot} function can also be passed to
\Rfunction{getBaseProfile}.
<<seqEffectAffyExpression, include=FALSE>>=
colors <- darkColors(4)
xL <- "Base Position"
yL <- "Effect"
pchs <- c("A", "C", "G", "T")
getBaseProfile(coefs[,1], plot=TRUE, pch=pchs, type="b", xlab=xL, ylab=yL,
               lwd=3, col=colors, ylim=c(-.4, .4))
@

\begin{figure}[!htp]
  \centering
  \includegraphics[width=.45\textwidth]{figure/seqEffectAffyExpression-1}
  \caption{Sequence effect for the first sample in the dataset. These
    results have been reported in detail elsewhere, but can be easily
    reproduced with the use of the \Biocpkg{oligo} package.}
  \label{fig:seqEffectAffyExpression}
\end{figure}

Tools implemented in other packages can be used in conjunction with
\Biocpkg{oligo} to investigate different hypothesis. The example below
shows how the \Rfunction{alphabetFrequency} function, defined by the
\Biocpkg{Biostrings} can be used to determine the GC content of the
probe sequences accessed by \Biocpkg{oligo}.
<<GCplots, results='hide'>>=
counts <- Biostrings::alphabetFrequency(pmSeq, baseOnly=TRUE)
GCcontent <- ordered(counts[, "G"]+counts[, "C"])
@

In addition to Figure~\ref{fig:seqEffectAffyExpression}, we can also
plot the $\log_2$-intensities as a function of the GC content computed
above. Figure~\ref{fig:gcBpAffyExpression} presents the strong
dependency of $\log_2$-intensities on GC contents for sample~1, which is
also present in all other samples.
<<intensityByContentAffyExpression, include=FALSE>>=
colors <- seqColors(nlevels(GCcontent))
xL <- "GC Frequency in 25-mers"
yL <- expression(log[2]~intensity)
boxplot(pmsLog2[,1]~GCcontent, xlab=xL, ylab=yL, range=0, col=colors)
@
\begin{figure}[!htp]
  \centering
  \includegraphics[width=.45\textwidth]{figure/intensityByContentAffyExpression-1}
  \caption{On this boxplot stratified by GC content, we can observe the
    strong dependency of $\log_2$-intensities on the number of G or C
    bases observed in the probe sequency.}
\label{fig:gcBpAffyExpression}
\end{figure}

\section{Probe Level Models}
\label{sec:plm}

Using the \Rfunction{fitProbeLevelModel} method, the user is able to fit Probe Level
Models (PLMs) with probe-level and sample-level parameters. The
resulting object is an \Rclass{oligoPLM} object, which stores parameter
estimates, residuals and weights.

\subsection{Fitting PLMs}
\label{sec:fitplm}

The simplest call to adjust a probe level model is as simple as
<<plm1, cache=TRUE>>=
fit1 <- fitProbeLevelModel(affyExpressionFS)
@
and will fit a model that accounts for probe (feature) and sample
effects, whose estimates and standard errors can be recovered,
respectively, with the \Rfunction{coef} and \Rfunction{se} methods, as
shown below.
<<coef1>>=
coef(fit1)[1:4, 1:2]
se(fit1)[1:4, 1:2]
@ 

\subsection{Visualizing \Rfunction{fitProbeLevelModel} Results}
\label{sec:visplm}

One of the most used QC metrics is the Relative Log Expression (RLE),
which is computed (for each sample on every probeset) by comparing the
expression level of one probeset against the median expression of that
probeset across samples.

The estimates obtained via RLE can be accessed by setting the argument \Rcode{type} to \Rcode{values}. By setting this argument to \Rcode{stats}, the user will be able to access the statistics (quantiles) for each sample.

<<rleStats>>=
RLE(fit1, type='stats')[, 1:2]
RLE(fit1, type='values')[1:4, 1:2]
@ 

Generating a boxplot of the RLE values is the default behavior of the method.
<<rle1, include=FALSE>>=
RLE(fit1)
@ 

Another useful tool for QC is the Normalized Unscaled Standard Errors
(NUSE). To determine NUSE, the standard error estimates are standardized
across arrays so that the median standard error for that probeset is 1
across all arrays. Therefore, arrays whose NUSE values are significantly
higher than other samples are often lower quality chips. Similarly to
RLE, the statistics, values and boxplot of NUSE can be obtained by
appropriately setting the \Robject{type} argument of the \Rfunction{NUSE} method.
<<nuseStats>>=
NUSE(fit1, type='stats')[, 1:2]
NUSE(fit1, type='values')[1:4, 1:2]
@ 
<<nuse1, include=FALSE>>=
NUSE(fit1)
@ 


\begin{figure}[!htp]
  \centering
  \subfigure[RLE]{\label{fig:plmplotsrle}\includegraphics[width=.4\textwidth]{figure/rle1-1}}
  \subfigure[NUSE]{\label{fig:plmplotsnuse}\includegraphics[width=.4\textwidth]{figure/nuse1-1}}
  \caption{Visualization of PLM results}
  \label{fig:plmplots}
\end{figure}

The use of PLMs also permits the inspection of the spatial distribution of
the data on the chip. The current implementation allows the
visualization of the estimated weights and residuals. Residuals can be
further decomposed in 4 types: residuals, positive residuals, negative
residuals and residual
signs. Figures~\ref{fig:imgplm1}-\ref{fig:imgplm5} show these plots for
the dataset used as example here.

<<imgplm, include=FALSE>>=
image(fit1, which=55, type='weights')
image(fit1, which=55, type='residuals')
image(fit1, which=55, type='pos.residuals')
image(fit1, which=55, type='neg.residuals')
image(fit1, which=55, type='sign.residuals')
@ 


\begin{figure}[!htp]
  \centering
  \subfigure[Weights]{\label{fig:imgplm1}\includegraphics[width=.4\textwidth]{figure/imgplm-1}}
  \subfigure[Residuals]{\label{fig:imgplm2}\includegraphics[width=.4\textwidth]{figure/imgplm-2}} \\
  \subfigure[Positive Residuals]{\label{fig:imgplm3}\includegraphics[width=.4\textwidth]{figure/imgplm-3}}
  \subfigure[Negative Residuals]{\label{fig:imgplm4}\includegraphics[width=.4\textwidth]{figure/imgplm-4}} \\
  \subfigure[Residual Signs]{\label{fig:imgplm5}\includegraphics[width=.4\textwidth]{figure/imgplm-5}}
  \caption{Pseudo-images for PLM objects}
  \label{fig:imgplm}
\end{figure}

\chapter{Preprocessing}
\label{chap:preproc}

Preprocessing refers to a series of complex statistical procedures
applied to microarray data prior to dowstream analyses. These steps are
required mainly for two reasons: A) technical artifacts are known to
affect results, so background subtraction and normalization are used to
minimize these issues; and B) there are multiple probes per probeset,
therefore summarization to the probeset level is needed, so downstream
analyses can be carried on.

\section{Background Subtraction}
\label{sec:bg}

The \Biocpkg{oligo} package implements background subtraction through
the \Rfunction{backgroundCorrect} command. The method currently available
is the one used in RMA, which treats the PM intensities as a convolution
of noise and true signal. Additional methods will be available on future
releases and choices will be made with the \Robject{method} argument
(currently, the default is \Robject{'rma'}).

<<bgrma, cache=TRUE>>=
backgroundCorrectionMethods()
bgData1 <- backgroundCorrect(affyExpressionFS)
bgData2 <- backgroundCorrect(affyExpressionFS, method='mas')
#bgData3 <- backgroundCorrect(affyExpressionFS, method='LESN')
@ 

Because the input was an \Rclass{ExpressionFeatureSet} object, the
output \Robject{bgData1} is also an
\Rclass{ExpressionFeatureSet}. Below, we show a boxplot of the corrected
data, which can be compared to Figure~\ref{fig-boxplot}.

<<bgbxp, fig.cap="Boxplot of background corrected data", fig.width=4, fig.height=4, fig.align='center'>>=
boxplot(bgData1)
@ 

\section{Normalization}
\label{sec:norm}

The Rmethod{normalize} method provided by \Biocpkg{oligo} allows the
user to normalize the input data. Different normalization methods are
available. The available options are given by
\Rcode{normalizationMethods} and the argument \Robject{method} in
\Rcode{normalize} is used to select the normalization approach to be used.

<<norm, cache=TRUE>>=
normData <- normalize(bgData1)
@ 

\section{Summarization}
\label{sec:summ}

\fixme{to finish}

\chapter{Workflows}
\label{sec:workflows}

\section{Preprocessing Affymetrix Expression Arrays}
\label{sec:prepr-affym-expr}

To preprocess expression data, \Biocpkg{oligo} implements the RMA
algorithm \cite{Irizarry2003a, Irizarry2003b}. The \Rfunction{rma}
method, as shown below, proceeds with background subtraction,
normalization and summarization using median-polish.
<<rmaAffyExpression, results='hide'>>=
ppData <- rma(affyExpressionFS)
@

The results are returned in an \Rclass{ExpressionSet} instance and used
in downstream analyses, as currently done by several strategies for
microarray data analysis and described elsewhere.
<<rmaResultClass>>=
class(ppData)
@

At this point, the user can proceed with, for example, differential
expression analyses. The methodologies involved in this step make use of
several other packages, like \Biocpkg{limma} and
\Biocpkg{genefilter}. When preprocessing the data, \Biocpkg{oligo}
stores the summaries in a matrix called \Robject{exprs}, present in the
\Robject{assayData} data slot of the \Rclass{ExpressionSet}
object. Therefore, the only restriction the additional strategies used
with the preprocessed data have is to be aware that the processed data
can be easily accessed with the \Rfunction{exprs} method.

\section{Preprocessing NimbleGen Expression Arrays}
\label{sec:prepr-nimbl-expr}

This section presents a non-trivial use of the \Biocpkg{oligo} Package for the
analysis of NimbleGen Expression data. This vignette follows the
structure of the chapter \textbf{From CEL files to a list of interesting
  genes} by R. A. Irizarry \textit{in} \textit{Bioinformatics and
  Computational Biology Solutions Using R and Bioconductor}, which shows
a case study for Affymetrix Expression arrays.

In order to analyze microarray data using \Biocpkg{oligo}, the user is expected
to have installed on the system a package with the annotation for the
particular array design on which the experiment was performed. For the
example in question here, the design is hg18\_60mer\_expr and the
annotation package associated to it is \Biocannopkg{pd.hg18.60mer.expr},
which is built by using the \Biocpkg{pdInfoBuilder} package.

\subsubsection{Initialization of the environment}
\label{sec:initNimblExpr}

On this particular example, we will read XYS files instead of loading
the \Rclass{FeatureSet} object already available through the
\Biocpkg{oligoData} package (the \Robject{maqc} object that we will
create below is exactly the \Robject{nimbleExpressionFS} data object
provided by the \Biocpkg{oligoData} package). We start by loading the
packages that are going to be used in this session. The
\Biocpkg{maqcExpression4plex} package provides a set of six samples on
the MAQC Study; the set is comprised of samples on two groups: universal
reference and brain. The remaining packages offer additional
functionality, like tools for filtering, plotting and visualization.
<<loading, echo=TRUE>>=
  library(oligo)
  library(maqcExpression4plex)
  library(genefilter)
  library(limma)
@

Once the package is loaded, we can easily get the location of the XYS
files that contain the intensities by calling \Rfunction{list.xysfiles},
which takes the same arguments as \Rfunction{list.files}. To minimize
the chance of problems, we strongly recommend the use of
\Rcode{full.names=TRUE}.
<<listing>>=
extdata <- system.file("extdata", package="maqcExpression4plex")
xys.files <- list.xysfiles(extdata, full.names=TRUE)
basename(xys.files)
@ %def

To read the XYS files, we provide the \Rfunction{read.xysfiles}
function, which also takes \Robject{phenoData}, \Robject{experimentData}
and \Robject{featureData} objects and returns an appropriate subclass of
\Rclass{FeatureSet}.
<<ExpressionFeatureSet>>=
theData <- data.frame(Key=rep(c("brain", "universal reference"), each=3))
rownames(theData) <- basename(xys.files)
lvls <- c("exprs", "_ALL_")
vMtData <- data.frame(channel=factor('exprs', levels=lvls),
                      labelDescription="Sample type")
pd <- new("AnnotatedDataFrame", data=theData, varMetadata=vMtData)
maqc <- read.xysfiles(xys.files, phenoData=pd)
class(maqc)
@ %def

\subsubsection{Exploring the feature-level data}
\label{sec:exploring}

The \Rfunction{read.xysfiles} function returns, in this case, an
instance of \Rclass{ExpressionFeatureSet} and the intensities of these
files are stored in its \Robject{exprs} slot, which can be accessed with
a method with the same name.
<<rawdata>>=
exprs(maqc)[10001:10010, 1:2]
@ %def

The \Rfunction{boxplot} method can be used to produce boxplots for the
feature-level data.
<<maqcBoxplot, include=FALSE>>=
boxplot(maqc, main="MAQC Sample Data")
@ %def
\begin{figure}[!htp]
  \centering
  \includegraphics[width=.45\textwidth]{figure/maqcBoxplot-1}
  \caption{Distribution of $\log_2$-intensities of samples on the MAQC dataset.}
\label{fig:maqcBp}
\end{figure}

Similarly, a smoothed histogram for the feature-level data can be
obtained with the \Rfunction{hist} method.
<<maqcHist, include=FALSE>>=
hist(maqc, main="MAQC Sample Data")
@ %def
\begin{figure}[!htp]
  \centering
  \includegraphics[width=.45\textwidth]{figure/maqcHist-1}
  \caption{Smoothed histogram of $\log_2$-intensities of samples on the MAQC dataset.}
\label{fig:maqcHist}
\end{figure}

\subsubsection{RMA algorithm}
\label{sec:rma}

The RMA algorithm can be applied to the raw data of expression
arrays. It is available via the \Rfunction{rma} method. The algorithm will
perform background subtraction, quantile normalization and summarization
via median polish. The result of \Rfunction{rma} is an instance of
\Rclass{ExpressionSet} class, which also contains an \Rfunction{exprs}
slot and method.
<<rma>>=
  eset <- rma(maqc)
  class(eset)
  show(eset)
  exprs(eset)[1:10, 1:2]
@ %def

The \Rfunction{boxplot} and \Rfunction{hist} methods are also implemented
for \Rclass{ExpressionSet} objects. Note that \Rfunction{rma}'s output is
in the $\log_2$ scale, so we call such methods using the argument
\Rcode{transfo=identity}, so the data are not transformed in any way.
<<rmaResults, include=FALSE>>=
  boxplot(eset, transfo=identity, main="After RMA")
  hist(eset, transfo=identity, main="After RMA")
@ %def
\begin{figure}[!htp]
  \centering
  \includegraphics[width=.45\textwidth]{figure/rmaResults-1}
  \includegraphics[width=.45\textwidth]{figure/rmaResults-2}
  \caption{Boxplot and smoothed histogram for MAQC data after preprocessing.}
\label{fig:maqcHist}
\end{figure}

\subsubsection{Assessing differential expression}
\label{sec:diffexp}

One simple approach to assess differential expression is to flag units
with log-ratios greater (in absolute value) than 1, i.e. a change
greater than 2-fold when comparing brain vs. universal reference.
<<naive>>=
  e <- exprs(eset)
  index <- which(eset[["Key"]] == "brain")
  d <- rowMeans(e[, index])-rowMeans(e[, -index])
  a <- rowMeans(e)
  sum(abs(d)>1)
@

Another approach is to use $t$-tests to infer whether or not there is
differential expression.
<<ttests>>=
  tt <- rowttests(e, factor(eset[["Key"]]))
  lod <- -log10(tt[["p.value"]])
@

The MA plot can be used to visualize the behavior of the log-ratio as a
function of average log-intensity. Features with log-ratios greater (in
absolute value) than 1 are candidates for being classified as
differentially expressed.
<<maplotNG, include=FALSE>>=
  smoothScatter(a, d, xlab="Average Intensity", ylab="Log-ratio", main="MAQC Sample Data")
  abline(h=c(-1, 1), col=2)
@
\begin{figure}[!htp]
  \centering
  \includegraphics[width=.45\textwidth]{figure/maplotNG-1}
  \caption{MA plot for Brain vs. Universal Reference. The red lines show
  the threshold for fold-change of 2, up or down, which correspond to
  log- fold-change of $1$ and $-1$, respectively.}
\label{fig:maplot}
\end{figure}


The use of $t$-tests allows us to use the volcano plot to visualize
candidates for differential expression. Below, we highlight, in blue,
the top 25 in log-ratio and, in red, the top 25 in effect size.
<<volcanoplot2, echo=FALSE, include=FALSE>>=
  o1 <- order(abs(d),decreasing=TRUE)[1:25]
  o2 <- order(abs(tt[["statistic"]]),decreasing=TRUE)[1:25]
  o <- union(o1,o2)
  smoothScatter(d, lod, main="A Better view")
  points(d[o1], lod[o1], pch=18, col="blue")
  points(d[o2], lod[o2], pch=1, col="red")
  abline(h=2, v=c(-1, 1), col=2)
@
\begin{figure}[!htp]
  \centering
  \includegraphics[width=.45\textwidth]{figure/volcanoplot2-1}
  \caption{Volcano plot for Brain vs. Universal Reference. The vertical
  red lines show the threshold for fold-change of 2 (up or down), while
  the horizontal red line shows the threshold for p-values at the
  $10^{-2}$ level. The probesets shown in solid blue diamonds are the
  top-25 probesets for log-ratio. The probesets highlighted in red are
  the top-25 in p-value.}
\label{fig:maplot}
\end{figure}

The \Biocpkg{limma} Package can also be used to assess difference in
expression between the two groups.
<<limma>>=
  design <- model.matrix(~factor(eset[["Key"]]))
  fit <- lmFit(eset, design)
  ebayes <- eBayes(fit)
  lod <- -log10(ebayes[["p.value"]][,2])
  mtstat<- ebayes[["t"]][,2]
@

The Empirical Bayes approach implemented in \Biocpkg{limma} provides
moderated $t$-statistic, shown to have a better performance when
compared to the standard $t$-statistic. Below, we reconstruct the
volcano plot, but using the moderated $t$-statisic.
<<volcanoplot3, include=FALSE>>=
  o1 <- order(abs(d), decreasing=TRUE)[1:25]
  o2 <- order(abs(mtstat), decreasing=TRUE)[1:25]
  o <- union(o1, o2)
  smoothScatter(d, lod, main="Moderated t", xlab="Log-ratio", ylab="LOD")
  points(d[o1], lod[o1], pch=18,col="blue")
  points(d[o2], lod[o2], pch=1,col="red")
  abline(h=2, v=c(-1, 1))
@
\begin{figure}[!htp]
  \centering
  \includegraphics[width=.45\textwidth]{figure/volcanoplot3-1}
  \caption{Volcano plot for Brain vs. Universal Reference using
  moderated t-tests. The vertical red lines show the threshold for
  fold-change of 2 (up or down), while the horizontal red line shows the
  threshold for p-values at the $10^{-2}$ level. The probesets shown in
  solid blue diamonds are the top-25 probesets for log-ratio. The
  probesets highlighted in red are the top-25 in p-value (for the
  moderated t-test). Note that there is more overlap between the top-25
  for both log-ratio and p-value.}
\label{fig:maplot}
\end{figure}

The \Rfunction{topTable} command provides us a way of ranking genes for
further evaluation. In the case below, we adjust for multiple testing by
FDR and look at the Top-10 genes.
<<toptable>>=
  tab <- topTable(ebayes, coef=2, adjust="fdr", n=10)
  tab
@

\section{Obtaining Genotype Calls from SNP Arrays}
\label{sec:genotyping}

The \Biocpkg{oligo} package can genotype, using the CRLMM algorithm,
several Affymetrix SNP arrays. To do so, the user will need, in addition
to the \Biocpkg{oligo} package, an annotation data package specific to
the designed used in the experiment. Although these annotation packages
are created using the \Biocpkg{pdInfoBuilder} package, the CRLMM
algorithm requires additional hand-curated data, which are included in
the packages made available through the BioConductor
website. Table~\ref{tab:snpPlatforms} describes the supported designs
and the respective annotation packages.
\begin{table}[!htp]
  \centering
  \begin{tabular}{|c|c|} \hline
    \textbf{Design}    & \textbf{Annotation Package} \\ \hline
    Mapping 50K XBA    & \Biocannopkg{pd.mapping50k.xba240} \\
    Mapping 50K HIND   & \Biocannopkg{pd.mapping50k.hind240} \\
    Mapping 250K NSP   & \Biocannopkg{pd.mapping250k.nsp} \\
    Mapping 250K STY   & \Biocannopkg{pd.mapping250k.sty} \\
    Genomewide SNP 5.0 & \Biocannopkg{pd.genomewidesnp.5} \\
    Genomewide SNP 6.0 & \Biocannopkg{pd.genomewidesnp.6} \\
    \hline
  \end{tabular}
  \caption{SNP array designs currently supported by the \Biocpkg{oligo} package and their respective annotation packages. These annotation packages are made available through the BioConductor website and contain hand-curated data, required by the CRLMM algorithm.}
  \label{tab:snpPlatforms}
\end{table}

As an example, we will use the 269 CEL files, on the XBA array,
available on the HapMap website\footnote{\url{http://www.hapmap.org}},
which were downloaded and saved, uncompressed, to a subdirectory called
\Robject{snpData}. Therefore, we need to instruct the software to look for
the files at the correct location. An output directory should also be
defined and that is the place where the summary files, including
genotype calls and confidences are stored. This output directory, which
we chose to call \Robject{crlmmResults}, must not exist prior to the CRLMM
call, the software will take care of this task.

\bioccomment{Given the restrictions for building this vignette, we
  replaced the big example containing 269 samples by a smaller example
  with 3 samples only.}

<<getRawSnpData, results='hide'>>=
library("oligo")
#fullFilenames <- list.celfiles("snpData", full.names=TRUE)
path <- system.file('celFiles', package='hapmap100kxba')
fullFilenames <- list.celfiles(path, full.names=TRUE)
outputDir <- file.path(getwd(), "crlmmResults")
@

Given the always increasing density of the SNP arrays, we developed
efficient methods to process these chips, reducing the required amount
of memory even for large studies. Using this approach, we process
batches of SNPs at a time, saving partial results to disk. We refer the
interested reader to \cite{Carvalho:2007p32} for detailed information on
the CRLMM algorithm. The genotyping strategy, in summary, has three
steps: A) quantile normalizes against a known reference distribution; B)
summarizes the data to the SNP-allele level using median polish; C) uses
estimated parameters to classify the samples in genotype groups using
Mahalanobis distance.

The summaries are average intensities and log-ratios, defined as across
allele and within strand, ie:
\begin{eqnarray}
  A_{s} & = & \frac{\theta_{A, s}+\theta_{B, s}}{2} \\
  M_{s} & = & \theta_{A, s} - \theta_{B, s},
\end{eqnarray}
where $s$ defines the strand (antisense or sense). On the genomewide
designs, SNP 5.0 and 6.0, the strand information is dropped. These
summaries can be obtained via \Rfunction{getA} and \Rfunction{getM} methods,
which return arrays with dimensions corresponding to SNPs, samples and
strands (if applicable), respectively. These measures are later used for
genotyping.

CRLMM involves running an EM algorithm to adjust for average intensity
and fragment length in the log-ratio scale. These adjustments may take
long time to run, depending on the combination of number of samples and
computer resources available. Below, we show the simplest way to call
CRLMM, which requires only the file names and output directory.
<<crlmm, results='hide'>>=
if (!file.exists(outputDir))
  crlmm(fullFilenames, outputDir)
@

The \Rfunction{crlmm} method does not return an object to the R session.
Instead, it saves the objects to disk, as not all systems are guaranteed
to meet the memory requirements that \Rclass{SnpSuperSet} objects might
need. For the user's convenience, the \Rfunction{getCrlmmSummaries} will
read the information from disk and make a \Rclass{SnpCallSetPlus} or
\Rclass{SnpCnvCallSetPlus} object available to the user.

<<theSummaries>>=
crlmmOut <- getCrlmmSummaries(outputDir)
calls(crlmmOut[1:5,1:2])
confs(crlmmOut[1:5,1:2])
@

The genotype calls are represented by 1 (AA), 2 (AB) and 3 (BB). The
confidence is the predicted probability that the algorithm made the
right call.

Summaries generated by the algorithm can also be accessed from the \R{}
session. The options for summaries are \Rcode{"alleleA"},
\Rcode{"alleleB"}, \Rcode{"alleleA-sense"},
\Rcode{"alleleA-antisense"}, \Rcode{"alleleB-sense"},
\Rcode{"alleleB-antisense"}. The options \Rcode{"alleleA"} and
\Rcode{"alleleB"} are only available for SNP 5.0 and SNP 6.0
platforms. The other options are to be used with 50K and 250K arrays.

Below, we choose two SNPs to show the different configurations of the
genotype groups.
<<settingPlotM>>=
snps <- paste("SNP_A-", c(1703121, 1725330), sep="")
LIM <- c(-4, 4)
@

Figure \ref{fig:goodSNP} represents a SNP for which genotyping is
simplified by the good discrimination of both strands. Figure
\ref{fig:senseSNP} shows a SNP for which features on the antisense
strand have very good discrimination power, while no information (for
classification) can be extracted from the sense strand.

<<goodSNP, include=FALSE>>=
gtypes <- as.integer(calls(crlmmOut[snps[1],]))
plotM(crlmmOut, snps[1], ylim=LIM, xlim=LIM, col=gtypes)
@
<<senseSNP, include=FALSE>>=
gtypes <- as.integer(calls(crlmmOut[snps[2],]))
plotM(crlmmOut, snps[2], ylim=LIM, xlim=LIM, col=gtypes)
@

\begin{figure}[!htp]
  \begin{center}
    \subfigure[SNP\_A-1703121 has very good discrimination on both
    strands and, as competing algorithms, CRLMM has excelent performance
    on scenarios like this. On this plot, genotype calls provided by
    oligo are represented in different colors (black: AA; red: AB;
    green: BB)]{\label{fig:goodSNP}
      \includegraphics[width=.45\textwidth]{figure/goodSNP-1}}
    \subfigure[SNP\_A-1725330 presents poor discrimination on the sense
    strand. Because CRLMM does not average across strands, it can
    perfectly predict the genotype cluster each sample belongs to. On
    similar scenarios, competing algorithms are known to fail. Color
    scheme follows Figure~\ref{fig:goodSNP}.]{\label{fig:senseSNP}
      \includegraphics[width=.45\textwidth]{figure/senseSNP-1}}
  \end{center}
\end{figure}

CRLMM was shown to outperform competing genotyping tools. We refer the
reader to \cite{Lin:2008p33} for further details on this subject. The
genotypes provided by CRLMM, and in this example stored in
\Robject{crlmmOut}, can be easily used with other BioConductor tools,
like the \Biocpkg{snpStats} package, for downstream analyses.

<<cleanCrlmm, echo=FALSE, results='hide'>>=
fns <- list.files('crlmmResults', full.names=TRUE)
sapply(fns, file.remove)
file.remove('crlmmResults')
@ 

\section{Preprocessing Exon Arrays}
\label{sec:exon}

On this section, we use colon cancer sample data for exon arrays,
available on the Affymetrix
website\footnote{\url{http://www.affymetrix.com/support/technical/sample_data/exon_array_data.affx}},
to demonstrate the use of the \Biocpkg{oligo} package to preprocess
these data. The interested reader can download the CEL files and use
\Rfunction{read.celfiles} to import the data. Here, however, we will use
the \Biocpkg{oligoData} package to load this dataset, as shown below.
<<loadingPackageFiles, results='hide'>>=
library(oligoData)
data(affyExonFS)
@

As already noted, \Biocpkg{oligo} implements different classes
depending on the nature of the data. Therefore, a quick inspection, as
in the snippet below, shows that \Robject{affyExonFS} is an
\Rclass{ExonFeatureSet} object. This is a especially interesting
feature, as it allows methods to behave differently depending on the
object class.

<<exonClass, results='hide'>>=
affyExonFS
@

Generally, RMA will background correct, quantile normalize and summarize
to the probeset level, as defined in the annotation packages. When
working with an \Rclass{ExonFeatureSet} object, processing to the
probeset level provides expression summaries at the exon level and can
be obtained by setting the argument \Rcode{target} to
\Robject{"probeset"}, as presented below.
<<rmaExonProbeset, results='hide'>>=
probesetSummaries <- rma(affyExonFS, target="probeset")
@

For Exon arrays, Affymetrix provides additional annotation files that
define meta-probesets (MPSs), used to summarize the data to the gene
level. These MPSs are classified in three groups -- core, extended and
full -- depending on the level of confidence of the sources used to
generate such annotations. Additional values allowed for the
\Rcode{target} argument are \Rcode{"core"}, \Rcode{"extended"}
and \Rcode{"full"}. The example below shows how gene level summaries
can be obtained through \Biocpkg{oligo}.
<<rmaExonGeneCore, results='hide'>>=
geneSummaries <- rma(affyExonFS, target="core")
@


The results obtained from analyses performed with \Biocpkg{oligo} can
be easily combined with features offered by other packages. As an
example, we use the \Biocpkg{biomaRt} package to obtain IDs of
probesets on the Human Exon array that map to Entrez Gene ID~10948
(ENSG00000131748).
<<probesetIDs2EntrezGeneID, results='hide'>>=
library(RCurl)
options(RCurlOptions=list(http.version=HTTP_VERSION_1_0))
library(biomaRt)
ensembl <- useMart("ENSEMBL_MART_ENSEMBL", dataset="hsapiens_gene_ensembl",
                   host='may2009.archive.ensembl.org')
theIDs <- getBM(attributes="affy_huex_1_0_st_v2", filters="entrezgene",
                values=10948, mart=ensembl)
names(theIDs) <- 'psets'
@

Combining this information with the annotation package associated to the
data in \Robject{affyExonFS}, we can get detailed facts on the
probesets found to map to Entrez Gene ID~10948. Below, we obtain,
respectively, the MPS IDs, probeset IDs, probe IDs and start/stop
positions for the probesets identified above.
<<getProbesetInfo, results='hide'>>=
library(AnnotationDbi)
conn <- db(affyExonFS)
fields <- 'meta_fsetid, pmfeature.fsetid, fid, start, stop'
tables <- 'featureSet, pmfeature, core_mps'
sql <- paste("SELECT", fields,
             "FROM", tables,
             "WHERE pmfeature.fsetid=featureSet.fsetid",
             "AND featureSet.fsetid=core_mps.fsetid",
             "AND pmfeature.fsetid=:psets")
probesetInfo <- dbGetPreparedQuery(conn, sql, theIDs)
@

The availability of start and stop positions of the probesets improves
the visualization of the summaries at the exon level. If genomic
coordinates were available for probes themselves, visualization could be
improved even more. To achieve this, we first obtain the sequences for
the probes identified above. We saw that the \Rfunction{pmSequence} method
provides the sequences for all PM probes identified on the chip but,
instead, we directly load the \Biocpkg{Biostrings} object used to store
the sequence information for these probes. This gives us access not only
to the sequences, but also to the probe IDs linked to them.
<<getSequences, results='hide'>>=
library(Biostrings)
data(pmSequence, package=annotation(affyExonFS))
@

Because probe IDs are available in the \Robject{pmSequence} object, we
can easily restrict our search to the probes listed in the
\Robject{probesetInfo} object.
<<restrictProbes, results='hide'>>=
idx <- match(probesetInfo[["fid"]], pmSequence[["fid"]])
pmSequence <- pmSequence[idx,]
@

<<clean00, echo=FALSE>>=
rm(idx)
@

The \Robject{pmSequence} object behaves like a \Rclass{data.frame}, but
it is comprised of complex data structures defined in
\Biocpkg{Biostrings}. Below, we modify its representation to make it a
regular \Rclass{data.frame} object.
<<pmSequenceDF, results='hide'>>=
pmSequence <- data.frame(fid=pmSequence[["fid"]],
                         sequence = as.character(pmSequence[["sequence"]]),
                         stringsAsFactors=FALSE)
@

By joining the \Robject{probesetInfo} and \Robject{pmSequence} objects,
we centralize the available probe annotation.
<<probeInfo>>=
probeInfo <- merge(probesetInfo, pmSequence)
@
<<clean01, echo=FALSE>>=
rm(probesetInfo, pmSequence)
@

The genomic coordinates in \Robject{probeInfo} refer to the probesets. To
better visualize the observed probe intensities, we would be better off
if the coordinates were relative to the probes. Below, we use the
\Biocpkg{BSgenome.Hsapiens.UCSC.hg18} to obtain up-to-date genomic
coordinates. The coordinates are found by aligning the probe sequences
to the reference genome made available through the package. Because
Entrez Gene ID~10948 is located on chromosome~17, the search is limited
to this region.
<<loadingBSgenome, results='hide'>>=
library("BSgenome.Hsapiens.UCSC.hg18")
chr17 <- Hsapiens[["chr17"]]
seqs <- complement(DNAStringSet(probeInfo[["sequence"]]))
seqs <- PDict(seqs)
matches <- matchPDict(seqs, chr17)
@

<<clean02, echo=FALSE>>=
rm(seqs, chr17)
@

After matching the sequences, we update the genomic coordinates.
<<>>=
probeInfo[["start"]] <- unlist(startIndex(matches))
probeInfo[["stop"]] <- unlist(endIndex(matches))
@

<<clean03, echo=FALSE>>=
rm(matches)
@

With the updated coordinates, we reorder the probe information object,
\Robject{probeInfo}, and extract the probe intensities in the same
order. The probe ID field, \Robject{fid} in \Robject{probeInfo},
provides direct access to the probes of interest. The \Rfunction{exprs}
method is used to access the intensity matrix of the
\Robject{affyExonFS} object and immediately subsetted to the probes of
interest. After subsetting the observed intensities, we
$\log_2$-transform the data.
<<reorderByCoordinates, results='hide'>>=
probeInfo <- probeInfo[order(probeInfo[["start"]]),]
probeData <- exprs(affyExonFS)[probeInfo[["fid"]],]
probeData <- log2(probeData)
@

We use the updated genomic to estimate the probeset coverage. This
information will be used when plotting the data and will provide
approximate delimiters of the probesets.
<<psRegion, results='hide'>>=
attach(probeInfo)
probesetStart <- aggregate(as.data.frame(start), list(fsetid=fsetid), min)
names(probesetStart) <- c("fsetid", "start")
probesetStop <- aggregate(as.data.frame(stop), list(fsetid=fsetid), max)
names(probesetStop) <- c("fsetid", "stop")
detach(probeInfo)
@

The \Robject{psInfo} object will store the probeset information
(probeset ID, start and stop positions), as shown below. After ordering
appropriately the data, the \Robject{psInfo} probeset is attached, to
simplify its usage during the \R{} session.
<<psInfo, results='hide'>>=
psInfo <- merge(probesetStart, probesetStop)
psInfo <- psInfo[order(psInfo[["start"]]),]
psInfo[["fsetid"]] <- as.character(psInfo[["fsetid"]])
attach(psInfo)
probesetData <- exprs(probesetSummaries[fsetid,])
detach(psInfo)
@

<<clean04, echo=FALSE>>=
rm(probesetStart, probesetStop)
@

To visualize the data processed by \Biocpkg{oligo}, we will use the
\Biocpkg{GenomeGraphs} package. To match the genome build used to
update the probe coordinates, an archived version of the database will
be queried.
<<loadingGenomeGraphs, results='hide'>>=
library(GenomeGraphs)
probeids <- as.character(probeInfo[["fsetid"]])
ensembl <- useMart("ENSEMBL_MART_ENSEMBL", dataset="hsapiens_gene_ensembl", host='may2009.archive.ensembl.org')
## ensembl = useMart("ensembl", dataset="hsapiens_gene_ensembl")
geneid <- "ENSG00000131748"
title <- makeTitle(text=geneid, color="darkred")
@

The raw data, in the $\log_2$ scale, will be represented by the
\Robject{raw} object below, created with the \Rfunction{makeExonArray}
constructor.
<<rawExon, results='hide'>>=
attach(probeInfo)
raw <- makeExonArray(intensity=probeData,
                     probeStart=start,
                     probeEnd=stop,
                     probeId=probeids,
                     nProbes=rep(1, nrow(probeInfo)),
                     dp=DisplayPars(color="blue", mapColor="dodgerblue2"),
                     displayProbesets=FALSE)
detach(probeInfo)
@

The summarized data is also represented through an object created by
\Rfunction{makeExonArray}. The structure is identical to the one used
above.
<<processedExon, results='hide'>>=
attach(psInfo)
exon <- makeExonArray(intensity=probesetData,
                      probeStart=start,
                      probeEnd=stop,
                      probeId=fsetid,
                      nProbes=rep(1, nrow(psInfo)),
                      dp=DisplayPars(color="seagreen",
                        mapColor="seagreen"),
                      displayProbesets=FALSE)
@

To represent the probesets designed by Affymetrix, we use an
\Rclass{AnnotationTrack} object.
<<annotationTrack, results='hide'>>=
affyModel <- makeAnnotationTrack(start = start,
                                 end = stop,
                                 feature = "gene_model",
                                 group = geneid,
                                 dp = DisplayPars(gene_model="darkgreen"))
detach(psInfo)
@

The gene and transcripts representations are build as
follows. Affymetrix probes will be represented in green, while the gene
will be in orange; transcripts are represented in blue.
<<geneTranscripts>>=
gene <- makeGene(id=geneid, biomart=ensembl)
transcript <- makeTranscript(id=geneid, biomart=ensembl)
legend <- makeLegend(c("Affymetrix", "Gene"), fill=c("darkgreen", "orange"))
@

Figure~\ref{fig:ENSG131748}, generated with the \Rfunction{gdPlot}
function, shows the representation of the $\log_2$-intensities and
summaries at the exon level. It also shows probesets, gene and
transcripts on the region of interest.
<<ENSG131748, include=FALSE>>=
gdPlot(list(title, raw, exon, affyModel, gene, transcript, legend))
@
\begin{figure}[!htp]
  \centering
  \includegraphics[width=.45\textwidth]{figure/ENSG131748-1}
  \caption{Visual representation of observed $\log_2$-intensities and
    summarized data at the exon level for gene ENSG00000131748. The
    probes, gene and transcript are also represented, respectively, in
    green, orange and blue.}
  \label{fig:ENSG131748}
\end{figure}

Below, we identify the meta-probeset ID associated to the probes used
above. Once that is known, we can extract the proper gene-level
summaries stored in \Robject{geneSummaries}.
<<mps>>=
mps <- unique(probeInfo[["meta_fsetid"]])
mps <- as.character(mps)
mps
@

Therefore, the standard accessors can be used to obtain the gene
summaries for the unit above. Figure~\ref{fig:scatterENSG131748} shows
the expressions for gene ENSG00000131748 across the 33 samples available
on this dataset.
<<geneLevelSummaries, include=FALSE>>=
gSummaries <- exprs(geneSummaries[mps,])
x <- 1:length(gSummaries)
plot(x, gSummaries, xlab="Sample", ylab="Expression", main=geneid)
@
\begin{figure}[!htp]
  \centering
  \includegraphics[width=.45\textwidth]{figure/geneLevelSummaries-1}
  \caption{Expression levels estimated through RMA at the gene level.}
  \label{fig:scatterENSG131748}
\end{figure}

\section{Interfacing with \Biocpkg{ACME} to Find Enriched Regions Using
  Tiling Arrays}
\label{sec:tiling}

On this Section, we demonstrate how \Biocpkg{oligo} can be easily
combined with tools that rely on the structure implemented in the
\Biocpkg{Biobase} package. Using a sample ChIP-chip dataset kindly
provided by NimbleGen, we could use the \Rfunction{getNgsColorsInfo}
function to obtain the information regarding channels and sample names
for the XYS files saved on disk. The \Rfunction{getNgsColorsInfo} parses
the file names and, using the \Robject{\_532} and \Robject{\_635}
strings in the names, suggests channels and sample names for each XYS
file available.
<<colorsInfoTiling, eval=FALSE>>=
library(oligo)
info <- getNgsColorsInfo("tilingData", full=TRUE)
@

Combining the results in \Robject{info} with \Rfunction{read.xyfiles2},
we read the XYS files using a data structure that simplifies the data
management across different channels.
<<readTiling, eval=FALSE>>=
nimbleTilingFS <- read.xysfiles2(info[,2], info[,1], sampleNames=info[,3])
@

However, on this example, we will load the aforementioned dataset from
the \Biocpkg{oligoData} package, as described below:
<<loadTilingNG, results='hide'>>=
library(oligoData)
data(nimbleTilingFS)
nimbleTilingFS
@

The user can access the channel specific data by calling the
\Rfunction{channel} method. The resulting object is an
\Rclass{ExpressionSet} object that the user can use as required.
<<getChannel>>=
c1 <- channel(nimbleTilingFS, "channel1")
c2 <- channel(nimbleTilingFS, "channel2")
@

Detailed information on the PM probes available on the array can be
obtained by directly querying the annotation package. The call below
will extract the \Robject{fid}, \Robject{fsetid}, \Robject{chromosome}
and \Robject{start} position of each probe from the annotation package
and order the results by chromosome and start position.
<<tilingProbeInfo, results='hide'>>=
fields <- 'fid, fsetid, chrom as chromosome, position as start'
sql <- paste("SELECT", fields,
             "FROM pmfeature INNER JOIN featureSet USING(fsetid)",
             "ORDER BY chrom, position")
annotPM <- dbGetQuery(db(nimbleTilingFS), sql)
@
<<cleanTiling, echo=FALSE>>=
rm(sql, c1, c2)
@

Using the probe sequence, the end position of the probe can be easily
obtained. We load the sequences directly, so the \Robject{fid} field can
be used to order the sequences appropriately.
<<tilingSequence, results='hide'>>=
data(pmSequence, package=annotation(nimbleTilingFS))
idx <- match(annotPM[["fid"]], pmSequence[["fid"]])
pmSequence <- pmSequence[idx,]
@

To obtain the end position, we use \Rfunction{width}, defined in the
\Biocpkg{Biostrings} package.
<<tilingEndPosition>>=
attach(annotPM)
library(Biostrings)
annotPM[["end"]] <- start+width(pmSequence[["sequence"]])-1
head(annotPM)
@

The \Robject{fid} field corresponds to the row number in the
\Robject{nimbleTilingFS} object. When applied to the raw data object, the
\Rfunction{getM} function returns a matrix with the $\log_2$-ratio of the
intensities. Below, we get the $\log_2$-ratios corresponding to the PM
probes described in the \Robject{annotPM} object.
<<tilingM>>=
ratioPM <- getM(nimbleTilingFS)[fid,]
dimnames(ratioPM) <- NULL
detach(annotPM)
class(ratioPM)
@

By converting \Robject{annotPM} to an \Rclass{AnnotatedDataFrame}, it
can be used in the \Rclass{featureData} slot of \Rclass{eSet}-like
objects.
<<tilingFeatureData, results='hide'>>=
annotPM <- as(annotPM, "AnnotatedDataFrame")
@

We will use the \Biocpkg{ACME} package to calculate enrichment, using
algorithms that are insensitive to normalization strategies and array
noise. To work with this package, we need to create, first, an
\Rclass{ACMESet} object, which contains \Robject{chromosome},
\Robject{start} and \Robject{end} positions in the \Robject{featureData}
slot.
<<prepACME, results='hide'>>=
library(ACME)
acme <- new("ACMESet", exprs=ratioPM, featureData=annotPM)
@

The \Rfunction{do.aGFF.calc} function processes the \Rclass{ACMESet}
object, using a window size and threshold to identify the positive
probes in the object.
<<tilingCalc, results='hide'>>=
calc <- do.aGFF.calc(acme, window=1000, thresh=0.95)
@

The \Robject{calc} object is then used to find enriched regions with the
\Rfunction{findRegions} function, as shown below.
<<tilingRegions>>=
regs <- findRegions(calc)
head(regs)
@

\chapter{High Performance Computing Features}

Starting on series 1.12.x, the \Biocpkg{oligo} package offers high
performance computing features:

\begin{itemize}
\item \textbf{Support to larger datasets; and}
\item \textbf{Support to parallel computing.}
\end{itemize}

These features are initially available for RMA methods on
Expression/Gene/Exon arrays and will be implemented in other methods as
necessity arrives.

The use of such features is as simple as loading the required packages
(and registering a parallel backend, if parallel computing is
desired). The methods themselves are able to detect if these
experimental features are enabled and use them if possible, without any
modification of the method call.

\section{Support to large datasets}

The \Biocpkg{oligo} package uses the features implemented by the \CRANpkg{ff}
package to provide a better support to large datasets. If the user
prefers not to use the \CRANpkg{ff} package, then regular R objects are
used and the usual memory restrictions apply.

The support to large datasets is enabled by simply loading the
\CRANpkg{ff} package. Once that is done, \Biocpkg{oligo} saves ff
files to the directory pointed by \Rcode{ldPath()}.

<<eval=FALSE>>=
library(oligo)
library(ff)
ldPath()
@ 

Methods (\Rfunction{rma}) uses batches to process data. When possible (eg.,
background correction), it uses at most \Rcode{ocSamples()} samples
simultaneously at processing. For procedures that process probes (like
summarization), a maximum of \Rcode{ocProbesets()} are used
simultaneously. Therefore, the user should tune these parameters for a
better performance.

<<eval=FALSE>>=
ocSamples()
ocSamples(50) ## changing default to 50
ocProbesets()
ocProbesets(100) ## changing default to 100
@ 

<<eval=FALSE>>=
  library(oligo)
  library(ff)
  rawData <- read.celfiles(list.celfiles())
  rmaRes <- rma(rawData)
  exprs(rmaRes)[1:10,]
@ 
 
\section{Parallel computing}

The \Biocpkg{oligo} package can make use of a parallel environment (with
\Rcode{rma} in the meantime) set via \CRANpkg{foreach} package, as long as
the user:

\begin{itemize}
  \item enables support to large datasets (load \CRANpkg{ff});
  \item loads the \CRANpkg{foreach} package;
  \item register a parallel backend (for example, through one of the
      \CRANpkg{doMPI}, \CRANpkg{doMC}, \CRANpkg{doSNOW} packages).
\end{itemize}

A simple example is shown below:

<<eval=FALSE>>=
library(ff)
library(foreach)
library(doMC)
registerDoMC(2)
library(oligo)
@ 
<<eval=FALSE>>=
rawData <- read.celfiles(list.celfiles())
rmaRes <- rma(rawData)
rmaRes
@ 

\section{Parallel Computing on Multicore Machines}

On multicore machines, one alternative for parallel preprocessing is
shown below. It assumes that the machine has enough RAM to deal with the
dataset and that the \CRANpkg{ff} package is \textbf{NOT} loaded. The
snippet compares the performance between a single-threaded run of
\Rfunction{rma}, although \Rfunction{fitProbeLevelModel} would also benefit
from it, and a run using 4 threads (which is enabled by setting the
\Rcode{R\_THREADS} environment variable).

<<>>=
library(oligoData)
data(affyExonFS)
t0 <- system.time(res0 <- rma(affyExonFS))
Sys.setenv(R_THREADS=4)
t1 <- system.time(res1 <- rma(affyExonFS))
all.equal(res0, res1)
t0
t1
@

\chapter{Session Info}

<<echo=FALSE>>=
sessionInfo()
@

\newpage

\bibliography{FrameworkPreprocessing}


\end{document}
