\documentclass[a4paper,11pt]{article}
\usepackage{amsmath}
\usepackage[authoryear,round]{natbib}
\usepackage{longtable}
\usepackage{threeparttable}
\usepackage{array}
\usepackage{hyperref}
\usepackage{geometry}
\usepackage{graphicx}

\usepackage{etoolbox}
\preto\tabular{\setcounter{magicrownumbers}{0}}
\newcounter{magicrownumbers}
\newcommand\rownumber{\stepcounter{magicrownumbers}\arabic{magicrownumbers}}

\usepackage{pdflscape}
\geometry{verbose,a4paper,tmargin=26mm,bmargin=26mm,lmargin=28mm,rmargin=28mm}

\usepackage[singlelinecheck=off]{caption}


\usepackage[latin1]{inputenc}
\usepackage{datetime}
\newdateformat{mydate}{\THEDAY-\monthname[\THEMONTH]-\THEYEAR}

\title{Benchmarking ADaCGH2 and comparison with previous versions}
\date{\mydate\today}
\author{Ramon Diaz-Uriarte$^{1}$}






\begin{document}
\maketitle

\begin{center}
1. Department of Biochemistry,
Universidad Autonoma de Madrid
Instituto de Investigaciones Biomedicas ``Alberto Sols'' (UAM-CSIC), Madrid
(SPAIN).
{\tt rdiaz02@gmail.com}
\end{center}

\tableofcontents
\listoftables
\listoffigures


\clearpage

\section{Introduction}
I provide here comparisons with the former version of ADaCGH2 (v.\ 1.10,
as available in BioConductor v.\ 2.12) as well as some comparisons against
non-parallelized executions and, finally, some details about recommended
patterns of usage. The benchmarks shown here total more than 2047 hours of
wall time (more than 85 days) and correspond to over 400 runs for reading
and 370 for analysis. The purpose of these benchmarks is to show the
differences in performance between the new and old versions, as well as to
illustrate the effects of changing several parameters in the new
version. Before showing the results, we provide information about the
hardware and data sets.



It should be noted that the very first version of ADaCGH (the one
documented in \cite{Diaz-Uriarte2007}) will no longer run in current
versions of R without tweaks, as it depends on a package (papply) that no
longer installs in current versions of R (it had problems at least since
v.\ 2.15.0 of R). The web-based application documented in that paper still
works because I did perform those tweaks, and because in some servers we
are still running version 2.9.0 of R. However, for the final user,
comparisons against that initial version are therefore of little
interest. Thus, the benchmarks that I show provide comparisons against the
version available from release 2.12 of BioConductor; this ``old'' version
(v.\ 1.10) still runs, but already incorporates several major advantages
over the initial one documented in the PLoS ONE paper, mainly:


\begin{itemize}

\item Clusters are not restricted to MPI cluster, whereas the initial
  version only allowed MPI clusters; so, for instance socket clusters
  (much easier to use in Windows than MPI clusters) were not available.

\item There are no dependencies on deprecated or orphaned packages so the
  package will install in current versions of R.

\item \textit{ff} objects start to be used. The very first version always
  required the complete data set to be in memory in the master process for
  the duration of the analysis, and MPI moved around actual columns of the
  data frame, and not just pointers to \textit{ff} objects. As a
  consequence, for the initial version, memory requirements for analysis
  were actually higher than the memory requirements of reading data of the
  ``old'' version; thus the largest possible data for analysis were
  smaller than the ones for v.\ 1.10, and analyses were also  slower.
\item Two algorithms, ACE and PSW, were eliminated because of little
  usage, and a new and very fast one, HaarSeg, included.
\item Input from, and output to, other BioConductor packages was added. 
\end{itemize}

Therefore, in what follows, ``old'' refers to v.\ 1.10, as available from
BioConductor 2.12, and ``new'' to versions $ge 2.3$, and those are the
versions that will be compared. 


\subsection{Data set and hardware}
\label{sec:data-set-hardware}

We will use a simulated data set that contains 6,067,433 rows and up to
4000 columns of data; these are, thus, data for an array with 6 million
probes and data for up to 4000 subjects. Many of the examples shown below
will use smaller subsets of the data, smaller in terms of the number of
subjects or samples (columns). There are 421,000 missing values per data
column. 

To give an idea of sizes, the ASCII file with the data for the 1000 column
data is about 96 GB\footnote{All sizes are computed from the reported size
  in bytes or megabytes, using 1024, or powers of 1024, as
  denominator.}. The directory with the data for 2000 columns occupies
about 198 GB and, when archived and compressed with bzip2, occupies 78
GB. The RData for the 1000 columns data is 46 GB (without compression; 41
GB with the standard R compression); in a freshly started R session,
loading the RData will use 46 GB (as reported by \texttt{gc()}). The RData
object with the 1000 columns, when loaded into R in the PowerEdges, takes
13 minutes to load and uses a total of about 46 GB (45.7 from calls to
\texttt{gc} before and after loading the object, and adding Ncells and
Vcells, or 45.6 as reported by \texttt{object.size}). Note that this is
not the result of the object being a data frame and having a column with
identifiers (a factor), instead of being a matrix; a similarly sized
matrix with just the numeric data for the probes (i.e., without the first
three columns of ID, chromosome, and location) has a size of 45.2 GB
(therefore, the difference of 300 MB due to the first column, ID, being a
factor with the identifiers, is minute relative to the size of the
matrix).  







 %%             used    (Mb) gc trigger    (Mb)   max used    (Mb)
 %% Ncells    6272458   335.0    9532693   509.2    6273371   335.1
 %% Vcells 6102207940 46556.2 6634368466 50616.3 6102370100 46557.4 









The examples below were run on a Dell PowerEdge C6145 chasis with two
nodes. Each node has 4 AMD Opteron 6276 processors; since each processor
has 16 cores, each node has 64 cores. One node has 256 GB RAM and the
other 384 GB of RAM. Both nodes are connected by Infiniband (40Gb/s).  For
the data presented here, when using a single node, the data live on an xfs
partition of a RAID0 array of SAS disks (15000 rpm) that are local to the
node doing the computations. When using the two nodes, the data live on
one of the xfs partitions, which is seen by the other node using a simple
NFS setup (we have also used distributed file systems, such as FhGFS, but
they have tended to be a lot slower in these experiments; your mileage
might vary). Therefore, in the table entries below, executions using both
nodes will indicate ``124 cores'' \footnote{124 is not a typo; it is 124,
  even if the total number of cores is $128 = 64 * 2$. This is due to the
  following documented issue with Open MPI and Infiniband:
  \url{http://www.open-mpi.org/community/lists/users/2011/07/17003.php},
  and since $128^2 = 16384$, we are hitting the limit, and we have not had
  a chance to correct this problem yet. Regardless, the penalty we would
  pay would be a difference of 4 process out of 124.}

  
We will also show some examples run on an HP Z800 workstation, with 2
Intel Xeon E5645 processors (each processor has six cores), and 64 GB of
RAM. The data live on an ext4 partition on a SATA disk (7200 rmp). % In the


In both systems, the operating system is Debian GNU/Linux (a mixture of
Debian testing and Debian unstable). The Dell PowerEdge nodes were running
version R-2.15.1 as available from the Debian repository (v.\ 2.15.1-5)
or, later, R-3.0.1, patched (different releases, as available through May
and June of 2013), and compiled from sources. The Xeon workstation was
running R-2.15.1, patched version (2012-10-02 r60861), compiled from
sources or, later R-3.0.1, patched (different releases, as available
through May and June of 2013). Open MPI is version 1.4.3-2 (as available
from Debian).


\subsection{Segmentation methods used}\label{methods}

The methods available in ADaCGH2 are (see further details in the help of
function \texttt{pSegment}):

\begin{itemize}

\item The popular Circular Binary Segmentation approach, described in its
  current implementation in \cite{CBS-paper} and implemented in package
  \texttt{DNAcopy}.

\item A wavelet-based method, proposed in \cite{waves-hsu}. This is called
  \texttt{pSegementWavelets} in ADaCGH2. 

\item Another wavelet-based method, HaarSeg, published in
  \cite{haarseg-paper}. This was later made available as an R-package as
  \cite{haar-package}.


\item HMM, as described by \cite{hmm-fridlyand} and implemented in package
  \cite{aCGHpackage}. 

\item BioHMM, a non-homogeneous HMM, described in \cite{biohmm} and
  implemented in package \cite{snapCGH}. 


\item The CGHseg method described in \cite{cghseg}. An implementation of
  \textbf{part} of this method is available in the R package
  \cite{tilingarray}, but ADaCGH2 is the first R implementation of the
  full description of the CGHseg procedure (see comments in the help of
  function \texttt{pSegmentCGHseg}).


\item GLAD, a method first described in \cite{glad-paper} and implemented
  in \cite{glad-package}.

\end{itemize}


\subsubsection{Difficulties of using some methods with large data sets}\label{methodproblems}

The tables below only show benchmarks for methods HMM, BioHMM, HaarSeg
(referred as Haar) and CBS. CGHseg and the wavelet approach described in
\cite{waves-hsu} cannot be used when any chromosome has a large number of
probes because of their memory use. With CGHseg the problem arises in the
underlying \texttt{tilingArray} package, in the internal step of computing
the ``costMatrix'', a function called by the function \texttt{segment} in
\texttt{tilingArray}.  When analyzing the first chromosome (for a single
subject), the request is for 1493 GB.  For the wavelet approach, the
problem shows up in the clustering step, when function \texttt{pam} (from
package \texttt{cluster}) is called. For instance, the memory requirements
for a chromosome of 350000 probes would exceeded 400 GB (the request is
for a vector of doubles of size $1 + (n * (n - 1))$).  It must be
emphasized that, in both cases, it is not the complete 6 million probes,
nor using multiple subjects, which causes the problems: neither of the
methods is capable of analyzing the first chromosome for a single subject.
GLAD seems capable of dealing with large data sets in terms of memory
usage, but it is extremely slow. After more than four days, the method had
not been able to finish the analysis of the 50-column data set in the
machines with 64 cores; on closer inspection, the problem lies in function
\texttt{OptimBkpFindCluster}, a C function internal to the package, and is
not attributable, therefore, to the initial segmentation method (we were
using, anyway, the recommended fast function, which uses HaarSeg).
Finally, to run method BioHMM we often had to increase the \texttt{ulimit}
(stack limit), by using \texttt{ulimit -u}, from the shell.



\subsection{Tables: column name explanation}
For the tables below, the meaning of columns is as follows:

\begin{description}

\item[Wall time (min.)] The ``elapsed'' entry returned by the command
  \texttt{system.time}. This is the real elapsed time, the wall time, in
  minutes, since the function was called.
  
  It is important to understand that these timings can be variable. In
  many cases, we show repeated executions with the exact same settings,
  that will help show the variability in those numbers.

\item[Memory (GB)] The memory used by the master R process. This is the
  sum of the two rows of the ``max used'' column reported by
  \texttt{gc()}, in R, at the end of the execution of the given
  function. This number cannot reflect all the memory used by the function
  if the function spawns other R processes (via MPI or forking, for
  example).

\item[$\Sigma$ Memory (GB)] A simple attempt to measure the memory used by
  all the processes\footnote{Just adding the entries given by \texttt{top}
    or \texttt{ps} will not do, and will overestimate, sometimes by a huge
    amount, the total memory used.}. Right before starting the execution
  of our function, we call the operating system command \texttt{free} and
  record the value reported by the ``-/+ buffers/cache'' row. Then, while
  the function is executing, we record, every 0.1 seconds (or every 0.05
  seconds), that same quantity. The largest difference between the
  successive measures and the original one is the largest RAM
  consumption. Note that this is an approximation. First, if other process
  start executing, they will lead to an overestimation of RAM usage; this,
  however, is unlikely to have had serious effects (the systems were
  basically idle, except for light weight cron jobs), though a few results
  in the tables suggest this happened in a few instances (related to
  backup processes). Second, sampling is carried out every 0.5 seconds, so
  we could miss short peaks in RAM usage but, again, this is unlikely to
  lead to a serious underestimation.
  % (at least one likely example of such underestimation shows in the
  % tables below)
  % . Finally, the coherence of repeated executions and the clear patterns
  % both within and among machines suggests that this procedure captures the
  % key

  Finally, note that for cases where we know that there is a single R
  process (e.g., reading with the old version), there is an excellent
  agreement between the ``Memory (GB)'' (whose value is reported from R
  itself) and ``$\Sigma$ Memory (GB)''. 
  
  \item[Columns] The number of data columns of the data set; the same as
    the number of arrays or the number of samples.
  \item[Method] The analysis method. ``Haar'' for HaarSeg, ``CBS'' for
    Circular Binary Segmentation (from package \texttt{DNAcopy}), ``HMM''
    for the HMM approach in package \texttt{aCGH}, ``BioHMM'' for the
    non-homogeneous HMM method in package \texttt{snapCGH}, and ``GLAD''
    for the method with the same name in package \texttt{GLAD}. See 
    section \ref{methodproblems} for why other methods are not shown in
    the tables.


  \item[MPI/Fork] Whether forking (via \texttt{mclapply}) or explicitly
    using an MPI cluster (using the facilities provided by package
    \textbf{Rmpi}, which are called from package \textbf{snow}) are used
    to parallelize execution.

    The ``NP'' entries in table \ref{nonparall} refer to non-parallelized
    execution, using the original packages\footnote{The packages are
      DNAcopy, as available from BioConductor, and the HaarSeg package,
      available from R-forge:
      \url{https://r-forge.r-project.org/projects/haarseg/}}.

    The entries marked as ``-LB'' correspond to the load-balanced options
    with the new version of ADaCGH2 (setting \texttt{loadBalance = TRUE},
    in v.\ $\ge 2.3.4$).


  \item[Cores] Number of cores used. In most cases, when running in the
    AMD Opteron machines we used all 64 cores, and when running on the
    Intel Xeon machine we used all 12 cores, but not always, to show the
    effects of changing the number of cores used. When running over both
    AMD Opterons we used 124 cores (see above).

  \item[Procs.\ per node] When using MPI, the total number of R processes
    that can run in a node; this is the parameter \texttt{npernode} passed
    to \texttt{mpirun} (from Open MPI). When running on a single node,
    that is the number of R slaves + 1.

  \item[Universe size] The number of slave nodes in the MPI universe (over
    all nodes in the universe). This is the parameter \texttt{count}
    passed to \texttt{makeMPIcluster} in R.

\item[Version] The version of ADaCGH2. For simplicity, ``Old'' means
  version 1.10 and ``New'' versions 2.1 and larger.
  
  The post-fix ``-noNA'' means the new version was run using option
  \texttt{certain\_noNA = TRUE}; note that the old version of ADaCGH2
  assumes there are no missing values in the data. Thus,
  \texttt{certain\_noNA = TRUE} is the closest to what the old version assumes. 

\item[ff/RAM] Where applicable in the tables, if the data for the analysis
  had been stored as an \textit{ff} object, or as a data frame inside an
  RData that was loaded before the analysis.
  

\end{description}



\clearpage
\section{Comparison with v.\ 1.10 of ADaCGH2}

\subsection{Main differences between the old (v.\ 1.10) and new versions
  (v.\ $\ge$ 2.3.4) of ADaCGH2}


The code for the new version of ADaCGH2 represent a major rewrite of most
of the code in the former version. Listed here are some of the major
advantages of the new version\footnote{Most of the new capabilities were
  already available in version 2.1.3; however, the vignettes have suffered
  major changes and there have been some changes in the code and help
  files. Thus, these comments all do apply to version $\ge$ 2.3.4.}; they
are shown in approximately decreasing order of importance from the user's
point of view.


\begin{description}

\item[Reading of large data sets] The new version of ADaCGH2 can read data
  sets much larger than the old one (see section \ref{comp-read}). In a
  machine with 64 GB RAM the old version cannot read data sets with 500
  columns (each with 6 million probes ---see section
  \ref{sec:data-set-hardware}), whereas data sets with 4000 columns can be
  read with the new version (see table \ref{read-coleonyx}) and the
  scaling of the memory consumption with number of columns suggests that
  much larger data sets could be read. Likewise, in machines with 256 and
  384 GB of RAM (tables \ref{read-gallotia} and \ref{read-lacerta}) data
  sets of 2000 columns could not be read with the old version of ADaCGH2,
  but data sets of 4000 columns are read with the new version and, again,
  the scaling of memory consumption with number of columns suggests (see
  Figure \ref{fig-read}) that much larger data sets could be read and,
  even for the sizes of data that can be read by the old version, reading
  is much faster with the new version because of the parallelized reading,
  which can make much better usage of available hardware (e.g., RAID
  arrays for disks). % In summary, given the scaling pattern, it seems the
  % size limits for reading data are not memory but hard drive space.



\item[Missing value handling] The old version of ADaCGH2 used row-wise
  deletion of missing values when reading data: a probe would be deleted
  from the data if it had one missing value in any
  subject/column. Analysis could be speed up, as no checks or provisions
  had to be taken for dealing with NAs, and all procedures are simplified,
  as the data are then known to be complete. However, row-wise deletion of
  missing values is probably not an appropriate approach, especially as
  the number of samples increases (because the probability that a given
  probe will then be left out of the analysis increases). The new version
  of ADaCGH2 deals with missing values column by column, so for each
  column (or subject) all available data (or probes) are used in the
  segmentation. Nevertheless, the new version incorporates a setting to
  provide speed ups when the user is certain that there are no missing
  values (\texttt{certain\_noNA = TRUE}).


\item[Analysis of large data sets] The old version of ADaCGH2 cannot
  analyze large data sets, as it cannot read them (and it cannot use data
  read by the new version since the old version assumes there are no
  missing values in the data after reading).

 In addition, although time increases, obviously, with number of samples
  to analyze, the scaling of memory consumption is modest and well below
  the memory available for the systems.

\item[Forking and clusters] The new version of ADaCGH2 allows for the
  usage of forking or an explicit cluster (e.g., MPI, sockets, etc) to
  parallelize reading and analysis. In POSIX operating systems (including
  Unix, GNU/Linux, and Mac OS), forking can be faster, less memory
  consuming, and much easier to use than using a cluster.


\item[Speed of analysis] The new version can be slightly faster than the
  old one for the default options. Further speed improvements can be
  achieved in some cases, for instance by not using load balancing with
  certain methods (e.g., HaarSeg).

\item[Flexibility of reading data] The new version of ADaCGH2 has not
  removed the mechanisms of reading data available in the old
  version. Thus, when data are small or memory is plentiful, reading data
  from a single RData is an available option. But the new version adds new
  mechanisms, mainly reading from a text file and from a directory of text
  files that, as discussed above, allow for reading much larger data sets.

\item[Usage of data read from the other version] The new version of
  ADaCGH2 can accept data read by the old version. However, the old
  version of ADaCGH2 cannot accept data from read by the new version
  unless the original data contained no missing values at all: the old
  version of ADaCGH2 assumes that data that have been read contain no
  missing values.



\item[Dependencies] The old version depends on package \texttt{snowfall}
  for parallelization, whereas the new version depends only on
  \texttt{multicore}. This makes the new version less likely to break in
  the future, as \texttt{multicore} is one of the core packages
  distributed with R (whereas, for instance, there were some problems with
  \texttt{snowfall} not building with the development versions of v.\ 3 of
  R around February 2013).

  % In addition, snowfall is difficult to use with Open MPI, probably the
  % most widely used approach for MPI, and often the recommended one for R.


\item[More flexible options for load balancing] The old version of ADaCGH2
  forced load balancing. Whether or not load-balancing is the best
  approach depends on the size and number of jobs relative to the number
  of cores. As shown in the tables (see tables \ref{anal-coleonyx},
  \ref{anal-lacerta}, \ref{anal-gallotia}), not using load balancing can
  sometimes lead to speed improvements. The new version of ADaCGH2 allows
  not to use load balancing with the argument \texttt{loadBalance = FALSE}.

\item[Limiting memory consumption] Memory usage is generally well below
  the available memory of the system. However, if it were necessary to
  limit memory usage during reading and analysis this is simple with the
  new version of ADaCGH2: limit the number of processes that are allowed
  to run simultaneously. This is not possible with the old version of
  ADaCGH2.

\item[Method availability] Two of the methods available, HMM and BioHMM,
  depend on packages \texttt{aCGH} and \texttt{snapCGH}. These two
  packages haven not been updated since 2010 and 2009, respectively, and
  aCGH will no longer be maintained (personal communication from the
  authors). There is code in the ADaCGH2 repositories (including both C
  and R code), taken and modified from those packages, that can be readily
  uncommented to make these two methods available in ADaCGH2 if either of
  those packages were not to pass checks in future versions of
  BioConductor.
 
\end{description}



\subsection{Data and code availability}

All scripts, data, and results from these benchmarks are available from
\url{http://www2.iib.uam.es/rduriarte\_lab/ADaCGH2-v2-suppl-files/public}. The
scripts include the R code to obtain the tables and figures shown
here. All of the code, scripts, and benchmark results are also available
from my personal web site at
\url{http://ligarto.org/rdiaz/Papers/ADaCGH2-v2-suppl-files/} (for space
restriction reasons, the more than 140 GB of data in the form of RData and
txt files are only available from the previous site.)



\clearpage
\subsection{Reading data}\label{comp-read}

The next three tables show time and memory consumption when reading data
using the recommended approach with each version of the package (an RData
object for the old version, a directory of single-column txt files for the
new version). Some of the major patterns and results are:

\begin{description}
\item[Size limits for old version] Table
\ref{read-coleonyx} \textbf{cannot} show reading benchmarks for the old
version with data sets of sizes 500, 1000, 2000, or 4000, as those could
not be read with the old version (R run out of memory). Likewise, tables
\ref{read-gallotia} and \ref{read-lacerta} show reading benchmarks for the
old version with up to 1000 columns, because the AMD Opteron machines with
$\ge$ 256 GB RAM could not read data sets of sizes 2000 or
4000, as R could not allocate the necessary memory.% \footnote{The details about how much memory can be
  % successfully requested by a single process depend on settings such as
  % the overcommit policy and the vm.overcommit_ratio and the RAM and swap
  % of the system and how much memory is already being used by other
  % processes. For the three systems used, the maximum memory that could be
  % requested, as shown in \texttt{CommitLimit} in \texttt{/proc/meminfo}
  % was 66050908 kB, 262878296 kB, and 395261068 kB, for Coleonyx, Gallotia,
  % and Lacerta. This numbers, when using a vm.overcommit setting of 2,
  % i.e., with overcommit disabled, are the same as those obtained from
  % vm.overcommit_ratio * RAM + swap. In all cases, the machines were under
  % very light load from other processes when running the benchmarks, and
  % thus R was not killed when requests were of about, or larger than, the
  % figures given above.}.

In contrast, the new version is capable of reading data sets of 4000
columns in all machines, without getting anywhere near the memory limits
of the machines. Moreover, the \textbf{scaling} (see also figure
\ref{fig-read}) shows that the total number of columns could be increased
to much larger numbers and, in addition, that the total memory used can be
limited by reducing the number of cores used (with little effects on speed
---see next point).% : for instance, compare row 32 with 33, and row 34


\item[Speed of new and old version] Reading is much faster with the new
  version. These differences are most likely inconsequential for small
  sized data sets (where differences are by a factor of about 2x), but can
  have large effects with a large number of columns.  As data sets grow
  larger in size, reading speeds are much faster with the new version than
  the old by factors of about 10x (this can only be verified in the
  machines with larger memory, as the old version will not read data sets
  of 500 columns or more in the smaller machine). There can,
  nevertheless, be quite a bit of variation in reading speeds of small
  data sets, specially in less capable machines; these variations,
  however, are most likely of little practical relevance.

  


\item[Speed and number of cores in new version] Reading speed does not
  always increase with the number of cores. In fact, for a range of number
  of cores, reading speeds show little variation with number of cores, as
  the most likely limiting factor is I/O, which is related to the number
  of spindles and the speed of the drives. Increasing the number of cores
  used, however, tends to make the system less responsive (higher loads)
  and thus using a reasonably small number of cores is recommended and the
  default option.


  % \ref{read-coleonyx}; rows 41 to 44 in table \ref{read-gallotia}; and
  % rows 30 to 33 in table \ref{read-lacerta}.
  % In particular, the number of spindles and the speed of the drives is
  % likely to have a large effect on the benefits of parallelizing
  % reading. In the Intel Xeon machine all the data are on an ext4 partition
  % that lives on a single SATA hard drive, whereas in the AMD Opteron
  % machines the data are on an xfs partition that is, physically, for each
  % machine, on four 600 GB SAS disks configured as a RAID0 array. In
  % addition, the reading step is generally faster the less full the
  % partition is and the type of file system is also likely to affect.

  If the reading operation is to be performed many times, or on very large
  set of data, it would pay off to experiment with the number of cores
  used for reading, which can be done with the option \texttt{mc.cores} to
  the function \texttt{inputToADaCGH}.

\end{description}

\clearpage
\begin{figure}[h!]
\begin{center}
  \includegraphics[width=16.1cm,keepaspectratio]{reading-benchmark-fig.pdf}
\end{center}
\caption[Wall time and memory usage when reading data: version
comparison]{\label{fig-read} Comparison between old and new versions in
  wall time and total memory usage (over all spawned processes) when
  reading data as a function of number of columns (or arrays or
  samples). Both axes shown in log scale. The figure shows the benchmarks
  using 12 cores in the Intel Xeon machine and 64 cores in the AMD
  Opterons; note that for some scenarios better speeds (and lower memory
  usage) can be achieved by decreasing the number of cores used (see
  tables). When more than one benchmark is available for a scenario, the
  median is shown.}

\end{figure}

\begin{center}
\input{read-coleonyx.tex}
\clearpage
\input{read-gallotia.tex}
\clearpage
\input{read-lacerta.tex}
\end{center}


\clearpage
\subsection{Analyzing data}\label{comp-anal}
The next four tables show time and memory consumption when analyzing
data. For the old version, the largest data sets analyzed are of 200
columns for the Intel Xeon machine with 64 GB of RAM, and 1000 columns for
the AMD Opteron machines (see details in section \ref{comp-read}). In
these benchmarks, runs were not allowed to run for more than 36 hours
(2160 minutes) except for a few cases that were allowed to run for longer
to either compare between methods (e.g., HMM in Coleonyx) or to verify
that the method is definitely not suitable for very large data, such as in
the case of GLAD, where two processes were allowed to run for four days
(see section \ref{methodproblems}). Finally, note that we do not compute
the time it takes to set up the MPI environment (with the old version or
with the new version, when using MPI), but only the time of the call for
the segmentation itself; setting up the cluster takes about half a minute
to a minute.


Some of the major patterns and results shown in tables \ref{anal-coleonyx}
to \ref{anal-lacerta-gallotia} (see also Figure\ref{fig-anal}) are:

\begin{description}

\item[Version comparison] There are small speed differences between the
  old and new versions, generally favoring the new version, specially with
  HaarSeg and CBS. The new version generally also uses less memory than
  the old version. The main difference, however, is that the new version
  can analyze much larger data sets, as the old version is limited by the
  size of the data sets that can be read (see section \ref{comp-read}).

\item[Load balancing] Load balancing is generally a good choice, but not
  with HaarSeg on a single multicore machine, because the individual
  analysis of HaarSeg are so fast that they rarely make it worth it the
  increased communication and processing overheads of load balancing.

\item[MPI vs.\ forking] Forking is faster than MPI when running on a
  single node, which is to be expected, and in some cases (e.g., HMM) the
  differences can be very large.

\item[Running over several nodes] Even with fast communication between
  nodes (as in this case) duplicating the number of cores might not result
  in significant decreases in wall time for the fastest methods. In
  particular, Wall time for HaarSeg is actually larger when run over two
  nodes. For CBS there is a slight advantage of running over two
  nodes. Running over more than one node to increase the number of
  cores/CPUs is, however, advantageous for the slower methods (e.g.,
  HMM).  
  
  Note that these results are \textbf{highly hardware dependent}: slower
  communication between nodes or slower I/O from shared storage will make
  running over several nodes less worth it. However, increasing the
  available number of cores/CPUs by larger factors (e.g., 4x or 8x) might
  make it worth it to use them even for fast methods such as HaarSeg.

\end{description}


\clearpage
\begin{figure}[h!]
\begin{center}
  \includegraphics[width=16.1cm,keepaspectratio]{anal-benchmark-fig.pdf}
\end{center}
\caption[Wall time and memory usage when analyzing data: version
comparison]{\label{fig-anal} Comparison between old and new version in
  wall time and total memory usage (over all spawned processes) as a
  function of number of columns (or arrays or samples). Both axes shown in
  log scale. The figure shows the default use cases: using 12 cores in the
  Intel Xeon machine and 64 cores in the AMD Opterons. Since the old -
  version assumes no missing data, when possible (i.e., when data read by
  the old version are available) the data without missing values have been
  used with option \texttt{certain\_noNA = TRUE}; these correspond to rows
  labeled ``New-noNA'' in tables \ref{anal-coleonyx} to
  \ref{anal-lacerta-gallotia}. When more than one benchmark is available
  for a scenario, the median is shown.}
\end{figure}



\begin{landscape}
\begin{center}
\input{anal-coleonyx.tex}
\clearpage
\input{anal-gallotia.tex}
\clearpage
\input{anal-lacerta.tex}
\clearpage
\input{anal-lacerta-gallotia.tex}
\end{center}
\end{landscape}


\clearpage
\section{Other comparisons}

\subsection{Comparison with non-parallelized executions}

ADaCGH2 was run without merging, to compare it to the canonical,
non-parallelized, implementations of CBS and Haar. Note that, as there are
missing values in the data, and the original HaarSeg code does not deal
with missing values, we are forced to remove NAs array-per-array, and make
repeated calls to the function.


\begin{center}
\begin{threeparttable}
  \caption[Time and memory usage of segmentation: comparison with
  non-parallized executions.]{Time and memory usage of segmentation
    without merging and comparison with non-parallized executions. These
    examples have all been run on the Dell Power Edges, except for the
    last two, run on the Intel machine (on the Intel machine
    non-parallelized runs with 1000 columns cannot be attempted as R runs
    out of memory loading the data).}
\label{nonparall}
  \begin{tabular}{lp{1.3cm}p{1.2cm}p{1.0cm}p{0.85cm}p{1.2cm}p{1.9cm}p{1.55cm}p{1.8cm}}

    \hline\hline

&Method& MPI /Fork & Cores & ff/ RAM & Columns & Wall time (min.) & Memory
(GB)& $\Sigma$ Memory (GB)\\

\hline
\rownumber&Haar & Fork &64  & ff & 100 & 1.2   &   0.13  &  24.5  \\
\rownumber&Haar & Fork &10  & ff & 1000 &23.5 & 0.142 & 5.9 \\
\rownumber&Haar & Fork &20 & ff & 1000 & 12.3 & 0.137 & 10.0 \\
\rownumber&Haar & Fork &40  & ff & 1000 & 7.4 & 0.142 & 17.6 \\
\rownumber&Haar & Fork &50  & ff & 1000 & 6.7 & 0.139 & 21.2 \\
\rownumber&Haar & Fork & 64  & ff & 1000 & 6.4 & 0.142 & 26.9 \\


\rownumber&Haar & Fork &10  & ff & 2000 &49.7 & 0.16 & 8.0 \\
\rownumber&Haar & Fork &20  & ff & 2000 & 26.3 & 0.16 & 11.9 \\
\rownumber&Haar & Fork &40  & ff & 2000 & 15.4& 0.16 & 19.5 \\
\rownumber&Haar & Fork &50  & ff & 2000 & 13.3& 0.16 & 23.2 \\
\rownumber&Haar & Fork &64  & ff & 2000 & 11.9& 0.16 & 28.4 \\


&&&&&&&\\
\rownumber&CBS    & Fork & 64 & ff  & 100  & 55.9  &   0.13   &  35.3  \\
\rownumber&CBS & Fork &10  & ff & 1000 &1855.4 & 0.135 & 8.6 \\
\rownumber&CBS & Fork &20 & ff & 1000 & 939.8 & 0.135 & 14.9 \\
\rownumber&CBS & Fork &40  & ff & 1000 & 513.5 & 0.136 & 27.0 \\
\rownumber&CBS & Fork &50  & ff & 1000 & 438.6 & 0.142 & 33.1 \\
\rownumber&CBS & Fork & 64  & ff & 1000 & 350.3 & 0.142 & 41.3 \\

\rownumber&CBS & Fork &10  & ff & 2000 &3770.9 & 0.15 & 11.1 \\
\rownumber&CBS & Fork &20 & ff & 2000 & 1878.8 & 0.16 & 16.5 \\
\rownumber&CBS & Fork &40  & ff & 2000 & 1007.0 & 0.15 & 28.8 \\
\rownumber&CBS & Fork &50  & ff & 2000 & 857.1 & 0.16 & 35.3 \\
\rownumber&CBS & Fork & 64  & ff & 2000 & 717.3 & 0.163 & 41.9 \\

&&&&&&&\\


\rownumber&Haar & NP &-  &RAM  & 100 &  25.2\tnote{a}   & 12.5 & 12.5  \\
\rownumber&CBS     & NP &-  &RAM  & 100 & 1706\tnote{b}  &   40   &  40\\
\rownumber&Haar & NP &-  &RAM  & 1000 &
\multicolumn{3}{c}{198.3\tnote{c}\quad \textit{\textbf{Cannot allocate memory}}} \\
\rownumber&CBS     & NP &-  &RAM  & 1000 &
\multicolumn{3}{c}{\textit{\qquad\textbf{Cannot allocate memory}}}\\

&&&&&&&\\
\rownumber&Haar & NP &-  &RAM  & 100 &  15.1\tnote{d}   & 14.1 & 14.1  \\
\rownumber&CBS     & NP &-  &RAM  & 100 & 1112\tnote{e}  &   38.3   &  38.3\\


\hline
   

  \end{tabular}

  \begin{tablenotes}
    {\footnotesize

     \item[a] 25.25 = 0.95 + 22.9 + 1.4: load data, analyze, and save
       results. %  Since
      % there are missing values in the data, and the original HaarSeg code
      % does not deal with missing values, we are forced to remove NAs
      % array-per-array, and make repeated calls to the function.
      % , both o
      %  which introduce an important penalty. We must do this to make
      % analysis comparable to those of \texttt{pSegmentHaarSeg} that
      % \textbf{always} checks, and allows, for missing values. 
      If there are no missing values in this data set, the total time of
      analysis (i.e., sending the whole matrix at once and not checking
      for, nor removing, NAs) is 3.3 minutes.
    \item[b] 1706 =  0.95 + 1698 + 6.7: load data, analyze, and save results. The
      analysis involves calling the \texttt{CNA} function to create the CNA object
      (5.3 min), calling the \texttt{smooth.CNA} function to smooth the data
      and detect outlier (83.2 minutes), and segmenting the data with the
      \texttt{segment} function (1609.5 minutes).
    \item[c] The analysis uses 113 GB, but results cannot be saved. This
      was in the machine with 256 GB of RAM.
     \item[d] 15.1 = 0.65 + 13.5 + 0.9: load data, analyze, and save
       results.
    \item[e] 1112 =  0.65 + 1106 + 4.9: load data, analyze, and save results. The
      analysis involves calling the \texttt{CNA} function to create the CNA object
      (0.95 min), calling the \texttt{smooth.CNA} function to smooth the data
      and detect outlier (20.2 minutes), and segmenting the data with the
      \texttt{segment} function (1085 minutes).

    % \item[f] These examples have been run on the Dell Power Edges, but
    %   changing the number of cores used.
      }
  \end{tablenotes}

\end{threeparttable}
\end{center}




\clearpage
\subsection{Reading from a directory of files vs.\ other options}

Here we show time and memory usage of options that are not the recommended
approach with large data set (using \textit{ff} objects and reading from a
directory of single-column files). All these benchmarks have been carried
out in the AMD Opteron machines. These data show the patterns discussed in
the main vignette: with large data sets the best approach is to read from
a directory of single-column files and store as \textit{ff} objects. Wall
time is much smaller when reading from a directory of single column files
(see also table \ref{read-coleonyx} for a comparison with former versions
of ADaCGH2, where original data where stored as RData and then read to
\textit{ff} objects). Moreover, storing as a RAM object, even when
possible, might result in a RAM object that can then not be successfully
used for analysis (see section \ref{ramobj}).


\renewcommand{\arraystretch}{1.8}

\begin{center}
\begin{threeparttable}
  \caption{Time and memory usage when reading data}    
\label{large-ex-read}

  \begin{tabular}{lp{3.62cm}p{1.7cm}p{1.8cm}p{1.8cm}p{3.99cm}}
 
 \hline   \hline
& Reading operation&Columns & Wall time (min)& Memory (GB) & $\Sigma$ Memory (GB)\\
\hline

\rownumber&Txt file to \textit{ff} & 1000 & 2630 & 1.3 & NA\\ % (not calculated) \\
\rownumber&RData to \textit{ff} & 1000 & 29.6   &   169 & 168.3 \\
\rownumber&Directory to data frame (RAM object) & 1000 & 22 + 2\tnote{a} & 96 & NA. Output unusable for analysis. See table \ref{ramobj}. \\
\rownumber&RData to data frame (RAM object) & 1000 & 22 + 2\tnote{b} &  139 & NA. Output unusable for analysis. See table \ref{ramobj}. \\
\rownumber&Directory to data frame (RAM object) & 200 & 7.7 + 0.4\tnote{c} & 20 & 38 \\
\rownumber&Directory to data frame (RAM object) & 100 & 8.7 + 0.3\tnote{d} & 10.5 & 30 \\
\rownumber&Directory to data frame (RAM object) & 50 & 5.8 + 0.1\tnote{e} & 5.8 & 27 \\



\hline
   
  \end{tabular}

  \begin{tablenotes}
    {
      \footnotesize
    \item[a] The 2 reflects the time needed to save the resulting data frame to an RData file.
    \item[b] The 2 reflects the time needed to save the resulting data frame to an RData file.
    \item[c] The 0.4 reflects the time  needed to save the resulting data frame to an RData file.
    \item[d] The 0.3 reflects the time  needed to save the resulting data frame to an RData file.
    \item[e] The 0.1 reflects the time  needed to save the resulting data frame to an RData file.
    }
    \end{tablenotes}
\end{threeparttable}
\end{center}








\clearpage
\subsection{Analyzing large data with RAM objects}
\label{ramobj}
Here we present some results from attempting to analyze large data sets,
with the new version of ADaCGH2, using RAM objects. Even moderately size
data sets (200 columns) cannot be analyzed when using RAM objects in a
machine with 384 GB of RAM; memory usage is already very large (140 GB)
with just 50 columns. Time of analysis is also much larger for the case
shown than for similarly sized problems when using \textit{ff} objects
(see tables \ref{anal-gallotia} and \ref{anal-lacerta}).


\renewcommand{\arraystretch}{1.4}

\begin{center}
  
\begin{threeparttable}

  \caption{Time and memory usage of segmentation with default options}    
  \begin{tabular}{lp{1.2cm}p{1.0cm}p{1.0cm}p{1.1cm}p{1.35cm}p{3.0cm}p{1.6cm}p{1.6cm}}

    \hline\hline

&Method& MPI /Fork & Cores & ff/RAM & Columns & Wall time (min.) & Memory (GB) & $\Sigma$ Memory (GB)\\

\hline
\rownumber&Haar & Fork &64  & RAM & 50 & 0.7 + 2.5 + 0.9\tnote{a}& 14.4 & 140\\
\rownumber&Haar & Fork &64  & RAM & 200 & NA & NA & \textbf{Cannot allocate memory}\\
\rownumber&Haar & Fork &64  & RAM & 1000 & NA & NA & \textbf{Cannot allocate memory} \\
\hline
   
  \end{tabular}

  \begin{tablenotes}
    {\footnotesize
    %% \item[a] With option \texttt{merging = "none"} to make it comparable
    %%   to the canonical code in the original packages.

    %  \item[b] 0.95 + 22.9 + 1.4: load data, analyze, and save results. Since
    %   there are missing values in the data, and the original HaarSeg code
    %   does not deal with missing values, we are forced to remove NAs
    %   array-per-array, and make repeated calls to the function.
    %   % , both o
    %   %  which introduce an important penalty. We must do this to make
    %   % analysis comparable to those of \texttt{pSegmentHaarSeg} that
    %   % \textbf{always} checks, and allows, for missing values. 
    %   If there are no missing values in this data set, the total time of
    %   analysis (i.e., sending the whole matrix at once and not checking
    %   for, nor removing, NAs) is 3.3 minutes.
    % \item[c] 0.95 + 1698 + 6.7: load data, analyze, and save results. The
    %   analysis involves calling the \texttt{CNA} function to create the CNA object
    %   (5.3 min), calling the \texttt{smooth.CNA} function to smooth the data
    %   and detect outlier (83.2 minutes), and segmenting the data with the
    %   \texttt{segment} function (1609.5 minutes).
    %% \item[d] Since only one process, same as previous column.
    \item[a] 0.7 + 2.5 + 0.9: load data, analyze, and save results.
    % \item[f] These examples have been run on the Dell Power Edges, but
    %   changing the number of cores used.
    }
  \end{tablenotes}

\end{threeparttable}
\end{center}




            





\clearpage
\section{Comments and recommended usage patterns}
\label{commentsend}


\subsection{Recommended usage: summary}\label{comments-summary}


\begin{enumerate}
\item For data analysis, use the defaults when running on a single
  multicore computer:
  \begin{enumerate}
  \item \textit{ff} objects for input and output.
  \item Forking (instead of explicit clusters).
  \item Load balancing, except when using HaarSeg.
  \end{enumerate}

\item If you have multiple machines available for analysis, try using them
  with explicit clusters (e.g., MPI). However, especially for methods such
  as HaarSeg, the gains could be modest unless you add many machines to
  the cluster. Use load balancing for all methods (i.e., override the
  defult if using HaarSeg).

\item When reading data, the fastest and least memory consuming is using a
  directory of single-column files. The best number of cores is likely to
  be strongly hardware (and possibly file system) dependent. The default
  \texttt{mc.cores} has been set to half the number of cores, but this is
  not necessarily a sensible default.
\end{enumerate}


\subsection{Recommended usage: details}\label{comments-detail}
\begin{enumerate}

\item Reading data and trying to save it as a RAM object, a usual
  in-memory data frame, will quickly exhaust available memory. For these
  data, we were not able to read data sets of 100 or more columns. Part of
  the problem lies on the way memory is handled and freed in the slaves,
  given that we are returning lists. In contrast, when saving as
  \textit{ff} objects, the slaves are only returning tiny objects (just
  pointers to a file).
  
\item Saving data as RData objects will also not be an option for large
  numbers of columns as we will quickly exhaust available memory when
  trying to analyze them. 
  
  
\item In a single machine, and for the same number of cores, analyzing
  data with MPI is often generally slower than using forking, which is not
  surprising. Note also that with MPI there is an overhead of spawning
  slaves and loading packages in the slaves (which, in our case, takes
  about half a minute to a minute). %% These overheads could be amortized if
  %% you continue using MPI for a session (but OpenMPI is not ideal for that
  %% scenario, in contrast to other versions such as LAM MPI).
  
  
\item When using two nodes (i.e., almost doubling the number of cores),
  MPI might, or might not, be faster than using forking on a single
  node. Two main issues affect the speed differences: inter-process
  communication and access to files. In our case, the likely bottleneck
  lies in access to files, which live on an array of disks that is
  accessed via NFS. With other hardware/software configurations, access to
  shared files might be much faster. Regardless, the MPI costs might not
  be worth if each individual calculation is fast; this is why MPI with
  HaarSeg does not pay off, but it does pay off with HMM and is borderline
  with CBS.
  
  
\item When using \textit{ff}, the exact same operations in systems with
  different RAM can lead to different amounts of memory usage, as
  \textit{ff} tries autotuning when starting. 

  You can tune parameters when you load the \textbf{ff} package, but even
  if you don't (and, by default, we don't), defaults are often sensible
  and will play in your favor.
  
\item Even for relatively small examples, using \textit{ff} can be faster
  than using RAM objects. Using RAM objects incurs overheads of loading
  and saving the RData objects in memory, but analyses also tend to be
  slightly slower. The later is somewhat surprising: with forking and RAM
  objects, the R object that holds the CGH data is accessed only for
  reading, and thus can be shared between all processes. We expected this
  to be faster than using \textit{ff}, because access to disk is several
  orders of magnitude slower than access to memory ---note that we made
  sure that memory was not virtual memory mapped to disk, as we had
  disabled all swapping. We suspect the main difference lies in
  bottlenecks and contention problems that result from accessing data in a
  single data frame simultaneously from multiple processes, compared to
  loading exactly just one column independently in each process, and/or
  repeated cache misses.
  
  
\item \texttt{inputToADaCGH} (i.e., transforming a directory of files into
  \textit{ff} objects) can be severely affected, of course, by other
  processes accessing the disk. More generally, since with
  \texttt{inputToADaCGH} several processes can try to access different
  files at once (we are trying to parallelize the reading of data), issues
  such as type of file system, configuration and type of RAID, number of
  spindles, quality of the RAID card, amount of free space, etc, etc, etc,
  can have an effect on all heavy I/O operations.  Note also that, as a
  general rule, it is better if the newly created \textit{ff} files from
  \texttt{inputToADaCGH} are written to an empty directory, and if the
  working directory for segmentation analysis is another empty directory
  if you are using \texttt{ff} objects.

  
  \texttt{inputToADaCGH} accepts an argument to reduce the number of cores used,
  which can help with contention issues related to I/O. A multicore
  machine (say, 12 cores) with a single SATA drive might actually
  complete the reading faster if you use fewer than 12 cores for the
  reading. But your mileage might vary. See also comments and full tables
  in section \ref{comp-read}.

   
\item Reordering data takes time (a lot if you do not use \textit{ff}
  objects) and can use a lot of memory. So it is much better if input data are
  already ordered (by Chromosome and Position within Chromosome).
  
\end{enumerate}



\newpage
\bibliographystyle{apalike}
\bibliography{benchmarks} %% a link to ADaCGH2.bib


\end{document}















