GNOS_PULL(1)

NAME
gnos_pull.pl - retrieve/update analysis flow results on local systems.

SYNOPSIS
./gnos_pull.pl [-h] -u http://pancancer.info/gnos_metadata/latest/ -c gnos_pull.ini -o local_mirror/

  Required input:

    --analysis  (-a)  ALIGNMENTS or CALLS

    --outdir    (-o)  Where to save jsonl and resulting GNOS downloads

    --config    (-c)  Mapping of GNOS repos to permissions keys

  Other options:

    --symlinks  (-s)  Rebuild symlinks only.

    --threads   (-t)  Number of parallel GNOS retrievals.

    --url       (-u)  The base URL to retrieve jsonl file from
                        [http://pancancer.info/gnos_metadata/latest/]

    --info      (-i)  Just prints how many donor's will be included in pull and some stats.

    --debug     (-d)  prints extra debug information

    --help      (-h)  Brief documentation

    --man       (-m)  More verbose usage info

OPTIONS

a|analysis
    What type of data should be retrieved in this run.

    ALIGNMENTS retrieves the mapped BAM for all samples linked to this donor. Only retrieves data if all BAMs are ready.

    CALLS retrieves the data for all variant calling workflows that have been run against the selected donors. If data is available in subsequent executions for a new caller then this will be retrieved without affecting existing results.

o|outdir
    The base output directory, this will need to be very large if pulling BAM files.

c|config
    The config.ini file to be used. There is a full example of this in the distribution. Here is a brief example:

     [GENERAL]
     # used in preference to path, optional
     gtbin=/software/genetorrent/bin

     [FILTERS]
     # filters are applied as a union, so all must be fulfilled
     # key must match a top level key in jsonl file
     # csv:
     dcc_project_code=BRCA-UK,BRCA-US

     # or multiline:
     donor_unique_id=<<EOT
     BRCA-UK::CGP_donor_1167078
     BRCA-UK::CGP_donor_1187030
     EOT

     gnos_study=

     [KEY_FILES]
     https://gtrepo-ebi.annailabs.com/=/fullpath/XXX.pem
     https://gtrepo-etri.annailabs.com/=/fullpath/XXX.pem
     https://gtrepo-dkfz.annailabs.com/=/fullpath/XXX.pem

     [TRANSFER]
     # When data at multiple sites use this list to preferentially choose source
     # when not found in list just try the one found in the json
     order=<<EOT
     https://gtrepo-dkfz.annailabs.com/
     https://gtrepo-ebi.annailabs.com/
     https://gtrepo-etri.annailabs.com/
     EOT

u|url
    Provide if you don't want to use the 'latest' jsonl file.

    Recommended if you want to complete a first pass on a fixed file.

i|info
    Find out how much data you will potentially retrieve. Can also be used to see how data is distributed.

    This doesn't take into account data already downloaded.

d|debug
    Turns on debug information which can aid in diagnosing why a particular donor is not retrieved.

METHODS

  load_config
Loads the provided *.ini file provided to '-c'. Embeds relevant data into options hash.

  pull_data
Generic control method for retrieving data for a donor from GNOS, see pull_alignments and pull_calls.

  pull_alignments
Abstracts calls to pull_bam to give a single method call which will retrieve all alignements associated with this donor.

  pull_bam
Performs GNOS retrieval for a single BAM file (and index) to the 'original/<DONOR>/' output area.

Generates symlinks in the 'donors' output area to simplify use.

A call to the related pull_alignments method for a donor with 2 tumour samples will result in the following directory structure.

 .../donors/DONOR/tumour/X.bam[.bai]
 .../donors/DONOR/tumour/Y.bam[.bai]
 .../donors/DONOR/normal/Z.bam[.bai]

  pull_calls
Performs GNOS retrieval for all available variant calling.

Generates symlinks in the 'donors' output area to simplify use, which would look like:

  .../donors/DONOR/sanger/TUMOUR.*
  .../donors/DONOR/broad/TUMOUR.*
  ...

  select_repo
If data is available from multiple GNOS repositories and TRANSFER.order is defined in the config.ini file this method selects the repo highest in the list (if not found defaults to first in data structure).

NOTE - if up to date transfer metric reports become available these should be used in preference.

  load_data
Parse the jsonl manifest, filter appropriately:

 Silently skip 'is_test' data.

If debug warn and skip when:

 Indicated by filters in *.ini
 Blacklisted
 Tumour or normal not aligned

If -a 'CALLS' skip when:

 No variant calling performed
 Not in Jamboree

  manifest
Retrieve the donor_pXXX.jsonl.gz from pancancer.info or if provided alternate source.

  option_builder
Parses the command line args and does basic checking of inputs.