htslib-s3-plugin(7)

NAME
s3 plugin - htslib AWS S3 plugin

DESCRIPTION
The S3 plugin allows htslib file functions to communicate with servers that use the AWS S3 protocol. Files are identified by their bucket and object key in a URL format e.g. s3://mybucket/path/to/file With path/to/file being the object key. Necessary security information can be provided in as part of the URL, in environment variables or from configuration files. The full URL format is: s3[+SCHEME]://[ID[:SECRET[:TOKEN]]@]BUCKET/PATH The elements are:

SCHEME
    The protocol used. Defaults to https.

ID  The user AWS access key.

SECRET
    The secret key for use with the access key.

TOKEN
    Token used for temporary security credentials.

BUCKET
    AWS S3 bucket.

PATH
    Path to the object under the bucket.

The environment variables below will be used if the user ID is not set.

AWS_ACCESS_KEY_ID
    The user AWS access key.

AWS_SECRET_ACCESS_KEY
    The secret key for use with the access key.

AWS_DEFAULT_REGION
    The region to use. Defaults to us-east-1.

AWS_SESSION_TOKEN
    Token used for temporary security credentials.

AWS_DEFAULT_PROFILE
    The profile to use in credentials, config or s3cfg files. Defaults to default.

AWS_PROFILE
    Same as above.

AWS_SHARED_CREDENTIALS_FILE
    Location of the credentials file. Defaults to ~/.aws/credentials.

HTS_S3_S3CFG
    Location of the s3cfg file. Defaults to ~/.s3cfg.

HTS_S3_HOST
    Sets the host. Defaults to s3.amazonaws.com.

HTS_S3_V2
    If set use signature v2 rather the default v4. This will limit the plugin to reading only.

HTS_S3_PART_SIZE
    Sets the upload part size in Mb, the minimum being 5Mb. By default the part size starts at 5Mb and expands at regular intervals to accommodate bigger files (up to 2.5 Tbytes with the current rate). Using this setting disables the automatic part size expansion.

HTS_S3_ADDRESS_STYLE
    Sets the URL style. Options are auto (default), virtual or path.

In the absence of an ID from the previous two methods the credential/config files will be used. The default file locations are either ~/.aws/credentials or ~/.s3cfg (in that order). Entries used in aws style credentials file are aws_access_key_id, aws_secret_access_key, aws_session_token, region and addressing_style. Only the first two are usually needed. Entries used in s3cmd style config files are access_key, secret_key, access_token, host_base, bucket_location and host_bucket. Again only the first two are usually needed. The host_bucket option is only used to set a path-style URL, see below.

NOTES
In most cases this plugin transforms the given URL into a virtual host-style format e.g. https://bucket.host/path/to/file. A path-style format is used where the URL is not DNS compliant or the bucket name contains a dot e.g. https://host/bu.cket/path/to/file. Path-style can be forced by setting one either HTS_S3_ADDRESS_STYLE, addressing_style or host_bucket. The first two can be set to path while host_bucket must not include the %(bucket).s string.

SEE ALSO
htsfile(1) samtools(1)

htslib website: <http://www.htslib.org/>