#

## Input Data Streams

### Overview

An input data stream is a time-series of input data files where all the fields in the stream are located in the same data file and all share the same spatial and temporal coordinates (ie. are all on the same grid and share the same time axis). Normally a time axis has a uniform dt, but this is not a requirement.

The data models can have multiple input streams.

The data for one stream may be all in one file or may be spread over several files. For example, 50 years of monthly average data might be contained all in one data file or it might be spread over 50 files, each containing one year of data.

The data models can loop over stream data -- repeatedly cycle over some subset of an input stream's time axis. When looping, the models can only loop over whole years. For example, an input stream might have SST data for years 1950 through 2000, but a model could loop over the data for years 1960 through 1980. A model cannot loop over partial years, for example, from 1950-Feb-10 through 1980-Mar-15.

The input data must be in a netcdf file and the time axis in that file must be CF-1.0 compliant.

There are two main categories of information that the data models need to know about a stream:

- data that describes what a user wants -- what streams to use and how to use them -- things that can be changed by a user.

- data that describes the stream data -- meta-data about the inherent properties of the data itself -- things that cannot be changed by a user.

Generally, information about what streams a user wants to use and how to use them is input via the strdata ("stream data") Fortran namelist, while meta-data that describes the stream data itself is found in an xml-like text file called a "stream description file."

### Stream Data

The strdata (short for "stream data") input is set via a fortran namelist called shr_strdata_nml. That namelist, the strdata datatype, and the methods are contained in the share source code file, share/util/shr_strdata_mod.F90. In general, strdata input defines an array of input streams and operations to perform on those streams. Therefore, many namelist inputs are arrays of character strings. Different variable of the same index are associated. For instance, mapalgo(1) spatial interpolation will be performed between streams(1) and the target domain.

The following namelist are available with the strdata namelist.

- dataMode - component specific mode
- domainFile- final domain
- streams - input files
- vectors - paired vector field names
- fillalgo - fill algorithm
- fillmask - fill mask
- fillread - fill mapping file to read
- fillwrite - fill mapping file to write
- mapalgo - spatial interpolation algorithm
- mapmask - spatial interpolation mask
- mapread - spatial interpolation mapping file to read
- mapwrite - spatial interpolation mapping file to write
- tintalgo - time interpolation algorithm
- taxMode - time interpolation mode
- dtlimit - delta time axis limit

The set of shr_strdata_nml namelist keywords are the same for all data models. As a result, any of the data model namelist documentation can be used to view a full description. For example, see stream specific namelist settings .

### Specifying What Streams to Use

The data models have a namelist variable that specifies which input streams to use and, for each input stream, the name of the corresponding stream description file, what years of data to use, and how to align the input stream time axis with the model run time axis. This input is set in the strdata namelist input.

General format:

    	&shr_strdata_nml
    	streams = 'stream1.txt year_align year_first year_last ',
            'stream2.txt year_align year_first year_last ',
            ...
            'streamN.txt year_align year_first year_last '
    	/

where:

`streamN.txt`

  the stream description file, a plain text file containing details about the input stream (see below)

`year_first`

  the first year of data that will be used

`year_last`

  the last year of data that will be used

`year_align`

  a model year that will be aligned with data for year_first

The stream text files for a given data model mode are automatically generated by the corresponding data model `build-namelist` with present names. As an example we refer to the following `datm_atm_in` example file (that would appear in both `$CASEROOT/CaseDocs` and `$RUNDIR`):

    	datamode   = 'CLMNCEP'
    	domainfile = '/glade/proj3/cseg/inputdata/share/domains/domain.lnd.fv1.9x2.5_gx1v6.090206.nc'
    	dtlimit    = 1.5,1.5,1.5,1.5
    	fillalgo   = 'nn','nn','nn','nn'
    	fillmask   = 'nomask','nomask','nomask','nomask'
    	mapalgo    = 'bilinear','bilinear','bilinear','bilinear'
    	mapmask    = 'nomask','nomask','nomask','nomask'
    	streams    = "datm.streams.txt.CLM_QIAN.Solar  1895 1948 1972  ",
            "datm.streams.txt.CLM_QIAN.Precip 1895 1948 1972  ",
            "datm.streams.txt.CLM_QIAN.TPQW   1895 1948 1972  ",
            "datm.streams.txt.presaero.trans_1850-2000 1849 1849 2006"
    	taxmode    = 'cycle','cycle','cycle','cycle'
    	tintalgo   = 'coszen','nearest','linear','linear'
    	vectors    = 'null'

As is discussed in the CESM1.2 User's Guide, to change the contents of `datm_atm_in`, you can edit `$CASEROOT/user_nl_datm` to change any of the above settings EXCEPT FOR THE NAMES `datm.streams.txt.CLM_QIAN.Solar`, `datm.streams.txt.CLM_QIAN.Precip`, `datm.streams.txt.CLM_QIAN.TPQW` and `datm.streams.txt.presaero.trans_1850-2000`. Note that any namelist variable from shr_strdata_nml and datm_nml can be modified by adding the appropriate keyword/value pairs to `user_nl_datm`. As an example, the following could be the contents of `$CASEROOT/user_nl_datm`:

    	!------------------------------------------------------------------------
    	! Users should ONLY USE user_nl_datm to change namelists variables
    	! Users should add all user specific namelist changes below in the form of
    	! namelist_var = new_namelist_value
    	! Note that any namelist variable from shr_strdata_nml and datm_nml can
    	! be modified below using the above syntax
    	! User preview_namelists to view (not modify) the output namelist in the
    	! directory $CASEROOT/CaseDocs
    	! To modify the contents of a stream txt file, first use preview_namelists
    	! to obtain the contents of the stream txt files in CaseDocs, and then
    	! place a copy of the  modified stream txt file in $CASEROOT with the string
    	! user_ prepended.
    	!------------------------------------------------------------------------
    	streams    = "datm.streams.txt.CLM_QIAN.Solar  1895 1948 1900  ",
            "datm.streams.txt.CLM_QIAN.Precip 1895 1948 1900  ",
            "datm.streams.txt.CLM_QIAN.TPQW   1895 1948 1900  ",
            "datm.streams.txt.presaero.trans_1850-2000 1849 1849 2006"

and the contents of shr_strdata_nml (in both $CASEROOT/CaseDocs and $RUNDIR) would be

    	datamode   = 'CLMNCEP'
    	domainfile = '/glade/proj3/cseg/inputdata/share/domains/domain.lnd.fv1.9x2.5_gx1v6.090206.nc'
    	dtlimit    = 1.5,1.5,1.5,1.5
    	fillalgo   = 'nn','nn','nn','nn'
    	fillmask   = 'nomask','nomask','nomask','nomask'
    	mapalgo    = 'bilinear','bilinear','bilinear','bilinear'
    	mapmask    = 'nomask','nomask','nomask','nomask'
    	streams    = "datm.streams.txt.CLM_QIAN.Solar  1895 1948 1900  ",
            "datm.streams.txt.CLM_QIAN.Precip 1895 1948 1900  ",
            "datm.streams.txt.CLM_QIAN.TPQW   1895 1948 1900  ",
            "datm.streams.txt.presaero.trans_1850-2000 1849 1849 2006"
    	taxmode    = 'cycle','cycle','cycle','cycle'
    	tintalgo   = 'coszen','nearest','linear','linear'
    	vectors    = 'null'

As is discussed in the User's Guide, you should use `preview_namelists` to view (not modify) the output namelist in `CaseDocs`.

### Stream Description File

The stream description file is not a Fortran namelist, but a locally built xml-like parsing implementation. Sometimes it is called a "stream dot-text file" because it has a ".txt." in the filename. Stream description files contain data that specifies the names of the fields in the stream, the names of the input data files, and the file system directory where the data files are located. In addition, a few other options are available such as the time axis offset parameter.

In CESM1.2, each data model's `build-namelist` utility (e.g. models/atm/datm/bld/build-namelist) automatically generates these stream description files. The directory contents of each data model will look like the following (using DATM as an example)

    	models/atm/datm/bld/build-namelist
    	models/atm/datm/bld/namelist_files/namelist_definition_datm.xml
    	models/atm/datm/bld/namelist_files/namelist_defaults_datm.xml

The `namelist_definition_datm.xml` file defines all the namelist variables and associated groups. The `namelist_defaults_datm.xml` provides the out of the box settings for the target data model and target stream. `build-namelist` utilizes these two files to construct the stream files for the given compset settings. You can modify the generated stream files for your particular needs by doing the following:

1. Call `setup` OR `preview_namelists`.

2. Copy the relevant description file from `$CASEROOT/CaseDocs` to `$CASEROOT` and pre-pend a "user_" string to the filename. Change the permission of the file to write. For example, assuming you are in `$CASEROOT`

       		cp CaseDocs/datm.streams.txt.CLM_QIAN.Solar  user_datm.streams.txt.CLM_QIAN.Solar
       		chmod u+w user_datm.streams.txt.CLM_QIAN.Solar

3. - Edit `user_datm.streams.txt.CLM_QIAN.Solar` with your desired changes.

   - Be sure not to put any tab characters in the file: use spaces instead.

   - In contrast to other user_nl_xxx files, be sure to set all relevant data model settings in the xml files, issue the `preview_namelist` command and THEN edit the `user_datm.streams.txt.CLM_QIAN.Solar` file.

   - `Once you have created a user_xxx.streams.txt.* file, further modifications to the relevant data model settings in the xml files will be ignored.`

   - If you later realize that you need to change some settings in an xml file, you should remove the user_xxx.streams.txt.* file(s), make the modifications in the xml file, rerun `preview_namelists,` and then reintroduce your modifications into a new user_xxx.streams.txt.* stream file(s).

4. Call `preview_namelists`

5. Verify that your changes do indeed appear in the resultant stream description file appear in `CaseDocs/datm.streams.txt.CLM_QIAN.Solar`. These changes will also appear in `$RUNDIR/datm.streams.txt.CLM_QIAN.Solar`.

The data elements found in the stream description file are:

`dataSource`

  A comment about the source of the data -- always set to GENERIC in CESM1.2 and not used by the model. This is there only for backwards compatibility.

`fieldInfo`

  Information about the field data for this stream...

  `variableNames`

    A list of the field variable names. This is a paired list with the name of the variable in the netCDF file on the left and the name of the corresponding model variable on the right. This is the list of fields to read in from the data file, there may be other fields in the file which are not read in (ie. they won't be used).

  `filePath`

    The file system directory where the data files are located.

  `fileNames`

    The list of data files to use. If there is more than one file, the files must be in chronological order, that is, the dates in time axis of the first file are before the dates in the time axis of the second file.

  `tInterpAlgo`

    The option is obsolete and no longer performs a function. Control of the time interpolation algorithm is in the strdata namelists, tinterp_algo and taxMode .

  `offset`

    The offset allows a user to shift the time axis of a data stream by a fixed and constant number of seconds. For instance, if a data set contains daily average data with timestamps for the data at the end of the day, it might be appropriate to shift the time axis by 12 hours so the data is taken to be at the middle of the day instead of the end of the day. This feature supports only simple shifts in seconds as a way of correcting input data time axes without having to modify the input data time axis manually. This feature does not support more complex shifts such as end of month to mid-month. But in conjunction with the time interpolation methods in the strdata input, hopefully most user needs can be accommodated with the two settings. Note that a positive offset advances the input data time axis forward by that number of seconds.

    The data models advance in time discretely. At a given time, they read/derive fields from input files. Those input files have data on a discrete time axis as well. Each data point in the input files are associated with a discrete time (as opposed to a time interval). Depending whether you pick lower, upper, nearest, linear, or coszen; the data in the input file will be "interpolated" to the time in the model.

    The offset shifts the time axis of the input data the given number of seconds. so if the input data is at 0, 3600, 7200, 10800 seconds (hourly) and you set an offset of 1800, then the input data will be set at times 1800, 5400, 9000, and 12600. so a model at time 3600 using linear interpolation would have data at "n=2" with offset of 0 will have data at "n=(2+3)/2" with an offset of 1800. n=2 is the 2nd data in the time list 0, 3600, 7200, 10800 in this example. n=(2+3)/2 is the average of the 2nd and 3rd data in the time list 0, 3600, 7200, 10800. offset can be positive or negative.

  `domainInfo`

    Information about the domain data for this stream...

    `variableNames`

      A list of the domain variable names. This is a paired list with the name of the variable in the netCDF file on the left and the name of the corresponding model variable on the right. This data models require five variables in this list. The names of model's variables (names on the right) must be: "time," "lon," "lat," "area," and "mask."

    `filePath`

      The file system directory where the domain data file is located.

    `fileNames`

      The name of the domain data file. Often the domain data is located in the same file as the field data (above), in which case the name of the domain file could simply be the name of the first field data file. Sometimes the field data files don't contain the domain data required by the data models, in this case, one new file can be created that contains the required data.

Actual example:

    	    <stream>
    	    <dataSource>
    	    GENERIC
    	    </dataSource>
    	    <domainInfo>
    	    <variableNames>
    	    time   time
    	    lon    lon
    	    lat    lat
    	    area   area
    	    mask   mask
    	    </variableNames>
    	    <filePath>
    	    /glade/proj3/cseg/inputdata/atm/datm7/NYF
    	    </filePath>
    	    <fileNames>
    	    nyf.ncep.T62.050923.nc
    	    </fileNames>
    	    </domainInfo>
    	    <fieldInfo>
    	    <variableNames>
    	    dn10  dens
    	    slp_  pslv
    	    q10   shnum
    	    t_10  tbot
    	    u_10  u
    	    v_10  v
    	    </variableNames>
    	    <filePath>
    	    /glade/proj3/cseg/inputdata/atm/datm7/NYF
    	    </filePath>
    	    <offset>
    	    0
    	    </offset>
    	    <fileNames>
    	    nyf.ncep.T62.050923.nc
    	    </fileNames>
    	    </fieldInfo>
    	    </stream>