$CTSMROOT/tools/mksurfdata_map/README.developers              Jun/08/2018

============================================================================
============================================================================
Developer's guide for mksurfdata_map
============================================================================
============================================================================

============================================================================
Table of Contents
============================================================================

I.  Adding a new raw data file

II. Adding mapping files for a raw data file with a new grid / landmask

III. Checks that should be done when making new surface datasets

============================================================================
I. Adding a new raw data file
============================================================================

Here is what you need to change when adding a new raw data file, with one or
more fields that need to be remapped to the CLM resolution. Note that
additional steps are needed (beyond what's listed here) when the field
you're adding specifies something about the subgrid breakdown (into
landunits, columns & pfts): for those fields, additional code is needed to
ensure that percentages add to 100%.

Note: The following assumes that the new file uses an existing set of
mapping files, or that you have already done everything necessary to add a
new set of mapping files. If your raw data file has a new grid, or a new
landmask on an existing grid, see the instructions for adding mapping files
in a separate section of this document.

- Add a new module in $CTSMROOT/tools/mksurfdata_map/src that provides a routine for
  remapping your new field(s). 

    Note that there is generally one module per input file; multiple fields
    can be regridded using a single subroutine or multiple subroutines.

- Add your new file in $CTSMROOT/tools/mksurfdata_map/src/Srcfiles

- Add new namelist options in $CTSMROOT/tools/mksurfdata_map/src/mkvarctl.F90; e.g., for a
  new field xxx:
    - mksrf_fxxx
    - map_fxxx

- Add output calls in $CTSMROOT/tools/mksurfdata_map/src/mkfileMod.F90; you need to add
  calls in 3 places:
    - raw data file name attribute (nf_put_att_text call)
    - mapping file name attribute (nf_put_att_text call)
    - variable definition (ncd_defvar calls)
        Make sure this goes in an 'if (.not dynlanduse)' conditional, if
        appropriate

- Add code in $CTSMROOT/tools/mksurfdata_map/src/mksurfdat.F90; you need to add the
  following:
    - add a 'use' statement to use your new regridding routine(s)
    - declare array(s) to hold data on the output grid
    - add your new mksrf_fxxx and map_fxxx variables to the 'namelist
      /clmexp/' declaration
    - document your new mksrf_fxxx and map_fxxx variables in the long
      comment following the 'namelist /clmexp/' declaration
    - add your new array(s) to the allocation statement under the heading
      "Allocate and initialize dynamic memory"
    - initialize your new array(s) in the initialization section following
      the allocation
    - add output to ndiag (the log file) documenting your new mksrf_fxxx and
      map_fxxx variables
    - add call(s) to your new subroutine(s) for regridding the data
    - add calls to nf_inq_varid & nf_put_var_double (or nf_put_var_int) for
      each new output variable; make sure to put these calls in the section
      for dynlanduse = false and/or true, as appropriate
    - add a deallocation statement for each new output variable

- Add your new file in $CTSMROOT/bld/namelist_files/namelist_definition_clm4_5.xml;
  e.g. (replace xxx with your new field):

    <entry id="mksrf_fxxx" type="char*256" category="mksurfdata"
           input_pathname="abs" group="clmexp" valid_values="" >
    XXX dataset for mksurfdata
    </entry>

- Add your new mksrf_fxxx variable to the list of valid_values for
  mksrf_filename in $CTSMROOT/bld/namelist_files/namelist_definition_clm4_5.xml

- Add defaults in $CTSMROOT/bld/namelist_files/namelist_defaults_clm4_5_tools.xml;
  note that the "type" attribute is a short code that can be used in
  mksurfdata.pl, and doesn't have to match the "xxx" that is used elsewhere
    - lmask
    - hgrid
    - mksrf_filename
    - mksrf_fxxx (including hgrid and lmask attributes)

- Add hooks to your new files in $CTSMROOT/tools/mksurfdata_map/mksurfdata.pl:
    - add new string in the list following 'foreach my $typ'
    - add the new mapping file to clmexp, as in:
        map_fxxx       = '$map{'xxx'}'
    - add the new raw data file to clmexp, as in:
        mksrf_fxxx     = '$datfil{'xxx'}'

- Add new raw data file to the inputdata repository: lnd/clm2/rawdata
    - locally
    - check in to the inputdata svn repository

- Add documentation for your new mksrf_fxxx in $CTSMROOT/doc/UsersGuide/tools.xml

============================================================================
II. Adding mapping files for a raw data file with a new grid / landmask
============================================================================

If your raw data file is on a new grid, or just has a new landmask on an
existing grid, you will need to perform a number of additional steps, as
laid out here.

- First, move your data file to the inputdata directory and give it its
  final name. (This will ensure that the appropriate metadata is put in the
  SCRIP grid file.)

- Make a scrip grid file from your data file using mkmapgrids, and move it
  to the inputdata directory

- Add a scripgriddata entry for the new scrip grid file in
  $CTSMROOT/bld/namelist_files/namelist_defaults_clm4_5_tools.xml

- If necessary, add other entries in
  $CTSMROOT/bld/namelist_files/namelist_defaults_clm4_5_tools.xml giving information about your
  scrip grid file:
    - If this is a high resolution grid (e.g., 3min or higher), add a
      scripgriddata_lrgfile_needed entry, saying we need 64bit_offset
      (or netcdf4) support for mapping files made with this scrip grid
      file
    - If the grid file is in UGRID format rather than SCRIP grid
      format, add scripgriddata_type and scripgriddata_meshname
      entries. If you don't know what I'm talking about, then your
      grid file is in SCRIP format and you can ignore this.

- If necessary, add new grid and/or landmask to lists of valid values for
  hgrid, res and lmask in $CTSMROOT/bld/namelist_files/namelist_definition_clm4_5.xml
    - Note that a new resolution currently needs to be added to both the hgrid
      and res lists of valid values, although in the future this
      should probably be changed so that these raw data grids just
      appear in hgrid

- Add the new grid-landmask combo to the 'mapgrids' list in
  $CTSMROOT/bld/namelist_files/checkmapfiles.ncl

- Add the new grid-landmask combo to the 'grids' list in
  $CTSMROOT/tools/shared/mkmapdata/mkmapdata.sh (in the clm4_5 branch of the
  conditional)

- Make mapping files, from $CTSMROOT/tools/shared/mkmapdata
    - Modify mkmapdata.sh:
        - edit the grids list so it only contains your new grid
    - Modify regridbatch.sh as desired, e.g.:
        - project number
        - number of processors (BSUB -n line, span, and the regrid_num_proc setting)
        - wall-clock limit
        - if ESMFBIN_PATH is in your environment, you may want to unset it;
          this can be important to allow mkmapdata.sh choose a different
          executable for mpi vs serial runs
        - if you renamed the mkmapdata.sh script, be sure to call the
          renamed script at the bottom of regridbatch.sh
    - Submit regridbatch.sh

- When mapping files have all been created, run createXMLEntries.pl from
  $CTSMROOT/tools/shared/mkmapdata (usage: just run the script with no arguments)

- Cut and paste the xml entries from mapping_entries.txt (created by
  createXMLEntries.pl) into $CTSMROOT/bld/namelist_files/namelist_defaults_clm4_5.xml,
  in the correct locations

- Move mapping files to correct location, either using mv_cmds.sh created by
  createXMLEntries.pl, or using $CTSMROOT/tools/shared/mkmapdata/mvNimport.sh. 
    - Note that the latter also imports to the inputdata directory; if you
      don't use that, you'll need to add the files to the inputdata
      directory yourself


============================================================================
III. Checks that should be done when making new surface datasets
============================================================================

Remaking all surface datasets carries the risk of introducing unintended
changes, particularly when you are expecting answer changes (so you
don't notice unintended answer changes that are mixed with the expected
changes).

Here are some things to check after making a new set of surface
datasets:

- For at least one global dataset (probably a production resolution
  rather than a low resolution that is just used for testing): Compare
  the new dataset against the previous version:

  - Compare header (via ncdump -h) and/or log file: ensure that the same
    source data were used, except where you expect differences

  - Compare all fields with a tool like cprnc: make sure that the only
    fields that differ are those you expect to differ

  - Visually compare all fields that differ: make sure differences look
    reasonable and as expected
    
And here are some things to check for when making new landuse.timeseries 
datasets (which often happens at the same time, and most of the above applies
as well):

- Compare one of the productive resolution datasets to a previous version.

  - If part of it should be identical (for example the historical period) make
    sure it is identical as expected (using cprnc make sure the historical period
    is identical and only the future scenario changes).
  
  - If the historical period should be identical, make sure the 1850 surface dataset
    created is identical to the previous one.
    
  - Visually compare all fields/times that differ: make sure differences look
    reasonable and as expected. Go through at least the first and last time to see
    that the change in time is as expected. 
    
  - Quickly going through the time differences for at least one field that changes
    can also be useful to see that there isn't a sudden jump for a particular time.
    
  - Go through the list of raw PFT files that were used to create the dataset and make
    sure it appears to be correct (ncdump -v input_pftdata_filename)
