README.pyCECT
=============
CESM Ensemble Consistency Test
------------------------------
The CESM-ECT package is used to compare the results of a set of new (modified)
CESM simulations against the accepted ensemble.  Current functionality includes 
CAM-ECT, which examines history files from CAM (atmosphere) or POP-ECT which 
examines files from POP (ocean).  CAM-ECT requires 
a summary file generated by pyEnsSum.py (see README.pyEnsSum), and POP-ECT 
requires  a summary file generated by pyEnsSumPop.py (see README.pyEnsSumPop).
An overall pass or fail is designated.

For details on CAM-ECT, see: 

A.H. Baker, D.M. Hammerling, M.N. Levy, H. Xu, J.M. Dennis, B.E. Eaton, J. Edwards, 
C. Hannay, S.A. Mickelson, R.B. Neale, D. Nychka, J. Shollenberger, J. Tribbia, 
M. Vertenstein, and D. Williamson,"A new ensemble-based consistency test for the 
community earth system model."Geoscientific Model Development, 8, pp. 2829-2840, 2015
(doi:10.5194/gmd-8-2829-2015).


For details on UF-CAM-ECT, see:

Daniel J. Milroy, Allison H. Baker, Dorit M. Hammerling, and Elizabeth R. Jessup, 
“Nine time steps: ultra-fast statistical consistency testing of the Community Earth
System Model (pyCECT v3.0)”, Geoscientific Model Development, 11, pp. 697-711, 
2018 (doi: 10.5194/gmd-11-697-2018).


For details on POP-ECT, see:  

A.H. Baker, Y. Hu, D.M. Hammerling, Y. Tseng, X. Hu, X. Huang, F.O. Bryan, and 
G. Yang, “Evaluating Statistical Consistency in the Ocean Model Component of the 
Community Earth System Model (pyCECT v2.0).” Geoscientific Model Development, 9, 
pp. 2391-2406, 2016 (doi:10.5194/gmd-9-2391-2016).


:AUTHORS: Haiying Xu, Allison Baker
:VERSION: 3
:COPYRIGHT: See the document entitled LICENSE.txt

Send questions and comments to Haiying Xu (haiyingx@ucar.edu).


Overview
--------
This package includes:

       pyCECT.py
                            A script which compares the new CAM or POP results to the 
                            accepted ensemble and issues an overall "pass" or "fail"
			    (runs CAM-ECT by default or POP-ECT if specified)

       pyEnsLib.py         Library python script used by pyCECT.py


Before you start to use the package, you need to load the following modules:

       - module load python
       - module load numpy
       - module load scipy
       - module load pynio

To see all options (and defaults):

       python pyCECT.py -h

Notes and examples:
--------------------------------------------

(A) Options for all CESM-ECT approaches:

     Required:

         To specify the summary file generated by pyEnsSum.py:
	    -sumfile  ens.summary.nc

     	 To specifying the directory path that contains the run(s) to be evaluated:
	    -indir  /glade/u/tdd/asap/verification/cesm1_3_beta11/mira

    Optional:
	 Verbose information:
	     --verbose

(B) CAM-ECT and UF-CAM-ECT specific options (and summary file generated by pyEnsSum.py)

    Note that CAM-ECT is the default test.

    Note that the parameters setting the pass/fail criteria are all set by 
    default (ie. sigMul, minPCFail, minRunFail, numRunFile, and nPC).  But 
    if the specified indir contains more files than the number (num) specified by 
    "--numRunFile <num>"  (default= 3), then <num> files will be chosen at random 
    from that directory. Ensemble Exhaustive Test (EET) is specified by --eet <num>. 
    This tool computes the failure rate of <num> tests taken <numRunFile> at a time.
    Therefore, when specifying --eet <num>, <num> must be greater than or equal to
    <numRunFile>. 

    To enable printing of extra variable information:
       --printVarTest

    By default, CAM-ECT looks at annual averages which is indictated by 

       --tslice 1  (For monthly files, --tslice 0.  Note that this 
       should correspond to what has been collected in the summary file.)

    To enable printing out sum of standardized mean of all variables and associated box plots
      
       --prn_std_mean
       requries Python seaborn package

    Example using the default settings:
    
    python pyCECT.py --sumfile  CAM.ens.summary.nc --indir  /glade/u/tdd/asap/verification/cesm1_3_beta11/mira 

    Example using EET:

    python pyCECT.py --sumfile  CAM.ens.summary.nc --indir  /glade/u/tdd/asap/verification/cesm1_3_beta11/mira --eet 10
         

 (C) POP-ECT specific options (and summary file generated by pyEnsSumPop.py)
      
      To use POP-ECT, you MUST add the following to enable this test 
      (which disables CAM-ECT):

           --popens 

      Be sure to use a POP-ECT summary file:
           
	   --sumfile /glade/p/cisl/iowa/pop_verification/summary/pop.ens.summary.nc
	    
      Directory path that contains the run(s) to be evaluated.
	    
	   --indir /glade/p/cisl/iowa/pop_verification/testcases/

      The above directory may contain many POP history files that following the standard 
      CESM-POP naming convention. To specific which file or files you wish to test, you 
      simply specifying the test case file prefix (like a wildcard expansion).  For  
example, to compare against all months in year 3 from the input directory above:

            --input_glob cvtd100000.0.pop.h.0003
           
      To compare only against month 12 in year 1:

            --input_glob cvtd100000.0.pop.h.0001-12
      (Note: if input_glob is not specified, all files in --indir will be compared)

      (Note: the recommendation is to just compare year 1, month 12)


     Be sure to specify the json file that includes the variables which will be run the test on:

            --jsonfile pop_ensemble.json

    Note that the parameters setting the pass/fail criteria are all set by 
    default (ie. pop_tol, pop_threshold) but may be modified:

         Specifying test tolerance (the minimum Z-score threshold):

            --pop_tol 3.0

    	 Specifying pop threshold (fraction of points that must satisfy the Z-score tolerance):
 
            --pop_threshold 0.9

    
    Example:
         
    python pyCECT.py --popens --sumfile /glade/p/cisl/iowa/pop_verification/summary/pop.ens.summary.nc --indir /glade/p/cisl/iowa/pop_verification/testcases/ --input_glob cvtd100000.0.pop.h.0001-12 --jsonfile pop_ensemble.json 
       	    
