Mark Taylor, 2016-02-19
Mark Taylor, 2010-09-23
Mike Levy, 2011-02-01

HOMME regression tests

These tests should be run in both the trunk and your branch
to ensure changes you have made have not broken existing code.
To check for bit-for-bit (BFB), output should be identical. To
check for rounding-error level changes, several statistics are
highlighted for each test below.

There are a couple of short BFB tests and an exact restart test
that should be sufficient for changes to the 3D code that are
BFB.  Non-BFB changes require verifing minimal changes in l2
solution errors,conservation errors, energy budget errors, and
checking plots of the vorticity field and and tracers.

To generate baselines:
   cd /working/path1
   cmake -C $MACHFILE -DUSE_NUM_PROCS=24  /path/to/homme/baseline/code
   make -j4 clean
   make -j4 baseline    make baselines in /working/path1/tests/baseline/
   make -j4 check       run tests, store results in /working/path1/tests
                        compare against self baselines 

To compare a different branch against the baselines:
   cd /working/path2
   cmake -C $MACHFILE -DUSE_NUM_PROCS=24  -DHOMME_BASELINE_DIR=/working/path1
   make -j4 clean
   make -j4 check      run tests, store results in /working/path2/tests
                       compare against results in $HOMME_BASELINE_DIR/tests/baseline



The above instructions assume you have queing disabled in your machine's cmake
file.  If your cluster is supported by HOMME's testing system, and queing is
enabled, then "make baseline" and "make check" above will submit jobs to the
que and monitor their progress.  In this case, -DUSE_NUM_PROCS is ignored and
the number of procs and nodes that will be used is configured for each test case.
The queue support code is currently not maintained and is most likley broken.  


Instructions to create a new test:
Determine executable (sweqx or preqx)
Create new executable configuration (if needed).
   i.e. homme/test_exec/swtcA
Create cmake description of test and namelist:
   homme/test/reg_test/run_tests/swtc2.cmake
   homme/test/reg_test/namelists/swtc2.nl
   homme/test/reg_test/run_tests/test-list.cmake   (update for new test)
   



 
=======================================================================
SHALLOW WATER TESTS -- DETAILS

swtc1.cmake Test case 1 from Williamson et al 1992 (advection)
            executable:     sweqx
            configuration:  homme/test_execs/swtcA
                NP=8, PLEV=4, USE_PIO=False  WITH_ENERGY=False
            namelist options:
               RK3-SSP, inviscid test
               Output using PIO_INTERP
               NE=10, dt=480, 12 days
               transport only, with limiter_option=4
            Verification:
            verify l1,l2,max errors have not changed significantly
            (3-4 digits?  need to estimate their sensitivity)
            k= 1 12.00 days  l1,l2,linf=  0.5347100E-02  0.3621730E-02  0.6553164E-02

            Using an RK3-SSP scheme with sign-preserving limiter, should verify 4 tracers
            (one per level) remain positive to machine precision.  


swtc2.cmake   Williamson et al 1992, test case 2 (steady state)
              This test ensures the RK3-SSP (which is mostly used for advection tests)
              also works for full shallow water.  
              We run with 4 levels on an unstructured grid to test the 3D, unstructured grid
              pack/boundary exchange/unpack code.  

              executable:     sweqx
              configuration:  homme/test_execs/swtcA
                  NP=8  PLEV=4, USE_PIO=FALSE, WITH_ENERGY=FALSE
              namelist options
                 RK3-SSP
                 Tensor-hyper-viscosity
                 Output using PIO_INTERP
                 grid: mountain_10_x2

                 Verify l1,l2,max errors have not changed significantly:
                 and that mass is conserved and identical on all 4 levels:
  1.00 days  l1,l2,linf=  0.1045449E-06  0.1258886E-06  0.3304282E-06

  u=   -0.496431514513851E-06   0.386396546491838E+02   0.153811206637787E+07
  v=   -0.263754879572531E-04   0.237167291254892E-04  -0.253359330740395E-01
  p=    0.108997400300042E+04   0.299811525752762E+04   0.119008435312488E+09
  p=    0.108997400300042E+04   0.299811525752762E+04   0.119008435312488E+09
  p=    0.108997400300042E+04   0.299811525752762E+04   0.119008435312488E+09
  p=    0.108997400300042E+04   0.299811525752762E+04   0.119008435312488E+09

level =  1  M-M0/M0      =-0.5776E-15 dM/dt   /M  = -0.267390E-19
level =  2  M-M0/M0      =-0.5776E-15 dM/dt   /M  = -0.267390E-19
level =  3  M-M0/M0      =-0.5776E-15 dM/dt   /M  = -0.267390E-19
level =  4  M-M0/M0      =-0.5776E-15 dM/dt   /M  = -0.267390E-19

                 



swtc5.cmake   Williamson et al 1992, test case 5 (isolated mtn)
              executable:     sweqx
              configuration:  homme/test_execs/swtcB
                  NP=4  PLEV=1, USE_PIO=FALSE, WITH_ENERGY=FALSE
              namelist options
                 Leapfrog-trapazoidal
                 hyper-viscosity
                 Output using PIO_INTERP
                 NE=30, dt=90, run for 15 days
              Verification:
                l1,l2,max errors have not changed (l1,l2,max errors 
                are computed in swtc5ref.ncl, which compares against
		NCAR's T213 reference sol)
                geop error: l1,l2,mx = 0.000531657 0.00074751 0.00837253


swtc6.cmake  short time, low resolution Rossby wave test
              executable:     sweqx
              configuration:  homme/test_execs/swtcC
                  NP=8  PLEV=1, USE_PIO=FALSE, WITH_ENERGY=FALSE
             executable:     sweqx
             namelist options:   homme/test/reg_test/namelists/swtc6-short.nl
                 NE=2, explicit leapfrog, no robert filter, no viscsoity, 
                 dt=6, run for 1 day    
                 cubed_sphere_map=2

             Run two simulations, one with the internal NE=2 grid, the
             second with a tempest NE=2 grid from a .g meshfile
             RMS error between the two runs should be: (final time):
             (computed via cprnc utility)
 RMS geop                             6.6086E-12            NORMALIZED  7.2158E-16
 RMS u                                2.1855E-13            NORMALIZED  6.7928E-15
 RMS v                                1.8779E-13            NORMALIZED  9.2013E-15

           Verification:
           verifty mass and energy conservatin to machine precision
           (timestep chosen very small to ensure energy conservation)
           verify enstrophy dissipation rate has not changed significantly
             (3-4 digits?).  Example output, last timestep: 
  M-M0/M0      = 0.0000E+00 dM/dt   /M  =  0.000000E+00
  ENS-ENS0/ENS0= 0.2010E-04 dENS/dt /ENS=  0.106396E-08
  E-E0/E0      = 0.9579E-12 dE/dt   /E  =  0.828421E-16  (-0.1025E-08, 0.1025E-08)




=======================================================================
3D PRIMITIVIE EQUATION TESTS -- DETAILS


When rebaselining, in addition to extra details specified below for
individual tests (i.e. exact restart, conservation), one should also:

   (1) look at stdout/log file and verify:
        tracer/tracer mass consistency:  tracer number 4 min/max 1.0 to 12 digits:
           qv=   0.999999999999636E+00  0.100000000000039E+01  0.561600000000003E+06
        check tracer mass conservation:  "Q diss" is near zero:
               (Q-Q0)/Q0   0.740187328509209E-14   Q1
               (Q-Q0)/Q0   0.146454099118228E-13   Q2
               (Q-Q0)/Q0   0.470825202983687E-14   Q3
               (Q-Q0)/Q0   0.625678319683976E-14   Q4
   (2) Run NCL script and verify that the l2 PS error has not changed:
             baro2b.sh:  l2 PS error at day 9:  1.13mb
             baro2d.sh:  l2 PS error at day 9:  1.25mb 
   (3) Example plots of PS, zeta and tracers produced by NCL script and
       compare with reference plots on github.    

   stdout will be stored:
     test output:     [cmake build directory]/tests/baro2d/movies
     baseline output: [cmake build directory]/tests/baseline/baro2d/movies
   NCL script to make plots and compute l2 error:
     ACME/models/atm/homme/test/reg_test/ncl/baro2.ncl 
   Reference solution plots and 'T340ref.nc' available from homme-archive repo:  
     https://github.com/ACME-Climate/homme-archive/tree/master/test/reg_test/ref_sol




baro2b.cmake  Test case from Jablonowski and Williamson 2006
              executable:     preqx
              configuration:  homme/test_execs/baroC
                  NP=4  PLEV=26, USE_PIO=FALSE, WITH_ENERGY=TRUE
              namelist options:  
                  dt=150
                  4 tracers, per ASP
                  Runge-Kutta timestepping with subcycling + hyper-viscosity 
                 (tstep_type=1, qsplit=4)
                 sign preserving limiter 
                 nu_p=nu
                 NE=15, NP=4, PLEV=26, dt=150, 9 days.  

               Performn 2 runs, 1 thread and 4 threads
               (2015/2 MT: disabled this slow running thread test, replaced by
                 thread tests in baroCamMoist below)

            verification:
              compare 1 thread vs4 thread runs (should be identical)
              Run the configuration in this script, visually
              compare plots produced in q.pdf (either run
              in branch AND trunk or compare to plots in
              baro2b-ref.pdf in ref_sol/)

baro2c.cmake   Test case from Jablonowski and Williamson 2006
            executable:     preqx
            configuration:  homme/test_execs/baroC
                  NP=4  PLEV=26, USE_PIO=FALSE, WITH_ENERGY=TRUE
            namelist options:  
               NE=6
               nmax = 2160 (9 days)
               qsize=1
               dt=360
               nu=nu_s=4e16  (nu_p,nu_top=0)
               monotone limiter

               tstep=5 should run with dt=1800
            restart run:  dt=30  LF
            

            first runs 9 days of the JW baroclinic test case (to get to an
            interesting flow field).  Then do a short restart run
            with a very small timestep to measure mass and energy
            conservation.  IE dissipation should be very small.
            KE dissipation should be matched by heating term, resulting
            in very small total energy dissipation.

            Verification:  mass and tracer mass conservation to machine precision.
            verify IE,KE balance to machine precision
            verify tracer variance dissipation has not changed significantly  
            This script will make two runs: run1: 9 days and writes a restart file
            run2: restarts from this file and runs 3 timesteps.       
            Example output from last timstep of run2:  
             
KE,d/dt,diss:  0.16556417255055E+07  0.38649033390296E-01 -0.25801357782547E-01
IE,d/dt,diss:  0.26216763769734E+10 -0.51174783706665E-01  0.25801414962755E-01
PE,d/dt,diss:  0.63765066746567E+03  0.12525807506476E-01  0.98007470195061E-11
 E,dE/dt       0.26233326563496E+10  0.63578287760417E-07
Q1,Q diss, dQ^2/dt:  0.65215821593973E+04 kg/m^2  0.0000000E+00 -0.4239988E-03
        
Original Leapfrog w/o Robert filter code    
(latest version does not support tracers, so Q stats no longer reported)
  KE diss = -.24e-1
  IE diss = +.24e-1
  PE diss = O(e-11)
  Q diss  = O(e-14)
  dE/dt   = .5e-7   
  dQ^2/dt = -.42e-3

2016-2/21: switch to RK5 for initial and restart run:
RK5 code dissipates more than leapfrog w/o Robert filter, as expected:
  KE diss = -.24e-1
  IE diss = +.24e-1
  PE diss = O(e-7)
  dE/dt   = .55e-5
  Q diss  = O(e-13)
  dQ^2/dt = -.26e-6








baro2d.sh   Test case from Jablonowski and Williamson 2006
            Same as baro2b.sh, but configred to match CAM-SE (Eulerian)
            executable:     preqx
            configuration:  homme/test_execs/baroC
                  NP=4  PLEV=26, USE_PIO=FALSE, WITH_ENERGY=TRUE
              namelist options:  
                subcycled tracers and hyperviscosity, nu_p=nu, nu_div=3*nu
                monotone limiter            
                Output using PIO_INTERP
                NE=15, NP=4, PLEV=26, dt=150, 9 days.
 
             (no threading tests)


baroCamMoist.cmake
              Similar to baro2d.sh, but configured to match CAM-SE Lagrangian
              moisture is turned on, so Q1 field feeds back to dynamics.  

              SE version
              NE=6 run for 3 days, dt=480

              SE-SL version
              NE=2 run for 1 days, dt=1200, qsplit=3
              on skybridge, this case is very very slow  - compiler issue with
              one of the subroutines.  Changing to NE=2 for 1 day until this is fixed.





======================================================================================
other tests
======================================================================================
templates.cmake 

       Test the HOMME grid template generation.  This runs 1
       timstep, and then outputs details about the grid and the GLL dual grid
 
       Not tested: From this output, NCL scripts are used to convert
       the data into a "*_latlon.nc" file used by CAM's interpic_new
       utility, and a "*_scip.nc" file used by ESMF and Tempest
       utilities that generate mapping files.

