The auxiliary state class will provide a container for the auxiliary variables
on a subdomain within OMEGA. It will provide methods for computing the
variables and it will register fields with IOStreams.

2.1 Requirement: Auxiliary state should register auxiliary variables with IOStreams

The auxiliary state class will register the auxiliary variables with IOStreams
in its constructor to facilitate their output.

2.1 requirement: auxiliary state should register auxiliary variables with iostreams

Methods to compute the auxiliary variables will be provided. At minimum, a
method to compute all of the variables is needed, but convenience functions
that compute just a subset of them might also be useful.

There will be methods to create and remove non-default auxiliary states, that
can be associated with different meshes.

The auxiliary state class will contain objects representing auxiliary variable
groups, strings for names of the auxiliary state and its metadata group, and a
pointer to the associated mesh. Additionally, it will have a static map of all
auxiliary states and a pointer to the default auxiliary state for quick
retrieval.

class AuxiliaryState {
 public:
   std::string Name;
   std::string GroupName;
   KineticAuxVars KineticAux;
   LayerThicknessAuxVars LayerThicknessAux;
   VorticityAuxVars VorticityAux;
   VelocityDel2AuxVars VelocityDel2Aux;
   // more variables in the future
 private:
   const HorzMesh *Mesh;
   static AuxiliaryState *DefaultAuxState;
   static std::map<std::string, std::unique_ptr<AuxiliaryState>> AllAuxStates;
};

There will be a constructor and destructor for the class and several public
methods. The constructor will be be private to make sure that every auxiliary
state is properly stored in the map. However, a static method to create new
auxiliary states, enforcing this requirement, will be provided.

AuxiliaryState(const std::string &Name, const HorzMesh *Mesh, int NVertLayers);

The create method will take the same arguments as the constructor, use it to
create a new auxiliary state, and put it in the static map of all auxiliary
states. It will return a pointer to the newly created state.

AuxiliaryState* AuxiliaryState::create(const std::string &Name, const HorzMesh *Mesh, int NVertLayers);

The init method will create the default auxiliary state and return an error code:

There will be methods for getting the default and non-default auxiliary states:

AuxiliaryState *AuxiliaryState::getDefault();
AuxiliaryState *AuxiliaryState::get(const std::string &Name);

The 'computeAll' method will compute all auxiliary variables using the ocean
state from a given time level:

The ‘computeAll’ method will compute all auxiliary variables using the ocean

void AuxiliaryState::computeAll(const OceanState *State, int TimeLevel);

The destructor will unregister fields and remove metadata. The erase method
will remove a named auxiliary state, whereas the clear method will remove all of
them. Both will call the destructor in the process.

void AuxiliaryState::erase(const std::string &Name);
void AuxiliaryState::clear();

There will be a test checking that multiple auxiliary state object can be
created and that removing them works properly.

There will be a test to ensure all of the variables are correctly computed by
the compute method.

Auxiliary variables are variables that are needed to compute the tendency terms
and advance the model state. While they are functions of the model state, they
differ from variables needed only for diagnostic purposes. Auxiliary variables
can often be bundled together into groups of variables that are logically
related, or can be efficiently computed together. Each auxiliary variable group
is implemented as a class containing array(s) storing the variable(s) and member
functions that compute them. The member functions compute the variable (or
variable group) over a chunk of vertical layers for a particular mesh location.
There may be more than one compute function, since some groups may contain
variables defined on different mesh elements. This approach allows flexible
groupings of computational work within larger cell/edge/vertex loops. This type
of flexibility will be important for optimizing the code across different device
architectures.

2.1 Requirement: Functions that compute auxiliary variables should have simple, intuitive calling arguments

Auxiliary variables take in constant data as constructor arguments, which
simplifies the arguments used to call the compute methods.

2.1 requirement: functions that compute auxiliary variables should have simple, intuitive calling arguments

2.2 Requirement: Auxiliary variables computations should be able to be grouped under different outer loops over mesh locations

The ability to fuse/separate auxiliary variables computations will allow for
optimization across different device architectures. Member functions will
implement a given variable (group) computation on a specific mesh location
(cell, edge, vertex).

2.2 requirement: auxiliary variables computations should be able to be grouped under different outer loops over mesh locations

Auxiliary variables computations have inner loops over a chunk of vertical
layers. The chunk size will be set to the vector length on CPU machines and 1
for GPUs. This will allow for the possibility of vectorization on CPUs.

Some auxiliary variables need to be computed differently based on configuration
options. The class constructor will retrieve possible options from the Config
and store them as enums. The compute functions will do different things based on
the stored options.

Auxiliary variables can also be useful as diagnostics. Each class will have a
method to register the variables with IOStreams to provide diagnostic output.

Some auxiliary variables are used only to compute another auxiliary variable,
and are not needed afterwards. It may be desirable to find a memory management
strategy that would allow to minimize the amount of persistent memory needed.

The algorithmic formulation for each auxiliary variable can be found in the
algorithms design document.

Each configurable auxiliary variable will define its parameters as an enum. Example:

The auxiliary variables group is a class that contains arrays for storing the
variables and methods to compute them. Configuration choices will also be
stored. The class also contains private member variables for any constant data
that are defined by the constructor.

class LayerThicknessAuxVars {
 public:
   Array2DReal FluxLayerThickEdge;
   Array2DReal MeanLayerThickEdge;
   FluxThickType FluxThickChoice;

   KOKKOS_FUNCTION void computeVarsOnEdge(int IEdge, int KChunk,
                                          const Array2DReal &LayerThickness,
                                          const Array2DReal &NormalVelocity) const;

 private:
   Array2DI4 CellsOnEdge;
};

   LayerThicknessAuxVars(const HorzMesh *mesh, int NVertLayers, const Config *Options)
       : FluxLayerThickEdge("FluxLayerThickEdge", mesh->NEdgesSize,
                            NVertLayers),
         MeanLayerThickEdge("MeanLayerThickEdge", mesh->NEdgesSize,
         CellsOnEdge(mesh->CellsOnEdge) {

             std::string FluxThickTypeStr;
             Config->get("fluxThicknessType", FluxThickTypeStr);
             switch (FluxThickTypeStr) {
                 case "Center":
                 FluxThickType = Center;
                 break;
                 case "Upwind":
                 FluxThickType = Upwind;
                 break;
                 default:
                 break;
             }

             defineIOFields();
    }

Compute methods implement the auxiliary variables computations for a chunk of
vertical layers at a given horizontal mesh location. The mesh location is
indicated in the method name. There may be more than one compute method to
compute different groups of variables over different mesh locations. Any
configurable computation options are handled inside the compute method. The
inner loop over a chunk of vertical layers enables CPU vectorization.

   KOKKOS_FUNCTION void
   computeVarsOnEdge(int IEdge, int KChunk, const Array2DReal &LayerThickCell,
                     const Array2DReal &NormalVelEdge) const {
      const int KStart = KChunk * VecLength;
      const int JCell0 = CellsOnEdge(IEdge, 0);
      const int JCell1 = CellsOnEdge(IEdge, 1);

      for (int KVec = 0; KVec < VecLength; ++KVec) {
         const int K = KStart + KVec;
         MeanLayerThickEdge(IEdge, K) =
             0.5_Real * (LayerThickCell(JCell0, K) + LayerThickCell(JCell1, K));
      }

      switch (FluxThickEdgeChoice) {
      case Center:
         for (int KVec = 0; KVec < VecLength; ++KVec) {
            const int K = KStart + KVec;
            FluxLayerThickEdge(IEdge, K) =
                0.5_Real *
                (LayerThickCell(JCell0, K) + LayerThickCell(JCell1, K));
         }
         break;
      case Upwind:
            if (NormalVelEdge(IEdge, K) > 0) {
               FluxLayerThickEdge(IEdge, K) = LayerThickCell(JCell0, K);
            } else if (NormalVelEdge(IEdge, K) < 0) {
               FluxLayerThickEdge(IEdge, K) = LayerThickCell(JCell1, K);
            } else {
               FluxLayerThickEdge(IEdge, K) = Kokkos::max(
                   LayerThickCell(JCell0, K), LayerThickCell(JCell1, K));
            }
         }
         break;
      }
   }

Each auxiliary variable will be used in a CTest which will compare its result
against a reference value.

The auxiliary variables for the shallow water equations will be tested on
existing test cases implemented in Polaris to verify they produce the correct
convergence rates.

In a parallel environment, it is often useful to broadcast values from
one MPI Rank to the others. In particular, the current design for
OMEGA configuration requires us to broadcast many aspects of that
configuration from the master rank to all others.

We must be able to broadcast variables of I4, I8, R4, R8, Real,
boolean, std::string.

Generally we will be broadcasting scalar values, but will require
vectors of values in some cases.

The most common use case will broadcast from the master rank but
we must be able to support broadcasting from any rank.

OMEGA plans to support subset partitions of the domain for sub-models,
so we must be able to broadcast only within these sub-groups using
the appropriate communicator.

It is desireable with any communication function to be able to
overlap communication and computation using non-blocking forms.

No algorithms needed beyond the MPI broadcast internal algorithm.

The OMEGA broadcast is mostly just wrappers around the MPI broadcast
function with simplified arguments.

For non-blocking broadcasts, we will alias the `MPI_request` type
to a Broadcast ID:

We will have four forms of the Broadcast function, each returning an
integer error code.  Note that in the interfaces below, we could
template based on data type, but prefer to overload to keep the
interfaces cleaner.

The first form is the simplest for a broadcast within the default
environment and from the master task:

where `[data type]` is one of the supported types (I4, I8 R4, R8, Real,
boolean, std::string). In actual use, this would look like:

int Err = Broadcast(MyIntValue);
int Err = Broadcast(MyRealValue);
[etc for all data types]

On the master task, the value would be broadcast and remain unchanged.
The remaining tasks would receive the broadcast and store the value.

This is similar to the above, but adds the additional argument
for the source rank to broadcast from.

int Broadcast([data type] Value,  ///< [in] value to be broadcast
              const int SrcRank   ///< [in] rank to broadcast from
              );      //

int Broadcast([data type] Value,     ///< [in] value to be broadcast
              const MachEnv *SubEnv, ///< [in] defined OMEGA environment
              );

4.2.4 Broadcast from another rank within a different environment

              const int SrcRank      ///< [in] rank to broadcast from
              );

4.2.4 broadcast from another rank within a different environment

There will be interfaces identical to the above with the value argument
replaced by `std::vector<type> value` to broadcast a vector of values.

Non-blocking forms of the above for all options will exist that
return a request id. Following the MPI standard, this will all
be named IBroadcast. In addition, an IBroadcastWait will be
included to wait for the request to complete. A non-blocking
sequence would look like:

BroadcastID myReqID = IBroadcast(MyVar);
[ perform other tasks/computation ]
int Err = IBroadcastWait(MyReqID);

A multi-processor test driver will initialize variables of all supported
types to zero or empty values. The master rank will then set a non-zero
value and broadcast to the remaining ranks. Each rank will verify the
new value is the expected one.

Repeat the above with vectors of all types using a set of non-zero
values.

Repeat the above two tests except broadcasting from a rank other
than the master rank.

Initialize non-blocking broadcasts in tandem with all the
above. After verification for the blocking forms, issue the
waits to complete the non-blocking forms and then perform the
verification on the non-blocking results.

Create a new MachEnv with a subset of ranks and repeat the above tests
with the extra environment argument. Check also that ranks not in
the environment are not corrupted/overwritten with the broadcast value.

Model configuration refers to all of the necessary variables to configure
and run the model. It contains all the input parameters, choices of methods,
physical coefficients for parameterizations and various options for I/O.
Typically, this is read as a single input file that can also serve as
a description of the configuration for provenance as well.

File or files containing the configuration must be easily understood and
modified by non-expert users. It is also desirable to have minimum markup
that would interfere with readability.

Configuration files must conform to a standard format for ease in parsing
and for potential interoperability. Examples of standards typically used
include YAML, JSON, XML.

The entire model configuration must be able to be saved and archived to
act as model provenance.

Although most input parameters will only be relevant to a particular
module or class, it is inevitable that some parameterizations or process
models will have dependencies on other configuration choices. All
configuration variables must therefore be accessible to all other
modules/classes.

Model configuration must support parameters in the standard data types,
including logical (boolean), integer, float/double, and string data types.
Limited support for vectors or arrays of the above types may be desirable.

Configuration should never consume significant resources, in either run
time or storage. For parallel execution, the configuration
will need to be replicated across MPI ranks, though in many cases, the
variables will be manipulated into a different form (eg string variables
describing choices will be converted to enums). This may happen before
or after replication.

For simplicity, we desire a single file for model configuration where users
can define all model configuration details and provenance can be maintained
more easily. If multiple files are required, there should be obvious links
or includes in the main configuration file to make the dependency clear.

For both readability and for easier encapsulation by modules/classes, the
configuration should be organized in a logical way to make it clear where
specific configuration variables are to be used. Parameters primarily or
specifically associated with a module or class should be in a grouping
of those parameters with the configuration.

Because the configuration will also be used for provenance, it is likely
that the configuration input file will need to be read by other languages
outside of the OMEGA C++ model. This requires an ability to parse a
configuration input file from other common languages (eg python).

If a configuration variable is missing from the configuration input file,
there must be an option to either supply a default value or throw an error
and exit, depending on the user choice. If a default value is supplied,
then the default value must be added to the configuration so that it is
included in the output and become part of the provenance.

If extra or unexpected values are encountered, they will be ignored.

2.13 Desired: Automated generation of default input and error checking

While the source code defines the configuration variables, it would
be desirable to have a means to extract from the source code what the
code is expecting into a default input config file. This would also
enable some external error checking for missing or extra entries.

2.13 desired: automated generation of default input and error checking

For users modifying an input configuration, it would be desirable
to document the acceptable values or range of values that each
variable can be assigned.

There are no specific algorithms needed for this other than those
used in the anticipated parsing/storage packages associated with
the standard format chosen.

We select the YAML format that meets the above requirements with
improved readability over other standard forms. Within the OMEGA
model, we use the yaml-cpp library, a third-party implementation for
parsing YAML input and efficiently storing/retrieving information
as YAML nodes. Many of the configuration variables will be stored as
maps in YAML, the package features a map syntax similar to the C++
standard template library map type.

Because extracting variables from a large and complex config
structure is less efficient, we plan for a single reading of the
full configuration. Each initialization routine in OMEGA will then
extract needed variables and manipulate them as needed for the later
forward integration. In this implementation, we will read/write the
configuration from a master rank and each initialization routine
within OMEGA will manipulate and broadcast necessary variables
across ranks for parallel execution. Because not all variables will
need to be broadcast and many others will be converted to more
efficient types (eg string options to logical or enums), we believe
this model will be more efficient than broadcasting the full config
structure and manipulating afterward.

We define a Config type, which is actually an alias of YAML::node:

from the yaml-cpp library. A YAML node is more fully and accurately
defined in the YAML specification, but for the purposes of this
document, our configuration is represented in YAML as a set of nested
map nodes, where a map is simply a keyword-value pair. At the lowest
level, these nodes are the simple `variable-name: value` maps. The next
level up is a map of the module name to the collection of maps associated
with the module. The root node corresponds to the full model configuration
and is simply a collection of all those module maps. I/O stream/file
configuration will be part of this configuration in a design TBD later
but will be a similar hierarchy under the full omega config node.

omega:
   timeManagement:
      doRestart: false
      restartTimestampName: restartTimestamp
      startTime: 0001-01-01_00:00:00
      stopTime: none
      runDuration: 0010_00:00:00
      calendarType: noleap

   [Other config options in a similar way]

   hmix:
      hmixScaleWithMesh: false
      maxMeshDensity: -1.0
      hmixUseRefWidth: false
      hmixRefWidth: 30.0e3

   [more config options]

   streams:

      mesh:
         type: input
         filenameTemplate: mesh.nc
         inputInterval: initial_only

      output:
         type: output
         filenameTemplate: output/output.$Y-$M-$D_$h.$m.$s.nc
         filenameInterval: 01-00-00_00:00:00
         referenceTime: 0001-01-01_00:00:00
         clobberMode: truncate
         precision: single
         outputInterval: 0001_00:00:00
         contents:
         - tracers
         - layerThickness
         - ssh
         - kineticEnergyCell
         - relativeVorticityCell
         - [other fields]

      [other streams in similar form]

All of the methods in the YAML::Node class are obviously supported,
but we will alias or wrap some of the most common in the OMEGA context
to be associated with Config.

The most common use case should be creating a Config by reading a
YAML configuration file using:

where the argument is the name for the YAML input file. In OMEGA,
we will retain this master configuration throughout the initialization
as omegaConfig.

Once the configuration has been read, we will need to retrieve variables
from the Config. Because our config is a hierarchy, there are really no
variables at the top level and we need to first retrieve the sub-config
associated with the local module/group. In the sample above, if we need
to retrieve a variable from the hmix group, we first retrieve the hmix
config and then the variable using:

Config hmixConfig = ConfigGet(omegaConfig,"hmix",iErr);
Real refWidth{0.0};
bool useRefWidth{false};
refWidth    = ConfigGet(hmixConfig, "hmixRefWidth",    iErr);
useRefWidth = ConfigGet(hmixConfig, "hmixUseRefWidth", iErr);

where there is a retrieval function for all supported Omega data types:
bool, I4, I8, R4, R8, Real, std::string. These retrievals are just
overloaded wrappers around the YAML form: `configName["varName"].as<type>`
with some error checking and reporting. Rather than a templated form,
we use simple overloading to keep a cleaner interface. If the variable
or config is missing, these functions will print an error message and
return with a non-zero error argument.

Another interface will allow the setting of a default value if the
variable is missing from the input config.  This interface simply
adds the default value as an additional argument, for example:

refWidth = ConfigGet(hmixConfig, "hmixRefWidth", defaultVal, iErr);

In this case, if the variable does not exist, it will not only
use the default value but print a warning that the default is
being used because the entry is missing.

While the intent is for all config variables to be set using the config
file read interface above, the capability modify a value is also
required. The syntax is essentially the inverse of the get/retrieval
above. Similar to that case, the sub-group will need to be retrieved first.

Config hmixConfig = ConfigGet(omegaConfig, "hmix", iErr);
ConfigSet(hmixConfig, "hmixRefWidth", 10.0e3, iErr);
ConfigSet(hmixConfig, "hmixUseRefWidth", true, iErr);

There will be overloaded interfaces for each supported type. For
literals (as in the example above), they will be cast to an appropriate
type according to C++ default type conversion and will be converted
to the desired type on retrieval (YAML internal storage is ignorant of
the type and only performs the type cast on retrieval).

It may be necessary to build up a configuration that does not yet
exist or add new entries to an existing group. We provide an Add
interface to distinguish this case from the Set case above.

// For an existing subgroup:
ConfigAdd(hmixConfig, "hmixRefWidth", 10.0e3, iErr);
ConfigAdd(hmixConfig, "hmixUseRefWidth", true, iErr);

// To add a new subgroup:
Config hmixConfig; // empty Config constructor
ConfigAdd(hmixConfig, "hmixRefWidth", 10.0e3, iErr); // build subgroup
ConfigAdd(omegaConfig, hmixConfig); // add new subgroup to parent

It is not expected that a user would test the existence since the
Get/Set functions will perform the test internally. However, to
satisfy requirement 2.11, we will add a function to test the
existence of an entry, given a config or sub-config.
Using the hmix example again:

if (ConfigExists(hmixConfig,"hmixRefWidth") {
   // variable exists, do stuff
}

Note that this can also be used to test the existence of a complete
sub-group as well:

While we may decide to save provenance a different way, a write interface
is supplied to write a configuration to an output YAML file:

A default constructor for an empty Config and destructor will be provided:

The destructor may be important to free up space since the Config is
likely to only be used during the init phase in the current plan.

In order to keep the source code, input files and documentation consistent
and avoid missing or extra entries, we propose inserting a block within
each source code header (where other interfaces will be documented).
This block would follow Doxygen-like format and look something like:

/// \ConfigInput
/// #
/// # Group description (eg. hmix: the horizontal mix configuration)
/// #
/// groupName:
///    #
///    # Parameter description (eg horizontal mixing coeff)
///    #    more description (units, acceptable values or range)
///    #
///    varName1: defaultValue1
///    #
///    # Parameter description
///    #
///    varName2: defaultValue2
///    [ continue for remaining vars in this block]
///
/// \EndConfigInput (might not be necessary?)

The block between the ConfigInput lines could be extracted verbatim
and written (with proper indenting) into a fully documented yaml
default input file or could be extracted and the `#`-delimited
comments stripped for a more concise yaml input file.
In addition, we may be able to similarly extract the same info for
the User and Developer Guides.

The selection of YAML automatically satisfies Requirements 2.1, 2.2 and
2.10. Requirement 2.8 will be enforced by Omega development. The other
requirements will be tested with a parallel unit test driver that performs
the following tests in order and will output the test name and a
PASS/FAIL for use with CTest or other testing frameworks.

Create an empty config on master rank using default constructor.

Add configuration variables to the empty config with at least one of
each supported type with a few extras to test behavior for other
requirements. Also add multiple levels of hierarchy and a large
enough number of subgroups and parameters to simulate a full omega
config.

tests function needed for requirements 2.3, 2.5, 2.9, 2.11, 2.12

Retrieve all the variables set in the test above and verify they
return identical values.

Broadcast variables from master after retrieval to provide test
of performance for this choice of parallel implementation.

Inquire for a missing variable and output a PASS if the variable
was not found. Also add the missing variable and assign a default
value, the retrieve to test this behavior.

Write a YAML output file using the above constructed Config. Then
read in the new file. Add an extra variable to the new file.
Verify that the newly read Config matches the original, ignoring
the extra variable.

Data type aliases are a useful means for managing precision within OMEGA and
to improve readability for for Kokkos array data types. This header file
defines a number of aliases to common data types used throughout OMEGA.

We wish to explore mixed precision and lower precision floating point within
OMEGA so require both a generic floating point type that can be switched at
compile time as well as explicit single and double-precision (32-bit, 64-bit)
types to enforce precision where needed.

OMEGA will be using Kokkos array data types which have a long syntax. For
readability, these should be aliased to a shorter, intuitive data type.

For better control over integer storage, it might be better to define
fixed-width integer data types rather than the generic int, long.

As much as possible, data types should not inhibit interoperability
with other codes (eg E3SM coupler, 3rd-party libs) or frameworks
(eg other portability or AI/ML frameworks).

Currently, language standard floating point conversion should be adequate,
but future mixed-precision implementations may require more reproducible
methods for implementation-dependent conversion.

No algorithms needed yet, though reproducibility in type conversion may
require reproducible algorithms in the future.

Currently, the data types can be defined in a single header file that sets
up aliases. Because the main data types are simply aliases to the standard
data types, interoperability (Req 2.4) should not be an issue. For the
Kokkos arrays, Kokkos provides means for interoperability with both other
languages and other frameworks and will be incorporated into any
interfaces with other codes.

We will use a global cpp define -D SINGLE_PRECISION at compile time as a
global switch for using single precision as the default real data type.
Otherwise, the default real will be double precision.

The data types will be defined using type aliases within a single header
file DataTypes.h We will use the "using" syntax rather than
the older typedef. For Kokkos arrays, we require both device arrays (default)
and host array types and will use C-ordering.

// Standard integer and floating point types
using I4 = std::int32_t;
using I8 = std::int64_t;
using R4 = float;
using R8 = double;
#ifdef SINGLE_PRECISION
using Real = float;
#else
using Real = double;
#endif

// Aliases for Kokkos arrays - by default on device and in
// C-ordering.
#define MAKE_OMEGA_VIEW_DIMS(N, V, T, ML, MS)  \
   using N##1D##T = Kokkos::V<T *, ML, MS>;    \
   using N##2D##T = Kokkos::V<T **, ML, MS>;   \
   using N##3D##T = Kokkos::V<T ***, ML, MS>;  \
   using N##4D##T = Kokkos::V<T ****, ML, MS>; \
   using N##5D##T = Kokkos::V<T *****, ML, MS>;

#define MAKE_OMEGA_VIEW_TYPES(N, V, ML, MS) \
   MAKE_OMEGA_VIEW_DIMS(N, V, I4, ML, MS)   \
   MAKE_OMEGA_VIEW_DIMS(N, V, I8, ML, MS)   \
   MAKE_OMEGA_VIEW_DIMS(N, V, R4, ML, MS)   \
   MAKE_OMEGA_VIEW_DIMS(N, V, R8, ML, MS)   \
   MAKE_OMEGA_VIEW_DIMS(N, V, Real, ML, MS)

// Aliases for Kokkos device arrays of various dimensions and types
MAKE_OMEGA_VIEW_TYPES(Array, View, MemLayout, MemSpace)

// Aliases for Kokkos host arrays of various dimensions and types
MAKE_OMEGA_VIEW_TYPES(HostArray, View, HostMemLayout, HostMemSpace)

Requirement 2.4 on interoperability will not have a unit test
and will be tested as part of larger system tests where coupling
with other codes is exercised.

Build test code with and without -D SINGLE_PRECISION and
verify size is as expected using sizeof

Create Kokkos arrays of each type on device. Create host arrays to
mirror each. Initialize Kokkos arrays on device and copy to host
using Kokkos. Create a non-Kokkos array on host initialized the same
way and compare values from each on the host.

For parallel execution, OMEGA will utilize a hierarchical parallelism with the
coarse-grained parallelism achieved via message passing. An OMEGA mesh will be
decomposed into subdomains so that computations on each sub domain can proceed
in parallel and messages will be passed to communicate data between domains.
This document describes the requirements and design of this domain
decomposition.

The primary requirement is to be able to partition a mesh across a
distributed memory machine. Many standard tools exist to perform
this partitioning (eg metis, parmetis, zoltan) and their use is
encouraged. This normally requires an input file with the primary
mesh indices and adjacency (see Requirement 2.4).

For backward compatibility and to facilitate bit-for-bit comparisons
with MPAS, at least one partition option must use the Metis library.

We require the partitioning to occur online (ie as part of model
initialization rather than a separate pre-processing step).
This is meant to avoid the past difficulties in maintaining a large
number of partition input files for each MPI configuration. Experience
has shown that even high-resolution partitioning is very inexpensive
and can be performed as part of the model run and does not need this
off-line step.

As input into the partitioning, we require a file with information
on cell adjacency. There are at least two potential design approaches
for this.  The first option is to use the global `graph.info` file
which is already in Metis input format with a line for each cell
listing the global indices of that cell's neighbors. A second option
is to use the input mesh netCDF file which has the same information
in the global cellsOnCell array. This second option would enable us to
eliminate all graph.info files after the mesh file has been created.

listing the global indices of that cell’s neighbors. A second option

While the primary partition is based on the cells and cell
adjacency, the decomposition must also partition the edge and vertex
index spaces.

For parallel efficiency, a halo containing copies of nearest neighbor
points will be defined. A separate design document describes the
capability needed for updating those halo points for field data.
Within the decomp type, we simply keep the halo index information.

Once the mesh is decomposed, each parallel process has a set of
local indices renumbered for the local space. The decomposition
must provide the ability to return the global index given the
processor's local index. It must also be able to
provide the location (MPI rank and local index) for all the
neighbor or halo points to facilitate later setup of halos and
other infrastructure. Some of this information may only be needed
for initialization (see Requirement 2.8 on release of memory).

At a minimum, the local addresses must be sorted with owned
indices first and each level of halo following. This is needed
for both optimal communication and for selectively computing
only what is needed. Additional sorting within those blocks
may be desireable for optimizing memory locality.

Once the partition information has been used to set up halos,
mesh variables, I/O and other quantities that depend on the
partition, all or parts of the partitioning data may be
released to free memory. Note that some of the information
may be transferred to mesh variables as part of later
mesh initialization.

For better load-balancing, support for supplying or computing weights
for each cell index based on estimated workload is desireable.

In the future, it may be desireable to perform parts of the calculation
(eg the communication-dominated barotropic mode) on a smaller
partition. We may need to support multiple partitions of the
same mesh on different numbers of MPI ranks.

For some future capabilities (eg local timestepping in regionally
focused mesh locations), it may be desireable to decompose the
global domain into smaller subdomains and over-subscribe subdomains
to a rank, combining low- and high-resolution subdomains for load
balancing. It is unclear whether this is best achieved with a multiple
sub-block structure (as in older MPAS) or accomodating this within a
weighted partitioning. Identifying the mesh regions using a coloring
of the mesh may be helpful.

Most of the algorithms required are embedded and documented in the
relevant packages (eg Metis multi-level k-way partitioning).

The design presented here is initially for decomposing the global
domain into a single block per MPI rank and does not yet include
weighted decompositions. However, it should not prevent these
options from being added later. The parmetis package will be
used initially to provide metis back compatibility while reducing
memory use over a serial metis approach. The cell mesh indices and
adjacency will be read from the non-decomposed graph.info files in
metis format, allowing the adjacency info to be read serially
before the parallel IO is set up.

Once the primary (cell) mesh is partitioned, the edge and vertex
index spaces will be assigned based on the adjacency to the local
cells, as in the current MPAS model. Because much of this information
will be later replicated in a mesh class with Kokkos arrays, this
decomposition structure will be destroyed after initialization and
any related setup of halos and other infrastructure.

The decomposition includeds a public enum to define the
supported partitioning method. Initially, this will support
the default Metis method, but others can be added:

    enum partMethod {
       partMethodUnknown,        ///< undefined for error checking
       partMethodMetisKWay,      ///< default KWay method in metis
       }

Other methods, like the Metis GeomKWay (combination of space-filling
curve with KWay) or Zoltan options may be added later.

Eventually, an enum defining options for computing weights for
weighted partitions may also be required, but will not appear in
this initial implementation.

Another parameter is the `haloWidth` that defines the width
of the halo needed to provide neighbor information for all
local operators. The minimum is typically 3 for current
algorithms.

These parameters and the mesh input file will be read as part of
the input configuration file in a decomp configuration group:

    decomp:
       meshInputFilename: 'omegaMeshFile.nc'
       partitionMethod: 'MetisKWay'
       haloWidth: 3

The Metis KWay partition method will be default and the haloWidth
will initially default to 3.

There will be a Decomp class that stores and defines a partitioning
of the address space.

    class Decomp {

       private:
          partMethod method;   ///< method to use for partitioning

       public:

          I4 haloDepth;    ///< depth of the halo

          I4 nCellsGlobal; ///< num cells in the global domain
          I4 nCellsAll;    ///< num cells on local rank (own+halos)
          I4 nCellsOwned;  ///< num cells exclusively owned by local rank

          std::vector<I4>[] nCellsHalo[]; ///< num of owned+halo points
                                          ///< for each level of halo

          std::vector<I4>[] cellGlobalID[]; ///< global index for each
                                            ///< local cell (own+halo)

          /// The adjacency (neighbor) info will be stored in the
          /// compact form that Metis uses. In this form the neighbor
          /// cell location for each owned and ghost cell in local cell
          /// order is stored as a linear vector of integer pairs
          /// (partition/processor number and local address). A second
          /// vector stores the starting address in this list for the
          /// neighbors associated with the local cell address.

          std::vector<I4>[] nbrStartCell ///< start addr in nbr array for cell
          std::vector<I4>[] nbrLocCell   ///< list of nbr addresses

          [ repeat identical variables for edge and vertex index spaces ]

          // methods described below

    }

Because most class members are public, only constructors and
destructors are really needed.

The main method will be a constructor that decomposes the mesh and
creates the public variables. The default decomposition will require
no arguments and will read options from the input config file:

For multiple decomposition of the same mesh, a second constructor
will be needed that starts from the default decomposition and
uses arguments for determining decomposition options. The exact form
will be determined later, but might look something like:

After most of the initialization is complete and the decomposition
information has been used to initialize meshes, halo, I/O, etc.,
we will supply a destructor to release the space in memory.

We will create a small sample graph/mesh for partitioning,
similar to examples in Metis documention and create a partition.
The resulting partition will be compared to that generated
by the standalone gpmetis partition.

We will create a decomposition based on an MPAS input mesh
and verify that the decomposition is the same as the older
MPAS Fortran code. This test should only be needed after initial
development and will not need to be repeated as part of
routine testing

To make sure all cells, etc. are accounted for, we can do a global
sum of the number of cells as well as a sum of global IDs across
the partition and ensure they the expected sum.

OMEGA can be used as either a standalone ocean model or as
the ocean component of E3SM. In either case, OMEGA requires a
top-level driver and interface layer. Here we describe the
requirements and design of both the standalone driver (main)
and the component interface for coupled simulation.

When running as part of a larger coupled model, OMEGA must
supply initialize, run and finalize methods. These routines
must export all variables needed by the parent model while also
importing all fields needed by OMEGA. In most cases, there will
also need to be a thin wrapper to translate data types between
the parent model and internal OMEGA data types.

When OMEGA is used as a standalone model, it must supply a
driver or main routine. For consistency with coupled
simulations, this driver must mimic a parent coupled model,
calling the same init, run and finalize methods and supplying
any needed data (eg surface forcing data).

Communication with the parent model or standalone driver must
be through the component interfaces as method arguments. All
other aspects of the OMEGA model and model state that need to
be retained across component calls or subsequent run intervals
must be stored internally within OMEGA as static variables.

Initializing and exiting environments like MPI and Kokkos must
take place at the driver level (standalone driver or coupled
model driver) as these environments are shared by other
components. The finalize method described below must clean up
all of Omega memory and data types so that these environments
can be exited cleanly.

The run method must advance the model one specified time
interval based on a set of inputs (the import state) and must
return a set of outputs (the export state) at the end of the
time interval. In a coupled simulation, this interval will be
the coupling interval. In standalone simulations, this interval
is typically the fastest forcing data interval.

The finalize method must provide a graceful exit, checkpointing
as needed and cleaning up all memory. It must not, however,
exit the MPI or other shared environments (eg Kokkos) per
requirement 2.4

An initialization method must initialize all model state, mesh
and other information needed by the model itself or the parent
coupled system. Variables needed by the parent model will be
returned as arguments while all other parts of the model state
will be retained in static variables for later retrieval by the
run method as described in Req 2.3. The model state on initialization
must correspond to the initial time for the simulation integration.

The design is essentially determined by the requirements above.
We define an init, run and finalize method. There will actually
be two layers of these functions. One will be the internal
Omega inteface used by the Omega standalone driver. A second
layer will be needed for translating between internal Omega
data types and E3SM (or other parent model) data types and
ensuring the model is synchronized correctly with the parent.

Within the directory structure of OMEGA, the `src/driver`
directory will contain two subdirectories called standalone
and E3SM. The standalone subdirectory will contain the
standalone driver (obviously) and the E3SM directory will
contain the wrapper interfaces that translate between the
E3SM components and data types and the Omega methods and data
types. The CMake build system will determine which directory
will be used in the build. The actual OcnInit, OcnRun, OcnFinalize
routines described below will reside in the `src/ocean` directory.

There are currently no additional parameters needed for this level.
Configuration is generally determined by other modules.

For standalone simulation, data types are determined by other modules
(eg state and mesh). No new data types are needed here.

E3SM data types are in flux (from MCT to MOAB). We will add the coupled
model data types here in the future.

The Init method (OcnInit) will call individual initialization
routines for every module in Omega. On input, it requires the
MPI communicator to use as the master ocean communicator
(`MPI_COMM_WORLD` for standalone, a coupler-partitioned
communicator for coupled simulations). On output, it will
return mesh information, the current model state and the
time instant associated with that state.

The precise interface awaits the design of various other
modules, but will look something like:

int OMEGA::OcnInit(
   MPIComm Comm, ///< [in] ocean MPI communicator
   OMEGA::TimeInstant &StartTime, ///< [out] sim start time
   OMEGA::TimeInterval &RunInterval, ///< [out] interval for run method
   OMEGA::State &CurrState, ///< [out] current model state
   other args as needed
);

The routine will return the initial state, the start time and
the run interval computed based on input options. An integer
return value will be non-zero if errors are encountered and zero
if successful.
It is also possible that a multi-stage initialization may
be needed, especially in coupled mode and would require
additional OcnInit interfaces. This will be determined during
integration with E3SM later.

The run method will advance the model one time interval,
typically the coupling time or the fastest forcing interval.
The interface will look like:

int OMEGA::OcnRun(
   OMEGA::TimeInstant &CurrTime, ///< [inout] current sim time
   OMEGA::TimeInterval &RunInterval, ///< [in] interval to advance model
   OMEGA::Alarm &EndAlarm, ///< [out] alarm to end simulation
   OMEGA::State &CurrState, ///< [inout] current model state
   other args as needed (eg forcing)
)

The model state, current time and the time interval will be input.
Other variables will be needed as well, like the surface forcing
fields, and will be added as needed. On return, the time instant
will contain the end time of the interval and the end alarm will
be ringing if the end of the simulation has been reached. The
CurrState will be the ocean state at the end of the run interval.
An integer error code will be zero if successful and non-zero if an
error was encountered. This interface will be modified as needed to
include other fields.

The finalize method will write a checkpoint/restart file
(if not already written by driver or run method on the
final timestep) and then clean up all arrays and classes by
calling the relevant routines for all Omega modules. The
interface is similar to the Init interface:

int OMEGA::OcnFinalize(
   OMEGA::TimeInstant &CurrTime, ///< [in] current sim time
   OMEGA::State &CurrState, ///< [in] current model state
   other args as needed for restart
);

An integer return value will be zero if successful and
non-zero if an error is encountered either writing a
restart or deallocating memory.

With the interfaces above, the standalone driver should look
something like the code below (details subject to change during implementation).

int main(int argc, char **argv) {

   MPI_Init(); // initialize MPI
   Kokkos::init(); // initialize Kokkos
   {

   OMEGA::State CurrState;
   OMEGA::TimeInstant CurrTime;
   OMEGA::TimeInterval RunInterval;
   OMEGA::Alarm EndAlarm;

   int Err = OcnInit(MPI_COMM_WORLD, CurrTime, RunInterval, CurrState,
                     EndAlarm, etc);
   if (Err != 0) LOG_ERROR("Error initializing OMEGA");

   while (Err == 0 && !(EndAlarm.isRinging()) ) {

      // call routines for forcing and other inputs

      // call run method
      Err = OMEGA::OcnRun(CurrTime, RunInterval, EndAlarm,
                          CurrState, etc);
      if (Err != 0) LOG_ERROR("Error advancing Omega one interval");

      // Other tasks if needed (eg IO could occur here or within run
      // method
   }

   int Err2 = OMEGA::OcnFinalize(CurrTime, CurrState, etc);
   if (Err2 != 0) LOG_ERROR("Error finalizing Omega");

   int ErrAll = abs(Err) + abs(Err2);
   if (ErrAll == 0){
      LOG_INFO("OMEGA successfully completed");
   } else {
      LOG_ERROR("OMEGA terminating due to error");
   }

   }
   Kokkos::finalize(); // Exit Kokkos
   MPI::Finalize(); // Exit MPI

   return ErrAll;
}

A forward model smoke test is included as part of the CTest
unit test suite. This test runs the standalone model in a
minimal configuration and only tests for successful completion.
Other forward model system testing (eg in Polaris) will be
inherently testing the driver layers.

Once Omega is integrated into E3SM, various E3SM system tests
will be run regularly and will test all coupled interfaces.
We will add an Omega developer test suite to include these tests.

The equation of state relates density to the prognostic state variables temperature and salinity. In the case of a non-Boussinesq model, it is also dependent on prognostic pressure. The prognostic temperature, $\theta$, is either potential temperature or Conservative Temperature, depending on the chosen equation of state, and  $S$ is the absolute salinity (linear eos could use practical salinity). Given that the Omega governing equations are  non-Boussinesq, the equation of state class will provide specific volume from the state variables. It will provide methods for computing the specific volume, and its first derivatives.

The equation of state relates density to the prognostic state variables temperature and salinity. In the case of a non-Boussinesq model, it is also dependent on prognostic pressure. The prognostic temperature, 

, is either potential temperature or Conservative Temperature, depending on the chosen equation of state, and  

 is the absolute salinity (linear eos could use practical salinity). Given that the Omega governing equations are  non-Boussinesq, the equation of state class will provide specific volume from the state variables. It will provide methods for computing the specific volume, and its first derivatives.

2.1 Requirement: calculate specific volume and density from prognostic temperature and salinity

Omega will use TEOS-10 as the primary equation of state. The reference implementation of TEOS-10 is contained in the Gibbs Sea Water (GSW) toolbox function. The Omega implementation will be equivalent to the computation in the GSW toolbox function: `gsw_specvol(sa,ct,p)`

Omega will use TEOS-10 as the primary equation of state. The reference implementation of TEOS-10 is contained in the Gibbs Sea Water (GSW) toolbox function. The Omega implementation will be equivalent to the computation in the GSW toolbox function: 

2.1 requirement: calculate specific volume and density from prognostic temperature and salinity

Omega-1 should include the option for two EOS options. The first is a linear eos, for benchmarking against MPAS-O and to use in simplified cases. The other option should be a well-established version of TEOS-10 (e.g. Roquet 75-term polynomial).

As one of the most expensive kernels, the EOS must be as efficient as possible. Developers should be discouraged from calling the EOS too frequently. The fraction of compute time used be the EOS within Omega1 will be compared to fraction used in MPAS-Ocean.

2.4 Requirement: abide by license for the GSW Oceanographic Toolbox

The GSW toolbox can only be used without modification, so a Omega-specific port of certain functions will be required to support Kokkos/GPU enabled computation. The toolbox without modifications can still be used for verification testing against the Omega implementation.

2.4 requirement: abide by license for the gsw oceanographic toolbox

The Omega implementation should be compared with the test value published in Roquet et al 2015, and the official GSW Oceanographic Toolbox (supported by TEOS-10) to verify correctness.

The EOS code should include a check on the range of ocean properties to assess use of TEOS-10 (equivalent to the ``ocean funnel'' check), and a truncation of state variables to use the polynomial only in the valid range (similar to the truncation in MPAS-Ocean).

The EOS code should include a check on the range of ocean properties to assess use of TEOS-10 (equivalent to the ``ocean funnel’’ check), and a truncation of state variables to use the polynomial only in the valid range (similar to the truncation in MPAS-Ocean).

The EOS code should be able to be used to calculate density adiabatically displaced to i) the layer below, ii) the surface. This is needed to calculate the Brunt-Vaisala and for mixed-layer parameterizations. This operation should be done as efficiently as possible (e.g. we will explore reusing parts of the polynomial or leveraging Kernel Fusion).

Later versions should include calculating the freezing temperature of seawater.

Later versions should include calculating the in situ  temperature needed for coupling (at surface including at non-zero pressure).

The EOS will need to be able to produce alpha and beta (drho/dT and drho/dS) for linear expansions in the higher-order pressure gradient and for some mixing parameterizations.

Later versions should include functions to convert between conservative and potential temperatures, and absolute and practical salinity. These will be used to convert the initial conditions or compare to ocean states from other sources (MPAS-Ocean, reanalysis, other models etc.) These functions may be used offline in pre- or post-processing but need to be consistent with the EOS implementation.

Later versions should include functions to calculate the layer potential enthalpy from enthalpy fluxes to convert coupling fluxes to changes in conservative temperature in a rigorous manner.

The formulation of the specific volume calculation will be the 75-term polynomial as documented in [Roquet et al. 2015](https://www.sciencedirect.com/science/article/pii/S1463500315000566). The implementation of the displaced density calculation may be altered to improve performance by reusing some polynomial coefficients, but should produce the same results as the full 75-term polynomial.

The formulation of the specific volume calculation will be the 75-term polynomial as documented in 

. The implementation of the displaced density calculation may be altered to improve performance by reusing some polynomial coefficients, but should produce the same results as the full 75-term polynomial.

The GSW-C toolbox will be incorporated into Omega as a submodule. In order to abide by the license of the toolbox, the toolbox will only be used in testing to check our implementation. The coefficients for the Roquet et al. 2015 expansion will be based on the published values (Appendix). A Kokkos implementation of the function to compute specific volume will be ported to Omega. The GSW-C toolbox submodule will serve as a baseline reference for our ports in unit tests.

To optimize performance, calls to the EOS function should be able to support whole array computation. The EOS class will include a public method that calculate specific volume given the state variables, and a public variable defining the chosen eos option (changeable by the user).

The `Eos` class will be used to perform the necessary eos operations such as calculating specific volume:

 class will be used to perform the necessary eos operations such as calculating specific volume:

class Eos{
    public:
       Array SpecVol;
       Array SpecVolDisplaced;
       void computeSpecVol();
       static Eos *create();
    private:
       EOSType eosChoice;
       I4 NCellsAll;
       I4 NChunks = NVertLayers / VecLength;
       void computeSpecVolTEOS10Poly75t();
       void computeSpecVolLinear();
       void truncateTempSal();
       static Eos *DefaultEos;
       static std::map<std::string, std::unique_ptr<Eos>> AllEos;

The user will select the EOS type at runtime in the input yaml file under an eos section:

An `enum class` will be used to specify options for the EOS used for an Omega simulation:

 will be used to specify options for the EOS used for an Omega simulation:

enum class EOSType{
   Linear,  /// Linear equation of state
   TEOS10Poly75t  /// Roquet et al. 2015 75 term expansion
}

We assume that the call to the eos is in a timestepping routine where input tracers for this timestep are defined, eg.

   Array2DReal ConservativeTemperature;
   Array2DReal AbsoluteSalinity;
   Tracers->getByName(ConservativeTemperature, VelTimeLevel, "temperature");
   Tracers->getByName(AbsoluteSalinity, VelTimeLevel, "salinity");

There will be a constructor and destructor for the class, as well as several public and private
methods. Modeled on the `Tendencies` class, the constructor will be private. A static `create` method is used to ensure every eos instance is properly stored in the static map of eos instances. This is to help ensure that the eos initialized in the `init` step of the model set-up is then retrievable during the `forward` step.

 method is used to ensure every eos instance is properly stored in the static map of eos instances. This is to help ensure that the eos initialized in the 

Eos::Eos(const HorzMesh *Mesh, int NVertLayers, Config *Options);

The create method will take the same arguments as the constructor plus a name. It calls the constructor to
create a new eos instance, and put it in the static map of all eos.
It will return a pointer to the newly created object.

Eos *Eos::create(const std::string &Name, const HorzMesh *Mesh, int NVertLayers, Config *Options);

The init method will create the default eos and return an error code:

There will be methods for getting the default and non-default eos instances:

Eos *Eos::getDefault();
Eos *Eos::get(const std::string &Name);

The public `computeSpecVol` method will rely on private methods for each specific EOS option (linear and TEOS-10).

 method will rely on private methods for each specific EOS option (linear and TEOS-10).

void Eos::computeSpecVol(const Array2DReal &SpecVol,
                         const Array2DReal &ConservativeTemperature,
                         const Array2DReal &AbsoluteSalinity,
                         const Array2DReal &Pressure) {
OMEGA_SCOPE(...)
    if (eosChoice == EOSType::Linear){
       parallelFor("eos-linear", {NCellsAll, NChunks},
       KOKKOS_LAMBDA(int ICell, int KChunk) {
          LocComputeSpecVolLinear(...);
       });
    }
    else if (eosChoice == EOSType::TEOS10Poly75t){
       parallelFor("eos-teos10", {NCellsAll, NChunks},
          LocComputeSpecVolTEOS10Poly75t(...);
        });
    }...

No operations are needed in the destructor as the Kokkos arrays are removed
when they are no longer in scope. The erase method
will remove a named eos instance, whereas the clear method will remove all of

A unit test will verify that the result of the GSW-C toolbox (used as a submodule, without modifications) matches the expected value of specific volume published in Roquet et al. 2015 within machine precision. The publication only includes one data point (Sa, Ct, P) to check.

5.2 Test: Verification of our TEOS-10 75-term polynomial implementation

A unit test will verify that the result of the Omega implementation of the Roquet et al. 2015 75-term polynomial calculation of the specific volume matches the expected value of specific volume published in Roquet et al. 2015 within machine precision. The publication only includes one data point (Sa, Ct, P) to check.

5.2 test: verification of our teos-10 75-term polynomial implementation

A unit test will verify that the result of the Omega and GSW-C toolbox implementations for the Roquet et al. 2015 expansion are within machine precision over a range of conservative temperature, absolute salinity, and pressure ranges.

The linear EOS should be close, but not equal to the TEOS-10 result. The calculated density should be exactly linear in T,S. It will be verified against known values from the MPAS-Ocean model for a range of temperature and salinity values.

The total potential enthalpy, or equivalently, the conservative temperature should be conserved in the absence of external energy fluxes. This could be based on the tracer conservation test (e.g. merry-go-round), applied to active tracers.

Any simulation code requires a means to check for errors,
report errors and abort if the error is critical. Omega
includes a logging facility that can perform some of these
functions. However, we desire a facility that can check
for errors in a manner that improves code readability and
that also provides a stack trace to better understand the
path taken that resulted in the error.

We require the ability to log an error with an error message
that includes the routine and line number. Note that the
current Omega Logging facility already supports this and
should be leveraged for this purpose.

Omega must support several levels of error severity. The
Logging facility already supports a common hierarchy with
debug, info, warn, error and critical levels. Because the
debug and info simply write information, they will continue
to be managed by the Logging interfaces.

The warn level may be shared by both the Logging facility and
error handler. For example, if only a warning message is required,
the Logging facility should manage the message. If we wish to
inform the calling routine that an error at the warning level
has occurred, the Error handler should manage the warning.

The error handler should be the primary interface for the more
severe errors at the error and critical levels. In a canonical
hierarchy, critical errors are those that cause the application
to fail, while the error level refers to errors that are severe
enough to be addressed but still allow the application to proceed
without crashing. In Omega, these latter errors typically cause
incorrect or unintended simulation results and are considered
an application failure. We prefer to use critical errors that
abort the simulation for most such errors. The error level
should be reserved for cases in which an error is encountered
but we must pass error information back to the calling routine
(eg through a return code) to judge the response to the error.

Note that at the initial creation of this document, many of
the current errors reported by Omega are treated at the error
level and should be promoted to critical errors following the
description above.

When a critical error is encountered, we must have the ability
to abort the simulation.

The error log must include a stack trace to determine and
output the calling sequence that resulted in the error being
reported.

We require a means of determining which parallel task is
encountering this error.

We require errors and the related stack trace to be logged
in an error log file. A separate error log file for each MPI
task may be desirable both for ease of implementation and to
better satisfy requirements 2.4 and 2.5 above. An abort error
message should also be sent to the default log file when
possible so that users know to look for an error log for
details.

The error handler should include a public success error code
(typically zero) to standardize a successful return code for
Omega routines. Any return code not equal to success should
be interpreted as a failure. However, a standard Fail code
can also be provided as a default failure code.

We require the ability to define an error message that will
be output to an error log for any error encountered. We also
require the ability to include some variable or run-time
information as part of the the logged message to better
describe the cause of the error, similar to the existing
Logging capability (eg "Value of MyVar is {} but expected
{}"). Ideally, the formatting would be identical to that
provided by the Logging facility and would leverage that
facility for the logging of error messages.

In some cases, we wish to accumulate (sum) non-critical errors
encountered within a routine rather than returning or aborting
immediately. For example, a unit test might prefer to report
all errors rather than aborting on the first error encountered.
Among other things, this is a requirment that error codes must
all have the same sign to avoid summing to zero (the nominal
success code).

We desire error checking to be very concise - ideally a
single line of code - so that the error checking does
not impair code readability.

To facilitate 2.10 or to provide unique error codes, it may
be desirable to pre-register an error code and related error
message during initialization of a module so that it can
be used later and referred to simply by the assigned code.

We will build the error handler on the existing Logging
functionality, the spdlog library and the cpptrace library
that provides a stack trace capability. Similar to the
Logging facility, we will make use of defined cpp macros
for many interfaces. These macros allow us to both condense
multi-line patterns into a single-line interface, but also
allow us to incorporate the LINE and FILE macros at the
actual line where the error is encountered.

There will be two public integer parameters (constexpr)
``Err::Success`` and ``Err::Fail``. Success will be the only
identifier for success and will be equal to zero. Any error code
not equal to success will denote failure. This allows multiple
failure error codes and accumulation of error. The ``Err::Fail``
code is defined to provide a default failure code if one is
not defined or if a specific value is not required.

There will be an Error class that primarily holds defined
error codes and associated messages.

class Error {
   public:

      static constexpr int Success = 0;
      static constexpr int Fail    = 1;

      // public methods (see below)

   private:

      // container for pre-defined error codes/messages
      static std::map<int, std::string> AllCodes;

      // private methods (see below)
};

We provide a number of macros for error checking. As described
in section 2.2, we expect the most common interface should be
the critical error macro:

The error message (ErrMsg) can be either a simple string or
can follow the format of the Logging facility in which ``{}``
placeholders can be used with additional arguments to
include variable information
(see the [Logging documentation](#omega-user-logging)).
If the input ErrCode is ``Err::Success`` nothing happens and
the code continues. If the ErrCode matches a code predefined
using the ``defineCode`` function described below, the ErrMsg
from that predefined code is used with any of the additional
arguments to fill in any ``{}`` placeholders. Finally, if the
ErrCode is not equal to Success and is not a predefined code,
the ErrMsg is used, again with the subsequent arguments to print
an error message. For the failure modes, the macro also prints
a stack trace using cpptrace calls and then aborts the
simulation using the ``Err::abort`` function described below.
Typical use cases might be:

// Checking an error condition
if (MyFailCondition) ERROR_CRITICAL(Err::Fail, "MySimpleMessage");

// or
if (MyFailCondition)
   ERROR_CRITICAL(Err::Fail, "Variable {} has value {}",
                  MyVarName, Value);

// or if the code is predefined
if (MyFailCondition) ERROR_CRITICAL(DefinedCode, " ");

// Checking a return code from another routine
Err = MyFunction(...);
ERROR_CRITICAL(Err, "Error encountered in MyFunction");

Two interfaces are provided in which a condition can be checked,
rather than the error code:

ERROR_REQUIRE(condition, ErrMsg, additional args for msg);
ERROR_ASSERT(condition, ErrMsg, additional args for msg);

where condition is any conditional expression that evaluates to
true/false. These have the same behavior as the critical errors
above with a false condition treated as an ``Err::Fail``. The
REQUIRE interface is always evaluated; the ASSERT interface behaves
similarly to the C++ assert function and is only checked for DEBUG
builds.

For less severe errors, we provide several different macros
to match the behavior desired when continuing the simulation.
The first interface is similar to the critical interface above:

ERROR_CHECK(ErrCode, ErrMsg, addition args for message);
ERROR_WARN(ErrCode, ErrMsg, addition args for message);

which simply prints the message when the ErrCode is not
equal to Success. The ``ERROR_WARN`` interface is used when
only a warning message is desired. Predefined
error codes can be used here as well. Unlike the
critical interface, this does not print a stack trace or abort
the simulation and the behavior is more similar to the
``LOG_ERROR`` or ``LOG_WARN`` macros.

If a routine has a return code and a return-on-error is
desired, we provide an interface:

ERROR_RETURN(ReturnCode, ErrCode, ErrMsg, added vars if needed);
ERROR_RETURN_WARN(ReturnCode, ErrCode, ErrMsg, added vars);

This behaves similar to the ``ERROR_CHECK`` except that after
the message is printed, the error code is added to the return
code and a return statement is executed to exit the routine.
A use case might be:

if (MyFailCondition) ERROR_RETURN(ReturnCode, Err::Fail, "MyErrMsg");

// which is equivalent to:

if (MyFailCondition) {
   LOG_ERROR("MyErrMsg");
   ReturnCode += Err::Fail;
   return ReturnCode;
}

which after printing the messages, adds the most recent
error code to the running sum stored in ``ErrSum``.

Note that in the latter two cases, care must be taken so that
the accumulated error or return codes do not match a
predefined code and cause confustion in a later use of the
code in an ``ERROR_CRITICAL`` or ``ERROR_CHECK`` call.

Some additional functions are needed to complement the macros
described above.

To allow more concise error checking, developers can pre-define
and error code and associated error message using:

This routine addes the ErrMsg to a map of predefined messages
and returns the new ErrCode assigned to that message for later
use in the macros described above. This function should be
called at some point during model (or module) initialization
and the ErrCode saved for use during the run sequence.

is provided to abort a simulation. A minimal version would
simply call the MPI abort function, but it is possible to
add additional cleanup tasks.

will clean up all of the predefined error codes. If the
ErrCode is not equal to success, it will set the error code
to the standard abnormal termination exit code (1) and avoid other
standard c++ exit codes for other run-time errors.

A unit test will test all non-critical error interfaces first
and then end with a critical failure. The ``WILL_FAIL`` property
in ctest will ensure the expected unit test failure will pass.
An inspection of the log and error files will be used to verify
messages are printed as expected.

In a distributed memory parallel environment, it is necessary to regularly
exchange data on the interfaces between adjacent partitions of the mesh
(i.e. halo elements). These halo exchanges will be accomplished utilizing
the MPI library.

Handle halo exchanges for arrays of each supported data type and dimensionality.

Meshes are composed of three types of elements (cells, edges, and vertices)
at which values can be defined, we must be able to handle exchanges at each
type of element.

The default minimum required halo depth for most of the algorithms we will
use is three cells. We should also be able to handle different halo depths.

Each MPI rank needs a send list and receive list for every neighboring rank and
each type of mesh element (cells, edges, and vertices) containing the local
indices for each owned element to be packed and sent to neighboring ranks and
each element owned by neighboring ranks to be received and unpacked into local
field arrays. Exchange lists should be organized by halo layer.

Each MPI rank needs two arrays for every neighboring rank to serve as a send
buffer and a receive buffer to be passed to the MPI send and receive routines.
On each rank, one buffer per neighbor will be sent concurrently, so there needs
to be one send and one receive buffer array for each neighbor to facilitate
communication.

To efficiently handle exchanges between each rank and all of its neighbors, we
will use the non-blocking MPI routines `MPI_Isend` and `MPI_Irecv`. The
`MPI_Test` routine will be used to determine when each communication is
completed. Boolean variables will be needed to track when messages have been
received and when buffers have been unpacked.

In environments with GPU devices, many arrays will reside solely on the device
and we will need to be able to manage halo exchanges for these arrays, either
via GPU-aware MPI or handling array transfers to host.

We may want the ability to communicate individual halo layers as opposed to
always exchanging the entire halo for a given array. Exchange lists should be
organized by halo layer to facilitate this.

Communications in MPAS are generally latency limited, so it may be advantageous
to exchange multiple arrays at the same time, packing the halo elements of
multiple arrays into the same buffer to reduce the number of messages.

OMEGA runs may contain multiple communication environments or mesh
decompositions, would need to be able to perform halo exchanges in these
circumstances.

If we allow OpenMP threading, halo exchanges would need to be carried out in a
thread-safe manner, and a process to carry out local exchanges may be needed.

It might be advantageous to have persistent buffer arrays that are large enough
to handle the largest halo exchange in a particular run, as opposed to
calculating the needed buffer size and reallocating the buffer arrays for each
halo exchange. Only a subset of the buffer array would need to be communicated
for smaller exchanges. A method for determining the largest necessary buffer
size for each neighbor would be needed, and would depend on which arrays are
active, the number of halo elements in each index space, the number of vertical
layers and tracers.

No specific algorithms needed outside of those provided by standard MPI library

The mesh decomposition will be performed during initialization via the Decomp
class based on the input mesh file and options defined in the configuration
file. A set of class objects described below will take the Decomp object and
store information about the decomposition as well as allocate buffer memory
necessary for performing halo exchanges.

An enum defining mesh element types would be helpful to control which exchange
lists to use for a particular field array:

An MPI datatype ```MPI_RealKind``` will be defined based on whether the default
real data type is single or double precision via the SINGLE_PRECISION compile
time switch:

MPI_Datatype MPI_RealKind = MPI_FLOAT;
#else
MPI_Datatype MPI_RealKind = MPI_DOUBLE;
#endif

Halo exchanges will depend on the ```haloWidth``` parameter defined in the
decomp configuration group.

The ExchList class will hold both send and receive lists. Lists are separated by
halo layer to allow for the possibility of exchanging individual layers. Buffer
offsets are needed to pack/unpack all the halo layers into/from the same buffer.

class ExchList {

   private:

      I4 nList[haloWidth];        ///< number of mesh elements in each
                                  ///<   layer of list
      I4 nTot;                    ///< number of elements summed over layers
      I4 offsets[haloWidth];      ///< offsets for each halo layer

      std::vector<I4> indices[haloWidth];   ///< list of local indices

   friend class Neighbor;
   friend class Halo;
};

The Neighbor class contains all the information and buffer memory needed for
carrying out each type of halo exchange with one neighboring rank. It contains
the ID of the neighboring MPI rank, an object of the ExchList class for sends
and receives for each mesh index space, a send and a receive buffer, MPI request
handles that are returned by `MPI_Irecv` and `MPI_Isend` which are
needed to test for completion of an individual message transfer, and boolean
switches to control the progress of a Halo exchange. To avoid the need for
separate integer and floating-point buffers, during the packing process integer
values can be recast as reals in a bit-preserving manner using
`reinterpret_cast` and then recast as integers on the receiving end.

class Neighbor {

   private:

      I4 rankID;                           ///< ID of neighboring MPI rank
      ExchList sendLists[3], recvLists[3]; ///< 0 = onCell, 1 = onEdge,
                                           ///< 2 = onVertex
      std::vector<Real> sendBuffer, recvBuffer;
      MPI_Request rReq, sReq;              ///< MPI request handles
      bool received = false;
      bool unpacked = false;

};

The Halo class collects all Neighbor objects needed by an MPI rank to perform
a full halo exchange with each of its neighbors. The total number of neighbors,
the local rank, and the MPI communicator are saved here. The mesh element type
for the current array being transferred is also stored here. This class will be
the user interface for halo exchanges, it is a friend class to the subordinate
classes so that it has access to all private data.

class Halo {

   private:

      I4 nNghbr;                           ///< number of neighboring ranks
      I4 myRank                            ///< local MPI rank ID
      MPI_Comm myComm;                     ///< MPI communicator handle
      meshElement elemType;                ///< index space of current array
      std::vector<Neighbor> neighbors;

   public:

      // methods

};

The constructor for the Halo class will be the interface for declaring an
instance of the Halo class and instances of all associated member classes,
and requires info from the Decomp and MachEnv objects.

The constructors for Neighbor and ExchList will be called from within the Halo
constructor and will create instances of these classes based on info from the
Decomp object.

The primary use for the Halo class will be a member function called
`exchangeFullArrayHalo` which will exchange halo elements for the input
array with each of its neighbors across all layers of the halo. The index space
the array is located in (cells, edges, or vertices) needs to be fetched from the
metadata associated with the array stored in the Halo object to determine which
exchange lists to use. It will return an integer error code to catch errors
(likewise for each subprocess below).

int exchangeFullArrayHalo(ArrayLocDDTT &array, meshElement elemType);

This will be an interface for different exchange funcitons for each type of
ArrayLocDDTT (where Loc is the array location, i.e. device or host, DD is the
dimensionality and TT is the data type) as defined in the DataTypes doc. If the
array is located on a GPU device, this function may tranfer arrays from device
to host. The ordering of steps for completing a halo exchange is:

A `startReceives` function will loop over all member Neighbor objects and
call `MPI_Irecv` for each neighbor. It takes no arguments because all the
info needed is already contained in the Halo object and its member objects.

Likewise, `startSends` will loop over all Neighbor objects and call
`MPI_Isend` to send the buffers to each neighbor.

For each type of ArrayDDTT, there will be a buffer pack function aliased to a
`packBuffer` interface:

where halo elements of the potentially multidimensional array ArrayDDTT are
packed into the 1D send buffer for a particular Neighbor in the member
`std::vector` neighbors using the associated ExchList based on the
meshElement the array is defined on. The `exchangeFullArrayHalo` will loop
over the member neighbors and call `packBuffer` for each Neighbor.

Similarly, buffer unpack functions for each ArrayDDTT type will be aliased to
an `unpackBuffer` interface:

A unit test where a set of halo exchanges are performed for each type of
ArrayDDTT and each index space could verify all requirements, except for 2.3.
The test should use a mesh decomposition across multiple MPI ranks, and define
an array for each type of ArrayDDTT and index space. The array elements could be
filled based on the global IDs for the cells, edges, or vertices. After an
exchange, all the elements in an array (owned+halo elements) would be checked
to ensure they have the expected value to verify a successful test. A similar
test utilizing a mesh decomposition with HaloWidth other than the default value
would satisfy requirement 2.3 as well.

The mesh class will contain the Kokkos arrays which describe the horizontal mesh and are used in the computation of the tendency terms in the discrete governing equations. OMEGA will separate the horizontal mesh variables from the vertical mesh information.

2.1 Requirement: OMEGA will use the previously established MPAS Mesh Spec

The OMEGA mesh information should be compatible with the [MPAS Mesh Specification](https://mpas-dev.github.io/files/documents/MPAS-MeshSpec.pdf).

2.1 requirement: omega will use the previously established mpas mesh spec

2.2 Requirement: Functionality is needed to read the mesh on the host and transfer relevant data to the device for computation

Not all mesh information is required in computing the tendency terms on the device, e.g. lonCell, latCell, etc.
However, other arrays will need to be allocated and copied to the device for use in tendency computation.
The mesh class will explicitly include host and device Kokkos arrays for each variable.
A class method will be included to copy the relevant mesh information to the device.

2.2 requirement: functionality is needed to read the mesh on the host and transfer relevant data to the device for computation

Although the existing MPAS Mesh spec uses a one-based mesh numbering, zero-based mesh numbering is required to be compatible with the zero-based indexing used for Kokkos arrays.

The mesh class will reference the partitioned connectivity arrays created by the Decomp class.

2.5 Requirement: Mesh variables will be associated with metadata to describe data

Following the Metadata and IO designs, the Kokkos arrays for the mesh variables will be associated with information about the represented values.

2.5 requirement: mesh variables will be associated with metadata to describe data

The Mesh class will have a method to read in the mesh information not obtained by the Decomp class.

2.7 Desired: Ability to support multiple independent mesh objects

This flexibility is required to support future implementations of spatially split barotropic and baroclinic modes that are computed on different resolution meshes.
Additionally, this flexibility can be used to support separate domain decompositions for the barotropic and baroclinic meshes, which can help optimize the communication frequency for the barotropic subcycling via wide barotropic halos.

2.7 desired: ability to support multiple independent mesh objects

2.8 Desired: OMEGA can read in a reduced number of mesh variables and compute the remaining array information online.

Many of the mesh variables are not independent, e.g.  areaCell, weightsOnEdge, etc., and can be computed from a reduced set of mesh variables.
This functionality could be used to reduce mesh/restart file size for high resolution meshes.
Building in this flexibility would allow for all mesh-related calculations to be available within the code base instead of spreading them amongst various utility programs in MPAS-Tools.
This will improve the ability to maintain and unit test the mesh calculations in MPAS-Tools.
The functions used to compute the dependent mesh information can also be used to verify any mesh information that is provided in the mesh input stream.

As standard practice, all necessary internally computed mesh information will be output in a single file for post-processing purposes.
A checksumming strategy will be implemented to avoid situations where simulation data and mesh information are mismatched during post processing.

Where appropriate, some additional derived quantities (e.g. reciprocals) will also be included to improve the performance of device-side calculations.

2.8 desired: omega can read in a reduced number of mesh variables and compute the remaining array information online.

The algorithms required for computing dependent mesh quantities are currently implemented in the MPAS Mesh Converter utility.

The horizontal mesh information will be organized in a class with public Kokkos arrays.
Arrays that require a device copy will have a explicit variable.
Connectivity arrays that are already contained in the Decomp class will be replicated in the horizontal mesh class via pointers.

class HorzMesh {

public:

  Array1DR8 AreaCell;
  HostArray1DR8 AreaCellH;

  Array2DI4 CellsOnCell;
  HostArray2DI4 CellsOnCellH;

}

There will be a constructor and destructor for the class with the constructor being responsible for calling several private methods.

use Decomp object to create reference to the decomposed connectivity arrays.

The mesh class requires a method to read in all other available mesh information that has been provided in the mesh file, but has not been initialized by the decomposition. This will be a private method called by the constructor.

The compute method will be a private method called by the constructor. It will be resonsible for calculating any dependent mesh information that is not provided in the mesh input file.

This method will be repsonsible for creating the device copies of the required mesh information on the host. It will be a private method called by the constructor.

The metadata associated with each mesh variable will be registred within the I/O infrastructure in a private method called by the constrctor.

The sample domain used for the Decomp test will be used to test obtaining the correct local values and the mesh computation routines.

The OMEGA model must be able to read and write data to a filesystem.
For performance at high resolution, much of this IO must occur in parallel
and interact with a high-performance filesystem. We describe here an IO
layer that provides interfaces to the underlying SCORPIO parallel I/O
library used in E3SM. It primarily provides a translation layer to
read/write OMEGA metadata and Kokkos arrays. It works together with the
OMEGA Metadata and IOStreams capabilities. Users will interact with
IO primarily through the IOStreams and should not need to access this
layer directly.

The model must be able to read desired data and metadata from
an input file into the internal model storage/decomposition.
Metadata requirements are described in the Metadata design document.

The model must be able to write desired data and metadata
to an output file for later use. Metadata requirements are
described in the Metadata design document.

All files must (eventually) be in a self-describing
format like HDF5 or netCDF. Note that alternative formats
can be supported as long as scripts are provided to convert
to netCDF/HDF5 efficiently as part of model run scripts.
SCORPIO supports a number of underlying formats and the
user must be able to select the appropriate format (eg
multiple netcdf formats and ADIOS format).

The model must support reading from and writing to an
arbitrary number of files with different properties
(eg precision, time frequencies, contents). There will
be a Streams capability described in a separate
design document that will manage many of these properties,
but the underlying I/O layer here must be able to
support multiple open files, each with unique
properties.

Performance at high resolution requires a parallel I/O
solution in which multiple processors can be writing/reading
data to maximize bandwidth to the filesystem and minimize
time spent in I/O.

Some properties of the parallel I/O must be configurable
to tune the parallel I/O for performance on a particular
architecture and filesystem. At a minimum, the user must
be able to specify the number of I/O tasks and the stride
of those tasks.

The I/O system must be able to read/write all supported
data types for metadata and all supported Kokkos array types.
In some cases, output files at reduced precision are required
so an option to convert data to reduced precision is needed.

Distributed data in OMEGA is stored as Kokkos array types.
We must be able to read/write Kokkos arrays and be able
to move data between host and device as needed.

If an output file exists, the user must be able to specify
whether the model should overwrite the existing file, exit
with an error message or append data to the existing file.
On input, if a file does not exist, the model must exit with
an appropriate error code.

For performance, it would be desireable to enable the model
to launch I/O tasks and resume computation while the actual
writing to file takes place. This will mask I/O costs by
overlapping I/O functions with model computations.

It may be desireable to apply data compression while
performing I/O to minimize data file sizes. This compression
may be lossless or not depending on the use for each
file.

Parallel I/O systems use various algorithms for rearranging
data in parallel decompositions to the decomposition required
for the I/O processor layout. These algorithms are described
in the I/O library (scorpio) documentation and related
publications.

The OMEGA model I/O will be built on top of the SCORPIO
parallel I/O library used across E3SM components. The
I/O interfaces here generally provide wrappers for
translating internal OMEGA metadata representations and
Kokkos array types to the form required by SCORPIO. OMEGA
users and developers will generally interact with I/O
through the IO Streams layer that manages all files and
associated file contents.

There will be a section in the input OMEGA configuration file
for managing overall parallel IO options. Currently, this
will include three variables:

    IO:
       ioTasks:  1
       ioStride: 1
       ioRearranger: box

The default for IO tasks and stride will be 1 for safety, but
should always be changed to appropriate values for a particular
simulation and machine architecture. For example, the user
should set the stride such that the number of cores performing
I/O is limited to the optimal number for a given multi-processor
node. Scorpio supports a few rearranger methods for moving the
data to IO tasks (see Scorpio documentation) but the box
rearranger is often preferred and is the default. The rearranger
is ignored for the ADIOS back-end since it writes separate files
from each task and rearranges the data in a post-processing step.
These input variables correspond to variables of the
same name.

We define a number of enums to support various options, including
the rearranger method, file format, operation (read/write) and
behavior when file exists:

   enum class IORearranger {
      box,    /// box rearranger (default)
      subset, /// subset rearranger
      unknown /// unknown or undefined
      };

   enum class IOFileFormat {
      undefined,
      default,
      netcdf,
      pnetcdf,
      netcdf4,
      pnetcdf5,
      hdf5,
      adios,
      };

   enum class IOMode {
      read,
      write,
      };

   enum class IOIfExists {
      overwrite,
      fail,
      };

   enum class IOPrecision {
      single,
      double,
      };

The PIO routines require some information about the decomposition,
so there will be one class containing environment information including
array decompositions for each data type, mesh location and array size.

   class IOEnv {

      private:

         /// track and store all defined environments internally
         static std::vector<std::shared_ptr<IOEnv>> definedEnvs;

         /// name for this environment
         std::string name;

         /// default file format and rearranger method
         IOFormat format;
         IORearranger rearranger;

         /// ptr to defined Scorpio IO system with MPI communicator info
         int* iosystem;

         /// pointers to defined decompositions for every multi-dim
         /// array, data type and cell location
         int* decompCell1DI4;
         int* decompCell1DR4;
         int* decompCell1DR8;
         int* decompCell2DI4;
         int* decompCell2DR4;
         int* decompCell2DR8;
         // continue for dimensions up to 5 and edge, vertex
         //   mesh locations
         // note that Scorpio does not support logical arrays
         //   or I8 types so these will be converted to an
         //   an appropriate type during read/write

         // Methods below will be private and only accessed
         // via the OMEGA IOstreams interfaces with IOstreams
         // as a friend class

      public:
         // Users will not access methods or vars directly
         // but only through the IOstreams interfaces

      friend class IOStreams;
   }

There will also be an IOField class to combine the Metadata
and a pointer to the array holding data. This will be used
to specify the fields in an IOStream.

   class IOField {

      private:

         /// track and store all defined fields internally
         static std::vector<std::shared_ptr<IOField>> definedFields;

         std::shared_ptr<Metadata> metaPtr;

         /// only one of the pointers will be defined based on array type
         std::shared_ptr<Array1DI4> data1DI4;
         [replicated up to 5D arrays of I4,I8,R4,R8]
         std::shared_ptr<HostArray1DI4> dataHost1DI4;

      public:
         // See methods below

   }

As noted above, these methods are actually private and accessed
only through the IOStreams interfaces.

There will be a constructor that initializes the IO system
and decompositions. Note that the defaults for file format
and rearranger can be overridden on a per-file basis.

   IOEnv(const std::string name, /// name for this env
         const MachEnv omegaEnv, /// machine env with MPI info
         const Decomp  omegaDecomp, /// mesh decomposition
         const int ioTasks,      /// number of IO tasks
         const int ioStride,     /// stride in MPI ranks for io tasks
         const IOFormat format,  /// default IO format
         const IORearranger rearranger, /// default rearranger method
         );

There will be interfaces for opening and closing files for reading and writing.
For the file open, a file id will be returned.

   /// Opens a file for reading or writing, returns an error code.
   /// If the IOFileFormat argument is `default`, the default
   /// format from IOEnv will be used. `ifexists` will be ignored for
   /// reads.
   int IOFileOpen(int& fileID, /// returned fileID for this file
                  const std::string filename,  /// name of file to open
                  const IOEnv& myEnv,          /// overall IO environment
                  const IOMode mode,           /// mode (read or write)
                  const IOPrecision precision, /// precision of floats
                  const IOIfExists ifexists,   /// behavior if file exists
                  const IOFileFormat format,   /// file format
                 );

   /// closes an open file using the fileID, returns an error code
   int IOFileClose(int& fileID  /// ID of the file to be closed
                  );

Because the IOStreams and metadata manager have aggregated all
metadata, dims and pointers to data arrays, a file can be written
with a single call using the aggregated data from the IOStream:

   /// writes all data associated with the file
   int IOWrite(const int fileID, /// id for the open file
               const std::vector<std::shared_ptr<IOField>> contents);

Unlike the write function, not all data within a file may need to
be read, so we read each field individually:

   /// reads a field the file
   int IORead(const int fileID, /// id for the open file
              std::shared_ptr<IOField> field);

Each field meant to be read or written should be defined, typically
at the same time a module defines the associated metadata. The IOField
is then constructed first with a pointer to the Metadata:

   /// IOField constructor
   IOField(std::shared_ptr<Metadata> fieldMeta);

Then a pointer to the array is attached using an attach function
(aliased to the various array types):

   /// IOField attach data
   IOField::attachData(std::shared_ptr<Array1DI4>);
   [replicate generic interface for all supported array types]

Because the functions here are only accessed via the IOStreams interfaces,
the testing of these routines will be part of the IOStreams unit test.

OMEGA must be able to read and write data throughout a simulation.
Data can be read or written either once or periodically. Files
will contain different fields to read/write. We consider each file
or file sequence an IO stream with its own unique set of properties.
We describe here the requirements and design of these IO streams.
The design relies on companion designs for managing metadata and a
lower-level IO functions for writing metadata and data stored in
Kokkos arrays.

An arbitrary number of input and output streams, each with its own
properties and precision, must be supported.

For some output, it is not necessary to retain the full precision
that the model supports. Reduced precision for each stream must
be supported to reduce file size when full precision is not
required.

Users must be able to supply a list of fields to be included in
each stream. This contents list should be checked against
available fields (from OMEGA defined metadata and metadata groups)
and exit if a field is not available.

There must be one stream for checkpointing and restarting and
must include all fields needed to exactly restart the model,
maintaining bit-for-bit when compared to a non-restarted simulation
over the same time interval. This also generally requires full
precision for all fields.

To avoid the need to modify configuration files with a new
input filename on every restart, a mechanism for determining
the last successful restart is needed. This is often implemented
by writing the name of the last successful restart file to a
standard location (pointer file) so that the model can read the
name of the last restart and continue the simulation.

The user must be able to specify a time interval for any
repeating input and output streams (eg monthly output).
The time interval can be any interval supported by the OMEGA
time manager.

For some streams, we may wish to supply a start and stop
time to sample the simulation only over a particular time
period. For example, requesting high-frequency output over a
short interval. The user must be able to optionally request
a start and stop time for the stream.

The user must be able to supply a filename associated with the
stream. For repeating I/O, the filename will be a template
that will specify how the filename will be modified to reflect
the time associated with the output. The template should
be able to support any conventions (eg E3SM) required. Filenames
should include the full path.

Streams must be able to support multiple time instances for
the requested fields in a single file. For example, monthly
forcing inputs may be provided in a single input file. Similarly,
for higher-frequency output, it may be desirable to include
several intervals in a single file. An appropriate filename
convention/template must also be provided for this situation.

To support multiple time slices per file and to support cases
where the model is re-run over the same interval (eg during
testing or repeating a failed run), the user must specify the
desired behavior when an output file of the same name already
exists. A minimum set of options should include replacing
the file, overwriting portions of a file (eg specific time
slices), appending to the existing file, or exiting with an
error message.

In some cases, it is desirable to accumulate the time average of
selected fields during the simulation in order to sample every
timestep (or other frequency higher than the output interval). While
the time-averaging capability itself will be implemented elsewhere
in the code, time-averaging over intervals longer than the typical
checkpoint/restart interval would require additional restart
capabilities to maintain these time averages and a filename
template should describe the averaging interval in some format.

In the future, it may be possible to apply lossy or loss-less
compression on data to save space. An option to compress
data would be desirable if it can be supported. Like
reduced-precision, this option would probably be on a per-stream
basis.

No algorithms are needed. Most functions carried out by other
modules and libraries.

IOStreams will be the primary mechanism used for all input and
output. The streams will be defined in a streams section of the input
Omega configuration file. Because YAML uses some common characters
to denote lists, etc., filenames should be in quotes. For example:

   IOStreams:

      mesh:
         mode: read
         name: '/mypath/meshFileName'
         freqUnits: initial
         freq: 1
         precision: double
         contents:
         - GRPmesh
         - [other fields?]

      restart-read:
         mode: read
         name: '/mypath/omega-restart.$Y-$M-$D_$h.$m.$s.nc'
         pointerFilename: '/mypath/pointerFileName'
         freq: 1
         contents:
         - GRPrestart
         - [other fields or groups as needed]

      restart-write:
         mode: write
         freqUnits: years
         freq: 1
         ifExists: overwrite
         contents:

      history:
         name: '/mypath/omega-history.$Y-$M-$D_$h.$m.$s.nc'
         freqUnits: months
         freq: 1
         contents:
         - temperature
         - salinity
         - normalVelocity

      # 10-day high-freq data stored in an annual file
      highFreq:
         name: '/mypath/omega-highfreq.$Y-$M.nc'
         freqUnits: days
         freq: 10
         ifExists: append
         startDate: yyyy-mm-dd
         endDate: yyyy-mm-dd
         contents:
         - salinity

Note that there is a stdlib iostreams so our use of IOStreams runs
the risk of inadvertent conflict. Use of both the Omega namespace and
the capitalized IOStreams will be used to distinguish the two.

There are no parameters specific to the IOStreams, though they will
utilize some parameters (like precision and other options) from a
lower-level IO module.

The main class is an IOStream that carries all the information
related to each input or output stream. Like other classes, all
instantiations of streams are managed within the class.

   class IOStream {

      private:
         std::string name;      /// name of stream
         std::string filename;  /// filename or template

         IOMode mode;           /// mode (read or write)
         IOPrecision precision; /// precision for floating point vars
         Alarm sAlarm;          /// time mgr alarm for read/write

         bool usePointer;       /// flag for using a pointer file
         std::string ptrFilename; /// name of pointer file

         bool useStartEnd;      /// flag for using start, end times
         TimeInstant startTime; /// start time for stream
         TimeInstant endTime;   /// end   time for stream

         /// Contents of stream in the form of a vector of
         /// pointers to defined IOFields
         std::vector<std::shared_ptr<IOField>> contents;

         /// Store and maintain all defined streams in this vector
         static int numStreams;
         static std::vector<std::shared_ptr<IOStream>> allStreams;

      public:
         [see methods described below]
   };

There will be a constructor that creates a stream and fills the
scalar variables. And a destructor that deletes the stream and removes
it from the list of defined streams.

   IOStream(int&  streamID, /// id of created stream
            const std::string name,     /// name of stream
            const std::string filename, /// file name or template
            const IOmode      mode,     /// read/write mode
            const IOPrecision precision, /// precision for floats
            const IOIfExists  ifExists,  /// action if file exists
            const std::string freqUnits, /// time frequency for I/O
            const int         freq,      /// freq in above units for I/O
            const std::string pointerFile, /// pointer filename if used
            const std::string startDate, /// optional start date string
            const std::string endDate.   /// optional end date string
            );

   ~IOStream();

Once an IOStream is created, the contents must be added using one of
two interfaces. If the streamId is still available, the first interface
can be used and is faster (the streamID is the index into the vector
of defined streams). Otherwise, an interface that takes the
stream name can be used. An error code is returned.

   int IOStreams::AddField(
                const int streamID, /// id of stream to be modified
                const std::string fieldName /// name of field
                );

                const std::string streamName, /// name of stream
                );

Generally, we want to write all streams at the end of the timestep
if it's time to write, so there will be one interface to simply
check the alarms for all streams and write the data if it's time.
After writing, the alarm will be reset.

check the alarms for all streams and write the data if it’s time.

However, if the user needs to write a stream from elsewhere in
the code, there will be an interface to write a single stream
by either streamID or by stream name after checking the alarm to
see if it's time to write. The alarm will be reset once the writing
is complete.

see if it’s time to write. The alarm will be reset once the writing

   int IOStreams::Write(const int streamID, /// id of stream to be modified
                       );

   int IOStreams::Write(const std::string streamName, /// name of stream
                       );

Unlike writing, each input stream is typically read from the
appropriate module, so only single-stream reads are provided and
with only the name interface since the id may no longer be
accessible by the calling routine.

   int IOStreams::Read(const std::string streamName, /// name of stream
                      );

A test driver and configuration file will create a number of streams
with fields, frequencies and other options that attempt to span all
possible configurations (though may not be able to test all file
formats). This driver will march in time to write streams at the
requested frequencies. At the end of the driver, input streams that
mirror the output streams will be read and the fields compared to
determine if they match.

The OMEGA logging system offers a standardized Application Programming
Interface (API) that facilitates the tracking of events occurring during
the runtime of all Omega sub-components. Each event comprises a descriptive
message along with an indicator denoting its severity level.

It is a requirement for the OMEGA logging system to furnish Omega
sub-components with a mechanism to store log messages onto a file system,
based on their corresponding severity levels.

It is a requirement for the OMEGA logging system to recognize the log root
filename defined in E3SM as a part of the input configuration.

It is a requirement that each process should have the capability to generate
log messages independently. Additionally, there should be a mechanism in
place to control the number of processes generating log messages
simultaneously. This functionality is desired to ensure efficient management
of system logs and prevent potential system overloading due to excessive
logging activities.

It is a requirement for the OMEGA logging system to provide thread-safe
logging capabilities. This functionality is necessary to ensure that
concurrent logging activities by multiple threads are executed in
a synchronized manner, without causing data corruption or inconsistencies
in the generated log files.

It is a requirement for the OMEGA logging system to support a subset of
Omega data types, in order to generate well-formatted log messages that
include these data types. This functionality is crucial to ensure that all
relevant information is captured in the logs and to facilitate effective
debugging and troubleshooting activities.

It is a requirement for the OMEGA logging system to support both buffered
and unbuffered file IO modes. Buffered mode reduces the overhead associated
with file IO operations, improving performance and efficiency. Unbuffered mode
is crucial for debugging purposes, as it allows for immediate writing of log
messages to the file, preventing potential loss of unwritten messages that
could lead to confusion. The API should provide an option for the user to
choose between buffered and unbuffered modes.

It is desired for the OMEGA logging system to support various use-cases of log
files by generating log messages through one of the installed message
formatters. These formatters should be switchable at both compile-time and
runtime to accommodate changing requirements. The default formatter should
generate an unstructured text output, while other possible formatters should
be able to generate structured output such as JSON. This functionality is
essential to ensure flexibility in the logging system, allowing users to
tailor the output format to their specific needs.

It is desired for the OMEGA logging system to be capable of transporting log
messages to various output sinks such as emails or databases. Thisfunctionality
is crucial for ensuring the accessibility and availability of log data, enabling
users to view and analyze the logs through multiple channels.

It is desired for the OMEGA logging system to provide each sub-component with
its own local logger, which can be used to set parameters such as "severity,"
"message format," and "sink type" independently from other sub-components. This
functionality is necessary to ensure that each sub-component has full control
over its own logging activities, allowing for more granular and targeted logging
output.

its own local logger, which can be used to set parameters such as “severity,”

“message format,” and “sink type” independently from other sub-components. This

The spdlog logging library has been chosen as the core logging capability for
the OMEGA logging system. spdlog is a feature-rich, header-only C++ logging
library that is widely used and highly regarded in the industry. It offers
a comprehensive set of features that satisfy most of the requirements, but
additional E3SM and Omega specific requirements must be addressed.

The software design will implement a global switch for setting the minimum log
severity and message format using predefined global C++ defines. The severity
level can be set at compile-time using `-D OMEGA_LOG_LEVEL=<level>`, where `<level>`
can be one of the predefined levels: `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR`,
or `CRITICAL`, where the severity increases from `TRACE` to `CRITICAL`. Log
messages that have severity equal to or higher than the specified level will be
stored in the log destination.

Furthermore, the message format can be set using `-D OMEGA_LOG_PATTERN=<pattern>` at
compile time, where `<pattern>` represents a format in the form of `%flag`,
similar to the strftime function. This pattern defines the layout of the log
message.

Users can control which MPI ranks generate log files using
`-D OMEGA_LOG_TASKS=<tasks-pattern>` at compile time. The `<tasks-pattern>` is
either all for all tasks to generate log files, or comma-separated MPI rank
numbers, or a range of MPI ranks with a dash. For example,
`-D OMEGA_LOG_TASKS=0,2-3` indicates that MPI ranks 0, 2, and 3 generate log
files.

The OMEGA logging system includes multiple macros that allow for logging at
various severity levels. These macros guarantee that log messages with severity
levels lower than the specified `OMEGA_LOG_LEVEL` will not be compiled.

The OMEGA logging system includes macros that allow the user to write log
messages with different severity levels. These macros follow the naming
convention `LOG_<severity level>`. For instance, the following macro can be
used to write an informational log message:

The OMEGA logging system provides the following macros for writing log messages
with different severity levels:

`LOG_CRITICAL`: Used to indicate a critical error or issue that requires
immediate attention and could potentially cause data loss or application
failure.

`LOG_ERROR`: Used to indicate an error or issue that could cause
application instability or incorrect behavior.

`LOG_WARN`: Used to indicate a warning or potential issue that should be
addressed but does not necessarily require immediate attention.

`LOG_INFO`: Used to provide general informational messages about the
application's operation or progress.

`LOG_DEBUG`: Used to provide detailed debugging information for
developers to track down issues or analyze behavior.

`LOG_TRACE`: Used to provide the most detailed level of information,
including function calls and variable values, and is typically used for deep
analysis and debugging.

In order to enhance the logging capabilities, OMEGA logging system provides
each macro with an optional argument of an Omega data type. If the data type
is recognized, it will be formatted automatically by the system and appended in
the appropriate format to the log message. As an example, the following macro
demonstrates how an argument of type `Array1DR8` can be appended to a log
message:

A unit test should be performed to verify all requirements, except for 2.2
(E3SM interoperability). The unit test should create multiple MPI ranks with
multiple threads to ensure that requirement coverage is complete. The unit
test should also include tests for both buffered and unbuffered logging
capability, with various log message sizes and frequencies, to ensure that both
modes work correctly and without loss of data.

A unit test should be performed to verify Requirement 2.5(Omega data types).

Requirement 2.2 will be tested as part of larger system tests and will not have
a dedicated unit test.

On startup, OMEGA will need to initialize the message-passing and other
environments and set up parameters related to machine layout, messaging,
and hardware for use throughout OMEGA.

In standalone mode, OMEGA will need to initialize the MPI environment
and define a default communicator.

When running as part of a coupled system, OMEGA will need to define a default
communicator based on a parent communicator sent by the calling routine
(coupler or parent model).

Each MPI rank will need to know its own rank id, number of ranks and
define a master rank.

Since we are using Kokkos for kernel launching and array types, we
will need to initialize Kokkos in standalone mode. It may also be
needed for coupled simulations.

To achieve the best CPU performance, especially when using
GPU-friendly loop forms, it is useful to explicitly size inner
loops based on a compile-time length (chunk size) that is a multiple
of the vector length.

While rank 0 is typically used as a master task, it is sometimes desirable
to assign a different rank as master to avoid overloading rank 0, especially
when running in coupled mode with other components also using the same rank.

In OMEGA, we may wish to run sub-components on different partitions. For
example, we might want to rearrange the communication-dominated
barotropic solve to run on fewer nodes or within a single node. We will
need to be able to create new environments based on a subset of an
existing environment. In setting up multiple environments, it will
be desireable to have some awareness of network topology for optimal
task placement.

If OpenMP threading is enabled for CPU, it may be useful to also define
similar master threads and thread layouts to enable some task parallelism
within an MPI rank.

In the future, it may be useful to define other machine parameters, like
the specific configuration of cores, accelerators and other devices, to
manage task assignments on hybrid nodes. It should be easy to modify
this class to add additional information.

No algorithms are needed beyond what is provided by standard MPI, OpenMP
libraries.

In general, this is a simple class to hold information for later

For improved vector performance on CPU and perhaps match thread block
sizes on GPU, we wish to block the inner loops with a compile-time
parameter. We define a CPP parameter: `OMEGA_VECTOR_SIZE`. This is
dependent on the machine and on whether GPU acceleration is enabled.
It will typically take on values like 16, 32, 64 for CPU-only builds
and 1 for GPU builds to maximize parallelism.

There will be a simple class MachEnv. We use a class here rather than
a struct so that we can make members private to prevent overwriting
these variables.

class MachEnv {

   private:
      int mComm;      ///< MPI communicator for this environment
      int mMyRank;    ///< rank ID for local MPI rank
      int mNumRanks;  ///< total number of MPI ranks
      int mMasterRank;///< rank ID for master rank
      bool mIsMaster; ///< true if the local rank is the master

   public:

      // Methods
      [define methods here - see below]
}

We will keep a default environment `OMEGA::defaultEnv` as a public
static instantiation that will be used by most of the OMEGA infrastructure.
If other environments are created, they must be maintained by the
defining routines or sub-components.

There will be two forms of the initialization routines. One of
these two must be called as early as possible in OMEGA initialization
(typically the first call). Both forms will return an integer
error code and will define the default environment `OMEGA::defaultEnv`.

// Initialization - standalone
int MachEnvInit();

// Initialization - coupled
int MachEnvInit(int inCommunicator, ///< [in] parent MPI communicator
                );

We provide several constructors for creating instances of the environment.
These will be used primarily by the above initialization routine, though
can also be used to create additional environments per requirement 2.7.

// Generic constructor that uses `MPI_COMM_WORLD`
MachEnv();

// Constructor that uses an assigned communicator (eg from coupler)
MachEnv(const int inCommunicator ///< [in] parent MPI communicator
        );

// Create a new environment from a contiguous subset of an
// existing environment
MachEnv(const int inCommunicator,///< [in] parent MPI communicator
        const int newSize        ///< [in] use first newSize ranks
        );

// Create a new environment from a strided subset of an
        const int newSize,       ///< [in] num ranks in new env
        const int begin,         ///< [in] starting parent rank
        const int stride         ///< [in] stride for ranks to incl
        );

// Create a new environment from a custome subset of an
// existing environment, supplying list of parent ranks to include
        const int ranks[]        ///< [in] vector of parent ranks to incl
        );

and would define `MPI_COMM_WORLD` as the communicator as well
as initialize various other environments as needed.  In coupled mode,
the second form would be used with the coupler passing the
appropriate communicator to be used.

To satisfy requirement 2.7, we provide three forms to create a new
environment based on subsets of the old. The first simply creates
a subset from the first newSize ranks of the parent. The second
creates a strided subset (every "n" ranks starting from a specified
beginning rank). A third is the most general and creates a subset from
a vector of specific ranks of the parent to include in the new environment.

creates a strided subset (every “n” ranks starting from a specified

We provide specific retrieval functions for each class member to mimic
using this environment as a struct:

int Comm() const;      ///< returns MPI communicator for this environment
int MyRank() const;    ///< returns local MPI rank
int NumRanks() const;  ///< total number of MPI ranks

myRank = OMEGA::defaultEnv.MyRank(); // retrieve local rank id
if (OMEGA::defaultEnv.IsMaster()){
   // do stuff on master rank
}

While most of the members should not be modified (read only using the get above), we will need a single set function to satisfy requirement 2.6.

Note that this should occur as soon as possible after the environment is
created. Resetting the master after other activities have already assumed
the default master rank could leave variables defined on the wrong rank
and undefined on the new master rank. The integer return value is a
success/fail return code.

We will test this with a simple test driver in an 8-rank
MPI configuration. Requirement 2.4 (Kokkos initialization) will
need to be verified by visual inspection but later tests using
Kokkos functions will determine whether this has been successful.

The test driver will call the standalone initialization, then
verify by retrieving the members and comparing to the equivalent
native MPI calls on `MPI_COMM_WORLD`

After the above test, use the set function to change the master
task to 1 and then verify using the retrieval function

Create three new environments using each of the three subset
constructors with the first creating a new environment using
the first 4 ranks, the second using every other rank (stride 2),
and the third using ranks 1,2,5,7. Verify that the members are
as expected when compared to the parent environment.

While we can't test this directly in a standalone test driver,
we can test the underlying constructor by passing one of the
subset communicators as the input and verifying the resulting
environment is the same as the subset environment.

Within the test driver, set an int variable to `OMEGA_VECTOR_SIZE`,
and build the test with `-D OMEGA_VECTOR_SIZE=16`. Verify the
internal variable is also 16 to test that the preprocessor properly
propagates the value internally.

All data files from OMEGA and E3SM must be in a self-describing format
to enable easier analysis, archiving and provenance. We describe here
requirements and design for managing metadata within the OMEGA code
itself to enable appropriate metadata in all model output. The capabilities
described here must interact with other parts of the I/O within OMEGA,
particularly the lower-level I/O interfaces that
perform the actual reading/writing to disk and the management of
files or streams and their contents.

Metadata can be in all supported OMEGA data types (I4, I8, R4, R8,
Boolean/logical, strings). Initially only scalar metadata
is required, but extending to vectors/arrays may be desirable.

Metadata refers not only to field/variable metadata but
also metadata associated with the model, code base, simulation
or data file. This capability must be able to track metadata
for these global/general metadata in addition to metadata
attached to particular fields or variables.

Where possible, all metadata must conform to the climate/forecast (CF)
metadata conventions at https://cfconventions.org.

To comply with conventions and E3SM practices, there will be a minimum
set of required metadata and interfaces must enforce this minimum set.
This minimum will be defined later, but for variables, this would
typically include a name (short), units, long name/description,
standard CF name (if exists), `_FillValue`, valid min/max and dimensions.

For array variables, a definition of each dimension used is required.
In an unstructured model like OMEGA, the dimension describes the
global extent of the index space for each dimension that is then paired
with mesh fields for the full description of OMEGA coordinate locations.
For time dimensions, we require support for both a fixed length time
dimension (e.g. to support multiple time levels in a restart) as well
as an unlimited dimension to support time series.

To support other I/O components, we require a means for maintaining
a list of fields available for I/O based on the requested model
configuration. This enables the I/O streams to define contents of
a given file as part of model configuration input by supplying a
list of variable names to be included.

Because lists of contents in the model config file can get long,
it would be useful to define and maintain a metadata group that
is simply a shorthand to a set of common fields (e.g. `meshFields` or
`prognosticVars`). Note that these would only be for the purpose
of shortening lists in the config file or managing lists internally.
The full list of fields would still appear in output file metadata.

The general philosophy is that modules will define the metadata for
each field they own and add it to a list of available fields. This
internally generated list ensures that the list of available fields
remains consistent with the code base and specific model configuration.
Note that this requires most initialization to occur before processing
output streams so that the requested contents can be checked against
the list of defined fields.

Internally, the metadata class will make use of the C++
`std::map` that provides a useful capability for name, value
pairs of each data type.

To enable general metadata associated with the code,
simulation or file, we define standard names for these
to be used as the "field" name during metadata definition.
For code and simulation metadata, we will use:

   constexpr std::string codeMeta{"code"};
   constexpr std::string simMeta {"simulation"};

For file metadata (eg. time stamps, convention, etc.),
we will use the stream name as the field name for metadata
definition.

The metadata class holds the collection of metadata associated
with a variable or the general category (code, sim, file) that we call a field.
This class implemented as a map (name, value pairs) using `std::any`
to accommodate all supported types. It also stores all the
defined metadata in a static map so the metadata can be
retrieved from anywhere.

   class MetaData {
     protected:
       /// metadata stored in map
       std::map<std::string, std::any> MetaMap;

       /// Store and maintain all defined metadata in this map
       static std::map<std::string, std::shared_ptr<MetaData>> AllFields;

     public:
      // [Methods described below]
   };

While the Metadata class provides a generic solution for storing/retrieving
metadata, we also provide specific solutions for important variable types.
ArrayMetaData is a class inherited from the Metadata class that helps users
input the required metadata for array-type variables.

    class ArrayMetaData : public MetaData {

        public:

        static std::shared_ptr<MetaData>
        create(const std::string Name,        /// Name of var
               const std::string Description, /// long Name or description
               const std::string Units,       /// units
               const std::string StdName,     /// CF standard Name
               const std::any ValidMin,       /// min valid field value
               const std::any ValidMax,       /// max valid field value
               const std::any FillValue,      /// scalar used for undefined entries
               const int NDims,               /// number of dimensions
               const std::vector<std::shared_ptr<MetaDim>> Dimensions // dim pointers
        );
    };

Another class is needed to describe dimensions for arrays
and vectors. Each is a name, length pair. Like the metadata
above, all defined dimensions are stored and maintained as a
static vector.

   class MetaDim{

      private:
         /// The length of the dimension. Use 0 for unlimited length.
         I4 Length;

         /// Store and maintain all defined dimensions
         static std::map<std::string, std::shared_ptr<MetaDim>> AllDims;

      public:
         [methods described below]
   };

Finally, to support common groups of metadata, we define a simple
class to define a list of variable names that are members of a
group. Like the classes above, we store and manage the group
definitions in a static map of defined groups.

   class MetaGroup{
      private:
         /// map of fields in group
         std::map<std::string, std::shared_ptr<MetaData>> Fields;

         /// Store and maintain all defined groups
         static std::map<std::string, std::shared_ptr<MetaGroup>> AllGroups;

      public:
   };

There will be two methods for creating generic metadata container and
one method for creating array-type metadata container. Because we
are managing all instances of metadata container within the class, these
are static functions rather than constructors. Also note in
the comments below that empty or zero values can be provided
in cases where input arguments don't make sense.

      ...
      public:

        /// Create an empty metadata container instance by name, primarily
        /// for global metadata not assigned to a variable/field. Returns a
        /// shared_ptr object. An exception is thrown if metadata of the same
        /// name already exists.
        create(
            const std::string Name /// [in] Name to assign
        );

        /// Create an instance by name and pairs of key-value.
        /// Returns a shared_ptr object. An exception is thrown if metadata
        /// of the same name already exists.
        create(
            const std::string Name,
            const std::initializer_list<std::pair<std::string, std::any>>& MetaPairs);
   }

   class ArrayMetaData : public MetaData {
   ...
   public
    /// Create an metadata container instance for a variable using all
    /// required metadata. Note that if input parameters don't exist
    /// (eg stdName) or don't make sense (eg min/max or fill) for a
    /// given field, empty or 0 entries can be provided. Similarly
    /// for scalars, nDims can be 0 and an emtpy MetaDim vector can be
    /// supplied.
    static std::shared_ptr<MetaData>
    create
         (const std::string name,        /// name of var
          const std::string description, /// long name or description
          const std::string units,       /// units
          const std::string stdName,     /// CF standard name
          const std::any    validMin,    /// min valid field value
          const std::any    validMax,    /// max valid field value
          const std::any    fillValue,   /// scalar used for undefined entries
          const int         nDims,       /// number of dimensions
          const std::vector<std::shared_ptr<MetaDims>> // dim pointers
          );
   }

A destructor will be provided to free metadata container and remove from the
defined list, though we anticipate most metadata will need to be
persistent through a given simulation.

In additiion, there will be a method to retrieve a metadata container, and
another method to check if a metadata container exists by name.

An interface for adding new metadata to a previously defined metadata
container instance will be provided. This can be used to add global metadata to
the code, simulation, file metadata, but also for variables if additional
metadata is desired.

    int addEntry(const std::string MetaName,   /// Name of new metadata to add
           const std::any Value          /// Value of new metadata to add
    );

where MetaName and Value are the usual name/value pair for the metadata
to be added. The value is a `std::any` type to support all types within
within the metadata map. This function will return 0 if successful and an error
if a metadata pair with that name already exists.

For symmetry, we will supply a remove function, though the
use case is likely rare.

    int removeEntry(const std::string MetaName /// Name of metadata to remove
    );

The most common use case will be retrieving metadata. For a single
metadata entry, there will be an explicit get function:

    int getEntry(const std::string MetaName,   /// Name of metadata to get
           I4 &Value                     /// I4 Value of metadata
    );

where MetaName is the name associated with the metadata
to be retrieved. Value argument is passed by reference so that
metadata value can be returned using the argument. This function is
overloaded with Omega data types.

During I/O stages, we will need a capability for retrieving
all metadata for a defined field or global category. We will
supply a function to retrieve the map associated with a given
data type. The `std::map` iterators and functionality can then
be used to extract all of the metadata for a given type. For
example, to retrieve all metadata from a defined field:

   auto MyVar = MetaData::create("MyVar");
   MyVar->addEntry("Name", 1);
   std::map<std::string, std::any> *VarMeta = MyVar->getAllEntries();
   std::string MetaName;
   std::any    MetaVal;

   // use std::map functionality to loop through all metadata
   for (const auto &Pair : *VarMeta) {
      MetaName = Pair.first;  // retrieve name part of meta pair
      MetaVal  = Pair.second; // retrieve value part of meta pair

      // do whatever needed with metadata
      // metaVal can be cast into the appropriate type using
      //   std::any_cast<type>(metaVal)
   }

For the dimension class, there will be a create function that will
both construct the instance and add it to the list of defined
dimensions. It will return a `shared_ptr` to the instance. An exception
will be thrown if the dimension is already defined.

    static std::shared_ptr<MetaDim>
    create(const std::string Name,     /// name of dimension
              const I4 Length         /// length of dimension
    );

   int MetaDim::destroy(std::string name // name of dimension
                       );

We use create/destroy functions rather than constructors so
that the class can store and manage all instances.

The only other function for a dimension will be a retrieval for
the dimension length:

4.2.6.1 Existence inquiry for metadata container and metadim container

For both metadata and dimensions, a function will be provided
to inquire whether the containers of metadata or dimensions
with a given name have already been defined:

   bool Metadata::has(const std::string Name);
   bool MetaDim::has (const std::string Name);

4.2.6.1 existence inquiry for metadata container and metadim container

For metadata, a function will be provided to inquire whether metadata

Common groups of fields can be defined as metadata groups (MetaGroup).
We supply create/destroy functions for defining a group of a given
name. Group names should start with GRP so that when groups are
included as contents in input config files, we can distinguish
a group from a field. Create will create an empty group to which
field names can be added. The functions will return a `shared_ptr` to
the MetaGroup instance.
Note that in the case of groups, if a group of the same name
already exists, no error is returned - the create does nothing.
This is to support cases where multiple modules may be adding their
fields to a group during initialization and these may be called
in an arbitrary order.

   int MetaGroup::create(std::string name /// name of group
                         );
   int MetaGroup::destroy(std::string name /// name of group
                         );

The members of a group can then be added individually. A remove
function is provided though use cases are rare. As in create,
if a field has already been added, the add function will do nothing.

   int MetaGroup::addField(std::string name /// name of field to add
                      );
   int MetaGroup::removeField(std::string name /// name of field to remove
                         );

Finally, we will need to be able to extract the names of all
fields that have been added to a group:

   auto MyGroup = MetaGroup::get(< group name >);
   std::map<std::string, std::shared_ptr<MetaData>> * FieldMap =
        MyGroup->getAllFields();

Create dimensions for a sample multi-dimensional field and then
create field metadata with metadata entries for all supported types.
Test by retrieving all meta data and dimension information and
ensuring it is identical.

Add metadata to supported global metadata fields "code" and
"simulation" and test by retrieving the same and comparing.

Test available field and available dimensions by inquiring if
above metadata and dimensions have been defined. Also test names
that don't exist to test failure modes.

Define one or two groups of metadata with both defined fields
and non-existent fields (to check error modes). Retrieve the
list of fields in each group and check against the expected
names.

To test CF compliance, we will test output files using a CF checking
code. This may be a part of Polaris testing rather than the unit test
framework.

This design document describes the first version of the Omega ocean model, Omega-0. Overall, Omega is an unstructured-mesh ocean model based on TRiSK numerical methods ([Thuburn et al. 2009](https://www.sciencedirect.com/science/article/pii/S0021999109004434)) that is specifically designed for modern exascale computing architectures. The algorithms in Omega will be nearly identical to those in MPAS-Ocean, but it will be written in c++ rather than Fortran in order to take advantage of libraries to run on GPUs, such as Kokkos ([Trott et al. 2022](https://ieeexplore.ieee.org/document/9485033)).

This design document describes the first version of the Omega ocean model, Omega-0. Overall, Omega is an unstructured-mesh ocean model based on TRiSK numerical methods (

) that is specifically designed for modern exascale computing architectures. The algorithms in Omega will be nearly identical to those in MPAS-Ocean, but it will be written in c++ rather than Fortran in order to take advantage of libraries to run on GPUs, such as Kokkos (

**Omega-0: Shallow water equations with identical vertical layers and inactive tracers.** There is no vertical transport or advection. The tracer equation is horizontal advection-diffusion, but tracers do not feed back to dynamics. Pressure gradient is simply gradient of sea surface height. Capability is similar to [Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780)

Omega-0: Shallow water equations with identical vertical layers and inactive tracers.

 There is no vertical transport or advection. The tracer equation is horizontal advection-diffusion, but tracers do not feed back to dynamics. Pressure gradient is simply gradient of sea surface height. Capability is similar to 

**Omega-1: Minimal stand-alone primitive equation eddy-permitting model.** This adds active temperature, salinity, density, and pressure as a function of depth. There is a true pressure gradient. Vertical velocity is from the continuity equation. An equation of state and simple vertical mixing scheme are needed. Capability is same as [Ringler et al. 2013](https://www.sciencedirect.com/science/article/pii/S1463500313000760)

Omega-1: Minimal stand-alone primitive equation eddy-permitting model.

 This adds active temperature, salinity, density, and pressure as a function of depth. There is a true pressure gradient. Vertical velocity is from the continuity equation. An equation of state and simple vertical mixing scheme are needed. Capability is same as 

**Omega-2: Eddying ocean with advanced parameterizations.** This model will have sufficient capability to run realistic global simulations, similar to E3SM V1 [Petersen et al. 2019](https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2018MS001373).

 This model will have sufficient capability to run realistic global simulations, similar to E3SM V1 

We will produce separate design documents for the time-stepping scheme and the tracer equations in Omega-0. Some of the requirements stated here are repeated or implied by the Omega framework design documents, but are included for clarity.

2.1  Omega-0 will solve the nonlinear shallow water equations, plus inactive tracers

The governing equations for Omega-0 are the shallow water equations, as described in [Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780) eqns 2-7, plus the tracer advection-diffusion equation. These equations are derived from conservation of momentum, volume, and tracers in a single layer. The code will solve a multi-layer formulation with independent, redundant layers in order to test performance with a vertical array dimension. The exact formulation is specified section 3.1 below, with variable definitions in section 3.2.

The governing equations for Omega-0 are the shallow water equations, as described in 

 eqns 2-7, plus the tracer advection-diffusion equation. These equations are derived from conservation of momentum, volume, and tracers in a single layer. The code will solve a multi-layer formulation with independent, redundant layers in order to test performance with a vertical array dimension. The exact formulation is specified section 3.1 below, with variable definitions in section 3.2.

Boundary conditions will include both no-slip and free-slip. The original MPAS-Ocean implementation had no-slip boundaries, implemented by setting vorticity to zero at the boundaries. The free-slip implementation is described by Darren Engwirda in a [github discussion](https://github.com/E3SM-Ocean-Discussion/E3SM/pull/49).

Boundary conditions will include both no-slip and free-slip. The original MPAS-Ocean implementation had no-slip boundaries, implemented by setting vorticity to zero at the boundaries. The free-slip implementation is described by Darren Engwirda in a 

2.1 omega-0 will solve the nonlinear shallow water equations, plus inactive tracers

The horizontal discretization will be taken from
[Thuburn et al. 2009](https://www.sciencedirect.com/science/article/pii/S0021999109004434) and [Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780), as described in the algorithmic formulation in Section 3 below. This is the same base formulation as in MPAS-Ocean. In addition, we will consider small alterations from the original MPAS-Ocean horizontal discretization and include them as options if they are of minimal additional effort and improve the simulated climate. This includes the recent AUST formulation in [Calandrini et al. 2021](https://www.sciencedirect.com/science/article/pii/S146350032100161X) and simple vorticity averaging considered by the Omega team this past year.

, as described in the algorithmic formulation in Section 3 below. This is the same base formulation as in MPAS-Ocean. In addition, we will consider small alterations from the original MPAS-Ocean horizontal discretization and include them as options if they are of minimal additional effort and improve the simulated climate. This includes the recent AUST formulation in 

 and simple vorticity averaging considered by the Omega team this past year.

We will continue to use the MPAS-format netcdf files for input and output, with the same mesh variable names and dimensions. This will facilitate ease of use and interoperability between MPAS-Ocean and Omega.

Omega mesh information will be stored in a separate mesh file, and not be included with initial condition, restart, or output files. This way there is never redundant mesh data stored in these files.

2.4  Omega-0 will interface with polaris for preprocessing and postprocessing

A substantial investment has already been made in the polaris tools. Continuing the use of polaris for Omega will speed development and encourage the documentation and long-term reproducibility of test cases.

The test cases relevant to this design document are in Section 5 below.

2.4 omega-0 will interface with polaris for preprocessing and postprocessing

2.5  Omega-0 will run portably on various DOE architectures (CPU and GPU nodes)

Omega will be able to run on all the upcoming DOE architectures and make good use of GPU hardware. This should occur with minimal alterations in the high-level PDE solver code for different platforms.

Options include: writing kernels directly for GPUs in CUDA; adding OpenACC pragmas for the GPUs; or calling libraries such as Kokkos ([Trott et al. 2022](https://ieeexplore.ieee.org/document/9485033)), YAKL ([Norman et al. 2023](https://link.springer.com/10.1007/s10766-022-00739-0)) or [HIP](https://github.com/ROCm-Developer-Tools/HIP) that execute code optimized for specialized architectures on the back-end, while providing a simpler front-end interface for the domain scientist.

Options include: writing kernels directly for GPUs in CUDA; adding OpenACC pragmas for the GPUs; or calling libraries such as Kokkos (

 that execute code optimized for specialized architectures on the back-end, while providing a simpler front-end interface for the domain scientist.

2.5 omega-0 will run portably on various doe architectures (cpu and gpu nodes)

This is with MPI and halo communication, as described in framework design documents. Results must be bit-for-bit identical across different number of partitions. This may be demonstrated with the ''QU240 partition test'' in Polaris.

This is with MPI and halo communication, as described in framework design documents. Results must be bit-for-bit identical across different number of partitions. This may be demonstrated with the ‘’QU240 partition test’’ in Polaris.

Each operator will be tested individually for the proper convergence rate: divergence, gradient, curl interpolated to cell centers, and tangential velocity are all second order; curl at vertices is first order. The details of the convergence tests are explained in section 5.1 below.

The total volume of the domain should be conserved to machine precision in the absence of surface volume fluxes. Likewise, the total tracer amount should be conserved to machine precision without surface fluxes. A simple test is to initialize an inactive test tracer with a uniform value of 1.0, and it should remain 1.0 throughout the domain.

2.9  Performance will be at least as good or better than MPAS-Ocean

The first two items will be compared to MPAS-Ocean with the same test configuration. For Omega-0, these may be tested first with the layered shallow water equations (momentum and thickness only) and be compared directly to the results in [Bishnu et al. 2023](https://gmd.copernicus.org/articles/16/5539/2023), using the inertia-gravity wave test case described in Section 6.2 below. After tracer transport capabilities are added, performance may be compared against MPAS-Ocean with test cases using active tracers in addition to momentum and thickness.

The first two items will be compared to MPAS-Ocean with the same test configuration. For Omega-0, these may be tested first with the layered shallow water equations (momentum and thickness only) and be compared directly to the results in 

, using the inertia-gravity wave test case described in Section 6.2 below. After tracer transport capabilities are added, performance may be compared against MPAS-Ocean with test cases using active tracers in addition to momentum and thickness.

For these comparisons, MPAS-Ocean will be set up with the identical terms and functionality as Omega-0. This means that vertical advection, vertical mixing, and all parameterizations will be disabled. Comparisons with these terms will be made with Omega-1 and higher and will be described in future design documents.

2.9 performance will be at least as good or better than mpas-ocean

2.10  Full-node GPU throughput will be comparable or better than full-node CPU throughput

For GPU throughput, comparisons should be made between full-node CPU throughput and full-node GPU throughput. For example, [Perlmutter at NERSC](https://docs.nersc.gov/systems/perlmutter/architecture/) has nodes with 64 and 128 CPU cores (AMD EPYC 7763) and 4 GPUs (NVIDIA A100). We expect the full-node GPU throughput to be at least as good as the full-node CPU throughput, and potentially a factor of four higher. These numbers depend on the performance specifications of the particular hardware.

For GPU throughput, comparisons should be made between full-node CPU throughput and full-node GPU throughput. For example, 

 has nodes with 64 and 128 CPU cores (AMD EPYC 7763) and 4 GPUs (NVIDIA A100). We expect the full-node GPU throughput to be at least as good as the full-node CPU throughput, and potentially a factor of four higher. These numbers depend on the performance specifications of the particular hardware.

Like the previous requirement, tests will first be conducted with layered shallow water equations and later with additional tracer advection. For reference, [Bishnu et al. 2023](https://gmd.copernicus.org/articles/16/5539/2023) was able to obtain nearly identical throughput between 64 CPU cores and a single GPU on Perlmutter using a Julia code and a layered shallow water test case.

Like the previous requirement, tests will first be conducted with layered shallow water equations and later with additional tracer advection. For reference, 

 was able to obtain nearly identical throughput between 64 CPU cores and a single GPU on Perlmutter using a Julia code and a layered shallow water test case.

2.10 full-node gpu throughput will be comparable or better than full-node cpu throughput

The algorithms for Omega-0 are given here in full detail, with all variables defined in section 3.2 below.
We begin with the standard shallow water equations, which are derived from conservation of momentum and mass in a single layer with uniform density in a rotating frame. The standard presentation may be found in standard textbooks on Geophysical Fluid Dynamics, such as those by Vallis (2017), Cushman‐Roisin and Beckers (2011), Pedlosky (1987), and Gill (2016). See references at bottom. In continuous form, these are

\frac{\partial \boldsymbol{u}}{\partial t} + \left(\boldsymbol{u}\cdot\nabla\right)\boldsymbol{u}  + f \boldsymbol{k}\times\boldsymbol{u}
= -g\nabla(h-b)
\hspace{1cm}

\frac{\partial h}{\partial t} + \nabla \cdot \left(h \boldsymbol{u}\right) = 0,

Next we replace the non-linear advection term with the right-hand side of the vector identity
$$
\boldsymbol{u} \cdot \nabla \boldsymbol{u} &= (\nabla \times \boldsymbol{u}) \times \boldsymbol{u} + \nabla \frac{|\boldsymbol{u}|^2}{2} \\
&= \{\boldsymbol{k} \cdot (\nabla \times \boldsymbol{u})\} \boldsymbol{k} \times \boldsymbol{u} + \nabla \frac{|\boldsymbol{u}|^2}{2} \\
&= \zeta \boldsymbol{u}^{\perp} + \nabla K.
$$

The governing equations for Omega-0 in continuous form are then
$$
\frac{\partial \boldsymbol{u}}{\partial t} + q\left(h\boldsymbol{u}^{\perp}\right) = -g\nabla(h-b) - \nabla K + \nu_2 \nabla^2 \boldsymbol{u} - \nu_4 \nabla^4 \boldsymbol{u} + \mathcal{D} + \mathcal{F}
\hspace{1cm}   (1)
$$

\hspace{1cm}   (2)

\frac{\partial h \phi}{\partial t} + \nabla \cdot \left(h \boldsymbol{u} \phi\right) = \kappa_2 h \nabla^2 \phi - \kappa_4 h \nabla^4 \phi.
\hspace{1cm}   (3)

These are typically referred to as the momentum equation (or velocity equation), the thickness equation, and the tracer equation.

The additional momentum terms in (1) are viscous dissipation (del2 and del4), and drag and forcing are as follows:
$$
\mathcal{D} = -Ra \, \boldsymbol{u}-C_D \frac{\boldsymbol{u}\left|\boldsymbol{u}\right|}{h}
$$

\mathcal{F} = - C_W \frac{(\boldsymbol{u}_W - \boldsymbol{u})\left|\boldsymbol{u}_W - \boldsymbol{u}\right|}{h}

The drag consists of simple Rayleigh drag for spin-up as well as quadratic bottom drag. The forcing is a quadratic wind forcing (see equation 1 in [Lilly et al. 2023](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2022MS003327)).

The drag consists of simple Rayleigh drag for spin-up as well as quadratic bottom drag. The forcing is a quadratic wind forcing (see equation 1 in 

Here $q$ is the potential vorticity, so that the term
$$
q\left(h\boldsymbol{u}^{\perp}\right) = \frac{\zeta + f}{h}\left(h\boldsymbol{u}^{\perp}\right)
 = \zeta \boldsymbol{u}^{\perp} + f \boldsymbol{u}^{\perp}
$$
is composed of the rotational part of the advection $\zeta \boldsymbol{u}^{\perp}$ and the Coriolis term $f \boldsymbol{u}^{\perp}$. This term is discussed in [Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780) sections 2.1 and 2.2.

The thickness equation (2) is derived from conservation of mass for a fluid with constant density, which reduces to conservation of volume. The model domain uses fixed horizontal cells with horizontal areas that are constant in time, so the area drops out and only the layer thickness $h$ remains as the prognostic variable.

The thickness equation (2) is derived from conservation of mass for a fluid with constant density, which reduces to conservation of volume. The model domain uses fixed horizontal cells with horizontal areas that are constant in time, so the area drops out and only the layer thickness 

The Tracer Equation (3) is the conservation equation for a passive tracer (scalar), with only advective and diffusive terms. It is not included in the textbook Shallow Water equations, but is useful for us to test tracer advection in preparation for a primitive equation model in Omega-1. For a tracer which is uniformly one ($\phi=1$), with no viscous terms, the tracer equation reduces to the thickness equation.
The tracer equation is thickness weighted, because the conserved quantity is the tracer mass. Here $(h\phi A)$ typically has units of mass of the tracer in [kg] while $\phi$ has units of concentration [kg m$^{-3}$]. Because the horizontal area is fixed, $A$ has been divided out making (3) thickness weighted, rather than volume weighted. For chemical tracers $\phi$ has units of [mmol m$^{-3}$]; salinity has units of Practical Salinity Units (PSU) and potential temperature has units of [C]. A derivation of the thickness-weighted tracer equation appears in Appendix A-2 of [Ringler et al. 2013](https://www.sciencedirect.com/science/article/pii/S1463500313000760).

The Tracer Equation (3) is the conservation equation for a passive tracer (scalar), with only advective and diffusive terms. It is not included in the textbook Shallow Water equations, but is useful for us to test tracer advection in preparation for a primitive equation model in Omega-1. For a tracer which is uniformly one (

), with no viscous terms, the tracer equation reduces to the thickness equation.

The tracer equation is thickness weighted, because the conserved quantity is the tracer mass. Here 

 has been divided out making (3) thickness weighted, rather than volume weighted. For chemical tracers 

]; salinity has units of Practical Salinity Units (PSU) and potential temperature has units of [C]. A derivation of the thickness-weighted tracer equation appears in Appendix A-2 of 

The Omega-0 governing equations (1-3) do not include any vertical advection or diffusion. Omega-0 will have a vertical index for performance testing and future expansion, but vertical layers will simply be redundant.

Further details of these derivations are given in [Thuburn et al. 2009](https://www.sciencedirect.com/science/article/pii/S0021999109004434) and [Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780) eqns. (2) and (7), and [Bishnu et al. 2024](https://doi.org/10.1029/2022MS003545), Section 2.1.
Additional information on governing equations may be found in chapter 8 of the MPAS User's Guide ([Petersen et al. 2024](https://zenodo.org/records/11098080)).
Publications that evaluate TRiSK against alternative formulations include [Weller et al. 2012](http://journals.ametsoc.org/doi/10.1175/MWR-D-11-00193.1),
[Thuburn and Cotter 2012](https://epubs.siam.org/doi/10.1137/110850293),
[Calandrini et al. 2021](https://www.sciencedirect.com/science/article/pii/S146350032100161X) and
[Lapolli et al. 2024](https://www.sciencedirect.com/science/article/pii/S1463500324000222).

Additional information on governing equations may be found in chapter 8 of the MPAS User’s Guide (

Publications that evaluate TRiSK against alternative formulations include 

\frac{\partial u_e}{\partial t} + \left[ \frac{{\boldsymbol k} \cdot \nabla \times u_e +f_v}{[h_i]_v}\right]_e\left([h_i]_e u_e^{\perp}\right)
= -g\nabla(h_i-b_i) - \nabla K_i + \nu_2 \nabla^2 u_e - \nu_4 \nabla^4 u_e + \mathcal{D}_e + \mathcal{F}_e
\hspace{1cm}   (4)

\frac{\partial h_i}{\partial t} + \nabla \cdot \left([h_i]_e u_e\right) = 0,
\hspace{1cm}   (5)

\frac{\partial h_i \phi_i}{\partial t} + \nabla \cdot \left(u_e [h_i \phi_i]_e \right) = \kappa_2 h_i \nabla^2 \phi_i - \kappa_4 h_i \nabla^4 \phi_i,
\hspace{1cm}   (6)

where subscripts $i$, $e$, and $v$ indicate cell, edge, and vertex locations ($i$ was chosen for cell because $c$ and $e$ look similar).  Here square brackets $[\cdot]_e$ and $[\cdot]_v$ are quantities that are interpolated to edge and vertex locations. The interpolation is typically centered, but may vary by method, particularly for advection schemes. For vector quantities, $u_e$ denotes the normal component at the center of the edge, while $u_e^\perp$ denotes the tangential component. In the discrete system, the normal component $u_e$ points positively from the lower cell index to the higher cell index, while the tangential component $u_e^\perp$ points positively $90^o$ to the left of $u_e$ (for unit vectors, ${\bf n}_e^\perp = {\bf k}\times {\bf n}_e$).

 are quantities that are interpolated to edge and vertex locations. The interpolation is typically centered, but may vary by method, particularly for advection schemes. For vector quantities, 

 denotes the tangential component. In the discrete system, the normal component 

 points positively from the lower cell index to the higher cell index, while the tangential component 

The discretized momentum drag and forcing terms are
$$
\mathcal{D}_e = - Ra \, u_e - C_D \frac{u_e\left|u_e\right|}{[h_i]_e}
$$

\mathcal{F}_e =  C_W \frac{(u_W - u_e)\left|u_W - u_e\right|}{[h_i]_e}

weights defined in [Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780) eqn 24

normalized weights, ${\tilde w}_{e,e'} = \tfrac{l_{e'}}{d_e} w_{e,e'}$

$\zeta={\boldsymbol k} \cdot \left( \nabla \times {\boldsymbol u}\right)$

\zeta={\boldsymbol k} \cdot \left( \nabla \times {\boldsymbol u}\right)

<!--- Note: Table created with [markdown table generator](https://www.tablesgenerator.com/markdown_tables) and original [google sheet](https://docs.google.com/spreadsheets/d/1rz-QXDiwfemq5NpSR1XsvomI7aSKQ1myTNweCY4afcE/edit#gid=0). --->

Table 2. Definition of geometric elements used to build the discrete system.

![Variable positions for MPAS mesh specification.](../_static/variablePosition.png "Variable positions")

![Areas for MPAS mesh specification.](../_static/areaPositionsMPAS.png "area specifications")

Table 3. Definition of element groups used to build the discrete system.

Set of primal mesh cells that form the vertices of dual mesh cell $D_v$

Set of primal mesh cells that form the vertices of dual mesh cell 

The set of dual-mesh cells that form the vertices of primal mesh cell $P_i$

The set of dual-mesh cells that form the vertices of primal mesh cell 

The definitions of geometric variables may be found in
[Thuburn et al. 2009](http://www.sciencedirect.com/science/article/pii/S0021999109004434),
[Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780), and
[Thuburn and Cotter 2012](https://epubs.siam.org/doi/10.1137/110850293).

The TRiSK formulation of the discrete operators are as follows. See [Bishnu et al. 2023](https://gmd.copernicus.org/articles/16/5539/2023) section 4.1 and Figure 1 for a description and documentation of convergence rates, as well as [Bishnu et al. 2021](https://doi.org/10.5281/zenodo.7439539). All TRiSK spatial operators show second-order convergence on a uniform hexagon grid, except for the curl on vertices, which is first order. The curl interpolated from vertices to cell centers regains second order convergence. The rates of convergence are typically less than second order on nonuniform meshes, including spherical meshes.

The TRiSK formulation of the discrete operators are as follows. See 

 section 4.1 and Figure 1 for a description and documentation of convergence rates, as well as 

. All TRiSK spatial operators show second-order convergence on a uniform hexagon grid, except for the curl on vertices, which is first order. The curl interpolated from vertices to cell centers regains second order convergence. The rates of convergence are typically less than second order on nonuniform meshes, including spherical meshes.

The divergence operator maps a vector field's edge normal component $F_e$ to a cell center ([Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780) eqn 21),

The divergence operator maps a vector field’s edge normal component 

\left( \nabla \cdot {\bf F}\right)_i
\equiv \nabla \cdot F_e
=  \frac{1}{A_i} \sum_{e\in EC(i)} n_{e,i} \, F_e \, l_e,

where $A_i$ is the area of cell $i$. The indicator function $n_{e,i}=1$ when the normal vector ${\bf n}_e$ is an *outward* normal of cell $i$ and $n_{e,i}=-1$ when ${\bf n}_e$ is an *inward* normal of cell $i$.
The notation $e\in EC(i)$ indicates all the edges surrounding cell $i$.
In the TRiSK formulation we assume that the divergence always occurs at the cell center, so the subscript $i$ is dropped and we simply write  $\left( \nabla \cdot {\bf F}\right)_i$ as $\nabla \cdot F_e$.

In the TRiSK formulation we assume that the divergence always occurs at the cell center, so the subscript 

The divergence operator above is applied to a general vector field $\bf F$.
In the actual formulations below, we substitute the velocity at the edge $F_e = u_e$ for the divergence variable in the momentum equation, and the thickness-weighted tracer
$F_e = u_e [h_i \phi_i]_e$
 in the tracer advection term. To be clear, when we refer to the divergence variable, rather than the operator, this means
$$
\left( \nabla \cdot {\bf u}\right)_i =
D_i
=  \frac{1}{A_i} \sum_{e\in EC(i)} n_{e,i} \, u_e \, l_e.
$$

The divergence operator above is applied to a general vector field 

In the actual formulations below, we substitute the velocity at the edge 

 for the divergence variable in the momentum equation, and the thickness-weighted tracer

in the tracer advection term. To be clear, when we refer to the divergence variable, rather than the operator, this means

D_i

The gradient operator maps a cell-centered scalar to an edge-normal vector component
([Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780) eqn 22),

\left( \nabla h \right)_e
\equiv
\nabla h_i = \frac{1}{d_e} \sum_{i\in CE(e)} -n_{e,i} \, h_i,

where $i\in CE(e)$ identifies the two cells that neighbor an edge. The edge subscript is dropped, so $\nabla h_i$ is understood to occur at an edge.

 identifies the two cells that neighbor an edge. The edge subscript is dropped, so 

For practical purposes the sign indicator variable $n_{e,i}$ is not used in the code, and we opt for the simpler formulation
$$
\equiv
\nabla h_i = \frac{h_{i2} - h_{i1} }{d_e},
$$
where $\left\{i_1, i_2\right\}$ are the cells neighboring edge $e$.  The indices $\left\{i_1, i_2\right\}$ are ordered such that the normal vector ${\bf n}$ points from cell $i_1$ to cell $i_2$. In the code ${\bf n}$ points from `CellsOnEdge(IEdge, 0)` to `CellsOnEdge(IEdge, 1)`.

 is not used in the code, and we opt for the simpler formulation

\equiv

The curl operator maps a vector's edge normal component $u_e$ to a scalar field at vertex
([Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780) eqn 23),

\left( {\bf k} \cdot \left( \nabla \times {\bf F} \right)\right)_v
\equiv
{\bf k} \cdot \left( \nabla \times F_e \right)
= \frac{1}{{\hat A}_v} \sum_{e\in EV(v)} t_{e,v} \, F_e \, d_e,

where ${\hat A}_v$ is the area of the dual mesh cell $v$, i.e. the triangle surrounding vertex $v$. Similar to $n_{e,i}$, the indicator function $t_{e,v}$ tracks whether a positive $F_e$ makes a positive or negative contribution to the curl function at vertex $v$. If the vector ${\bf k} \times {\bf n}_e$ is directed toward
${\bf x}_v$, then $t_{e,v}=1$. The summation is over $e\in EV(v)$, which are the edges that terminate at vertex $v$. There are *always* three edges that terminate at each vertex for Voronoi Tessellations, and four edges on each vertex for quadrilateral meshes, unless neighboring cells are missing for land boundaries. Again, the subscript $v$ is dropped and a vertex is assumed for the location of the curl.

 makes a positive or negative contribution to the curl function at vertex 

 three edges that terminate at each vertex for Voronoi Tessellations, and four edges on each vertex for quadrilateral meshes, unless neighboring cells are missing for land boundaries. Again, the subscript 

 is dropped and a vertex is assumed for the location of the curl.

The perpendicular component a vector field is defined as
$$
 {\bf F}^\perp \equiv
 {\bf k} \times {\bf F}
$$
so that it points 90$^o$ to the left of ${\bf F}$ in a right-handed coordinate system.

In TRiSK, the native component for the prognostic velocity field is the edge-normal velocity $u_e$. The perpendicular component is also positioned at the middle of the edge, but points 90$^o$ to the left, tangential to the edge, from one vertex to another. The perpendicular velocity component is also called the tangential velocity, with the variable name `TangentialVelocity`. It is computed diagnostically as
([Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780) eqn 24)

In TRiSK, the native component for the prognostic velocity field is the edge-normal velocity 

. The perpendicular component is also positioned at the middle of the edge, but points 90

 to the left, tangential to the edge, from one vertex to another. The perpendicular velocity component is also called the tangential velocity, with the variable name 

u_e^\perp = \frac{1}{d_e} \sum_{e'\in ECP(e)} w_{e,e'} \, l_{e'} \, u_{e'}.

The perpendicular vector is computed from the edge-normal components of all the edges of the two neighboring cells of edge $e$, denoted as $e'\in ECP(e)$. For example, on a mesh with hexagons, there are a total of 10 edges $e'$ on the two cells neighboring edge $e$ (the original edge $e$ is not included).  The tangential velocity $u_e^\perp$ is used in the Coriolis force, which appears in the potential vorticity advection term.  The weighting coefficients $w_{e,e'}$ are carefully chosen to conserve potential vorticity, as described in
[Thuburn et al. 2009](https://www.sciencedirect.com/science/article/pii/S0021999109004434)
and
[Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780).

The perpendicular vector is computed from the edge-normal components of all the edges of the two neighboring cells of edge 

. For example, on a mesh with hexagons, there are a total of 10 edges 

 is used in the Coriolis force, which appears in the potential vorticity advection term.  The weighting coefficients 

 are carefully chosen to conserve potential vorticity, as described in

In the code, the cell indices are ordered such that the edge-normal vector (like $u_e$) points positively from the lower to higher cell index value. Likewise, the vertex indices are ordered such that the edge-tangent vector (like $u_e^\perp$) points from the lower to higher vertex index value.

In the code, the cell indices are ordered such that the edge-normal vector (like 

) points positively from the lower to higher cell index value. Likewise, the vertex indices are ordered such that the edge-tangent vector (like 

In order to reduce redundant operations, the weights in the standard MPAS mesh files are pre-multiplied by the edge length ratio $l_{e'}/d_e$. Thus the MPAS variable `WeightsOnEdge` is
$$
{\tilde w}_{e,e'} = \frac{l_{e'}}{d_e} w_{e,e'}
$$
and the revised calculation of the tangential velocity is
$$
u_e^\perp = \sum_{e'\in ECP(e)} {\tilde w}_{e,e'} \, u_{e'}.
$$
This simple operation can be seen in MPAS-Ocean in the subroutine `ocn_diagnostic_solve_vortVel` in the file `mpas_ocn_diagnostics.F`.

In order to reduce redundant operations, the weights in the standard MPAS mesh files are pre-multiplied by the edge length ratio 

This simple operation can be seen in MPAS-Ocean in the subroutine 

The gradient of a scalar at the middle of an edge, pointing tangentially along the edge (from one vertex to the other), is sometimes used. For example, the del2 formulation requires the perpendicular gradient of vorticity. This is called the perpendicular gradient because the standard gradient is normal to the edge.

The perpendicular gradient maps a scalar at vertices to an edge-tangential vector component,
$$
\left(\nabla \zeta_v\right)_e^\perp = \frac{1}{l_e} \sum_{v\in VE(e)} -t_{e,v}\zeta_v.
$$
Like the standard gradient, in practice we drop the sign indicator $t_{e,v}$ and rewrite it using the index ordering,
$$
\left(\nabla \zeta_v\right)_e^\perp = \frac{\zeta_{v2} - \zeta_{v1}}{l_e},
$$
where the positive vector ${\bf n}^\perp$ is 90$^o$ to the left of ${\bf n}$. The indices are ordered such that ${\bf n}^\perp$ points from $v_1$ to $v_2$, which corresponds to `VerticesOnEdge(IEdge, 0)` and `VerticesOnEdge(IEdge, 1)` in the code.

Like the standard gradient, in practice we drop the sign indicator 

The mid-point average of a scalar from cell centers to the adjoining edge is
$$
[h_i]_e = \frac{1}{2} \sum_{i\in CE(e)} h_i.
$$
In a Voronoi tessellation the edge is defined to be at the mid-point between the two cell centers, so this formula is identical to an area-weighted average using the triangles between edge $e$ and the neighboring cell centers.

In a Voronoi tessellation the edge is defined to be at the mid-point between the two cell centers, so this formula is identical to an area-weighted average using the triangles between edge 

The mid-point average of a scalar from vertices to the middle of the connecting edge is
$$
[\zeta_v]_e = \frac{1}{2} \sum_{v\in VE(e)} \zeta_v.
$$
The is a distance-weighted average, since the edge quantity lives at the mid-point between the vertices. One could alternatively compute an area-weighted average using the dual-mesh cell area ${\hat A}_v$ surrounding each vertex, but that is not done and would result in very small differences.

The is a distance-weighted average, since the edge quantity lives at the mid-point between the vertices. One could alternatively compute an area-weighted average using the dual-mesh cell area 

 surrounding each vertex, but that is not done and would result in very small differences.

The area-weighted average of a scalar at a vertex from the three surrounding cells is
$$
[h_i]_v = \frac{1}{{\hat A}_v} \sum_{i\in CV(v)} h_i {\tilde A}_{v,i}.
$$
Here ${\hat A}_v$ is the dual triangle area centered on the vertex, and ${\tilde A}_{v,i}$ is the area of the kite that extends from $x_i$ to the vertex $x_v$.
The three kites sum to the triangle area,
$$
{\hat A}_v = \sum_{i\in CV(v)} {\tilde A}_{v,i}.
$$

The area-weighted average of a scalar at a cell from the surrounding vertices is
$$
[h_v]_i = \frac{1}{A_i} \sum_{v\in VC(i)} h_v {\tilde A}_{v,i}
$$

The prognostic velocity variable on the edge is the edge normal velocity, $u_e$. The tangential velocity $u_e^\perp$ is computed diagnostically from $u_e$. In addition, the full vector may be computed at the cell center from the edge normal velocities $u_e$ of the edges on that cell. That is done in MPAS with radial basis functions, and is explained in this [previous design document](https://github.com/MPAS-Dev/MPAS-Documents/blob/master/shared/rbf_design/rbf.pdf).

The prognostic velocity variable on the edge is the edge normal velocity, 

. In addition, the full vector may be computed at the cell center from the edge normal velocities 

 of the edges on that cell. That is done in MPAS with radial basis functions, and is explained in this 

The computation of each term in (4-6) is now described in detail, along with alternative formulations.

The kinetic energy gradient term is the non-rotational part of the nonlinear advection. Fundamentally, it maps the prognostic edge-normal velocity from edges back to an edge scalar quantity. We use the standard gradient formulation from cell center to edge,

There are several methods to compute the cell-centered kinetic energy. The first uses the edge-normal velocities as in
[Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780) equation 63,

K_i = \frac{1}{A_i} \sum_{e\in EC(i)} \frac{{\bar A}_{e}}{4}u_e^2.

Here ${\bar A}_{e}$ is the rectangle about the edge defined by ${\bar A}_{e} = l_e d_e$ so that
$$
A_i = \sum_{e\in EC(i)} \frac{{\bar A}_{e}}{4}
$$
as stated in
[Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780) equations 48, 64, and footnote 4.

One might expect an additional factor of one-half in the $K_i$ formula, as in the standard formula for kinetic energy, $K = \tfrac{1}{2} m v^2$. However, the one-half is missing because $u_e$ is only one component of the velocity vector on the edge. One could replace $u_e^2$ with $\tfrac{1}{2}(u_e^2 + {u_e^{\perp}}^2)$, but $u^\perp_e$ is already computed from the surrounding values of $u_e$, so we don't do that because it would just add redundant information. As a side note, the mass $m$ is missing because the mass is already divided out of the momentum equation (1).

 is only one component of the velocity vector on the edge. One could replace 

, so we don’t do that because it would just add redundant information. As a side note, the mass 

 is missing because the mass is already divided out of the momentum equation (1).

The kinetic energy may also be computed at the vertex.  This uses the velocity on the three connected edges,
$$
K_v = \frac{1}{{\hat A}_{v}} \sum_{e\in EV(v)} \frac{{\bar A}_{e}}{4}u_e^2.
$$
Note that this normalization works because
$$
{\hat A}_{v} = \sum_{e\in EV(v)} \frac{{\bar A}_{e}}{4},
$$
since ${\bar A}_{e}/4$ is exactly the portion of the triangle contributed to ${\hat A}_{v}$ by each edge.

The computation of $K_v$ can be found in the MPAS-Ocean code with the `kineticEnergyVertex` variable. The final kinetic energy at the cell center may be chosen to be either $K_i$ in the first equation above, or a linear combination of $K_i$ and $K_v$ interpolated to cell centers. In MPAS-Ocean, the flag `config_include_KE_vertex` uses the value

 variable. The final kinetic energy at the cell center may be chosen to be either 

for the final kinetic energy at the cell center. Note that addition of $[K_v]_i$ enlarges the stencil. One could also use $u_e^{\perp}$ and compute the kinetic energy at the edge itself in order to enlarge the stencil, but that method is not used here.
See [Calandrini et al. 2021](https://www.sciencedirect.com/science/article/pii/S146350032100161X) section 2.3 for more information.

for the final kinetic energy at the cell center. Note that addition of 

 and compute the kinetic energy at the edge itself in order to enlarge the stencil, but that method is not used here.

The potential vorticity term, $q\left(h\boldsymbol{u}^{\perp}\right)$, includes the rotational part of the advection. It may be computed in two ways,

, includes the rotational part of the advection. It may be computed in two ways,

\text{Option 1:  } \left[ \frac{\zeta_v +f_v}{[h_i]_v}\right]_e\left([h_i]_e u_e^{\perp}\right)

The first computes the potential vorticity $q_v$ at the vertex and interpolates that quantity to the edge, which is what is done in MPAS-Ocean. One may also cancel the thickness $h$ (ignoring the interpolated locations) and use option 2. Additional interpolation options and results are presented in [Calandrini et al. 2021](https://www.sciencedirect.com/science/article/pii/S146350032100161X)

 at the vertex and interpolates that quantity to the edge, which is what is done in MPAS-Ocean. One may also cancel the thickness 

 (ignoring the interpolated locations) and use option 2. Additional interpolation options and results are presented in 

The sea surface height (SSH) gradient uses the standard gradient formulation from cell center to edge,

-g\nabla(h_i-b_i) &= -g\frac{1}{d_e} \sum_{i\in CE(e)} -n_{e,i}(h_i-b_i)\\
&= -g\frac{(h_{i2}-b_{i2}) - (h_{i1}-b_{i1}) }{d_e}.

The Del2, or Laplacian operator, viscous momentum dissipation maps edge-normal velocity back to the edge-normal component of the Laplacian. In TRiSK this is done with the vorticity-divergence formulation, which may be written as

\nabla^2 {\bf u} &=  \nabla \left( \nabla\cdot{\bf u}\right) - \nabla \times \left( \nabla \times {\bf u} \right) \\
 &=  \nabla {\bf D} - \nabla \times \zeta \\
 &=  \nabla {\bf D} - {\bf k} \times \nabla \zeta \\
 &=  \nabla {\bf D} - \nabla^\perp \zeta.

This formulation is also mentioned in [Gassmann 2011](https://linkinghub.elsevier.com/retrieve/pii/S0021999111000325) (equation 17), [Gassman 2018](https://onlinelibrary.wiley.com/doi/10.1002/qj.3294) (equation 44) and [Lapolli et al. 2024](https://www.sciencedirect.com/science/article/pii/S1463500324000222) (section 4.5.2).

For our discretization, the full Del2 term is written as
$$
\nu_2 \nabla^2 u_e &= \nu_2 \left( \nabla D_i - \nabla^\perp \zeta_v\right) \\
&= \nu_2 \left( \frac{1}{d_e} \sum_{i\in CE(e)} -n_{e,i}D_i - \frac{1}{l_e} \sum_{v\in VE(e)} -t_{e,v}\zeta_v\right) \\
&= \nu_2 \left( \frac{D_{i2} - D_{i1}}{d_e} - \frac{\zeta_{v2} - \zeta_{v1}}{l_e} \right),
$$
where the ordering of indices $\{i_i, i_2\}$ and $\{v_1, v_2\}$ are explained in the gradient operator sections above.

An alternative formulation for the Del2 dissipation on an unstructured mesh is presented in section 4.2 and Appendix B of
[Gassman 2018](https://onlinelibrary.wiley.com/doi/10.1002/qj.3294).

The Del4, or biharmonic, momentum dissipation also maps edge-normal velocity back to the edge-normal component of the Laplacian. This is done with two applications of the Del2 operator above.

- \nu_4 \nabla^4 u_e = - \nu_4 \nabla^2 \left( \nabla^2 u_e \right)

Rayleigh drag is simply linear drag, applied to all levels. It is typically only used during the spin-up process to damp large velocities during the initial adjustment. The Rayleigh coefficient $Ra$ is simply a scalar constant,

Rayleigh drag is simply linear drag, applied to all levels. It is typically only used during the spin-up process to damp large velocities during the initial adjustment. The Rayleigh coefficient 

Bottom drag is more relevant to layered models than to shallow water systems, but is included here for completeness. It is a quadratic drag applied to the edge velocity,

Wind forcing has the same form as the bottom drag, but the forcing is the difference between the current velocity and the wind $u_W$, interpolated and projected to the edge normal direction,

Wind forcing has the same form as the bottom drag, but the forcing is the difference between the current velocity and the wind 

There are many schemes available for tracer advection. Simple schemes include centered advection and upstream. MPAS-Ocean uses Flux Corrected Transport, which is fourth order under normal conditions, and reduces to third order to preserve monotonicity.

The tracer advection term is
$$
\nabla \cdot \left(u_e [h_i \phi_i]_e \right)
$$

and the stencil for divergence makes this
$$
\nabla \cdot \left( u_e [h_i \phi_i]_e \right)  =  \frac{1}{A_i} \sum_{e\in EC(i)} n_{e,i}u_e [h_i \phi_i]_e l_e.
$$
The question is how to interpolate to the edge to obtain $[h_i \phi_i]_e$. Centered advection simply uses the midpoint rule. Upwind takes the value from the upstream cell. Higher order schemes cast a wider stencil to compute the value at this edge. This includes Flux Corrected Transport, used in MPAS-Ocean, and described in [Skamarock and Gassmann 2011](https://journals.ametsoc.org/view/journals/mwre/139/9/mwr-d-10-05056.1.xml).

. Centered advection simply uses the midpoint rule. Upwind takes the value from the upstream cell. Higher order schemes cast a wider stencil to compute the value at this edge. This includes Flux Corrected Transport, used in MPAS-Ocean, and described in 

Note that the thickness advection is identical to the tracer advection when $\phi$ is uniformly one. Thus the same algorithm may be used for both. In fact, this is required for global conservation of tracers, so that the volume flux and tracer flux across each edge are consistent.

Note that the thickness advection is identical to the tracer advection when 

 is uniformly one. Thus the same algorithm may be used for both. In fact, this is required for global conservation of tracers, so that the volume flux and tracer flux across each edge are consistent.

More details of the tracer advection scheme will be given in a future design document.

Tracer diffusion is applied with a Laplacian operator on the cell-centered tracer phi, and the product of the operator is also at the cell center. The Laplacian may be written as the divergence of the gradient,

\kappa_2 h_i \nabla^2 \phi_i
=
\kappa_2 h_i \nabla \cdot \left( \nabla \phi_i \right).

and the stencils in Section 3.3 are used. Here $\kappa_2$ is the del2 diffusion coefficient, and the operator is thickness-weighted by $h_i$. The position of a gradient is assumed to be at an edge, and a divergence at the cell center. This could be written explicitly as

 is the del2 diffusion coefficient, and the operator is thickness-weighted by 

. The position of a gradient is assumed to be at an edge, and a divergence at the cell center. This could be written explicitly as

\kappa_2 h_i \left( \nabla^2 \phi_i \right)_i
=
\kappa_2 h_i \left( \nabla \cdot \left( \nabla \phi_i \right)_e \right)_i.

The del4 tracer diffusion is simply the Laplacian operator applied twice,

- \kappa_4 h_i \nabla^4 \phi_i
=
- \kappa_4 h_i \nabla^2 \left( \nabla^2 \phi_i \right).

The del2 operator using the divergence of the gradient in the last section is used, with the simple stencils from Section 3.3.

The design details may be found in other design documents, such as those for the state, auxiliary variables, and tendency terms.

<!-- To do in later PR: Add all design details.

1. Index ordering will be: (time, tracer, horizontal index, vertical index) Note this is c-style indexing, with last index being fastest, i.e. contiguous in memory.
2. Time stepping will be multi-level, with an arbitrary number of levels for multistage methods. -->

<!-- to do: List and define any configuration parameters or public constants. -->

<!-- to do: Describe any public data types and/or the class definition -->

<!-- to do: List and describe all public methods and their interfaces (actual code for  -->

<!-- interface that would be in header file). Describe typical use cases. -->

The following terms can be tested with sine waves on periodic domains on a cartesian regular-hexagon mesh, as described in [Bishnu et al. 2024](https://doi.org/10.1029/2022MS003545) and [Bishnu et al. 2021](https://doi.org/10.5281/zenodo.7439539).

The following terms can be tested with sine waves on periodic domains on a cartesian regular-hexagon mesh, as described in 

Curl operator at vertices, i.e. vorticity from a vector field (1st order)

Tangential velocity at edges, computed from normal velocity (2nd order)

These tests can be conducted at the earliest stages of dycore development, in tandem with the implementation of each operator. In the longer term, these tests will be subsumed by the inertia gravity wave, which tests the order of convergence of all these operators together.

Operator convergence tests may additionally be conducted on the sphere using spherical harmonics for the analytic solution. This was conducted on MPAS-Ocean by Hyun Kang in 2023.

The inertia gravity wave test provides an exact solution in time for the linearized shallow water equations (momentum and thickness). It tests the time stepping scheme along with the linearized advection terms, the SSH gradient. It does not test diffusive terms or bottom drag. It is conducted on a doubly periodic cartesian mesh, so does not test boundary conditions. The numerical solution should converge to the analytic solution at 2nd order, as shown in [Bishnu et al. 2024](https://doi.org/10.1029/2022MS003545), Section 2.6, Figures 11 and 15. Also see the inertia gravity test case in Polaris.

The inertia gravity wave test provides an exact solution in time for the linearized shallow water equations (momentum and thickness). It tests the time stepping scheme along with the linearized advection terms, the SSH gradient. It does not test diffusive terms or bottom drag. It is conducted on a doubly periodic cartesian mesh, so does not test boundary conditions. The numerical solution should converge to the analytic solution at 2nd order, as shown in 

, Section 2.6, Figures 11 and 15. Also see the inertia gravity test case in Polaris.

This test should be conducted as soon as momentum and thickness equations and time-stepping is in place. It does not require any tracer infrastructure.

5.3 Manufactured Solution: full nonlinear shallow water, no tracers

The manufactured solution test provides an exact solution in time for the full nonlinear shallow water equations (momentum and thickness). It tests the time stepping scheme along with the full advection terms and the SSH gradient.  It is conducted on a doubly periodic cartesian mesh, so does not test boundary conditions. Error should converge at 2nd order, as shown in [Bishnu et al. 2024](https://doi.org/10.1029/2022MS003545), Section 2.10 and Figures 13 and 19. Also see the manufactured solution test case in Polaris.

The manufactured solution test provides an exact solution in time for the full nonlinear shallow water equations (momentum and thickness). It tests the time stepping scheme along with the full advection terms and the SSH gradient.  It is conducted on a doubly periodic cartesian mesh, so does not test boundary conditions. Error should converge at 2nd order, as shown in 

, Section 2.10 and Figures 13 and 19. Also see the manufactured solution test case in Polaris.

5.3 manufactured solution: full nonlinear shallow water, no tracers

A test suite will be used to test horizontal transport schemes on the sphere comprised of test case 1 from [Williamson et al. (1992)](https://www.sciencedirect.com/science/article/pii/S0021999105800166) and several tests from [Lauritzen et al. (2012)](https://www.geosci-model-dev.net/5/887/2012/).
They test the tracer advection and tracer time stepping. Velocity and thickness are predefined fields and remain fixed, so this does not exercise those equations.

A test suite will be used to test horizontal transport schemes on the sphere comprised of test case 1 from 

produces visualization that allows the user to evaluate whether numerical mixing resembles real mixing (preserves functional relationships between tracers)

tracer min and max values should remain bounded by initial bounds.

Tests can be conducted with inertia-gravity wave tests but with full non-linear terms and 100 identical layers. Domain will be cartesian resolutions from 64x64 up to 512x512. See performance test results using MPAS-Ocean on Perlmutter, using this setup, in [Bishnu et al. 2023](https://gmd.copernicus.org/articles/16/5539/2023). Additionally, we could test spherical cases with a Williamson test case or similar.

Tests can be conducted with inertia-gravity wave tests but with full non-linear terms and 100 identical layers. Domain will be cartesian resolutions from 64x64 up to 512x512. See performance test results using MPAS-Ocean on Perlmutter, using this setup, in 

. Additionally, we could test spherical cases with a Williamson test case or similar.

Single CPU performance should be as good or better than single CPU MPAS-Ocean.

GPU tests on single node should be better than single node CPU tests.

Scaling on CPUs multi-node, up to 4096 cores or higher, should be close to perfect scaling for 512x512 mesh, and as good or better than MPAS-Ocean.

Scaling on GPUs for multi-node should be close to perfect scaling for 512x512 mesh.

5.6 Shallow water tests on spherical domains and with realistic coastlines

Global cases will facilitate the testing of shallow water dynamics on the sphere and of boundary conditions with realistic coastlines, which are not included in any of the previous tests. This includes testing conservation of volume and tracers in global domains.

Surface gravity wave: The speed of the surface gravity wave can be compared to the theoretical expectation, as shown in [Pal et al. 2023](https://gmd.copernicus.org/articles/16/1297/2023) Appendix A. This could use an aquaplanet domain (flat bottom, no coastlines) to measure the gravity wave speed, and a realistic domain to test boundary conditions and variable-depth bathymetry.

Surface gravity wave: The speed of the surface gravity wave can be compared to the theoretical expectation, as shown in 

 Appendix A. This could use an aquaplanet domain (flat bottom, no coastlines) to measure the gravity wave speed, and a realistic domain to test boundary conditions and variable-depth bathymetry.

Stommel double gyre:  This may be compared to an exact solution in the Cartesian case, as in [Pal et al. 2023](https://gmd.copernicus.org/articles/16/1297/2023) Appendix B, or qualitative comparisons for the spherical case using either idealized boundaries or an isolated Atlantic Basin domain.

Stommel double gyre:  This may be compared to an exact solution in the Cartesian case, as in 

 Appendix B, or qualitative comparisons for the spherical case using either idealized boundaries or an isolated Atlantic Basin domain.

Tests of realistic global circulation cannot be done with the shallow water equations of Omega-0, but will be part of Omega-1 development with the layered primitive equation model.

5.6 shallow water tests on spherical domains and with realistic coastlines

Potential additional test cases include the following. These are useful to explore and validate model behavior, but are optional, depending on the time available.

[Williamson et al. 1992](https://www.sciencedirect.com/science/article/pii/S0021999105800166) as test case 2 or 3, global steady state solutions.

Global case with unstable jet [Galewsky et al. 2003](https://doi.org/10.3402/tellusa.v56i5.14436)

Cushman‐Roisin, B., & Beckers, J.‐M. (2011). Introduction to geophysical fluid dynamics: Physical and numerical aspects. Academic press.

Pedlosky, J. (1987). Geophysical fluid dynamics (Vol. 710). Springer.

Vallis, G. K. (2017). Atmospheric and oceanic fluid dynamics. Cambridge University Press.

Hyperlink target "omega-design-shallow-water-omega0" is not referenced.

Hyperlink target "thickness-and-tracer-terms" is not referenced.

<!--
Add later, if it seems necessary. There is a toc on the left bar.
**Table of Contents**
1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Algorithmic Formulation](#3-algorithmic-formulation)
4. [Design](#4-design)
5. [Verification and Testing](#5-verification-and-testing)
-->

This design document describes the governing equations for Omega, the Ocean Model for E3SM Global Applications. Overall, Omega is an unstructured-mesh ocean model based on TRiSK numerical methods ([Thuburn et al. 2009](https://www.sciencedirect.com/science/article/pii/S0021999109004434)) that is specifically designed for modern exascale computing architectures. The algorithms in Omega will be mostly identical to those in MPAS-Ocean, but it will be written in c++ rather than Fortran in order to take advantage of the Kokkos performance portability library to run on GPUs ([Trott et al. 2022](https://ieeexplore.ieee.org/document/9485033)). Significant differences between MPAS-Ocean and Omega are:

This design document describes the governing equations for Omega, the Ocean Model for E3SM Global Applications. Overall, Omega is an unstructured-mesh ocean model based on TRiSK numerical methods (

) that is specifically designed for modern exascale computing architectures. The algorithms in Omega will be mostly identical to those in MPAS-Ocean, but it will be written in c++ rather than Fortran in order to take advantage of the Kokkos performance portability library to run on GPUs (

Omega is non-Boussinesq. This means that the full 3D density is used everywhere, and results in a mass-conserving model. MPAS-Ocean and POP were Boussinesq, so that a reference density $\rho_0$ is used in the pressure gradient term, and were therefore volume-conserving models. In Omega the layered mass-conservation equation is in terms of pseudo-thickness ($\tilde{h}=-\Delta p / \rho_0 g$). In MPAS-Ocean the simple thickness ($h=\Delta z$) is the prognostic volume variable (normalized by horizontal cell area).

Omega is non-Boussinesq. This means that the full 3D density is used everywhere, and results in a mass-conserving model. MPAS-Ocean and POP were Boussinesq, so that a reference density 

 is used in the pressure gradient term, and were therefore volume-conserving models. In Omega the layered mass-conservation equation is in terms of pseudo-thickness (

) is the prognostic volume variable (normalized by horizontal cell area).

Omega will use the updated equation of state TEOS10, while MPAS-Ocean used the Jackett-McDougall equation of state.

**Omega-0: Shallow water equations with identical vertical layers and inactive tracers.** In his first version, there is no vertical transport or advection. The tracer equation is horizontal advection-diffusion, but tracers do not feed back to dynamics. Pressure gradient is simply gradient of sea surface height. Capability is similar to [Ringler et al. 2010](https://www.sciencedirect.com/science/article/pii/S0021999109006780)

 In his first version, there is no vertical transport or advection. The tracer equation is horizontal advection-diffusion, but tracers do not feed back to dynamics. Pressure gradient is simply gradient of sea surface height. Capability is similar to 

**Omega-1.0: Layered ocean, idealized, no surface fluxes.** This adds active temperature, salinity, and density as a function of pressure in the vertical. Vertical advection and diffusion terms are added to the momentum and tracer equations. An equation of state and simple vertical mixing, such as constant coefficient, are needed. Capability and testing are similar to [Petersen et al. 2015](http://www.sciencedirect.com/science/article/pii/S1463500314001796). Tests include overflow, internal gravity wave, baroclinic channel, seamount, and vertical merry-go-round.

 This adds active temperature, salinity, and density as a function of pressure in the vertical. Vertical advection and diffusion terms are added to the momentum and tracer equations. An equation of state and simple vertical mixing, such as constant coefficient, are needed. Capability and testing are similar to 

. Tests include overflow, internal gravity wave, baroclinic channel, seamount, and vertical merry-go-round.

**Omega-1.1: Layered ocean, idealized, with surface fluxes.** Addition of simple vertical mixing scheme such as Pacanowski & Philander; nonlinear equation of state (TEOS10); tracer surface restoring to a constant field; constant-in-time wind forcing; and flux-corrected transport for horizontal advection. Testing will be with the baroclinic gyre and single column tests of surface fluxes and vertical mixing.

 Addition of simple vertical mixing scheme such as Pacanowski & Philander; nonlinear equation of state (TEOS10); tracer surface restoring to a constant field; constant-in-time wind forcing; and flux-corrected transport for horizontal advection. Testing will be with the baroclinic gyre and single column tests of surface fluxes and vertical mixing.

**Omega-2.0: Coupled within E3SM, ocean only.** Ability to run C cases (active ocean only) within E3SM. Requires addition of E3SM coupling infrastructure; simple analysis (time-averaged output of mean, min, max); split baroclinic-barotropic time; global bounds checking on state. Testing and analysis similar to [Ringler et al. 2013](https://www.sciencedirect.com/science/article/pii/S1463500313000760), except Omega uses a non-Boussinesq formulation.

 Ability to run C cases (active ocean only) within E3SM. Requires addition of E3SM coupling infrastructure; simple analysis (time-averaged output of mean, min, max); split baroclinic-barotropic time; global bounds checking on state. Testing and analysis similar to 

**Omega-2.1: E3SM fully coupled** Ability to run G cases (active ocean and sea ice) and B cases (all components active) within E3SM. This will include: a full vertical mixing scheme, such as KPP; frazil ice formation in the ocean; and a submosescale parameterization. Omega 2.1 will mostly be run at eddy-resolving resolutions, and will not include parameterizations for lower resolutions such as Gent-McWilliams and Redi mixing.  Simulations will be compared to previous E3SM simulations, including [Caldwell et al. 2019](https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2019MS001870) and [Petersen et al. 2019](https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2018MS001373).

 Ability to run G cases (active ocean and sea ice) and B cases (all components active) within E3SM. This will include: a full vertical mixing scheme, such as KPP; frazil ice formation in the ocean; and a submosescale parameterization. Omega 2.1 will mostly be run at eddy-resolving resolutions, and will not include parameterizations for lower resolutions such as Gent-McWilliams and Redi mixing.  Simulations will be compared to previous E3SM simulations, including 

This document describes the governing equations for the layered ocean model, which are applicable for Omega-1.0 onwards. Specific terms, such as the pressure gradient, vertical mixing, and parameterizations, are left in their general form here but are described in more detail in other design documents.

The requirements in the [Omega-0 design document](OmegaV0ShallowWater) still apply. Additional design requirements for Omega-1 are:

Omega will adopt a non-Boussinesq formulation, meaning it retains the full, spatially and temporally varying fluid density $\rho({\bf x}, t)$ in all governing equations. This ensures exact mass conservation, as opposed to volume conservation in Boussinesq models that approximate density as a constant reference value $\rho_0$ outside the pressure gradient. The equations are derived in terms of layer-integrated mass per unit area (see [layered equations](#Layer Equations)), and no approximation is made that filters out compressibility or density variations.

Omega will adopt a non-Boussinesq formulation, meaning it retains the full, spatially and temporally varying fluid density 

 in all governing equations. This ensures exact mass conservation, as opposed to volume conservation in Boussinesq models that approximate density as a constant reference value 

 outside the pressure gradient. The equations are derived in terms of layer-integrated mass per unit area (see [layered equations](#Layer Equations)), and no approximation is made that filters out compressibility or density variations.

The model also assumes hydrostatic balance in the vertical momentum equation, which is a standard and well-justified simplification for large-scale geophysical flows. In such regimes, vertical accelerations are typically small compared to the vertical pressure gradient and gravitational forces. This assumption simplifies the dynamics and removes most sound waves while retaining fidelity for the mesoscale to planetary-scale ocean circulation that Omega is designed to simulate.

Omega-1.0 will add new terms for the pressure gradient, vertical mixing, and vertical advection.

See forthcoming design documents on the pressure gradient, [vertical mixing](VerticalMixingCoeff), and vertical advection.

omega-1.0 will add new terms for the pressure gradient, vertical mixing, and vertical advection.

We begin with the continuous, control-volume form of the conservation equations. Consider an arbitrary control volume $V(t)$ with a bounding control surface $\partial V(t)$. The fluid has density $\rho({\bf x},t)$ and velocity ${\bf v}({\bf x},t)$, where $\mathbf{x}$ is the three-dimensional position vector. The control surface is moving at a velocity ${\bf v}_r({\bf x},t)$.  The vector ${\bf n}$ denotes the outward-facing unit normal vector on the control surface $\partial V(t)$, and is used in all surface integrals to represent the direction of fluxes across the boundary. The conservation equations are

We begin with the continuous, control-volume form of the conservation equations. Consider an arbitrary control volume 

 is the three-dimensional position vector. The control surface is moving at a velocity 

 denotes the outward-facing unit normal vector on the control surface 

, and is used in all surface integrals to represent the direction of fluxes across the boundary. The conservation equations are

\frac{d}{dt} \int_{V(t)} \rho({\bf x},t) \, dV
+ \int_{\partial V(t)}\rho({\bf x},t)\left({\bf v}({\bf x},t) - {\bf v}_r({\bf x},t) \right) \cdot {\bf n} \, dA
= 0

\frac{d}{dt} \int_{V(t)} \rho({\bf x},t) \, \varphi({\bf x},t) \, dV
+ \int_{\partial V(t)}\rho({\bf x},t)\, \varphi({\bf x},t) \left({\bf v}({\bf x},t) - {\bf v}_r \right) \cdot {\bf n} \, dA
= 0

&\frac{d}{dt} \int_{V(t)} \rho({\bf x},t)\,  {\bf v}({\bf x},t) \, dV
+ \int_{\partial V(t)}\rho({\bf x},t)\, {\bf v}({\bf x},t) \left({\bf v}({\bf x},t) - {\bf v}_r \right) \cdot {\bf n} \, dA
\\ & \; \; \; =
\int_{V(t)} \rho({\bf x},t) \, {\bf b}({\bf x},t)\, dV
+ \int_{\partial V(t)} {\bf \tau}({\bf n},{\bf x},t)  \, dA

The operator $\frac{d}{dt}$ used here denotes the rate of change within a moving control volume, sometimes referred to as a Reynolds transport derivative. It differs from the partial derivative $\frac{\partial}{\partial t}$, which represents the local rate of change at a fixed point in space (Eulerian frame), and from the material derivative $\frac{D}{Dt}$, which follows an individual fluid parcel (Lagrangian frame). The use of $\frac{d}{dt}$ allows for conservation laws to be expressed in a general framework that includes both stationary and moving control volumes, consistent with the Reynolds transport theorem.

 used here denotes the rate of change within a moving control volume, sometimes referred to as a Reynolds transport derivative. It differs from the partial derivative 

, which represents the local rate of change at a fixed point in space (Eulerian frame), and from the material derivative 

, which follows an individual fluid parcel (Lagrangian frame). The use of 

 allows for conservation laws to be expressed in a general framework that includes both stationary and moving control volumes, consistent with the Reynolds transport theorem.

These equations are taken from [Kundu et al. 2016](https://doi.org/10.1016/C2012-0-00611-4) (4.5) for mass conservation and (4.17) for momentum conservation.
All notation is identical to Kundu, except that we use ${\bf v}$ for the three-dimensional velocity (${\bf u}$ will be used below for horizontal velocities) and ${\bf v}_r$ and $\partial V$ match the notation in [Ringler et al. 2013](https://www.sciencedirect.com/science/article/pii/S1463500313000760) Appendix A.2.

 (4.5) for mass conservation and (4.17) for momentum conservation.

The tracer equation is simply mass conservation, where the conserved quantity is the tracer mass in the control volume $\rho \varphi$, as $\varphi$ is the tracer concentration per unit mass.
In all three equations, the first term is the change of the quantity within the control volume; the second term is the flux through the moving boundary.
If the control surface moves with the fluid in a Lagrangian fashion, then ${\bf v}_r={\bf v}$ and the second term (flux through the boundary) vanishes---there is no net mass, momentum, or tracer transport across the moving surface.

The tracer equation is simply mass conservation, where the conserved quantity is the tracer mass in the control volume 

If the control surface moves with the fluid in a Lagrangian fashion, then 

 and the second term (flux through the boundary) vanishes—there is no net mass, momentum, or tracer transport across the moving surface.

The momentum equation is an expression of Newton's second law and includes two types of external forces.
The first is the body force, represented here as ${\bf b}({\bf x}, t)$, which encompasses any volumetric force acting throughout the fluid, such as gravitational acceleration or the Coriolis force. In some contexts, body forces may be expressible as the gradient of a potential, ${\bf b} = -\nabla_{3D} \Phi$, but this is not assumed in general.
The second is the surface force ${\bf \tau}$, which acts on the boundary of the control volume and includes pressure and viscous stresses. These forces appear as surface integrals over the boundary and drive momentum exchange between adjacent fluid parcels or between the fluid and its environment.
The derivation of the momentum equation may also be found in [Leishman 2025](https://eaglepubs.erau.edu/introductiontoaerospaceflightvehicles/chapter/conservation-of-momentum-momentum-equation/#chapter-260-section-2), Chapter 21, equation 10.

The momentum equation is an expression of Newton’s second law and includes two types of external forces.

, which encompasses any volumetric force acting throughout the fluid, such as gravitational acceleration or the Coriolis force. In some contexts, body forces may be expressible as the gradient of a potential, 

, which acts on the boundary of the control volume and includes pressure and viscous stresses. These forces appear as surface integrals over the boundary and drive momentum exchange between adjacent fluid parcels or between the fluid and its environment.

In large-scale geophysical flows, vertical accelerations are typically much smaller than the vertical pressure gradient and gravitational forces. As a result, we adopt the **hydrostatic approximation**, which simplifies the vertical momentum equation by assuming a leading-order balance between pressure and gravity:

In large-scale geophysical flows, vertical accelerations are typically much smaller than the vertical pressure gradient and gravitational forces. As a result, we adopt the 

, which simplifies the vertical momentum equation by assuming a leading-order balance between pressure and gravity:

This balance allows us to express the pressure at any depth $z$ in terms of an integral over the density field above:

p(x,y,z,t) = p^{\text{surf}}(x,y,t) + \int_{z}^{z^{\text{surf}}} \rho(x,y,z',t) g \, dz'

Here, $p^{\text{surf}}$ is the pressure at the free surface $z^{\text{surf}}(x, y, t)$, $\rho(z')$ is the local fluid density and $z'$ is a dummy variable of integration. This relation holds pointwise in space and time and provides a foundation for defining vertical coordinates based on pressure.

 is a dummy variable of integration. This relation holds pointwise in space and time and provides a foundation for defining vertical coordinates based on pressure.

The hydrostatic approximation also simplifies vertical pressure forces in control-volume integrations. For example, in a finite-volume cell bounded above and below by sloping surfaces, the vertical pressure force reduces to:

because the area correction from the slope and the projection of the pressure gradient into the vertical direction cancel to leading order.  [](#vert-pressure-balance) allows us to create simpler relations between pressure and pseudo-height in the following section.

because the area correction from the slope and the projection of the pressure gradient into the vertical direction cancel to leading order.  

 allows us to create simpler relations between pressure and pseudo-height in the following section.

This hydrostatic balance underlies the definition of **pseudo-height**, introduced next, and ensures consistency between the prognostic vertical coordinate and the model’s vertical force balance.

, introduced next, and ensures consistency between the prognostic vertical coordinate and the model’s vertical force balance.

In our non-Boussinesq, hydrostatic ocean model, we adopt **pseudo-height** $\tilde{z}$ as the prognostic vertical coordinate. This choice is motivated directly by the hydrostatic approximation introduced in [](#hydrostatic).
Under this balance, we define the pseudo-height as a normalized pressure:

 as the prognostic vertical coordinate. This choice is motivated directly by the hydrostatic approximation introduced in 

Here, $\rho_0$ is a constant reference density and $g$ is gravitational acceleration. Although $\tilde{z}$ is essentially a pressure coordinate, the normalization ensures it has units of length and retains an intuitive geometric interpretation.

 is essentially a pressure coordinate, the normalization ensures it has units of length and retains an intuitive geometric interpretation.

The inverse relation, giving $\tilde{z}$ in terms of geometric height, follows from the hydrostatic pressure integral:

 in terms of geometric height, follows from the hydrostatic pressure integral:

\tilde{z}(x, y, z, t) = -\frac{1}{\rho_0 g} \left( p(x, y, z, t)^{\text{surf}} + \int_{z}^{z^{\text{surf}}} \rho(z') g \, dz' \right)

This expression shows that pseudo-height varies in space and time like pressure. It also makes clear that $z$ is no longer a prognostic variable but a derived quantity, recoverable via hydrostatic inversion.

This expression shows that pseudo-height varies in space and time like pressure. It also makes clear that 

 is no longer a prognostic variable but a derived quantity, recoverable via hydrostatic inversion.

Taking a differential, we relate pseudo-height and geometric height:

d\tilde{z} = -\frac{1}{\rho_0 g} \, dp = \frac{\rho}{\rho_0} \, dz

This motivates the definition of **pseudo-velocity** in the vertical direction:

\tilde{w} \equiv \frac{d\tilde{z}}{dt} = \frac{\rho}{\rho_0} \, w

$\tilde{w}$ is the vertical **mass flux per unit reference density**, a key variable in a non-Boussinesq framework.

Vertical transport of mass, tracers, and momentum will use $\tilde{w}$ rather than the vertical velocity in height coordinates, $w$.

[Griffies 2018](https://doi.org/10.2307/j.ctv301gzg) p. 37  argues for the use of pseudo-velocities, which he calls the density-weighted velocity, for non-Boussinesq models.
Griffies recommends a value of $\rho_0=1035$ kg/m$^3$, following p. 47 of [Gill (1982)](https://doi.org/10.1016/S0074-6142(08)60028-5), because ocean density varies less than 2% from that value.

 p. 37  argues for the use of pseudo-velocities, which he calls the density-weighted velocity, for non-Boussinesq models.

The use of a constant $\rho_0$ in defining pseudo-height does not imply the Boussinesq approximation. In Boussinesq models, $\rho$ is set to $\rho_0$ everywhere except in the buoyancy term (i.e., the vertical pressure gradient or gravitational forcing). Here, by contrast, we retain the full $\rho$ in all terms, and use $\rho_0$ only as a normalization constant—for example, so that $d\tilde{z} \approx dz$ when $\rho \approx \rho_0$. This preserves full mass conservation while making vertical units more intuitive.

 in defining pseudo-height does not imply the Boussinesq approximation. In Boussinesq models, 

 everywhere except in the buoyancy term (i.e., the vertical pressure gradient or gravitational forcing). Here, by contrast, we retain the full 

. This preserves full mass conservation while making vertical units more intuitive.

This approach ensures consistency with mass conservation and simplifies the derivation of the discrete equations.

**Note:** This definition does not invoke the Boussinesq approximation. The full, time- and space-dependent density $\rho(x, y, z, t)$ is retained in all conservation laws. The use of $\rho_0$ is solely for normalization.

 This definition does not invoke the Boussinesq approximation. The full, time- and space-dependent density 

Alternative definitions of pseudo-height could add an arbitrary offset, e.g., setting $\tilde{z} = 0$ at mean sea level. However, the adopted form:

Alternative definitions of pseudo-height could add an arbitrary offset, e.g., setting 

Units of $\tilde{z}$, $\tilde{h}$, and $\tilde{w}$ are intuitive.

It aligns layer thickness with pseudo-thickness: $\tilde{h} = \Delta \tilde{z} \propto \Delta p$

$\tilde{z}^{\text{floor}} \propto p^{\text{bot}}$, which aids barotropic pressure gradient calculation.

This makes $\tilde{z}$ a natural coordinate for a mass-conserving hydrostatic model.

6. Horizontal and Vertical Flux Separation in Pseudo-Height Coordinates

In geophysical flows, the vertical and horizontal directions are treated differently due to rotation and stratification, which lead to distinct characteristic scales of motion. To reflect this, we separate horizontal and vertical fluxes explicitly in the governing equations.

We partition the control surface of a finite-volume cell into the side walls $\partial V^{\text{side}}$ (which are fixed in space and time) and the upper and lower pseudo-height surfaces $\partial V^{\text{top}}(t)$ and $\partial V^{\text{bot}}(t)$, which evolve in time.

We partition the control surface of a finite-volume cell into the side walls 

 (which are fixed in space and time) and the upper and lower pseudo-height surfaces 

As an example, we consider the tracer equation and drop explicit notation for spatial and temporal dependence for clarity. We write the control-volume form as:

\frac{d}{dt} \int_{V(t)} \rho \, \varphi \, dV
& + \int_{\partial V^{\text{side}}} \rho \varphi ({\bf v} - {\bf v}_r) \cdot {\bf n} \, dA \\
& + \int_{\partial V^{\text{top}}(t)} \rho \varphi ({\bf v} - {\bf v}_r) \cdot {\bf n}^{\text{top}} \, dA
+ \int_{\partial V^{\text{bot}}(t)} \rho \varphi ({\bf v} - {\bf v}_r) \cdot {\bf n}^{\text{bot}} \, dA
= 0

We now express the top and bottom surfaces using the pseudo-height variable $\tilde{z}(x,y,t)$ rather than geometric height $z$. The unit normals are therefore:

We now express the top and bottom surfaces using the pseudo-height variable 

{\bf n}^{\text{top}} \approx (-\nabla \tilde{z}^{\text{top}}, 1), \quad
{\bf n}^{\text{bot}} \approx (-\nabla \tilde{z}^{\text{bot}}, 1)

Here we apply a small-slope approximation, assuming $|\nabla \tilde{z}| \ll 1$, and omit normalization factors for clarity. This approximation retains leading-order effects of slope while simplifying the surface geometry.

, and omit normalization factors for clarity. This approximation retains leading-order effects of slope while simplifying the surface geometry.

To facilitate integration over a fixed horizontal domain, we introduce the following notation:

$A$ is the horizontal footprint (in the $x$–$y$ plane) of the control volume $V(t)$.

$\partial A$ is the boundary of $A$, and $dl$ is the line element along this boundary.

${\bf n}_\perp$ is the outward-pointing unit normal vector in the horizontal plane, defined on $\partial A$. It lies in the $x$–$y$ plane and is orthogonal to $dl$.

 is the outward-pointing unit normal vector in the horizontal plane, defined on 

We now project the tracer equation [](#tr-vh-split-pseudo) onto a horizontal domain $A$ (the footprint of the control volume), over which the top and bottom boundaries vary in height. The side walls remain fixed in time and space, but not vertical extent.

 (the footprint of the control volume), over which the top and bottom boundaries vary in height. The side walls remain fixed in time and space, but not vertical extent.

The pseudo-height surfaces $\tilde{z}^{\text{top}}(x,y,t)$ and $\tilde{z}^{\text{bot}}(x,y,t)$ are not fixed in time, so their motion must be accounted for when computing vertical fluxes. We define $\tilde{w}_r$ as the pseudo-height velocity of the interface:

 are not fixed in time, so their motion must be accounted for when computing vertical fluxes. We define 

\tilde{w}_{r} = \left.\frac{d\tilde{z}}{dt}\right |_{\text interface}

This gives the relative pseudo-mass flux through the interface per unit reference density. Vertical flux terms are written using this difference to ensure conservation in the presence of interface motion.

In the flux integrals below, we express all surface-normal transport using $\tilde{w}_{tr}$, i.e.,

In the flux integrals below, we express all surface-normal transport using 

\rho_0 \varphi \left[ \tilde{w}_{tr} - \tilde{ u} \right]
= \rho_0 \varphi \left[ \tilde{w} - \tilde{w}_r - \tilde{ u} \right]

so that the vertical flux terms correctly account for both the local motion of the fluid and the motion of the interface itself.

The quantity $\tilde{w}$ in equation [](#w-tilde) is the pseudo-velocity. The second term, $\tilde{ u} = {\bf u} \cdot \nabla \tilde{z}^{\text{top}}$, captures the component of horizontal velocity advecting material across a sloping pseudo-height interface. Together, the expression in brackets represents the **net vertical transport** through a surface of constant $\tilde{z}$. A schematic of how $\tilde{w}$, $\tilde{w}_{tr}$, and $\tilde{u}$ relate to each other is shown in {numref}`pseudo-vel-schematic`:

, captures the component of horizontal velocity advecting material across a sloping pseudo-height interface. Together, the expression in brackets represents the 

Because both terms are scaled by $1/\rho_0$ in [](#vertical-flux), the full flux is multiplied by $\rho_0$ to recover a physical mass flux. This ensures dimensional consistency and physical equivalence to the traditional form $\rho w - \rho {\bf u} \cdot \nabla z$ expressed in geometric-height coordinates.

 to recover a physical mass flux. This ensures dimensional consistency and physical equivalence to the traditional form 

$\tilde{W}_{tr}$ will be introduced and defined in [](#notational-simplifications). Using this projection, we obtain:

\frac{d}{dt} \int_A \int_{\tilde{z}^{\text{bot}}}^{\tilde{z}^{\text{top}}} \, \varphi \, d\tilde{z} \, dA
& + \int_{\partial A} \left( \int_{\tilde{z}^{\text{bot}}}^{\tilde{z}^{\text{top}}} \varphi \, {\bf u} \, d\tilde{z} \right) \cdot {\bf n}_\perp \, dl \\
& + \int_A \rho_0 \left\{\varphi \left[ \tilde{w}_{tr} - \tilde{ u} \right]\right\}_{\tilde{z} = \tilde{z}^{\text{top}}} dA \\
& - \int_A \rho_0 \left\{\varphi \left[ \tilde{w}_{tr} - \tilde{ u} \right]\right\}_{\tilde{z} = \tilde{z}^{\text{bot}}} dA
= 0

This equation is structurally identical to its $z$-based counterpart but with all references to geometric height replaced by pseudo-height quantities.

-based counterpart but with all references to geometric height replaced by pseudo-height quantities.

This formulation allows vertical mass and tracer transport to be computed directly in terms of prognostic variables, without reconstructing $\rho$ or $z$ explicitly.

This formulation allows vertical mass and tracer transport to be computed directly in terms of prognostic variables, without reconstructing 

Similar expressions to [](#tr-vh-separation-pseudo) hold for mass and momentum:

\frac{d}{dt} \int_A \int_{\tilde{z}^{\text{bot}}}^{\tilde{z}^{\text{top}}} \, d\tilde{z} \, dA
& + \int_{\partial A} \left( \int_{\tilde{z}^{\text{bot}}}^{\tilde{z}^{\text{top}}} \, {\bf u} \, d\tilde{z} \right) \cdot {\bf n}_\perp \, dl \\
& + \int_A \rho_0 \left\{\left[ \tilde{w}_{tr} - \tilde{ u} \right]\right\}_{\tilde{z} = \tilde{z}^{\text{top}}} dA \\
& - \int_A \rho_0 \left\{\left[ \tilde{w}_{tr} - \tilde{ u} \right]\right\}_{\tilde{z} = \tilde{z}^{\text{bot}}} dA
= 0

\frac{d}{dt} \int_A \int_{\tilde{z}^{\text{bot}}}^{\tilde{z}^{\text{top}}}  \varphi \, d\tilde{z} \, dA
& + \int_{\partial A} \left( \int_{\tilde{z}^{\text{bot}}}^{\tilde{z}^{\text{top}}}  \varphi \, {\bf u} \, d\tilde{z} \right) \cdot {\bf n}_\perp \, dl & \\
& + \int_A \rho_0 \left\{\varphi \left[\tilde{w}_{tr} - \tilde{ u} \right] \right\}_{\tilde{z} = \tilde{z}^{\text{top}}} \, dA & \\
& - \int_A \rho_0 \left\{\varphi \left[\tilde{w}_{tr} - \tilde{ u} \right] \right\}_{\tilde{z} = \tilde{z}^{\text{bot}}}\, dA
& = 0

\frac{d}{dt} \int_{A} \int_{\tilde{z}^{\text{bot}}}^{\tilde{z}^{\text{top}}} \, {\bf u} \, d\tilde{z} \, dA
&+
\int_{\partial A} \left( \int_{\tilde{z}^{\text{bot}}}^{\tilde{z}^{\text{top}}} \, {\bf u} \otimes {\bf u} \, d\tilde{z} \right) \cdot {\bf n}_\perp \, dl \\
&+
\int_A \rho_0 \, \left\{{\bf u} \left[ \tilde{w}_{tr} - \tilde{ u} \right]\right\}_{\tilde{z} = \tilde{z}^{\text{top}}} \, dA
-
\int_A \rho_0 \, \left\{{\bf u} \left[ \tilde{w}_{tr} - \tilde{ u} \right]\right\}_{\tilde{z} = \tilde{z}^{\text{bot}}} \, dA \\
&=
\int_A \int_{\tilde{z}^{\text{bot}}}^{\tilde{z}^{\text{top}}} {\bf b}_{\perp} \, d\tilde{z} \, dA
+
\int_{\partial A} \left( \int_{\tilde{z}^{\text{bot}}}^{\tilde{z}^{\text{top}}} \frac{{\bf \tau}}{\rho} \, d\tilde{z} \right) dl \\
&\quad
+ \int_A \left[ \frac{{\bf \tau}}{\rho} \right]_{\tilde{z} = \tilde{z}^{\text{top}}} \, dA
- \int_A \left[ \frac{{\bf \tau}}{\rho} \right]_{\tilde{z} = \tilde{z}^{\text{bot}}} \, dA

In [](#vh-momentum-pseudo), $\mathbf{b}_\perp$ is defined similarly to $\mathbf{n}_\perp$, it is the body force normal to $dl$. These equations express horizontal and vertical fluxes naturally in terms of pseudo-height, enabling fully consistent discretization in the vertical coordinate used for prognostic evolution.

. These equations express horizontal and vertical fluxes naturally in terms of pseudo-height, enabling fully consistent discretization in the vertical coordinate used for prognostic evolution.

6. horizontal and vertical flux separation in pseudo-height coordinates

The previous equation set [](#vh-mass-pseudo) to [](#vh-momentum-pseudo) is for an arbitrary layer bounded by $\tilde{z}^{\text{top}}$ above and $\tilde{z}^{\text{bot}}$ below. We now provide the details of the vertical discretization. The ocean is divided vertically into $K_{max}$ layers, with $k=0$ at the top and increasing downwards (opposite from $z$). Layer $k$ is bounded between $\tilde{z}_k^{\text{top}}$ above and $\tilde{z}_{k+1}^{\text{top}}$ below (i.e. $\tilde{z}_k^{\text{bot}} = \tilde{z}_{k+1}^{\text{top}}$).

 below. We now provide the details of the vertical discretization. The ocean is divided vertically into 

{\tilde h}_k(x,y,t)
&= \int_{{\tilde z}_{k}^{\text{bot}}}^{{\tilde z}_k^{\text{top}}} d\tilde{z} \\
&= \frac{1}{\rho_0} \int_{z_{k}^{\text{bot}}}^{z_k^{\text{top}}} \rho \, dz \\
&= \frac{1}{\rho_0 g} \left( p_{k}^{\text{bot}} - p_k^{\text{top}} \right)

which is the mass per unit area in the layer, normalized by $\rho_0$. This pseudo-thickness and layer-averaging will be used to express conservation laws in a mass-weighted coordinate system.  Pseudo-thickness, rather than geometric thickness will be the prognostic variable in Omega.

. This pseudo-thickness and layer-averaging will be used to express conservation laws in a mass-weighted coordinate system.  Pseudo-thickness, rather than geometric thickness will be the prognostic variable in Omega.

The density-weighted average of any variable $\varphi({\bf x},t)$ in layer $k$ is

{\overline \varphi}^{\tilde{z}}_k(x,y,t) =
\frac{\int_{z_{k}^{\text{bot}}}^{z_k^{\text{top}}} \rho \varphi dz}
     {\int_{z_{k}^{\text{bot}}}^{z_k^{\text{top}}} \rho dz}
     =
\frac{\frac{1}{\rho_0}\int_{z_{k}^{\text{bot}}}^{z_k^{\text{top}}} \rho \varphi dz}
     {\frac{1}{\rho_0}\int_{z_{k}^{\text{bot}}}^{z_k^{\text{top}}} \rho dz}
     =
\frac{\int_{{\tilde z}_{k}^{\text{bot}}}^{{\tilde z}_k^{\text{top}}} \varphi d{\tilde z}}
     {\int_{{\tilde z}_{k}^{\text{bot}}}^{{\tilde z}_k^{\text{top}}} d{\tilde z}}
=
\frac{1}{{\tilde h}_k}\int_{{\tilde z}_{k}^{\text{bot}}}^{{\tilde z}_k^{\text{top}}} \varphi d{\tilde z}.

\int_{\tilde{z}_{k}^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \varphi d\tilde{z}
 = {\tilde h}_k {\overline \varphi}^{\tilde{z}}_k.

This relation is frequently used in discretized fluxes and conservation equations to replace integrals with layer-mean quantities.

In this document, two decompositions will be critical.  The first decomposition is

\varphi(x,y,z,t) = \overline{\varphi}^{\tilde{z}}_k(x,y,t) + \delta \varphi(x,y,z,t)

which is a density weighted vertical integral based upon [](#def-layer-average) and the deviation from this value within the layer.

which is the traditional Reynolds' average and deviation from this value.

which is the traditional Reynolds’ average and deviation from this value.

The fundamental ocean model equations are most often Reynolds' averaged to derive the sub gridscale stresses. Each variable is decomposed into an average over a sufficiently large sample of turbulent processes, which allows for a meaningful fluctuating component [[](#reynolds-definition)]. Commonly, the Reynolds' average is thought of as a time average, but an ensemble average is equally valid.  In fact, the ensemble and time averaging can be thought of as equivalent, given that a sufficiently long time average will effectively average over variations in the flow that could be thought of as an ensemble.  Given the functions are continuous, the averaging operator can be passed through derivatives and integrals without corrections and an average of products with a single perturbation quantity is zero.

The fundamental ocean model equations are most often Reynolds’ averaged to derive the sub gridscale stresses. Each variable is decomposed into an average over a sufficiently large sample of turbulent processes, which allows for a meaningful fluctuating component [

]. Commonly, the Reynolds’ average is thought of as a time average, but an ensemble average is equally valid.  In fact, the ensemble and time averaging can be thought of as equivalent, given that a sufficiently long time average will effectively average over variations in the flow that could be thought of as an ensemble.  Given the functions are continuous, the averaging operator can be passed through derivatives and integrals without corrections and an average of products with a single perturbation quantity is zero.

The Reynolds' average is most commonly denoted by an overbar. However, to disambiguate from the definition of the bar as the vertical density weighted average, the Reynolds' average herein is denoted by $< . >$.

The Reynolds’ average is most commonly denoted by an overbar. However, to disambiguate from the definition of the bar as the vertical density weighted average, the Reynolds’ average herein is denoted by 

The Reynolds' approach is an attractive approach for Boussinesq ocean models since the fundamental equations do not include products of spatially variable density and tracer, pressure, or momentum.  When the ocean model is non Boussinesq, products of spatially varying density and other fields (e.g., tracer) arise and create difficulties, producing a term like $\left<\rho^\prime {\mathbf u}^\prime\right>$ which is difficult to parameterize. However, given the definition of our chosen pseudo-height, the density terms are wrapped up in $\tilde{z}$ and once again a Reynolds' approach can be cleanly used.

The Reynolds’ approach is an attractive approach for Boussinesq ocean models since the fundamental equations do not include products of spatially variable density and tracer, pressure, or momentum.  When the ocean model is non Boussinesq, products of spatially varying density and other fields (e.g., tracer) arise and create difficulties, producing a term like 

 which is difficult to parameterize. However, given the definition of our chosen pseudo-height, the density terms are wrapped up in 

The quantity being averaged in [](#def-layer-average) is arbitrary. For example, this equation can apply equally to a Reynolds' averaged quantity, e.g.,

 is arbitrary. For example, this equation can apply equally to a Reynolds’ averaged quantity, e.g.,

\overline{\left<\varphi\right>}^{\tilde{z}}_k(x,y,t) =
\frac{1}{\tilde{h}_k} \int_{\tilde{z}_{k}^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \left<\varphi\right> d\tilde{z}.

Additionally, for the decomposition related to vertical averaging, we will encounter terms such as ${\overline \varphi}^{\tilde{z}}_k \delta\varphi$ that need to be vertical averaged.  Using [](#def-layer-average) we have

Additionally, for the decomposition related to vertical averaging, we will encounter terms such as 

\overline{{\overline \varphi}^{\tilde{z}}_k \delta\varphi}^{\tilde{z}}_k &= \frac{\int_{{\tilde z}_{k}^{\text{bot}}}^{{\tilde z}_k^{\text{top}}} {\overline \varphi}^{\tilde{z}}_k \delta \varphi d{\tilde z}}
     {\int_{{\tilde z}_{k}^{\text{bot}}}^{{\tilde z}_k^{\text{top}}} d{\tilde z}} \\
     &= {\overline \varphi}^{\tilde{z}}_k \frac{\int_{{\tilde z}_{k}^{\text{bot}}}^{{\tilde z}_k^{\text{top}}} \delta \varphi d{\tilde z}}
     &= 0,

We first Reynolds' average [](#vh-tracer-pseudo) and given the definition of the operator we can move the averaging past the derivatives and integrals without correction terms to yield

 and given the definition of the operator we can move the averaging past the derivatives and integrals without correction terms to yield

\frac{d}{dt} \int_A \int_{\tilde{z}_{k+1}^{\text{top}}}^{\tilde{z}_k^{\text{top}}} \left<\varphi\right> \, d\tilde{z} \, dA
& + \int_{\partial A} \left( \int_{\tilde{z}_{k+1}^{\text{bot}}}^{\tilde{z}_k^{\text{bot}}} \left<\varphi \, {\bf u} \right> \, d\tilde{z} \right) \cdot {\bf n}_\perp \, dl & \\
& + \int_A \left<\left\{ \rho_0 \varphi \left[\tilde{w}_{tr} - \tilde{ u} \right] \right\}\right>_{\tilde{z} = \tilde{z}_k^{\text{top}}} \, dA & \\
& - \int_A \left<\left\{ \rho_0 \varphi \left[\tilde{w}_{tr} - \tilde{ u} \right] \right\}\right>_{\tilde{z} = \tilde{z}_k^{\text{bot}}}\, dA
& = 0.

Next, we use a Reynolds' decomposition on any terms involving products inside a Reynolds' average,

Next, we use a Reynolds’ decomposition on any terms involving products inside a Reynolds’ average,

\frac{d}{dt} \int_A \int_{\tilde{z}_{k}^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \left<\varphi \right> \, d\tilde{z} \, dA
& +
\int_{\partial A} \left( \int_{\tilde{z}_{k}^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \left(\left<{\varphi}\right>\left<{\bf u}\right> + \left<\varphi^{\prime}{\bf u}^{\prime}\right> \right) d\tilde{z} \right) \, \cdot {\bf n}_\perp \, dl & \\
& + \int_A \left\{ \rho_0 \left[ \left<\varphi\right>\left<\tilde{w}_{tr}\right> + \left<\varphi^{\prime} \tilde{w}_{tr}^\prime \right> - \left<\varphi\right> \left<\tilde{ u}\right> -\left<\varphi^{\prime} \tilde{ u}^{\prime}\right> \right] \right\}_{\tilde{z} = \tilde{z}_k^{\text{top}}} \, dA & \\
& - \int_A \left\{ \rho_0 \left[ \left<\varphi\right>\left<\tilde{w}_{tr}\right> + \left<\varphi^{\prime} \tilde{w}_{tr}^\prime \right> - \left<\varphi\right> \left<\tilde{ u}\right> - \left<\varphi^{\prime} \tilde{ u}^{\prime}\right> \right] \right\}_{\tilde{z} = \tilde{z}_k^{\text{bot}}}\, dA
& = 0.

We can rewrite the first two terms in [](#Aintegral-tracer-reynolds) using [](#def-layer-average-reynolds)

\frac{d}{dt} \int_A \tilde{h}_k \, \overline{\left<\varphi\right>}^{\tilde{z}}_k \, dA
& + \int_{\partial A} \left( \tilde{h}_k \, \left[\overline{\left<\varphi\right> \left<{\bf u}\right> + \left<\varphi^{\prime}{\bf u}^{\prime}\right>}^{\tilde{z}}_k \right] \right) \cdot {\bf n}_\perp \, dl & \\
& = 0.

The second term is expanded utilizing [](#vertical-decomposition)

& + \int_{\partial A} \left( \tilde{h}_k \, \overline{\left(\overline{\left<\varphi\right>}^{\tilde{z}}_k + \delta \varphi\right)\left(\overline{\left<{\bf u}\right>}^{\tilde{z}}_k + \delta {\bf u}\right)}^{\tilde{z}}_k + \tilde{h}_k \overline{\left<\varphi^{\prime}{\bf u}^{\prime}\right>}^{\tilde{z}}_k \right) \cdot {\bf n}_\perp \, dl & \\
& = 0.

& + \int_{\partial A} \left[ \tilde{h}_k \, \left(\overline{\left<\varphi\right>}^{\tilde{z}}_k \overline{\left<{\bf u}\right>}^{\tilde{z}}_k + \overline{\delta \varphi \delta {\bf u}}^{\tilde{z}}_k + \overline{\left<\varphi^{\prime}{\bf u}^{\prime}\right>}^{\tilde{z}}_k \right)\right] \cdot {\bf n}_\perp \, dl & \\
& = 0.

Given that the horizontal area is not changing in time, we can move the time derivative through the first integral.  We also invoke Green's Theorem for the second integral to convert that from a surface integral to an area integral.  With these changes, the equation becomes

Given that the horizontal area is not changing in time, we can move the time derivative through the first integral.  We also invoke Green’s Theorem for the second integral to convert that from a surface integral to an area integral.  With these changes, the equation becomes

\int_A  \bigl\{ \frac{\partial \tilde{h}_k \, \overline{\left<\varphi\right>}^{\tilde{z}}_k}{\partial t}
& + \nabla \cdot \left[ \tilde{h}_k \, \left(\overline{\left<\varphi\right>}^{\tilde{z}}_k \overline{\left<{\bf u}\right>}^{\tilde{z}}_k + \overline{\delta \varphi \delta {\bf u}}^{\tilde{z}}_k + \overline{\left<\varphi^{\prime}{\bf u}^{\prime}\right>}^{\tilde{z}}_k \right)\right] & \\
& + \left\{ \rho_0 \left[ \left<\varphi\right>\left<\tilde{w}_{tr}\right> + \left<\varphi^{\prime} \tilde{w}_{tr}^\prime \right> - \left<\varphi\right> \left<\tilde{ u}\right> - \left<\varphi^{\prime} \tilde{ u}^{\prime}\right> \right] \right\}_{\tilde{z} = \tilde{z}_k^{\text{top}}} & \\
& - \left\{ \rho_0 \left[ \left<\varphi\right>\left<\tilde{w}_{tr}\right> + \left<\varphi^{\prime} \tilde{w}_{tr}^\prime \right> - \left<\varphi\right> \left<\tilde{ u}\right> - \left<\varphi^{\prime} \tilde{ u}^{\prime}\right> \right] \right\}_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \bigr\}  \, dA
& = 0.

A few comments on the last two lines of [](#reynolds-tracer-final).  If only the first two terms of the integrals are retained, this represents the vertical advective and turbulent fluxes for a finite volume framework.  The second two terms represent the projection of the horizontal advective and turbulent fluxes into the direction normal to the sloping surface.  When the coordinate surfaces are flat $\nabla \tilde{z}^{\text{top}} = \nabla \tilde{z}^{\text{bot}} = 0$ and the vertical fluxes are aligned with the normal to the coordinate surface reducing the equation to the expected z-coordinate, finite volume, formulation.

.  If only the first two terms of the integrals are retained, this represents the vertical advective and turbulent fluxes for a finite volume framework.  The second two terms represent the projection of the horizontal advective and turbulent fluxes into the direction normal to the sloping surface.  When the coordinate surfaces are flat 

\nabla \tilde{z}^{\text{top}} = \nabla \tilde{z}^{\text{bot}} = 0

 and the vertical fluxes are aligned with the normal to the coordinate surface reducing the equation to the expected z-coordinate, finite volume, formulation.

Finally, given that the area integral operates on the entire equation, it is valid for any area and we can take the limit as $A \rightarrow 0$ to arrive at the final tracer equation

Finally, given that the area integral operates on the entire equation, it is valid for any area and we can take the limit as 

\frac{\partial {\tilde h}_k \overline{\left<\varphi\right>}^{\tilde{z}}_k  }{\partial t}  & + \nabla \cdot \left({\tilde h}_k \overline{\left<\varphi\right>}^{\tilde{z}}_k \overline{\left<{\bf u}\right>}^{\tilde{z}}_k\right) \\
& + \rho_0 \left\{ \left[ \left<\varphi\right> \left<{\tilde w}_{tr}\right> - \left<\varphi\right>\left<\tilde{ u}\right> \right]_{{\tilde z}={\tilde z}_k^{\text{top}}} - \left[ \left<\varphi\right> \left<{\tilde w}_{tr}\right> - \left<\varphi\right>\left<\tilde{ u}\right> \right]_{{\tilde z}={\tilde z}_k^{\text{bot}}} \right\} \\
& = - \nabla \cdot \left({\tilde h}_k \overline{\left<\varphi^{\prime} {\bf u}^{\prime}\right>}^{\tilde{z}}_k \, + {\tilde h}_k \overline{\delta \varphi \delta {\bf u}}^{\tilde{z}}_k \right) \\
& - \rho_0 \left\{ \left[\left< \varphi^\prime {\tilde w}_{tr}^{\prime} \right> - \left< \varphi^\prime \tilde{ u}^{\prime}\right> \right]_{{\tilde z}={\tilde z}_{k}^{\text{top}}} - \left[\left< \varphi^\prime {\tilde w}_{tr}^{\prime} \right> - \left< \varphi^\prime \tilde{ u}^{\prime}\right> \right]_{{\tilde z}={\tilde z}_{k}^{\text{bot}}}\right\}.

A few notes on the layer-averaged tracer equation. In this complete form, it includes three types of fluctuating quantities that must be dealt with: (1) the layer averaged, Reynolds' averaged horizontal turbulent flux $\left( \overline{\left<\varphi^{\prime} {\bf u}^{\prime}\right>}^{\tilde{z}}_k \right)$, (2) the Reynolds' average vertical turbulent flux $\left( \left< \varphi^\prime {\tilde w}_{tr}^{\prime} \right> - \left< \varphi^\prime \tilde{u}^{\prime}\right> \right)$, and (3) the layer-averaged product of deviations from the layer-integrated variables $\left(\overline{\delta \varphi \delta {\bf u}}^{\tilde{z}}_k \right)$. The details of the first two quantities will be discussed later in this document and follow-on design documents. The terms involving perturbations from the layer integrated quantity are necessary to extend beyond piecewise constant represenation of variables. In this equation, variables with no overline are the full field variable at the interfaces.

A few notes on the layer-averaged tracer equation. In this complete form, it includes three types of fluctuating quantities that must be dealt with: (1) the layer averaged, Reynolds’ averaged horizontal turbulent flux 

\left( \overline{\left<\varphi^{\prime} {\bf u}^{\prime}\right>}^{\tilde{z}}_k \right)

\left( \left< \varphi^\prime {\tilde w}_{tr}^{\prime} \right> - \left< \varphi^\prime \tilde{u}^{\prime}\right> \right)

, and (3) the layer-averaged product of deviations from the layer-integrated variables 

\left(\overline{\delta \varphi \delta {\bf u}}^{\tilde{z}}_k \right)

. The details of the first two quantities will be discussed later in this document and follow-on design documents. The terms involving perturbations from the layer integrated quantity are necessary to extend beyond piecewise constant represenation of variables. In this equation, variables with no overline are the full field variable at the interfaces.

The mass equation is identical to the tracer equation with $\varphi=1$.

\frac{\partial {\tilde h}_k }{\partial t}
+ \nabla \cdot \left({\tilde h}_k \overline{\left<{\bf u}\right>}^{\tilde{z}}_k\right)
+ \rho_0 \left[ \left<{\tilde w}_{tr}\right> - \left<\tilde{ u}\right> \right]_{{\tilde z}={\tilde z}_k^{\text{top}}}
- \rho_0 \left[ \left<{\tilde w}_{tr}\right> - \left<\tilde{ u}\right> \right]_{{\tilde z}={\tilde z}_{k}^{\text{bot}}}
= 0.

We now derive the horizontal momentum equation in our non-Boussinesq, hydrostatic framework, following the same finite-volume approach used for mass and tracer conservation.

We begin from [](#vh-momentum-pseudo) and specify the body forces ${\bf b}$ and surface forces ${\bf \tau}$

\frac{d}{dt} \int_A \int_{\tilde{z}_k^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} {\bf u} \, d\tilde{z} \, dA
& + \int_{\partial A} \left( \int_{\tilde{z}_k^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} {\bf u} \otimes {\bf u} \, d\tilde{z} \right) \cdot {\bf n}_\perp \, dl \\
& + \int_A \rho_0 \, {\bf u} \left[ \tilde{w}_{tr} - \tilde{\bf u} \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \, dA
- \int_A \rho_0 \, {\bf u} \left[ \tilde{w}_{tr} - \tilde{\bf u} \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \, dA \\
& = -\int_A \int_{\tilde{z}_k^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \rho_0 \left({\bf f} \times \mathbf{u} + \nabla \Phi \right) \, d\tilde{z} \, dA
- \int_{\partial A} \left( \int_{\tilde{z}_k^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \frac{\rho_0 p}{\rho} \, d\tilde{z} \right) dl \\
& - \int_A \left[ \frac{\rho_0 p}{\rho} \nabla \tilde{z}_k^{\text{top}} \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \, dA
+ \int_A \left[ \frac{\rho_0 p}{\rho} \nabla \tilde{z}_k^{\text{bot}} \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \, dA.

The first term on the right hand side is the **Coriolis force**, where $ \mathbf{f} $ is the vector Coriolis vector (e.g., $ f \hat{\mathbf{z}} $ on the sphere).

The second term on the right hand side represents the **gravitational force**, expressed in terms of the two-dimensional gradient of the gravitational potential $ \Phi(x, y, z, t) $, which may include effects such as tides and self-attraction and loading.  This gradient is along layer (e.g. constant $\tilde{z}$) such that the gradient of $\Phi$ doesn't disappear.

, expressed in terms of the two-dimensional gradient of the gravitational potential 

, which may include effects such as tides and self-attraction and loading.  This gradient is along layer (e.g. constant 

The final terms are the **pressure force**, which acts on the boundary surfaces and is naturally expressed as a surface integral. It gives rise to both horizontal pressure gradients and contributions from sloping surfaces.

, which acts on the boundary surfaces and is naturally expressed as a surface integral. It gives rise to both horizontal pressure gradients and contributions from sloping surfaces.

A negative sign is included for the pressure terms derived from the surface force terms. According to [Leishman 2025](https://eaglepubs.erau.edu/introductiontoaerospaceflightvehicles/chapter/conservation-of-momentum-momentum-equation/#chapter-260-section-2), Chapter 21, equation 2, "the negative sign indicates that the force is inward and in the opposite direction to the unit normal vector area, which always points *outward* by convention". Also see discussion in [Kundu et al. 2016](https://doi.org/10.1016/C2012-0-00611-4) page 104 and equation 4.26.

A negative sign is included for the pressure terms derived from the surface force terms. According to 

, Chapter 21, equation 2, “the negative sign indicates that the force is inward and in the opposite direction to the unit normal vector area, which always points 

As with the tracer derivation, we next Reynolds' average [](#vh-momentum-forces),

\frac{d}{dt} \int_A \int_{\tilde{z}_k^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \left< {\bf u} \right> \, d\tilde{z} \, dA
& + \int_{\partial A} \left( \int_{\tilde{z}^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \left< {\bf u} \otimes {\bf u} \right> \, d\tilde{z} \right) \cdot {\bf n}_\perp \, dl \\
& + \int_A \rho_0 \,\left< {\bf u} \left[ \tilde{w}_{tr} - \tilde{\bf u} \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \right> \, dA
- \int_A \rho_0 \, \left< {\bf u} \left[ \tilde{w}_{tr} - \tilde{\bf u} \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \right> \, dA \\
& = -\int_A \int_{\tilde{z}_k^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \rho_0 \, \left<\left( {\bf f} \times \mathbf{u} + \nabla \Phi \right) \right> \, d\tilde{z} \, dA
- \int_{\partial A} \left( \int_{\tilde{z}_k^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \rho_0 \left< \frac{p}{\rho} \right> \, d\tilde{z} \right) dl \\
& - \int_A \rho_0 \left[ \left< \frac{p}{\rho} \nabla \tilde{z}_k^{\text{top}} \right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \, dA
+ \int_A \rho_0 \left[ \left< \frac{p}{\rho} \nabla \tilde{z}_k^{\text{bot}} \right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \, dA.

Here we have also moved the Reynolds' average through the spatial integrals given the properties of the averaging.  Next we do a Reynolds' decomposition, this yields

Here we have also moved the Reynolds’ average through the spatial integrals given the properties of the averaging.  Next we do a Reynolds’ decomposition, this yields

& + \int_{\partial A} \left( \int_{\tilde{z}_k^{\text{bot}}}^{\tilde{z}^{\text{top}}} \left( \left< {\bf u} \right> \otimes \left< {\bf u} \right> + \left< {\bf u}^\prime \otimes {\bf u}^\prime \right> \right) \, d\tilde{z} \right) \cdot {\bf n}_\perp \, dl \\
& + \int_A \rho_0 \, \left[ \left(\left< {\bf u} \right> \left< \tilde{w}_{tr} \right> + \left<{\bf u}^\prime \tilde{w}_{tr}^\prime \right> \right) - \left(\left<{\bf u}\right> \left<\tilde{\bf u}\right> + \left< {\bf u}^\prime \tilde{\bf u}^\prime \right> \right) \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \, dA \\
& - \int_A \rho_0 \, \left[ \left(\left< {\bf u} \right> \left< \tilde{w}_{tr} \right> + \left<{\bf u}^\prime \tilde{w}_{tr}^\prime \right> \right) - \left(\left<{\bf u}\right> \left<\tilde{\bf u}\right> + \left< {\bf u}^\prime \tilde{\bf u}^\prime \right> \right) \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \, dA  \\
& = - \int_A \int_{\tilde{z}_k^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \rho_0 \, \left( {\bf f} \times \left<\mathbf{u}\right> + \nabla \left<\Phi\right> \right) \, d\tilde{z} \, dA \\
& - \int_{\partial A} \left( \int_{\tilde{z}_k^{\text{bot}}}^{\tilde{z}_k^{\text{top}}} \rho_0 \left(\left< \alpha \right> \left<p \right> + \left<\alpha^\prime p^\prime\right> \right) \, d\tilde{z} \right) dl \\
& - \int_A \rho_0 \left[ \left< \alpha \right> \left<p \nabla \tilde{z}_k^{\text{top}} \right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{top}}\right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \, dA \\
& + \int_A \rho_0 \left[ \left< \alpha \right> \left<p \nabla \tilde{z}_k^{\text{bot}} \right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{bot}}\right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \, dA.

In [](#vh-momentum-reynolds2) we have also used $\alpha = \frac{1}{\rho}$ for notation conciseness.  The definition of the layer average [](#def-layer-average-reynolds) is now utilized on terms with vertical integrals to yield

 for notation conciseness.  The definition of the layer average 

\frac{d}{dt} \int_A \tilde{h}_k \overline{\left< {\bf u} \right>}^{\tilde{z}}_k  \, dA
& + \int_{\partial A} \tilde{h}_k \left( \overline{\left< {\bf u} \right> \otimes \left< {\bf u} \right>}^{\tilde{z}}_k + \overline{\left< {\bf u}^\prime \otimes {\bf u}^\prime \right>}^{\tilde{z}}_k  \right) \cdot {\bf n}_\perp \, dl \\
& = - \int_A \rho_0 \, \tilde{h}_k \overline{\left( {\bf f} \times \left<\mathbf{u}\right> + \nabla \left<\Phi\right> \right)}^{\tilde{z}}_k \, dA \\
& - \int_{\partial A} \rho_0 \tilde{h}_k \left( \overline{\left< \alpha \right> \left<p \right>}^{\tilde{z}}_k + \overline{\left<\alpha^\prime p^\prime\right>}^{\tilde{z}}_k \right) dl \\
& - \int_A \rho_0 \left[ \left< \alpha \right> \left<p \nabla \tilde{z}_k^{\text{top}} \right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{top}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \, dA \\
& + \int_A \rho_0 \left[ \left< \alpha \right> \left<p \nabla \tilde{z}_k^{\text{bot}} \right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{bot}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \, dA.

The next step is to decompose vertical averages of products. This yields

\frac{d}{dt} \int_{A} \tilde{h}_k \overline{\left< {\bf u} \right>}^{\tilde{z}}_k  \, dA
& + \int_{\partial A} \tilde{h}_k \left( \overline{\left< {\bf u} \right>}^{\tilde{z}}_k \otimes \overline{\left< {\bf u} \right>}^{\tilde{z}}_k + \overline{\delta {\bf u} \otimes \delta {\bf u}}^{\tilde{z}}_k + \overline{\left< {\bf u}^\prime \otimes {\bf u}^\prime \right>}^{\tilde{z}}_k  \right) \cdot {\bf n}_\perp \, dl \\
& - \int_{\partial A} \rho_0 \tilde{h}_k \left( \overline{\left< \alpha \right>}^{\tilde{z}}_k \overline{\left<p \right>}^{\tilde{z}}_k + \overline{\delta \alpha \delta p}^{\tilde{z}}_k+ \overline{\left<\alpha^\prime p^\prime\right>}^{\tilde{z}}_k \right) dl \\
& - \int_A \rho_0 \left[\left< \alpha \right> \left<p \nabla \tilde{z}_k^{\text{top}} \right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{top}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \, dA \\
& + \int_A \rho_0 \left[\left< \alpha \right> \left<p \nabla \tilde{z}_k^{\text{bot}} \right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{bot}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \, dA.

We next use the Divergence theorem on the surface integrals and combine terms into fewer integrals on each side of the equation.

\int_A \bigl\{ \frac{\partial\tilde{h}_k \overline{\left< {\bf u} \right>}^{\tilde{z}}_k}{\partial t}
& + \nabla \cdot \left[ \tilde{h}_k \left(\overline{\left< {\bf u} \right>}^{\tilde{z}}_k \otimes \overline{\left< {\bf u} \right>}^{\tilde{z}}_k + \overline{\delta {\bf u} \otimes \delta {\bf u}}^{\tilde{z}}_k + \overline{\left< {\bf u}^\prime \otimes {\bf u}^\prime \right>}^{\tilde{z}}_k  \right) \right] \\
& + \rho_0 \left[ \left(\left< {\bf u} \right> \left< \tilde{w}_{tr} \right> + \left<{\bf u}^\prime \tilde{w}_{tr}^\prime \right> \right) - \left(\left<{\bf u}\right> \left<\tilde{\bf u}\right> + \left< {\bf u}^\prime \tilde{\bf u}^\prime \right> \right) \right]_{\tilde{z} = \tilde{z}^{\text{top}}} \\
& - \rho_0 \left[ \left(\left< {\bf u} \right> \left< \tilde{w}_{tr} \right> + \left<{\bf u}^\prime \tilde{w}_{tr}^\prime \right> \right) - \left(\left<{\bf u}\right> \left<\tilde{\bf u}\right> + \left< {\bf u}^\prime \tilde{\bf u}^\prime \right> \right) \right]_{\tilde{z} = \tilde{z}^{\text{bot}}} \bigr\} \, dA \\
& = - \int_A \bigl\{ \rho_0 \, \tilde{h}_k \overline{\left( {\bf f} \times \left<\mathbf{u}\right> + \nabla \left<\Phi\right> \right)}^{\tilde{z}}_k \\
& - \nabla \left[ \rho_0 \tilde{h}_k \left( \overline{\left< \alpha \right>}^{\tilde{z}}_k \overline{\left<p \right>}^{\tilde{z}}_k + \overline{\delta \alpha \delta p}^{\tilde{z}}_k+ \overline{\left<\alpha^\prime p^\prime\right>}^{\tilde{z}}_k \right) \right] \\
& - \rho_0 \left[ \left< \alpha \right> \left<p \nabla \tilde{z}_k^{\text{top}} \right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{top}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \\
& + \rho_0 \left[ \left< \alpha \right> \left<p \nabla \tilde{z}_k^{\text{bot}} \right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{bot}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}}\bigr\} \, dA.

Since the equation is fully inside the integral, the equation is true for any area and therefore we can write the layer averaged momentum equation as

\frac{\partial\tilde{h}_k \overline{\left< {\bf u} \right>}^{\tilde{z}}_k}{\partial t}
& + \nabla \cdot \left[ \tilde{h}_k \left( \overline{\left< {\bf u} \right>}^{\tilde{z}}_k \otimes \overline{\left< {\bf u} \right>}^{\tilde{z}}_k + \overline{\delta {\bf u} \otimes \delta {\bf u}}^{\tilde{z}}_k + \overline{\left< {\bf u}^\prime \otimes {\bf u}^\prime \right>}^{\tilde{z}}_k  \right) \right] \\
& + \rho_0 \left[ \left(\left< {\bf u} \right> \left< \tilde{w}_{tr} \right> + \left<{\bf u}^\prime \tilde{w}_{tr}^\prime \right> \right) - \left(\left<{\bf u}\right> \left<\tilde{\bf u}\right> + \left< {\bf u}^\prime \tilde{\bf u}^\prime \right> \right) \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \\
& - \rho_0 \left[ \left(\left< {\bf u} \right> \left< \tilde{w}_{tr} \right> + \left<{\bf u}^\prime \tilde{w}_{tr}^\prime \right> \right) - \left(\left<{\bf u}\right> \left<\tilde{\bf u}\right> + \left< {\bf u}^\prime \tilde{\bf u}^\prime \right> \right) \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \\
& = - \rho_0 \, \tilde{h}_k \overline{\left( {\bf f} \times \left<\mathbf{u}\right> + \nabla \left<\Phi\right> \right)}^{\tilde{z}}_k \\
& - \rho_0 \left[ \left< \alpha \right> \left<p \cdot \nabla \tilde{z}_k^{\text{top}}\right> + \left<\alpha^\prime \left(p \cdot \nabla \tilde{z}_k^{\text{top}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \\
& + \rho_0 \left[ \left< \alpha \right> \left<p \cdot \nabla \tilde{z}_k^{\text{bot}}\right> + \left<\alpha^\prime \left(p \cdot \nabla \tilde{z}_k^{\text{bot}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}}.

The product rule is used on the first two terms of [](#vh-momentum-v1) and then we multiply [](#layer-mass) by $\overline{\mathbf{u}}^{\tilde{z}}_k$ and subtract it from [](#vh-momentum-v1).  This yields

\tilde{h}_k \frac{\partial \overline{\left< {\bf u} \right>}^{\tilde{z}}_k}{\partial t}
& + \tilde{h}_k \overline{\left< {\bf u} \right>}^{\tilde{z}}_k \cdot \nabla \overline{\left< {\bf u} \right>}^{\tilde{z}}_k + \nabla \cdot \left[ \tilde{h}_k \left(\overline{\delta {\bf u} \otimes \delta {\bf u}}^{\tilde{z}}_k + \overline{\left< {\bf u}^\prime \otimes {\bf u}^\prime \right>}^{\tilde{z}}_k  \right) \right] \\
& + \rho_0 \left(\left<\mathbf{u}\right> - \overline{\left<\mathbf{u}\right>}^{\tilde{z}}_k\right) \left\{ \left[ \left<\tilde{w}_{tr}\right> - \left<\tilde{\bf u}\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} - \left[ \left<\tilde{w}_{tr}\right> - \left<\tilde{\bf u}\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \right\} \\
& + \rho_0 \left\{ \left[ \left<\mathbf{u}^\prime \tilde{w}_{tr}^\prime \right> - \left< \mathbf{u}^\prime \tilde{\bf u}^\prime \right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} - \left[ \left<\mathbf{u}^\prime \tilde{w}_{tr}^\prime \right> - \left< \mathbf{u}^\prime \tilde{\bf u}^\prime \right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \right\} \\
& +  \rho_0 \left[ \left< \alpha \right> \left<p \nabla \tilde{z}_k^{\text{bot}} \right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{bot}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}}.

\frac{\partial \overline{\left< {\bf u} \right>}^{\tilde{z}}_k}{\partial t}
& + \overline{\left< {\bf u} \right>}^{\tilde{z}}_k \cdot \nabla \overline{\left< {\bf u} \right>}^{\tilde{z}}_k + \frac{1}{\tilde{h}_k} \nabla \cdot \left[ \tilde{h}_k \left(\overline{\delta {\bf u} \otimes \delta {\bf u}}^{\tilde{z}}_k + \overline{\left< {\bf u}^\prime \otimes {\bf u}^\prime \right>}^{\tilde{z}}_k  \right) \right] \\
& + \frac{\rho_0}{\tilde{h}_k} \left(\left<\mathbf{u}\right> - \overline{\left<\mathbf{u}\right>}^{\tilde{z}}_k\right) \left\{ \left[ \left<\tilde{w}_{tr}\right> - \left<\tilde{\bf u}\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} - \left[ \left<\tilde{w}_{tr}\right> - \left<\tilde{\bf u}\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \right\} \\
& + \frac{\rho_0}{\tilde{h}_k} \left\{ \left[ \left<\mathbf{u}^\prime \tilde{w}_{tr}^\prime \right> - \left< \mathbf{u}^\prime \tilde{\bf u}^\prime \right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} - \left[ \left<\mathbf{u}^\prime \tilde{w}_{tr}^\prime \right> - \left< \mathbf{u}^\prime \tilde{\bf u}^\prime \right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \right\} \\
& = - \rho_0 \, \overline{\left( {\bf f} \times \left<\mathbf{u}\right> + \nabla \left<\Phi\right> \right)}^{\tilde{z}}_k \\
& - \frac{1}{\tilde{h}_k} \nabla \left[ \rho_0 \tilde{h}_k \left( \overline{\left< \alpha \right>}^{\tilde{z}}_k \overline{\left<p \right>}^{\tilde{z}}_k + \overline{\delta \alpha \delta p}^{\tilde{z}}_k+ \overline{\left<\alpha^\prime p^\prime\right>}^{\tilde{z}}_k \right) \right] \\
& - \frac{\rho_0}{\tilde{h}_k} \left[ \left< \alpha \right> \left<p \nabla \tilde{z}^{\text{top}}\right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{top}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} \\
& + \frac{\rho_0}{\tilde{h}_k} \left[ \left< \alpha \right> \left<p \nabla \tilde{z}^{\text{bot}}\right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{bot}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}}.

The $\overline{\bf u}^{\tilde{z}}_k \cdot \nabla \overline{\bf u}^{\tilde{z}}_k$ may be replaced with the vector identity

\overline{\bf u}^{\tilde{z}}_k \cdot \nabla \overline{\bf u}^{\tilde{z}}_k

\begin{aligned}
\overline{\left<{\bf u}\right>}^{\tilde{z}}_k \cdot \nabla \overline{\left<{\bf u}\right>}^{\tilde{z}}_k
& = (\nabla \times \overline{\left<{\bf u}\right>}^{\tilde{z}}_k) \times \overline{\left<{\bf u}\right>}^{\tilde{z}}_k + \nabla \frac{\left|\overline{\left<{\bf u}\right>}^{\tilde{z}}_k\right|^2}{2} \\
& = \left( \hat{k} \cdot (\nabla_\perp\times \overline{\left<{\bf u}\right>}^{\tilde{z}}_k)\right)
\left( \hat{k} \times \overline{\left<{\bf u}\right>}^{\tilde{z}}_k \right) + \nabla_\perp\frac{\left|\overline{\left<{\bf u}\right>}^{\tilde{z}}_k\right|^2}{2} \\
& = \zeta {\overline{\left<{\bf u}\right>}^{\tilde{z}}_k}^{\perp} + \nabla K.
\end{aligned}

where $\zeta$ is relative vorticity and $K$ is kinetic energy.  This step separates the horizontal advection into non-divergent and non-rotational components, which is useful in the final TRiSK formulation.

 is kinetic energy.  This step separates the horizontal advection into non-divergent and non-rotational components, which is useful in the final TRiSK formulation.

The Coriolis term in [](#vh-momentum-v3), when projected into a local coordinate system can be written as

, when projected into a local coordinate system can be written as

{\bf f} \times \left<\mathbf{u}\right> = f \left<\mathbf{u}\right>^\perp + 2 \Omega \tilde{w}_{tr} \cos \phi

We expect the second term above to be negligible for the ocean.  Ocean vertical velocities are largest ($\approx 10$ cm) in the high latitudes, where $\cos \phi$ is small.  Further, these large vertical velocities only arise in non-hydrostatic ocean models.  Therefore we expect this second term to always be small in Omega and will neglect it.

We expect the second term above to be negligible for the ocean.  Ocean vertical velocities are largest (

 is small.  Further, these large vertical velocities only arise in non-hydrostatic ocean models.  Therefore we expect this second term to always be small in Omega and will neglect it.

Defining the absolute vorticity $\zeta_a$ as $\zeta + f$, where $f$ is the Coriolis parameter, which results from [](#coriolis-expansion), [](#vh-momentum-v3) becomes,

& + \zeta_a {\overline{\left<{\bf u}\right>}^{\tilde{z}}_k}^{\perp} + \nabla K + \frac{1}{\tilde{h}_k} \nabla \cdot \left[ \tilde{h}_k \left(\overline{\delta {\bf u} \otimes \delta {\bf u}}^{\tilde{z}}_k + \overline{\left< {\bf u}^\prime \otimes {\bf u}^\prime \right>}^{\tilde{z}}_k  \right) \right] \\
& + \frac{\rho_0}{\tilde{h}_k} \left\{ \left[\left(\left<\mathbf{u}\right> - \overline{\left<\mathbf{u}\right>}^{\tilde{z}}_k\right) \left\{\left<\tilde{w}_{tr}\right> - \left<\tilde{\bf u}\right> \right\} \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} - \left[  \left(\left<\mathbf{u}\right> - \overline{\left<\mathbf{u}\right>}^{\tilde{z}}_k\right) \left\{\left<\tilde{w}_{tr}\right> - \left<\tilde{\bf u}\right> \right\} \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \right\} \\
& + \frac{\rho_0}{\tilde{h}_k}  \left\{ \left[ \left<\mathbf{u}^\prime \tilde{w}_{tr}^\prime \right> - \left< \mathbf{u}^\prime \tilde{\bf u}^\prime \right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} - \left[ \left<\mathbf{u}^\prime \tilde{w}_{tr}^\prime \right> - \left< \mathbf{u}^\prime \tilde{\bf u}^\prime \right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \right\} \\
& = - \rho_0 \, \overline{\nabla \left<\Phi\right>}^{\tilde{z}}_k - \frac{1}{\tilde{h}_k} \nabla \left[ \rho_0 \tilde{h}_k \left( \overline{\left< \alpha \right>}^{\tilde{z}}_k \overline{\left<p \right>}^{\tilde{z}}_k + \overline{\delta \alpha \delta p}^{\tilde{z}}_k+ \overline{\left<\alpha^\prime p^\prime\right>}^{\tilde{z}}_k \right) \right] \\
& - \frac{\rho_0}{\tilde{h}_k} \left\{ \left[ \left< \alpha \right> \left<p \nabla \tilde{z}_k^{\text{top}}\right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{top}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}}
-  \left[ \left< \alpha \right> \left<p \nabla \tilde{z}_k^{\text{bot}}\right> + \left<\alpha^\prime \left(p \nabla \tilde{z}_k^{\text{bot}} \right)^\prime\right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \right\}.

The $\delta X$ terms in the layered equations are vertical deviations from the vertical layer average of a given quantity $X$.  We will assume these deviations are small and products of these deviations are even smaller.  Thus we will ignore most of these terms in Omega.  The exception is the pressure gradient force.  At this time we will assume piecewise constant approximations of the vertically continuous system, which is appropriate for the simple pressure gradient force targeted for early versions of Omega.  This assumption will be revisited at a later date.  We also note that these terms could potentially serve as a bridge to multiscale fluxes, as resolution is increased, these $\delta X$ terms would get larger, but likely only for significantly higher resolution (e.g. 10s of meters).  However, these terms would have to be further analyzed and developed as these terms are only deviations from layer averages, not temporal averages as in the Reynolds' approach.

 terms in the layered equations are vertical deviations from the vertical layer average of a given quantity 

.  We will assume these deviations are small and products of these deviations are even smaller.  Thus we will ignore most of these terms in Omega.  The exception is the pressure gradient force.  At this time we will assume piecewise constant approximations of the vertically continuous system, which is appropriate for the simple pressure gradient force targeted for early versions of Omega.  This assumption will be revisited at a later date.  We also note that these terms could potentially serve as a bridge to multiscale fluxes, as resolution is increased, these 

 terms would get larger, but likely only for significantly higher resolution (e.g. 10s of meters).  However, these terms would have to be further analyzed and developed as these terms are only deviations from layer averages, not temporal averages as in the Reynolds’ approach.

Drop the $< >$ notation around single variables and note that all variables are assumed to be Reynolds' averaged.  Turbulent correlations will retain the angle bracket notation.

 notation around single variables and note that all variables are assumed to be Reynolds’ averaged.  Turbulent correlations will retain the angle bracket notation.

Change the vertical density weighted average notation from $\overline{\varphi}^{\tilde{z}}_k$ to the more simple $\varphi_k$.  Thus, any subscript $k$ implies a layer average.

Define a total vertical velocity across the pseudo height surface as $\tilde{W}_{tr} \equiv \tilde{w}_{tr} - \tilde{u}$.  As a reminder $\tilde{u}$ is the projection of the normal velocity onto the normal vector to the pseudo height surface, in many cases this can be a very small correction to $\tilde{w}_{tr}$.

Define a total vertical velocity across the pseudo height surface as 

 is the projection of the normal velocity onto the normal vector to the pseudo height surface, in many cases this can be a very small correction to 

With these three simplifications, the final continuous equations become

\frac{\partial {\tilde h}_k \varphi_k}{\partial t}  & + \nabla \cdot \left({\tilde h}_k \varphi_k {\bf u}_k\right) + \rho_0 \left\{ \left[ \varphi \tilde{W}_{tr} \right]_{{\tilde z}={\tilde z}_k^{\text{top}}} - \left[ \varphi \tilde{W}_{tr} \right]_{{\tilde z}={\tilde z}_{k}^{\text{bot}}} \right\} \\
& = - \nabla \cdot \left({\tilde h}_k \left<\varphi^{\prime} {\bf u}^{\prime}\right>_k\right) \\

+ \nabla \cdot \left({\tilde h}_k {\bf u}_k\right)
+ \rho_0 \left[ \tilde{W}_{tr} \right]_{{\tilde z}={\tilde z}_{k}^{\text{top}}}
- \rho_0 \left[ \tilde{W}_{tr} \right]_{{\tilde z}={\tilde z}_{k}^{\text{bot}}}
= 0.

\frac{\partial {\bf u}_k}{\partial t}
& + \zeta_a {{\bf u}_k}^{\perp} + \nabla K + \frac{1}{\tilde{h}_k} \nabla \cdot \left[ \tilde{h}_k \left< {\bf u}^\prime \otimes {\bf u}^\prime \right>_k \right] \\
& + \frac{\rho_0}{\tilde{h}_k} \left\{ \left[\left(\mathbf{u} - \mathbf{u}_k\right) \left\{\tilde{W}_{tr} \right\} \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} - \left[  \left(\mathbf{u} - \mathbf{u}_k\right) \left\{\tilde{W}_{tr} \right\} \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \right\} \\
& + \frac{\rho_0}{\tilde{h}_k}  \left\{ \left[ \left<\mathbf{u}^\prime \tilde{w}_{tr}^\prime \right> - \left< \mathbf{u}^\prime \tilde{ u}^\prime \right> \right]_{\tilde{z} = \tilde{z}_k^{\text{top}}} - \left[ \left<\mathbf{u}^\prime \tilde{w}_{tr}^\prime \right> - \left< \mathbf{u}^\prime \tilde{ u}^\prime \right> \right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \right\} \\
& = - \left(\nabla \Phi \right)_k - \frac{1}{\tilde{h}_k} \nabla \left[ \tilde{h}_k \alpha_k p_k \right] \\
& - \frac{1}{\tilde{h}_k} \left\{ \left[ \alpha p \nabla \tilde{z}_k^{\text{top}}\right]_{\tilde{z} = \tilde{z}_k^{\text{top}}}
-  \left[ \alpha p \nabla \tilde{z}_k^{\text{bot}}\right]_{\tilde{z} = \tilde{z}_k^{\text{bot}}} \right\}.

In [](#layer-tracer-final-simple) and [](#layer-momentum-final-simple) we have not combined turbulent flux terms (e.g., $\left<\varphi^\prime \tilde{W}_{tr}^\prime \right>$) as the two terms that comprise this term ($\left< \varphi^\prime {\tilde w}_{tr}^{\prime} \right>$ and $\left< \varphi^\prime \tilde{ u}^{\prime}\right>$) are fundamentally different processes that must be modeled separately.

) are fundamentally different processes that must be modeled separately.

In the following equations, the subscripts $i$, $e$, and $v$ indicate cell, edge, and vertex locations and subscript $k$ is the layer.  Square brackets $[\cdot]_e$ and $[\cdot]_v$ are quantities that are interpolated to edge and vertex locations. For vector quantities, $u_{e,k}$ denotes the normal component at the center of the edge (where the normal component is the local ${\bf n}_\perp$ described in above sections), while $u_{e,k}^\perp$ denotes the tangential component. We have switched from $\varphi_{i,k}^{bot}$ to the identical $\varphi_{i,k+1}^{top}$ for all variables in order for the notation to match the array names in the code. It is important to note that any term without a subscript $k$ are quantities evaluated at that location and are ***not*** individual layer averages, but will be reconstructed as a function of neighboring layer average values.  For example, Eq (44) of [White and Adcroft (2008)](https://www.sciencedirect.com/science/article/pii/S0021999108002593) shows a third order reconstruction.  Finally, we note that in the equations below, it is assumed that $k$ increases away from the surface.

 are quantities that are interpolated to edge and vertex locations. For vector quantities, 

 denotes the normal component at the center of the edge (where the normal component is the local 

 for all variables in order for the notation to match the array names in the code. It is important to note that any term without a subscript 

 individual layer averages, but will be reconstructed as a function of neighboring layer average values.  For example, Eq (44) of 

 shows a third order reconstruction.  Finally, we note that in the equations below, it is assumed that 

\frac{\partial {\tilde h}_{i,k} }{\partial t}
+ \nabla \cdot \left(\left[{\tilde h}_k\right]_e {\bf u}_{e,k}\right)
+ \rho_0 \left[ \tilde{W}_{tr} \right]_k^\text{top}
- \rho_0 \left[ \tilde{W}_{tr} \right]_{k+1}^\text{top}
= 0.

In this equation, mass source term ($Q$), such as sources like river runoff, sea ice freshwater fluxes, precipitation, and evaporation are bound up in the surface value of $\tilde{W}_tr^{\text{top}}$.  The mass sources will be normalized by $\rho_0$ to achieve consistent units.

), such as sources like river runoff, sea ice freshwater fluxes, precipitation, and evaporation are bound up in the surface value of 

\frac{\partial {\tilde h}_{i,k} \varphi_{i,k}}{\partial t}  & + \nabla \cdot \left(\left[{\tilde h}_{i,k} \varphi_{i,k}\right]_e {\bf u}_{e,k}\right) + \rho_0 \left\{ \left[ \varphi \tilde{W}_{tr} \right]_k^\text{top} - \left[ \varphi \tilde{W}_{tr} \right]_{k+1}^\text{top} \right\} \\
& = - \nabla \cdot \left({\tilde h}_k \left<\varphi^{\prime} {\bf u}^{\prime}\right>_k \right) - \rho_0 \left\{ \left[\left< \varphi^\prime {\tilde w}_{tr}^{\prime} \right> - \left< \varphi^\prime \tilde{ u}^{\prime}\right> \right]_k^\text{top} - \left[\left< \varphi^\prime {\tilde w}_{tr}^{\prime} \right> - \left< \varphi^\prime \tilde{ u}^{\prime}\right> \right]_{k+1}^\text{top} \right\}.

In the tracer equation, we note that surface fluxes (e.g. latent heat fluxes) will appear in the surface value of the vertical turbulent tracer flux (e.g., $\left[\left<\varphi^\prime \tilde{w}_{tr}^\prime\right>\right]_{k=0}^\text{top}$).  The surface fluxes will also include effects from volumetric surface fluxes that carry a non zero tracer value. The terms on the second line represent the small scale turbulence and the form used in Omega to represent these terms will be discussed in the next section.

In the tracer equation, we note that surface fluxes (e.g. latent heat fluxes) will appear in the surface value of the vertical turbulent tracer flux (e.g., 

).  The surface fluxes will also include effects from volumetric surface fluxes that carry a non zero tracer value. The terms on the second line represent the small scale turbulence and the form used in Omega to represent these terms will be discussed in the next section.

Omega will only predict the layer average normal velocity, so we drop the bold face on the $u$ terms except for the product of primes, which is specified in the next section.

Omega will only predict the layer average normal velocity, so we drop the bold face on the 

 terms except for the product of primes, which is specified in the next section.

\frac{\partial u_{e,k}}{\partial t}
& + \left[ {\bf k} \cdot \nabla \times u_{e,k} +f_v\right]_e\left(u_{e,k}^{\perp}\right) + \left[\nabla K\right]_e  \\
& + \frac{\rho_0}{\left[\tilde{h}_{i,k}\right]_e} \left\{ \left[\left(u - u_k\right) \left\{\tilde{W}_{tr} \right\} \right]_{e,k}^\text{top} - \left[  \left(u - u_k\right) \left\{\tilde{W}_{tr} \right\} \right]_{e,k+1}^\text{top} \right\} \\
& = - \left(\nabla \Phi \right)_{e,k} - \frac{1}{\left[\tilde{h}_k\right]_e} \nabla \left( \tilde{h}_k \alpha_k p_k \right) - \frac{1}{\left[\tilde{h}_k\right]_e} \left\{ \left[ \alpha p \nabla \tilde{z}^{\text{top}}\right]_{e,k}^\text{top} -  \left[ \alpha p \nabla \tilde{z}^{\text{bot}}\right]_{e,k+1}^\text{top} \right\} \\
&  - \frac{1}{\left[\tilde{h}_{i,k}\right]_e} \nabla \cdot \left( \tilde{h}_k \left< {\bf u}^\prime \otimes {\bf u}^\prime \right>_k \right) - \frac{\rho_0}{\left[\tilde{h}_{i,k}\right]_e^\text{top}}  \left\{ \left[ \left<\mathbf{u}^\prime \tilde{w}_{tr}^\prime \right> - \left< \mathbf{u}^\prime \tilde{ u}^\prime \right> \right]_{e,k}^\text{top} - \left[ \left<\mathbf{u}^\prime \tilde{w}_{tr}^\prime \right> - \left< \mathbf{u}^\prime \tilde{ u}^\prime \right> \right]_{e,k+1}^\text{top} \right\}.

 p_{i,k} = p_{i}^{surf} + g\rho_0 \sum_{k'=1}^{k-1} \tilde{h}_{i,k'} + \frac{1}{2} g\rho_0 \tilde{h}_{i,k}
 
z_{i,k}^{top} = z_{i}^{floor} + \rho_0 \sum_{k'=k}^{K_{max}} \alpha_{i,k'} \tilde{h}_{i,k'}

We refer to these as the discrete equations, but time derivatives remain continuous. The time discretization is described in the [time stepping design document](TimeStepping.md). The velocity, mass-thickness, and tracers are solved prognostically using [](discrete-momentum), [](discrete-mass), [](discrete-tracer). At the new time, these variables are used to compute pressure [](discrete-pressure), specific volume [](discrete-eos), and z-locations [](discrete-z). Additional variables are computed diagnostically at the new time: $\mathbf{u}^{\perp}$, $K$, $\zeta_a$, $z^{mid}$, $\Phi$, etc. The initial geopotential is simply $\Phi=gz$, but additional gravitational terms may be added later.

We refer to these as the discrete equations, but time derivatives remain continuous. The time discretization is described in the 

. The velocity, mass-thickness, and tracers are solved prognostically using 

. At the new time, these variables are used to compute pressure 

. Additional variables are computed diagnostically at the new time: 

The horizontal operators $\nabla$, $\nabla\cdot$, and $\nabla \times$ are now in their discrete form. In the TRiSK design, gradients ($\nabla$) map cell centers to edges; divergence ($\nabla \cdot$) maps edge quantities to cells; and curl ($\nabla \times$) maps edges to vertices. The exact form of operators and interpolation stencils remain the same as those given in {ref}`Omega-0 operator formulation <33-operator-formulation>` The discrete version of terms common with Omega-0, such as advection, potential vorticity, and $\nabla K$, can be found in {ref}`Omega-0 Momentum Terms <34-momentum-terms>` and {ref}`Omega-0 Thickness and Tracer Terms <35-thickness-and-tracer-terms>`.

 are now in their discrete form. In the TRiSK design, gradients (

) maps edges to vertices. The exact form of operators and interpolation stencils remain the same as those given in 

 The discrete version of terms common with Omega-0, such as advection, potential vorticity, and 

:ref:`Omega-0 Thickness and Tracer Terms <35-thickness-and-tracer-terms>`

There are two terms related to horizontal momentum dissipation in [](#discrete-momentum) that need to be parameterized, $\left<\mathbf{u}^\prime \tilde{u}^\prime \right>_k$ and $\frac{1}{\left[\tilde{h}_{i,k}\right]_e} \nabla \cdot \left( \tilde{h}_k \left< {\bf u}^\prime \otimes {\bf u}^\prime \right>_k \right)$.  The former only arises from the layer integration in pseudo-height coordinates, we interpret this term as the projection of the horizontal momentum dissipation that crosses $\tilde{z}$ interfaces.  Given this, we discuss the form of the horizontal dissipation parameterization first and return to the second term in a later subsection.

There are two terms related to horizontal momentum dissipation in 

\frac{1}{\left[\tilde{h}_{i,k}\right]_e} \nabla \cdot \left( \tilde{h}_k \left< {\bf u}^\prime \otimes {\bf u}^\prime \right>_k \right)

.  The former only arises from the layer integration in pseudo-height coordinates, we interpret this term as the projection of the horizontal momentum dissipation that crosses 

 interfaces.  Given this, we discuss the form of the horizontal dissipation parameterization first and return to the second term in a later subsection.

As in MPAS-Ocean, parameterization of the horizontal momentum dissipation is through Laplacian or Biharmonic dissipation,

\frac{1}{\left[\tilde{h}_{i,k}\right]_e} \nabla \cdot \left( \tilde{h}_k \left< {\bf u}^\prime \otimes {\bf u}^\prime \right>_k \right) =  \nu_{2,e} \nabla^2 u_{e,k} - \nu_{4,e} \nabla^4 u_{e,k}.

Again, the quantities in [](#discrete-mom-diss) are layer averaged.  The gradient of $\tilde{h}$ is assumed to be small relative to the stress tensor to allow the utilization of traditional parameterization of the dissipation. In [](#discrete-mom-diss), the viscosities are defined as

 is assumed to be small relative to the stress tensor to allow the utilization of traditional parameterization of the dissipation. In 

Where the $\nu_4$ and $\nu_2$ are the constant Biharmonic and Laplacian coefficients specified in the YAML configuration file.  The meshScaling variable is constructed to appropriately scale the viscosity with changes in mesh spacing (for details, see [Hecht et al 2008](https://www.researchgate.net/profile/Mark-Petersen-2/publication/255570421_Lateral_Mixing_in_the_Eddying_Regime_and_a_New_Broad-Ranging_Formulation/links/55886e1c08ae347f9bda9f04/Lateral-Mixing-in-the-Eddying-Regime-and-a-New-Broad-Ranging-Formulation.pdf)).  This formulation implies that for regionally refined configurations, we cannot strictly pull the $\nu_{2,e}$ and $\nu_{4,e}$ variables through the gradient operator.  To do this, we assume that the spatial gradient of $\nu$ is small locally and we can treat them as constant and pass them through the gradient operator.

 are the constant Biharmonic and Laplacian coefficients specified in the YAML configuration file.  The meshScaling variable is constructed to appropriately scale the viscosity with changes in mesh spacing (for details, see 

).  This formulation implies that for regionally refined configurations, we cannot strictly pull the 

 variables through the gradient operator.  To do this, we assume that the spatial gradient of 

 is small locally and we can treat them as constant and pass them through the gradient operator.

 \nu_{2,e} \nabla^2 u_{e,k} = \nu_{2,e} \left( \nabla D_{i,k} - \nabla^{\perp} \zeta_{v,k} \right)

where $D$ is divergence and $\zeta$ is relative vorticity. See {ref}`Omega V0 Section 3.4.4 <344-del2-momentum-dissipation>` for further details.

As in {ref}`Omega V0 Section 3.4.5 <345-del4-momentum-dissipation>`, biharmonic momentum dissipation is computed with two applications of the Del2 operator above.

, biharmonic momentum dissipation is computed with two applications of the Del2 operator above.

 - \nu_{4,e} \nabla^4 u_{e,k}
= - \nu_{4,e} \nabla^2 \left( \nabla^2 u_{e,k} \right)

We interpret $\left<\mathbf{u}^\prime \tilde{u}^\prime \right>$ as the dissipation of momentum across the sloping $\tilde{z}$ surface.

\left<\mathbf{u}^\prime \tilde{u}^\prime \right> = \left\{\left[\nu_{2,e} \left( \nabla \tilde{D}_{i} - \nabla^{\perp} \tilde{\zeta}_{v} \right)\right]_k - \left[\nu_{4,e} \nabla^2 \left( \nabla^2 \tilde{u}_{e,k} \right)\right]_k\right\}

While it looks very similar to [](#discrete-mom-del2) - [](#discrete-mom-del4), there are a few critical differences.  First, the normal velocities in the divergence and relative vorticity in [](#discrete-mom-flux-sloping) are the reconstructed velocity at the top of the cell along an edge, not the layer average.  Second, the velocities in the divergence and relative vorticity are only the projection across the interface (hence the tilde on $D$ and $\zeta$), computed in a discrete sense following

, there are a few critical differences.  First, the normal velocities in the divergence and relative vorticity in 

 are the reconstructed velocity at the top of the cell along an edge, not the layer average.  Second, the velocities in the divergence and relative vorticity are only the projection across the interface (hence the tilde on 

in this relation, we have moved the subscript $k$ off the variable itself to prevent confusion with the layer average.  With this definition, [](#discrete-mom-flux-sloping) goes to zero for flat layer surfaces.

 off the variable itself to prevent confusion with the layer average.  With this definition, 

The vertical turbulent momentum stress is most commonly parameterized as a down-gradient process, i.e.,

\left[ \left<\mathbf{u}^\prime \tilde{w}_{tr}^\prime \right> \right]_{e,k} = -\frac{\nu_v \rho}{\rho_0} \left[\frac{\partial u}{\partial \tilde{z}}\right]_{e,k}

Plugging this relation into the last part of [](#discrete-momentum)

\frac{1}{\left[\tilde{h}_{i,k}\right]_{e,k}}  \left\{ \left[ \nu_v \left[\frac{\partial u}{\partial \tilde{z}}\right]_{e,k} \right]_{e,k} - \left[ \nu_v \left[\frac{\partial u}{\partial \tilde{z}}\right]_{e,k} \right]_{e,k+1}  \right\}

Since, Omega will predict layer average quantities (e.g., $u_k$), it's not immediately clear that a traditional discretization is appropriate as there is a discontinuity in the predicted variables at layer interfaces.  To circumvent this problem, we turn to weak derivatives instead of traditional pointwise forms.  A weak derivative of $\varphi$ is defined as

), it’s not immediately clear that a traditional discretization is appropriate as there is a discontinuity in the predicted variables at layer interfaces.  To circumvent this problem, we turn to weak derivatives instead of traditional pointwise forms.  A weak derivative of 

\int_{\tilde{z}_{k+1/2}^{\text{top}}}^{\tilde{z}_{k-1/2}^{\text{top}}} \varphi^\prime \psi d\tilde{z} = \varphi \psi - \int_{\tilde{z}_{k+1/2}^{\text{top}}}^{\tilde{z}_{k-1/2}^{\text{top}}} \varphi \psi^\prime d\tilde{z}.

Here $\psi$ is a smooth test function.  If we choose $\psi$ such that it has compact support on the interval $(\tilde{z}_{k+1/2}^{\text{top}},\tilde{z}_{k-1/2}^{\text{top}})$ then the first term on the right hand side of [](#weak-derivative-defn) is zero.  Using the fact that Omega predicts layer averages, the integral on the right hand side can be broken into

 is zero.  Using the fact that Omega predicts layer averages, the integral on the right hand side can be broken into

-\int_{\tilde{z}_{k+1/2}^{\text{top}}}^{\tilde{z}_{k-1/2}^{\text{top}}} \varphi \psi^\prime d\tilde{z} = - \int_{\tilde{z}_{k+1/2}^{\text{top}}}^{\tilde{z}_{k}^{\text{top}}} \varphi_{k+1} \psi(z)^\prime d\tilde{z} - \int_{\tilde{z}_{k}^{\text{top}}}^{\tilde{z}_{k-1/2}^{\text{top}}} \varphi_k \psi(z)^\prime d\tilde{z}.

Since $\varphi$ is constant over the integral domain, it can be pulled outside the integral and we use the fundamental theorem to yield

 is constant over the integral domain, it can be pulled outside the integral and we use the fundamental theorem to yield

-\varphi_{k+1} (\psi(k)-\psi(k+1/2)) - \varphi_k (\psi(k-1/2) - \psi(k)).

Since we have assumed $\psi$ has compact support within the interval but not including the endpoints, $\psi(k-1/2) = \psi(k+1/2) = 0$ and we are left with

 has compact support within the interval but not including the endpoints, 

\int_{\tilde{z}_{k+1/2}^{\text{top}}}^{\tilde{z}_{k-1/2}^{\text{top}}} \varphi^\prime \psi d\tilde{z} = \psi(k) \left(\varphi_k - \varphi_{k+1}\right).

Since we are able to pick any test function that satisfies our assumptions, we choose the dirac delta function and rewrite the previous equation as

\varphi^\prime(z) = \left(\varphi_k - \varphi_{k+1}\right) \delta \left(z-\tilde{z}_{k}^{\text{top}}\right).

If [](#weak-derivative) is averaged between $\tilde{z}_{k-1/2}^{\text{top}}$ and $\tilde{z}_{k+1/2}^{\text{top}}$, we arrive at the expected form of the gradient centered on layer interfaces

, we arrive at the expected form of the gradient centered on layer interfaces

\left[\frac{\partial \varphi}{\partial z}\right]_{avg} = \frac{\left(\varphi_k - \varphi_{k+1}\right)}{0.5 \left(\tilde{h}_k + \tilde{h}_{k+1}\right)}

where $0.5 \left(\tilde{h}_k + \tilde{h}_{k+1}\right)$ is the distance between $\tilde{z}_{k-1/2}^{\text{top}}$ and $\tilde{z}_{k+1/2}^{\text{top}}$.  A similar derivation can be followed to compute gradients across layer centers.  This will form discrete derivatives in Omega.

.  A similar derivation can be followed to compute gradients across layer centers.  This will form discrete derivatives in Omega.

With this, we can now fully discretize [](#discrete-mom-vert-diff) as

-\frac{\rho_0}{\left[\tilde{h}_k\right]_e} \left\{ \left[\left<u^\prime \tilde{w}_{tr}^\prime\right> \right]_{e,k} - \left[\left<u^\prime \tilde{w}_{tr}^\prime\right> \right]_{e,k+1} \right\} = -\frac{\rho_0}{\left[\tilde{h}_k\right]_e} \left\{ \frac{\left(u_{e,k-1} - u_{e,k}\right)}{0.5 \left(\tilde{h}_{k-1} + \tilde{h}_{k}\right)} - \frac{\left(u_{e,k} - u_{e,k+1}\right)}{0.5 \left(\tilde{h}_k + \tilde{h}_{k+1}\right)} \right\}.

This form can be interfaced with the Omega [tridiagonal solver](TridiagonalSolver.md) routine.

The discretized momentum and tracer forcing appear as the surface value of the vertical turbulent fluxes of tracer and momentum.  We note that surface forcings (e.g. river runoff) could also extend beyond the first layer.  This will be discussed further in a later design document.  Omega also includes a ocean floor vertical turbulent flux of momentum.

The wind forcing is applied as a top boundary condition during implicit vertical mixing as

where $\tau$ is the wind stress in Pa. Since the mass-thickness $\tilde{h}$ is in kg/s/m$^2$, this results in the desired units of m/s$^2$ for a momentum tendency term.

Bottom Drag is applied as a bottom boundary condition during implicit vertical mixing as

- C_D \frac{u_{e,k}\left|u_{e,k}\right|}{[\alpha_{i,k}\tilde{h}_{i,k}]_e}.

The units of specific volume times mass-thickness $\alpha h$ are length (m), so that the full term has units of m/s$^2$.

Rayleigh drag is a simple drag applied to every cell.  It is used to ensure stability during spin-up.

Direct forcing of temperature, e.g. from latent or sensible heat fluxes take a form similar to MPAS-Ocean

where $\rho_1$ is the density in the top layer of Omega. This gives units of mK/s.

 is the density in the top layer of Omega. This gives units of mK/s.

Since Omega is a non-Boussinesq ocean, surface sources of water will be mass fluxes instead of being converted into thickness fluxes.  Similar to MPAS-Ocean, Omega will include an ability to spread certain fluxes (e.g., river runoff) over a depth specified in the YAML configuration file.

As with momentum dissipation, the horizontal tracer diffusion arises from the $\left<\mathbf{u}_k^\prime \varphi_k \right>$ and $\left< \tilde{u}^\prime \varphi^\prime \right>$.  As in MPAS-Ocean, the former term can be parameterized either as Laplacian or Biharmonic diffusion,

As with momentum dissipation, the horizontal tracer diffusion arises from the 

.  As in MPAS-Ocean, the former term can be parameterized either as Laplacian or Biharmonic diffusion,

D^\varphi_{i,k} =  \kappa_{2,e} \nabla^2 \varphi_{i,k} - \kappa_{4,e} \nabla^4 \varphi_{i,k}.

As in [](#discrete-mom-diss), we have defined $\kappa_{2,e} = \kappa_2 \times \text{meshScaling}_{4,e}$ and $\kappa_{2,e} = \kappa_2 \times \text{meshScaling}_{4,e}$, where $\kappa_2$ and $\kappa_4$ are the constant values specified in the YAML configuration file.  We also assume that $\varphi_{i,k} \nabla^2 \kappa_{2,e}$ is small relative to $\kappa_{2,e} \nabla^2 \varphi_{i,k}$.

 are the constant values specified in the YAML configuration file.  We also assume that 

\nabla \cdot \left(\tilde{h}_k \left<\varphi^\prime u^\prime \right>_k \right) = \nabla \cdot \left( \tilde{h}_{i,k} \kappa_{2,e} \nabla \varphi_{i,k} \right).

See {ref}`Omega V0 Section 3.5.2 <352-del2-tracer-diffusion>` for details of this calculation.

 -  \nabla \cdot \left( \kappa_{4,e} \nabla
\right[
\nabla \cdot \left( \tilde{h}_{i,k} \nabla \varphi_{i,k} \right)
\left]
 \right).

Each of these operators are written as horizontal stencils in the {ref}`Omega V0 Operator Formulation Section <33-operator-formulation>`.  Again we note that the variables in these equations are the layer average.

Each of these operators are written as horizontal stencils in the 

:ref:`Omega V0 Operator Formulation Section <33-operator-formulation>`

.  Again we note that the variables in these equations are the layer average.

As with horizontal momentum dissipation, there is a turbulent flux of tracer across a sloping $\tilde{z}$ interface.  We interpret the $\left< \tilde{z}^\prime \varphi^\prime \right>$ as the projection of the horizontal turbulent flux across the sloping interface.  The form of the diffusion is similar, taking Laplacian diffusion as an example

As with horizontal momentum dissipation, there is a turbulent flux of tracer across a sloping 

 as the projection of the horizontal turbulent flux across the sloping interface.  The form of the diffusion is similar, taking Laplacian diffusion as an example

 \nabla \cdot \left( \tilde{h}_{i} \kappa_{2,e} \nabla \varphi_{i} \right)_k.

While this is similar in form, this uses the reconstruction at the top of the layer and not the layer averages directly as in [](#discrete-tracer-del2).

While this is similar in form, this uses the reconstruction at the top of the layer and not the layer averages directly as in 

The vertical tracer diffusion arises from the $\rho_0\left(\left[\left<\varphi^\prime \tilde{w}_{tr}^\prime \right> \right]_k - \left[\left<\varphi^\prime \tilde{w}_{tr}^\prime \right> \right]_{k+1} \right)$ term. Discretization of this term differs from the vertical velocity diffusion. In the tracer equation, the quantity being updated is the mass-weighted tracer $\tilde{h}_k \varphi$, while in the momentum equation, the variable is simply the velocity $\bf u$, without multiplication by pseudo-thickness $\tilde{h}_k$. This difference affects how vertical mixing is handled. For the momentum equation, the vertical mixing term includes an explicit $1/\tilde{h}_k$ factor, which makes it easy to rearrange the equation into a standard tridiganal form that can be solved implicitly ([](#final-vert-vel-dissipation)). However, in the tracer equation [](#layer-tracer-final-simple), because $\tilde{h}_k$ is included in the prognostic variable $\tilde{h}_k \varphi_k$ and not divided through as in the velocity equation.  Thus isolating $\varphi_k^{n+1}$ becomes more difficult, where $n$ is an index of timestep. In particular, if $\tilde{h}_k^{n+1}$ were treated implicitly, the left-hand side of [](#layer-tracer-final-simple) would involve the product of two unknowns $\tilde{h}_k^{n+1}$ and $\varphi_k^{n+1}$, resulting in a non-linear system. This nonlinearity prevents the formation of the standard tridiagonal matrix for $\varphi_k^{n+1}$, unlike in the vertical momentum diffusion. To address this, Omega follows the approach used in the MPAS-Ocean: the tracer update is split into two steps. First, a provisional tracer field is computed with all processes except vertical mixing (e.g. advection, horizontal mixing). Then, vertical tracer diffusion is applied in a separate implicit step using this provisional field.

\rho_0\left(\left[\left<\varphi^\prime \tilde{w}_{tr}^\prime \right> \right]_k - \left[\left<\varphi^\prime \tilde{w}_{tr}^\prime \right> \right]_{k+1} \right)

 term. Discretization of this term differs from the vertical velocity diffusion. In the tracer equation, the quantity being updated is the mass-weighted tracer 

, while in the momentum equation, the variable is simply the velocity 

. This difference affects how vertical mixing is handled. For the momentum equation, the vertical mixing term includes an explicit 

 factor, which makes it easy to rearrange the equation into a standard tridiganal form that can be solved implicitly (

 and not divided through as in the velocity equation.  Thus isolating 

, resulting in a non-linear system. This nonlinearity prevents the formation of the standard tridiagonal matrix for 

, unlike in the vertical momentum diffusion. To address this, Omega follows the approach used in the MPAS-Ocean: the tracer update is split into two steps. First, a provisional tracer field is computed with all processes except vertical mixing (e.g. advection, horizontal mixing). Then, vertical tracer diffusion is applied in a separate implicit step using this provisional field.

During the tracer update, Omega first computes a provisional tracer field that excludes vertical turbulent mixing to obtain $\varphi^{n+1}$. This temporary tracer update is given by

During the tracer update, Omega first computes a provisional tracer field that excludes vertical turbulent mixing to obtain 

\varphi^{*} = \frac{\tilde{h}_k^{n} \varphi^{n} + \Delta t  (\text{Adv} + \text{Diff}_{\text{H}})}{\tilde{h}_k^{n+1}}.

where the term "$\text{Adv}$" represents resolved tracer advection and includes both horizontal transport and resolved vertical transport by the projected vertical velocity $\tilde{W}_{tr}$, and the "$\text{Diff}_{\text{H}}$" term represents horizontal tracer diffusion including the harmonic and biharmonic diffusion.
Once the provisional tracer field ($\varphi^{*}$) is formed, vertical diffusion is applied using an implicit tridiagonal solver. The non mass weighted tracer can now be used to compute the diffusive tendency and compute the final tracer at the next timestep. Given diffusion is done on the non mass weighted tracer, the diffusive term is written as

” represents resolved tracer advection and includes both horizontal transport and resolved vertical transport by the projected vertical velocity 

” term represents horizontal tracer diffusion including the harmonic and biharmonic diffusion.

) is formed, vertical diffusion is applied using an implicit tridiagonal solver. The non mass weighted tracer can now be used to compute the diffusive tendency and compute the final tracer at the next timestep. Given diffusion is done on the non mass weighted tracer, the diffusive term is written as

\frac{\rho_0}{\tilde{h}_k} \left\{\left[\left<\varphi^\prime \tilde{w}_{tr}^\prime \right>\right]_k - \left[\left<\varphi^\prime \tilde{w}_{tr}^\prime \right>\right]_{k+1} \right\}

Taking a traditional down gradient approximation and using [](#weak-deriv-final) the vertical turbulent flux can be applied using a tridiagonal solver ($\mathcal{L}$) in the
[tridiagonal solver](TridiagonalSolver) in the implicit vertical mixing step:
$$
\varphi^{n+1}=\mathcal{L}_\varphi^{-1} (\varphi^*)
$$
Additional enhancements, such as non local tracer fluxes will be discussed in future design documents.

 the vertical turbulent flux can be applied using a tridiagonal solver (

The MPAS-Ocean layered formulation are provided here for reference. MPAS-Ocean solves for momentum, thickness, and tracers at layer $k$. These are continuous in the horizontal and discrete in the vertical.

The MPAS-Ocean layered formulation are provided here for reference. MPAS-Ocean solves for momentum, thickness, and tracers at layer 

. These are continuous in the horizontal and discrete in the vertical.

+ \frac{1}{2}\nabla \left| {\bf u}_k \right|^2
+ ( {\bf k} \cdot \nabla \times {\bf u}_k) {\bf u}^\perp_k
+ f{\bf u}^{\perp}_k
+ \frac{w_k^{bot}{\bf u}_k^{bot} - w_k^{top}{\bf u}_k^{top}}{h_k}
=
- \frac{1}{\rho_0}\nabla p_k
- \frac{\rho g}{\rho_0}\nabla z^{mid}_k
+ \nu_h\nabla^2{\bf u}_k
+ \frac{\partial }{\partial z} \left( \nu_v \frac{\partial {\bf u}_k}{\partial z} \right),

\frac{\partial h_k}{\partial t} + \nabla \cdot \left( h_k^e {\bf u}_k \right) + w_k^{bot} - w_k^{top} = 0,

\frac{\partial h_k\varphi_k}{\partial t} + \nabla \cdot \left( h_k^e\varphi_k^e {\bf u}_k \right)
+ \varphi_k^{bot} w_k^{bot} - \varphi_k^{top} w_k^{top}
= \nabla\cdot\left(h_k^e \kappa_h \nabla\varphi_k \right)
+ h_k \frac{\partial }{\partial z} \left( \kappa_v \frac{\partial \varphi_k}{\partial z} \right).

The layer thickness $h$, vertical velocity $w$, pressure $p$, and tracer $\varphi$, are cell-centered quantities, while the horizontal velocity ${\bf u}$ and $e$ superscript are variables interpolated to the cell edges.

Table 1. Definition of variables. Geometric variables may be found in the {ref}`Omega V0 design document, Table 1 <32-variable-definitions>`

Table 1. Definition of variables. Geometric variables may be found in the 

:ref:`Omega V0 design document, Table 1 <32-variable-definitions>`

$f = 2\Omega sin(\phi)$, $\Omega$ rotation rate, $\phi$ latitude

total velocity across a pseudo height surface $\tilde{W}_{i,k} \equiv $\tilde{w}_{i,k} - \tilde{u}_{i,k}$

variable that holds the scaling factor for biharmonic and laplacian mixing

horizontal biharmonic (del4) viscosity scaled by mesh resolution

Capability and testing are similar to [Petersen et al. 2015](http://www.sciencedirect.com/science/article/pii/S1463500314001796). The following tests are in idealized domains and do not require surface fluxes or surface restoring. For the following tests to show results comparable to those published with other models, the full dynamic sequence of density, pressure, momentum, and advection must work correctly. The successful completion of the following tests is a validation of the primitive equation functions in Omega 1.0. All of the following tests may exercise a linear equation of state or the nonlinear TEOS10. The first four tests quantify the anomalous mixing caused by the numerical schemes. The first five are on cartesian planes with regular hexagon meshes.

. The following tests are in idealized domains and do not require surface fluxes or surface restoring. For the following tests to show results comparable to those published with other models, the full dynamic sequence of density, pressure, momentum, and advection must work correctly. The successful completion of the following tests is a validation of the primitive equation functions in Omega 1.0. All of the following tests may exercise a linear equation of state or the nonlinear TEOS10. The first four tests quantify the anomalous mixing caused by the numerical schemes. The first five are on cartesian planes with regular hexagon meshes.

The Lock Exchange is the simplest possible test of a primitive equation model. There is an analytic formulation for the wave propagation speed. It is listed as optional because the Overflow tests the same dynamics.
See [Petersen et al. 2015](http://www.sciencedirect.com/science/article/pii/S1463500314001796) and the compass `lock_exchange` case.

The Overflow test case adds bathymetry to the Lock Exchange. It is a particularly effective test of vertical mass and tracer advection, and vertical mixing. It is useful to compare different vertical coordinates, like level (z- or p-level) versus terrain-following (sigma).
See [Petersen et al. 2015](http://www.sciencedirect.com/science/article/pii/S1463500314001796) and the compass `overflow` case.

The internal gravity wave tests horizontal and vertical advection.
See [Petersen et al. 2015](http://www.sciencedirect.com/science/article/pii/S1463500314001796) and the `internal_wave` case in both compass and polaris.

This is the first test to add the Coriolis force and uses a three-dimensional domain. It is designed to result in an eddying simulation at sufficiently high resolution. This tests the combination of Coriolis and pressure gradient forces that produce geostrophic balance, as well as horizontal advection and dissipation for numerical stability.
See [Petersen et al. 2015](http://www.sciencedirect.com/science/article/pii/S1463500314001796) and the `baroclinic_channel` case in both compass and polaris.

This is a 3D domain with a seamount in the center, where temperature and salinity are stratified in the vertical and constant in the horizontal. The test is simply that an initial velocity field of zero remains zero. For z-level layers the velocity trivially remains zero because the horizontal pressure gradient is zero. For tilted layers, this is a test of the pressure gradient error and the velocity is never exactly zero. This is a common test for sigma-coordinate models like ROMS because the bottom layers are extremely tilted along the seamount, but it is a good test for any model with tilted layers. Omega will use slightly tilted layers in p-star mode (pressure layers oscillating with SSH) and severely tilted layers below ice shelves, just like MPAS-Ocean. See [Ezer et al. 2002](https://www.sciencedirect.com/science/article/pii/S1463500302000033), [Haidvogel et al. 1993](https://journals.ametsoc.org/view/journals/phoc/23/11/1520-0485_1993_023_2373_nsofaa_2_0_co_2.xml), [Shchepetkin and McWilliams 2003](https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2001JC001047), and previous MPAS-Ocean [confluence page](https://acme-climate.atlassian.net/wiki/spaces/OCNICE/blog/2015/11/19/40501447/MPAS-O+Sigma+coordinate+test+sea+mount).

This is a 3D domain with a seamount in the center, where temperature and salinity are stratified in the vertical and constant in the horizontal. The test is simply that an initial velocity field of zero remains zero. For z-level layers the velocity trivially remains zero because the horizontal pressure gradient is zero. For tilted layers, this is a test of the pressure gradient error and the velocity is never exactly zero. This is a common test for sigma-coordinate models like ROMS because the bottom layers are extremely tilted along the seamount, but it is a good test for any model with tilted layers. Omega will use slightly tilted layers in p-star mode (pressure layers oscillating with SSH) and severely tilted layers below ice shelves, just like MPAS-Ocean. See 

This test uses a fixed horizontal velocity field to test horizontal tracer advection. It is repeated from [Omega-0 design document](OmegaV0ShallowWater) and is important to conduct again as we convert Omega to a layered primitive-equation model. See `cosine_bell` case in both compass and polaris.

This test uses a fixed horizontal velocity field to test horizontal tracer advection. It is repeated from 

 and is important to conduct again as we convert Omega to a layered primitive-equation model. See 

This is an exact test for horizontal and vertical tracer advection. A fixed velocity field is provided, and a tracer distribution is advected around a vertical plane. See the `merry_go_round` test in compass, and the results on the [merry-go-round pull request](https://github.com/MPAS-Dev/compass/pull/108) and [compass port pull request](https://github.com/MPAS-Dev/compass/pull/452).

This is an exact test for horizontal and vertical tracer advection. A fixed velocity field is provided, and a tracer distribution is advected around a vertical plane. See the 

This section is for references without webpage links. These are mostly textbooks.

Cushman‐Roisin, B., & Beckers, J.M. (2011). Introduction to Geophysical Fluid Dynamics: Physical and Numerical Aspects. Academic Press.

Kundu, P.K., Cohen, I.M., Dowling D.R. (2016) Fluid Mechanics 6th Edition, Academic Press.

Pedlosky, J. (1987). Geophysical Fluid Dynamics (Vol. 710). Springer.

Hyperlink target "omega-design-governing-eqns-omega1" is not referenced.

Hyperlink target "equation-continuous-tracer" is not referenced.

Hyperlink target "equation-continuous-momentum" is not referenced.

Hyperlink target "equation-vert-pressure-balance" is not referenced.

Hyperlink target "equation-pseudo-heigh-from-z" is not referenced.

Hyperlink target "equation-tr-vh-split-pseudo" is not referenced.

Hyperlink target "equation-top-bot-normal-pseudo" is not referenced.

Hyperlink target "equation-interface-velocity" is not referenced.

Hyperlink target "equation-tr-vh-separation-pseudo" is not referenced.

Hyperlink target "equation-vh-momentum-pseudo" is not referenced.

Hyperlink target "equation-def-pseudo-thickness" is not referenced.

Hyperlink target "equation-def-layer-average" is not referenced.

Hyperlink target "equation-vertical-decomposition" is not referenced.

Hyperlink target "equation-reynolds-definition" is not referenced.

Hyperlink target "equation-def-layer-average-reynolds" is not referenced.

Hyperlink target "equation-delta-vert-average" is not referenced.

Hyperlink target "equation-aintegral-tracer-first-reynolds" is not referenced.

Hyperlink target "equation-aintegral-tracer-reynolds" is not referenced.

Hyperlink target "equation-aintegral-tracer2" is not referenced.

Hyperlink target "equation-reynolds-tracer-final" is not referenced.

Hyperlink target "equation-vh-momentum-forces" is not referenced.

Hyperlink target "equation-vh-momentum-reynolds1" is not referenced.

Hyperlink target "equation-vh-momentum-reynolds2" is not referenced.

Hyperlink target "equation-vh-momentum-reynolds-lay-avg" is not referenced.

Hyperlink target "equation-vh-momentum-reynolds-lay-avg2" is not referenced.

Hyperlink target "equation-vh-momentum-reynolds-lay-avg3" is not referenced.

Hyperlink target "equation-advection-identity" is not referenced.

Hyperlink target "equation-coriolis-expansion" is not referenced.

Hyperlink target "equation-layer-momentum-final" is not referenced.

Hyperlink target "notational-simplifications" is not referenced.

Hyperlink target "equation-layer-tracer-final-simple" is not referenced.

Hyperlink target "equation-layer-mass-final-simple" is not referenced.

Hyperlink target "equation-layer-momentum-final-simple" is not referenced.

Hyperlink target "equation-discrete-momentum" is not referenced.

Hyperlink target "equation-discrete-pressure" is not referenced.

Hyperlink target "equation-discrete-mom-diss" is not referenced.

Hyperlink target "equation-discrete-mom-del2" is not referenced.

Hyperlink target "equation-discrete-mom-del4" is not referenced.

Hyperlink target "equation-discrete-mom-flux-sloping" is not referenced.

Hyperlink target "equation-discrete-mom-vert-diff" is not referenced.

Hyperlink target "equation-weak-derivative-defn" is not referenced.

Hyperlink target "equation-final-vert-vel-dissipation" is not referenced.

Hyperlink target "equation-discrete-mom-bottom" is not referenced.

Hyperlink target "equation-discrete-tracer-diff" is not referenced.

Hyperlink target "equation-discrete-tracer-del2" is not referenced.

Hyperlink target "equation-discrete-tracer-del4" is not referenced.

Hyperlink target "equation-provision-tracer-update" is not referenced.

Hyperlink target "equation-mpaso-continuous-momentum" is not referenced.

Hyperlink target "equation-mpaso-continuous-thickness" is not referenced.

Hyperlink target "equation-mpaso-continuous-tracer" is not referenced.

<!--- OMEGA global reductions requirements and design ------------------------->

In a parallel application like OMEGA, it is often necessary to perform
averages, sums or other global collective operations on the distributed
data. These reductions are described here.

The ability to perform the sum of all elements of a distributed array
is required

The ability to find the minimum or maximum value in a distributed
array is required.

There must be reductions for all supported arithmetic types
(integers, floating point) and all array dimensions from
0-d (scalars) to 5-d.

In many cases, a global sum is often performed on a product of two
arrays. For example, a sum can be performed with a multiplicative
mask or a sum may be a dot product of two vectors within a solver.
A global sum interface that multiplies two arrays before the sum
is required.

In order to compute reductions over only owned and active
cells/edges or for regional diagnostics, there must be an option
to restrict the address space over which the local portion of sums
are computed.

For performance reasons, it is often beneficial to compute reductions
for multiple arrays at the same time to reduce the number of
messages and associated message latency. An interface for computing
independent reductions for multiple arrays at once is needed. Note
that the size of the arrays and any associated options must be the
same for all arrays in a multi-field reduction. It would be possible
to relax this restriction in the future.

Because OMEGA supports multiple machine environments (communicators
and processor sets) and decompositions, the reductions must be
callable from environments other than the default.

The results (esp sums) must be bit-for-bit reproducible on a given
machine with respect to processor and thread count

When running on accelerated architectures, the ability to specify
whether the reduction occurs on host or device is needed.

For reductions performed on the CPU, the operations must support
and MPI+OpenMP model while still meeting reproducibility requirements.

It will be desireable in the future to add other reduction operations
like minloc/maxloc.

For integers, the sum is straightforward. For single-precision floating
point, we will perform sums in double precision and convert back to
single before returning. In each of the above cases, the local Kokkos
versions of sum, minval, maxval will be used for the local sum before
accumulating the MPI sum across ranks.

For double-precision sums, we will begin with
a version of the double-double (DDPDD) algorithm of Donald Knuth,
further improved by David A Bailey and outlined in He and Ding (2001
J. of Supercomputing, 18, 259). In the future, we will also support
the more robust version by Pat Worley in the E3SM reprosum routines.

In the future, if we support multiple reproducible algorithms,
we will need an enum to define choice of algorithm.

There are no new classes or data types associated with this
functionality.

For each supported array type ArrayTTDD (where TT is the data
type I4, I8, R4, R8, Real and DD is the dimension 1D thru 5D)
or supported scalar data type, there will be a sum function
aliased to the globalSum interface:

    sum = globalSum(const ArrayTTDD array,
                    const I4 mpiComm,
                    const std::vector<I4> indxRange);

where mpiComm is the MPI communicator extracted from the
relevant MachEnv (eg defaultEnv.comm) and indxRange is a
`std::vector` of length 2x the number of array dimensions.
The entries of this vector will be the min, max index of
each array dimension. So, for example an array dimensioned
`(nCellsAll+1, nVertLayers+1)` might need to be summed over
the `indxRange{0, nCellsOwned-1, 0, nVertLayers-1}`. Note
that the indxRange supports a constant scalar values only so
does not support a variable index range like
minLayerCell/maxLayerCell. That is best managed either
by masking with the sum-product interface below or
ensuring the array has been set to zero in non-active
entries.

For scalar sums, the interface is the same with the
indxRange argument absent.

We will assume that Kokkos host arrays will be summed on the
host and device arrays will be summed on the device.

    sum = globalSum(const ArrayTTDD array1,
                    const ArrayTTDD array2,

that returns the sum of `array1*array2`. The two arrays
must be of the same data type. It is not required that
the two arrays be of exactly the same size but both arrays
must be of appropriate size to accomodate the indxRange
and the indices must align appropriately. This typically
means that if the sizes differ, any extra entries occur
at the end of the respective dimension.

To sum multiple fields at once, a `std::vector` of
arrays must be constructed and the result will be
stored in a `std::vector` of appropriate type:

    std::vector<type> sum =
         globalSum(const std::vector<ArrayTTDD> arrays,
                   const I4 mpiComm,
                   const std::vector<I4> indxRange)

As is the case with sum-product, each array must be
of the same data type but can have slightly different
sizes as long as the dimension lengths can accomodate
the indxRange and that the indices align appropriately
with the indxRange requested.

The multi-field sum with product is still to be determined, but
should have an interface like:

         globalSum(const std::vector<ArrayTTDD> arrays1,
                   const std::vector<ArrayTTDD> arrays2,

The implementation detail that needs to be determined is how
to manage two use cases. A typical use case is for the second
array in the product to be the same for all fields (eg a mask or
an area array), but we may wish to support a case where each
array product has a different array for both operands.
Because Kokkos arrays contain metadata and a pointer to
the data, it may be possible to create a std::vector of
the same array, allowing both the case of a fixed array
to be used for all or for all the array products to have
different arrays. This requires some testing of approaches.

The global minval will support interfaces similar to the
global sums, but will return the minimum value instead.

    varMin = globalMinval(const ArrayTTDD array,
                          const I4 mpiComm,
                          const std::vector<I4> indxRange);

    varMin = globalMinval(const ArrayTTDD array1,
                          const ArrayTTDD array2,

    std::vector<type> varMin =
         globalMinval(const std::vector<ArrayTTDD> arrays,
                      const I4 mpiComm,
                      const std::vector<I4> indxRange)

Note that the minval routine will not be cognizant of any
special values so any inactive entries should either be
masked or set to an appropriately high value.

The maxval will have the identical form of minval but will
return the maximum value. The same restrictions and comments
on special values apply.

For each data type, a set of reference arrays will be created
with random numbers that extend across the entire range of that
type (for later reproducibility tests). A reference sum will
be computed serially using the same reproducible algorithm
as the global implementation. The basic test will compare the
global sum with the reference serial sum. Because the data types
include both host and device arrays, this will also test reductions
on either location.

A MachEnv and decomposition for a subset of MPI ranks will be
created and the reference arrays above will be distributed across
the new decomposition. Global sums in each case will be compared
to the serial reference value and the full MPI rank case
for bit reproducibility. Similarly, a test with CPU threading
on will test reproducibility under threading.

The sums will be tested with a reduced index range on the same
reference arrays from above and compared with the
similarly-restricted serial reference sums.

A mask array will be defined for each type and used with the
sum-with-product interface and compared against a serial
version of the same.

Additional arrays similar to the reference arrays will be created
with reference serial sums. Then the multi-field interface will
be tested against these reference sums.

A similar approach to the sums above will be used to test the
min and max functions in all combinations of interfaces.

Hyperlink target "omega-design-global-reductions" is not referenced.

The state class will provide a container for the prognostic (normal velocity and layer thickness) variables on a subdomain within OMEGA.
It will have functionality for registering fields with IOStreams and performing time level updates.
This will also be a data type that can be used for tendency terms and provis variables within the timestepping scheme.

2.1 Requirement: State should register prognostic variables with IOStreams

The state class will have a method to register the progostic variables with IOStreams to provide initial condition input and solution output.

2.1 requirement: state should register prognostic variables with iostreams

A method will be responsible for advancing the prognostic variables prior to the beginning of a new timestep.
The left-most index of the prognostic variables will be a time index, which will facilitate flexibility with timestepping schemes.
The time level update will include a halo exchange and an exchange of time level.

2.3 Requirement: Can be used as a common data type for tendency and provis variables

The timestepping scheme will require storage for the tendency terms and the intermediate stage. The state class can also be used to fulfill this need.

2.3 requirement: can be used as a common data type for tendency and provis variables

The State class will contain public Kokkos host and device views for the normal velocity and layer thickness variables.

class State{

public:

  Array3DR8 LayerThickness;
  Array3DR8 NormalVelocity;

}

There will be a constructor and destructor for the class and several public methods. Some of these methods are only applicable to the prognostic state and not to the tendency and provis contexts.

The time level update method will perform a halo exchange for the prognostic variables and advance the state variables for the next timestep.

The sample domain used for the Decomp test will be used to test obtaining the correct local values from the IO infrastructure

There will be a test to ensure the time level update functions correctly.

<!--- use table of contents if desired for longer documents  -->

State requirement clearly and simply, followed by justification.
Avoid getting into design or implementation details

Clear statement of desired feature and reason for including (e.g. design needs
to take this into account for future capability).

Describe any algorithms or point to references that are important for implementation

(a + b)^2  &=  (a + b)(a + b) \\
           &=  a^2 + 2ab + b^2

List and define any configuration parameters or public constants.

List and describe all public methods and their interfaces (actual code for
interface that would be in header file). Describe typical use cases.

Describe test including conditions for pass/fail
List which requirements it tests:

The tendencies class will provide a container for the tendency terms
terms for the layer thickness and normal velocity tendencies.

Methods to compute the tendency terms will be provided. This will include
functions to compute the layer thickness tendencies, normal velocity tendencies,
and all tendencies. This will also involve calling the compute function for the
auxiliary state.

There will be methods to create and remove non-default tendencies instances, that

The tendencies class is an object representing a group of tendency terms and
arrays for the accumulated tendency calculations. Additionally, it will have
a static map of all tendencies and a pointer to the default tendencies instance for quick

class Tendencies{
 public:
   Array2DReal LayerThicknessTend;
   Array2DReal NormalVelocityTend;
   ThicknessFluxDivOnCell ThicknessFluxDiv;
   PotentialVortHAdvOnEdge PotientialVortHAdv;
   KEGradOnEdge KEGrad;
   SSHGradOnEdge SSHGrad;
   VelocityDiffusionOnEdge VelocityDiffusion;
   VelocityHyperDiffOnEdge VelocityHyperDiff;
 private:
   static Tendencies *DefaultTendencies;
   static std::map<std::string, std::unique_ptr<Tendencies>> AllTendencies;
};

methods. The constructor will be be private to make sure that every tendencies instance
is properly stored in the map. However, a static method to create new
tendencies instance, enforcing this requirement, will be provided.

Tendencies(const std::string &Name, const HorzMesh *Mesh, int NVertLayers, Config *Options);

create a new tendencies instance, and put it in the static map of all tendencies.

Tendencies* Tendencies::create(const std::string &Name, const HorzMesh *Mesh, int NVertLayers, Config *Options);

The init method will create the default tendencies and return an error code:

There will be methods for getting the default and non-default tendencies instances:

Tendencies *Tendencies::getDefault();
Tendencies *Tendencies::get(const std::string &Name);

The 'computeAll' method will compute and accumulate all layer thickness and normal velocity tendencies using the ocean
state and auxiliary state from a given time level:

The ‘computeAll’ method will compute and accumulate all layer thickness and normal velocity tendencies using the ocean

void Tendencies::computeAllTendencies(const OceanState *State, const AuxilaryState *AuxState, int TimeLevel);

void Tendencies::computeThicknessTendencies(const OceanState *State, const AuxilaryState *AuxState, int TimeLevel);

void Tendencies::computeVelocityTendencies(const OceanState *State, const AuxilaryState *AuxState, int TimeLevel);

will remove a named tendencies instance, whereas the clear method will remove all of

void Tendencies::erase(const std::string &Name);
void Tendencies::clear();

There will be a test checking that multiple tendencies objects can be

There will be a test to ensure all of the tendencies are correctly computed by

The tendency terms in OMEGA are implemented as functors, which define an operation over a number of vertical layers for particular a cell, edge, or vertex.
Tendency functors take information which remains constant during the forward simulation (such as `HorzMesh` and `Config`)  as constructor arguments.
The `operator()` method is overloaded with the relevant discrete /parameterization.
This approach allows for a modularization of the tendency terms that enables flexible groupings of work within larger cell/edge/vertex loops.
This type of flexibility will be important for optimizing the code across different device architectures.

Tendency functors take information which remains constant during the forward simulation (such as 

 method is overloaded with the relevant discrete /parameterization.

2.1 Requirement: Tendencies should be able to be grouped under different outer loops over mesh locations.

The ability to fuse/separate tendency computations will allow for optimization across different device architectures.
Each functor will implement a given tendency operation on a specific mesh location (cell, edge, vertex).

2.1 requirement: tendencies should be able to be grouped under different outer loops over mesh locations.

2.2 Requirement: Tendency operator calls should have simple, intuitive calling arguments.

Tendency functors take in constant data as constructor arguments, which simplifies the arguments used to call the operator method.

2.2 requirement: tendency operator calls should have simple, intuitive calling arguments.

2.3 Requirement: Tendencies must allow for vectorization on CPU architectures.

Tendency operations have inner loops over a chunk of vertical layers.
The chunk size will be set to the vector length on CPU machines and 1 for GPUs.
This will allow for the possibility of vectorization on CPUs.

2.3 requirement: tendencies must allow for vectorization on cpu architectures.

The algorithmic formulation for each tendency can be found in the algorithms design document.

The tendency functor is a class the overrides the `operator()` method.
The class contains private member variables for any constant data that are defined by the constructor.
Each tendency term will have a `bool` variable which can be set to enable/disable the computation of the tendency.

 variable which can be set to enable/disable the computation of the tendency.

class ThicknessFluxDivergenceOnCell {
  public:

    bool enabled = false;

    ThicknessFluxDivergenceOnCell(const HorzMesh *Mesh, Config *Options);

    KOKKOS_FUNCTION void operator()(Array2DReal &Tend
                                    int ICell,
                                    int KChunk,
                                    const Array2DReal &ThicknessFlux);

  private:
    Array1DI4 NEdgesOnCell;
    Array2DI4 EdgesOnCell;
    Array1DR8 DvEdge;
    Array1DR8 AreaCell;
    Array2DR8 EdgeSignOnCell;
};

Tendency functor constructors are responsible for initializing the private member variables:

ThicknessFluxDivergenceOnCell::ThicknessFluxDivergenceOnCell(HorzMesh const *Mesh, Config *Options)
    : NEdgesOnCell(Mesh->NEdgesOnCell), EdgesOnCell(Mesh->EdgesOnCell),
      DvEdge(Mesh->DvEdge), AreaCell(Mesh->AreaCell),
      EdgeSignOnCell(Mesh->EdgeSignOnCell) {

    enabled = Options->get("ThicknessFluxTendencyEnable")
}

The operator method implements the tendency computation for a chunk of vertical layers at a given horizontal mesh location.
The inner loop over a chunk of vertical layers enables CPU vectorization.

KOKKOS_FUNCTION void ThicknessFluxDivergenceOnCell::operator()(Array2DReal &Tend
                                                               int ICell,
                                                               int KChunk,
                                                               const Array2DReal &ThicknessFlux)  const {
   const Real InvAreaCell = 1. / AreaCell(iCell);
   for (int J = 0; J < NEdgesOnCell(ICell); ++J) {
      const int JEdge = EdgesOnCell(ICell, J);
      for (int K = KChunk * VecLength; K < (KChunk + 1) * VecLength; ++K) {
         Tend(ICell, K) -= DvEdge(JEdge) * EdgeSignOnCell(ICell, J) * ThicknessFlux(JEdge, K) * InvAreaCell;
      }
   }
}

Each tendency operator will be used in a CTest which will compare its result against a reference value.

The tendency terms for the shallow water equations will be tested on existing test cases implemented in Polaris to verify they produce the correct convergence rates.

When simulating the Earth System, a model must track the simulation time
as it steps forward with a given time step. Time must be kept in the
appropriate calendar system, accounting for leap years and other
adjustments as needed. Because these simulations may extend over
millions of time steps or more, time must be accumulated without roundoff
so that accumulated time does not drift and events will occur at precise
time intervals. This document describes various classes needed to track the
model's simulation time accurately and manage events in the appropriate
calendar conventions.

model’s simulation time accurately and manage events in the appropriate

The time manager must be able to track the current simulation time
as the model integration steps forward.

Because integrations may extend over millions of time steps or
longer, the model must track time without accumulating roundoff.

The time manager must be able to track time in supported calendars,
starting with:

No calendar (some test cases only need elapsed time in sec)
Time for standard calendars will assume Coordinated Universal
Time (UTC).

The model must be able to compute time intervals, not only to
represent the model time step itself, but also to be able to
compute time between events, track periodic events or perform
time averaging. These intervals can be both absolute time (seconds)
or represented as calendar intervals (monthly, yearly, number of
days). Conversion from a calendar interval to absolute interval
(in seconds) will be needed to compute time-related quantities
like time averaging.

The time manager must be able to trigger events that occur
at specific times or at specified periodic time intervals.
This requires being able to compare time instants and trigger an
alarm if a time instant or interval has been reached. It will
also be useful to track when a previous or next event occurs in a
periodic sequence.

To support very long time integrations or potential paleoclimate
simulations, the model must support years that extend beyond the
4-digit representation and support negative years to represent
years Before the Common Era (BCE)

For model output or metadata, the model must be able to
represent a time in accepted formats. Following ISO standards
(eg ISO 8601 and prior), these will typically represent time
in decreasing intervals, eg `YYYYYY-MM-DD HH:MM:SS.SSSSS` though
variations on this format can be supported to use, for example,
different separators and different widths for years and fractional
seconds.

The requirement for no roundoff accumulation implies the need to
use an integer fraction representation as in the ESMF time
manager. Similarly, calendar days are best tracked using the
Julian Day that counts days since a specified start time
(Noon UTC on 1 Jan, 4713 BCE in the Gregorian calendar -
a beautiful Monday by most accounts) with conversion to
various calendars following the algorithms in:

Fliegel, H. F., and Van Flandern, T. C., 1968, A Machine Algorithm
for Processing Calendar Dates, Communications of the Association of
Computing Machines, 11, 657.

Hatcher, D.A., 1984. Simple Formulae for Julian Day Numbers and
Calendar Dates, Quart. J. of R. Astr. Soc., 25, 53-55.

The time manager design will follow an ESMF-like approach,
though simplified and not requiring linking to the full
ESMF library. Previous simplifications were performed for
WRF by John Michalakes in Fortran and are used in MPAS, E3SM.
A similar simplification was performed by Phil Jones from the
ESMF C++ version for the SciDAC CANGA project and will be used
here.

`Alarm`: alarms that trigger at time instants or periodic intervals

`Clock`: a clock that keeps track of model time as it marches forward

A number of parameters are defined among the above classes. For
all classes, it will be useful to define times and intervals with
a number and units, so we define an enum class for time units:

enum class TimeUnits{
                 None = 0, ///< value for undefined units
                 Seconds,  ///< time units in seconds (typical)
                 Minutes,  ///< time units in minutes
                 Hours,    ///< time units in hours
                 Days,     ///< time units in days
                 Months,   ///< time units in months
                 Years,    ///< time units in years
                 };

For calendars, there will be an enum for supported calendars as
well as a string name for each.

#define NUM_SUPPORTED_CALENDARS=9

enum CalendarKind {
   CalendarGregorian=1,   ///< usual Gregorian calendar
   CalendarNoLeap,        ///< Gregorian, but without leap yrs
   CalendarJulian,        ///< Julian
   CalendarJulianDay,     ///< Julian day
   CalendarModJulianDay,  ///< modified Julian day
   Calendar360Day,        ///< 12 months, 30 days each
   CalendarCustom,        ///< user defined
   CalendarNoCalendar,    ///< track elapsed time only
   CalendarUnknown};      ///< uninitialized or invalid

const std::string CalendarKindName[CALENDAR_KIND_COUNT] = {
     "Gregorian",
     "No Leap",
     "Julian",
     "Julian Day",
     "Modified Julian Day",
     "360 Day",
     "Custom",
     "No Calendar",
     "Invalid" };

There will be a TimeFrac class for the base fractional time
representation:

class TimeFrac {

   // private variables
   private:
      long long whole;  ///< whole seconds
      long long numer;  ///< fractional second (n/d) numerator
      long long denom;  ///< fractional second (n/d) denominator

   public:
      [methods described below]
};

The calendar class holds useful information for the calendar to be
used:

#define MONTHS_PER_YEAR=12

class Calendar {

  // private variables
  private:

     int id;                   ///< unique id for quick checks
     static int numCalendars;  ///< number of calendars created
     std::string name;         ///< name of calendar
     CalendarKind calKind;     ///< enum for calendar kind
     std::string calKindName;  ///< name of calendar kind

     // variables defining calendar characteristics for time
     int daysPerMonth[MONTHS_PER_YEAR]; ///< days in each month
     int monthsPerYear;             ///< num months in year
     int secondsPerDay;             ///< seconds per day
     int secondsPerYear;            ///< seconds per normal year
     int daysPerYear;               ///< days per normal year

     // public methods
     public:
        [methods described below]
};

The time instant class represents a point in time within a
given calendar:

class TimeInstant {

   private:
      TimeFrac elapsedTime; ///< Fractional seconds since reference time
      Calendar *calPtr;  ///< Pointer to calendar in which time is based

   public:
};

The time interval is used for both time steps and to compute
differences between time instants. The time interval can either
be an time interval in seconds or it can be used to hold a
time interval in calendar units (eg number of days, months or
years). The latter is useful for periodic events that occur
once per year, month or ndays.

class TimeInterval {

   private:
      TimeFrac interval; ///< Non-calendar interval in fractional seconds
      bool isCalendar;       ///< True if calendar interval
      long long calInterval; ///< Calendar interval length
      TimeUnits units;       ///< Calendar interval units

   public:
};

The alarm class allows a user to set either one-time or periodic
alarms to trigger events, like forcing updates, I/O, etc. that
occur at specific times.

class Alarm {

   private:
      std::string name; ///< name for the alarm

      bool ringing;   ///< alarm is currently ringing
      bool periodic;  ///< alarm rings periodically on interval
      bool stopped;   ///< alarm has been stopped and not reset

      TimeInstant  ringTime;     ///< time at/after which alarm rings
      TimeInterval ringInterval; ///< interval at which this alarm rings
      TimeInstant  ringTimePrev; ///< previous alarm time for interval alarms

   public:
};

The clock class is meant to track and manage the time for a model
advancing in time. Alarms can be attached to the clock so that
the ringing status can be updated as the clock marches forward.

class Clock {

   private:

      TimeInstant  startTime; ///< initial time for this clock
      TimeInstant  currTime;  ///< current time
      TimeInstant  prevTime;  ///< time at previous timestep
      TimeInstant  nextTime;  ///< time at next timestep
      TimeInterval timeStep;  ///< interval at which this clock advances

      int numAlarms; ///< current number of attached alarms

      std::vector<Alarm *> alarms; ///< pointers to alarms associated with this clock

   public:
};

The classes above have a number of methods associated with them for
managing time.

The TimeFrac class is a base class that is not meant to be
accessed by users but provides the integer fraction time
capability needed for accumulating time. It contains a number
of operators to perform arithmetic on fractional integers
and a number of accessor functions to convert units into
a fractional time.

// Accessor methods

/// Single call to set all native base time components
/// \return error code
int set(const long long whole, ///< [in] whole seconds
        const long long numer, ///< [in] fractional second numerator
        const long long denom  ///< [in] fractional second denominator
        );
/// Set base time by converting from integer hours, minutes, seconds
int setHMS(const int hours,   ///< [in] integer hours
           const int minutes, ///< [in] integer minutes
           const int seconds  ///< [in] integer seconds

/// Set base time by converting from a real number of seconds
int setSeconds(const double seconds ///< [in] Time in real seconds
              );
/// Set base time by converting from a real number of hours
int setHours(const double hours ///< [in] Time in real hours
            );
/// Set base time by converting from a real number of minutes
int setMinutes(const double minutes ///< [in] Time in real minutes
              );

/// Set whole seconds separately
int setWhole(
       const long long whole ///< [in] Whole number of seconds
       );
/// Set numerator of fractional seconds separately
int setNumer(
       const long long numer ///< [in] Numerator of fractional seconds
       );
/// Set denominator of fractional seconds separately
int setDenom(
       const long long denom ///< [in] Denominator of fractional seconds
       );

/// Single call to retrieve native base time components
int get(long long &whole, ///< [out] whole seconds
        long long &numer, ///< [out] fractional second numerator
        long long &denom  ///< [out] fractional second denominator
        ) const;
/// Get base time converted to integer hours, minutes, seconds
int getHMS(int &hours,   ///< [out] integer hours
           int &minutes, ///< [out] integer minutes
           int &seconds  ///< [out] integer seconds
           ) const;
/// Get base time and convert to a real number of seconds
/// \return Time in real seconds
double getSeconds(void) const;
/// Get base time and convert to a real number of hours
/// \return Time in real hours
double getHours(void) const;
/// Get base time and convert to a real number of minutes
/// \return Time in real minutes
double getMinutes(void) const;
/// Retrieve the whole seconds component of base time
/// \return Whole number of seconds
long long getWhole(void) const;
/// Retrieve the numerator component of fractional base time
/// \return Numerator of fractional seconds
long long getNumer(void) const;
/// Retrieve the denominator component of fractional base time
/// \return Denominator of fractional seconds
long long getDenom(void) const;

// constructors/destructors
/// Default base time constructor
TimeFrac(void);
/// Copy constructor for base time
TimeFrac(const TimeFrac& ///< [in] existing base time to be copied
        );
/// Construct base time by component
TimeFrac(
    const long long whole, ///< [in] whole seconds
    const long long numer, ///< [in] fractional second numerator
    const long long denom  ///< [in] fractional second denominator
    );
/// Construct base time by converting from a real number of seconds
TimeFrac(const double seconds ///< [in] Time in real seconds
        );
/// Destructor for base time
~TimeFrac(void);

// operators
/// Equivalence comparison operator for TimeFrac
bool operator==(const TimeFrac &) const;
/// Non-equivalence comparison operator for TimeFrac
bool operator!=(const TimeFrac &) const;
/// Less than comparison operator for TimeFrac
bool operator< (const TimeFrac &) const;
/// Greater than comparison operator for TimeFrac
bool operator> (const TimeFrac &) const;
/// Less than or equal comparison operator for TimeFrac
bool operator<=(const TimeFrac &) const;
/// Greater than or equal comparison operator for TimeFrac
bool operator>=(const TimeFrac &) const;
/// Addition operator for TimeFrac
TimeFrac  operator+ (const TimeFrac &) const;
/// Subtraction operator for TimeFrac
TimeFrac  operator- (const TimeFrac &) const;
/// Increment operator for TimeFrac
TimeFrac& operator+=(const TimeFrac &);
/// Decrement operator for TimeFrac
TimeFrac& operator-=(const TimeFrac &);
/// Multiplication by integer scalar
TimeFrac  operator* (const int multiplier) const;
/// Multiplication in place by integer scalar
TimeFrac& operator*=(const int multiplier);
/// Multiplication by real scalar
TimeFrac  operator* (const double multiplier) const;
/// Multiplication in place by real scalar
TimeFrac& operator*=(const double multiplier);
/// Divide TimeFrac by integer scalar
TimeFrac  operator/ (const int divisor) const;
/// Divide TimeFrac in place by integer scalar
TimeFrac& operator/=(const int divisor);
/// Divide two TimeFracs and return a real result
double   operator/ (const TimeFrac &) const;
/// Modulus method for TimeFrac
TimeFrac  operator% (const TimeFrac &) const;
/// Modulus method in place
TimeFrac& operator%=(const TimeFrac &);
/// Assignment operator for TimeFrac
TimeFrac& operator=(const TimeFrac &);

// Other utility methods
/// Convert a time fraction to new denominator
int convert(const long long denom ///< [in] new denominator
           );
/// Reduce a time fraction to simplest form
int simplify(void);

The calendar class is a mostly immutable class that
holds information about the chosen calendar for use by other
time manager classes. It mostly constructs an instance based
on the user-selected calendar. Retrieval functions and query
functions are provided. An equivalence/non-equivalence operator
is needed for the later TimeInstant equivalence. Utility functions
to convert between elapsed time and calendar dates and incrementing
calendar time are supplied for use by other time manager routines.

// accessor functions
// this is (mostly) an immutable class so use constructors
// and provide only one set accessor for renaming

// the only set function is for renaming
/// Renames a Calendar to the input string
/// \return Error code
int rename(const std::string inName ///< [in] name to use for calendar
           );

/// Retrieve any/all calendar properties
int get(int           *outId,   ///< [out] id assigned to calendar
        std::string   *outName, ///< [out] Name of calendar
        CalendarKind  *outKind, ///< [out] Kind of calendar
        int           *outDaysPerMonth,  ///< [out] Days per month
        int           *outMonthsPerYear, ///< [out] Months per year
        int           *outSecondsPerDay, ///< [out] Seconds per day
        int           *outSecondsPerYear,///< [out] Seconds per year
        int           *outDaysPerYear    ///< [out] Days per year (DPY)
        ) const;

// Might also add specific retrievals for individual
// components, eg seconds per day/year?

/// Default constructor
Calendar(void);
/// Copy constructor
Calendar(const Calendar &calendar);
/// Constructor based on kind of calendar
Calendar(std::string inName,       ///< [in] name of calendar
         CalendarKind calKind      ///< [in] choice of calendar kind
         );
/// Constructs custom calendar based in inputs
Calendar(const std::string inName, ///< [in] name of calendar
         int *inDaysPerMonth,      ///< [in] array of days per month
         int inSecondsPerDay,      ///< [in] seconds per day
         int inSecondsPerYear,     ///< [in] seconds per year
         int inDaysPerYear         ///< [in] days per year (dpy)
         );
/// Calendar destructor
~Calendar(void);

/// Calendar equivalence operator
bool operator==(const Calendar &calendar) const;
/// Calendar non-equivalence operator
bool operator!=(const Calendar &calendar) const;

/// Checks whether input year is a leap year
/// \return true if year is a leap year, false otherwise
bool isLeapYear(long long year, ///< [in]  year to check
                int &rc         ///< [out] return code to flag errors
                ) const;

/// Computes the total elapsed time in seconds (in TimeFrac form)
/// since the calendar reference time, given a calendar date, time.
/// \return Elapsed time in TimeFrac form
TimeFrac getElapsedTime(
    const long long year,   ///< [in] calendar year
    const long long month,  ///< [in] calendar month
    const long long day,    ///< [in] calendar day
    const long long hour,   ///< [in] time of day-hour
    const long long minute, ///< [in] time of day-min
    const long long whole,  ///< [in] time of day-whole seconds
    const long long numer,  ///< [in] time of day-frac secs (numerator)
    const long long denom   ///< [in] time of day-frac secs (denom)
    ) const;

/// Determines the calendar date and time of day, given an
/// elapsed time since the calendar reference time.
int getDateTime(
    const TimeFrac elapsedTime, ///< [in] time in secs from ref time
    long long &year,   ///< [out] calendar year
    long long &month,  ///< [out] calendar month
    long long &day,    ///< [out] calendar day
    long long &hour,   ///< [out] time of day-hours
    long long &minute, ///< [out] time of day-minutes
    long long &whole,  ///< [out] time of day-whole seconds
    long long &numer,  ///< [out] time of day-frac secs (numerator)
    long long &denom   ///< [out] time of day-frac secs (denom)
    ) const;

/// Increments (or decrements) a calendar date by a specified
/// interval, supplied by an integer interval in given time units.
/// Only calendar based intervals (years, months or days) are
/// supported. This is primarily meant to be called by other time
/// manager routines (eg to add/subtract time instants) for those
/// time intervals that are dependent on date and sensitive to
/// calendar features like leap years and varying days of the month.
int incrementDate(
    const long long interval, ///< [in] time interval to advance date
    const TimeUnits units,    ///< [in] time units for interval
    long long &year,   ///< [in,out] calendar year of time to be changed
    long long &month,  ///< [in,out] calendar month of ...
    long long &day     ///< [in,out] calendar day
    ) const;

The time instant class represents a single instant in time.
Most of the methods are accessor methods for getting/setting
a time instant or its components. In addition, a number of
operators are defined for performing basic arithmetic with
time instants and time intervals (see following interval class).
Finally, a method for creating a time string for a time
instant is supplied.

/// Default constructor creates empty time instant
TimeInstant(void);

/// Construct time instant from date, time, calendar
/// Where seconds is supplied as real number.
TimeInstant(Calendar       *Cal,    ///< [in] Calendar to use
            const long long year,   ///< [in] year
            const long long month,  ///< [in] month
            const long long day,    ///< [in] day
            const long long hour,   ///< [in] hour
            const long long minute, ///< [in] minute
            const double    rSecond ///< [in] second (real)
            );

/// Where seconds is supplied in integer fractional seconds.
TimeInstant(      Calendar *Cal,    ///< [in] Calendar to use
            const long long whole,  ///< [in] second (whole integer)
            const long long numer,  ///< [in] second (fraction numerator)
            const long long denom   ///< [in] second (fraction denominator)
            );

/// Destructor for time interval
~TimeInstant(void);

/// Set time instant calendar
int set(Calendar *Cal   ///< [in] Calendar to use for this time
        );

/// Set time instant from date and time, where seconds is supplied
/// as a real number.
int set(const long long year,   ///< [in] year
        const long long month,  ///< [in] month
        const long long day,    ///< [in] day
        const long long hour,   ///< [in] hour
        const long long minute, ///< [in] minute
        const double    rSecond ///< [in] second (real)
        );

/// in integer fractional seconds.
        const long long whole,  ///< [in] second (whole integer)
        const long long numer,  ///< [in] second (fraction numerator)
        const long long denom   ///< [in] second (fraction denominator)
        );

/// Retrieve calendar from time instant
int get(Calendar *&cal ///< [out] Calendar ptr in which instant defined
        ) const;

/// Retrieve time in date, time form with real seconds.
int get(long long &year,   ///< [out] year   of this time instant
        long long &month,  ///< [out] month  of this time instant
        long long &day,    ///< [out] day    of this time instant
        long long &hour,   ///< [out] hour   of this time instant
        long long &minute, ///< [out] minute of this time instant
        double    &second  ///< [out] second of this time instant
        ) const;

/// Retrieve time in date, time form with fractional integer seconds.
        long long &whole,  ///< [out] whole seconds of this time
        long long &numer,  ///< [out] frac second numerator
        long long &denom   ///< [out] frac second denominator
        ) const;

// Operators on time instants
/// Equivalence comparison for TimeInstant
bool operator==(const TimeInstant &) const;
/// Non-equivalence comparison operator for TimeInstant
bool operator!=(const TimeInstant &) const;
/// Less than comparison operator for TimeInstant
bool operator< (const TimeInstant &) const;
/// Greater than comparison operator for TimeInstant
bool operator> (const TimeInstant &) const;
/// Less than or equal comparison operator for TimeInstant
bool operator<=(const TimeInstant &) const;
/// Greater than or equal comparison operator for TimeInstant
bool operator>=(const TimeInstant &) const;
/// Increment time by adding a time interval
TimeInstant  operator+ (const TimeInterval &) const;
/// Decrement time by subtracting a time interval
TimeInstant  operator- (const TimeInterval &) const;
/// Create a time interval by subtracting two time instants
TimeInterval operator- (const TimeInstant &) const;
/// Increment time in place by adding time interval
TimeInstant& operator+=(const TimeInterval &);
/// Decrement time in place by subtracting time interval
TimeInstant& operator-=(const TimeInterval &);

/// Get time as a string in the format
/// 'YYYYYY-MM-DD{separator}HH:MM:SS.SSSSSS' where the number
/// of digits in year, number of digits after the decimal in
/// seconds and the character to use as separator are all input
/// by the user.
/// \return time string
std::string getString(
    const int yearWidth,   ///< [in] number of digits in year
    const int secondWidth, ///< [in] num digits after decimal in seconds
    std::string separator  ///< [in] string(char) to separate date/time
    ) const;

The time interval class manages differences or increments in time.
The user can create or set a time interval with a value and units.
Retrieval functions for the time interval in various forms are
also supplied. A number of operators are defined for a variety
of mathematical operations on time intervals (add, subtract,
multiply, equivalence, absolute value). Finally, the time instant
class is treated as a friend class so that time instants can
be incremented by a time interval and time intervals can be created
by subtracting two time instants.

/// Default time interval constructor
TimeInterval(void);

/// Construct time interval from base time fractional integer seconds
TimeInterval(const long long whole, ///< Whole integer seconds
             const long long numer, ///< Fractional seconds numerator
             const long long denom  ///< Fractional seconds denominator
             );

/// Construct time interval from an integer length and units
TimeInterval(const int       length, ///< length of time interval
             const TimeUnits units   ///< unit of time for interval
             );

/// Construct time interval from a long long integer length and units
TimeInterval(const long long length, ///< length of time interval
             );

/// Construct time interval from an real length and units
TimeInterval(const double length,    ///< length of time interval
             );

~TimeInterval(void);

/// Set a non-calendar interval in native fractional integer seconds
int set(const long long whole, ///< Whole integer seconds
        const long long numer, ///< Fractional seconds numerator
        const long long denom  ///< Fractional seconds denominator
        );

/// Set a time interval from integer length and units
int set(const int       length, ///< length of time interval
        const TimeUnits units   ///< unit of time for interval
        );

/// Set a time interval from long long integer length and units
int set(const long long length, ///< length of time interval
        );

/// Set a time interval from an real length and units
/// Real length is only supported for non-calendar intervals since
/// a non-integral length has ambiguous meaning when months, years
/// have variable length.
int set(const double length,    ///< length of time interval
        );

/// Retrieve non-calendar interval in native fractional integer form
        ) const;

/// Retrieve a time interval in integer length in specified units.
/// To avoid roundoff issues during conversions, integer retrieval
/// is only permitted in the same units in which the interval was
/// defined.
int get(long long &length, ///< [out] requested integer length of interval
        const TimeUnits units ///< [in] unit of time for interval
        ) const;

/// Retrieve a time interval in real length and specified units.
/// For calendar intervals, the units must match the units in which
/// the interval was defined. For non-calendar intervals, only conversions
/// between hours, minutes and seconds are allowed.
int get(double &length, ///< [out] Requested time interval length
       ) const;

/// Equivalence comparison operator for TimeInterval
bool operator==(const TimeInterval &) const;
/// Non-equivalence comparison operator for TimeInterval
bool operator!=(const TimeInterval &) const;
/// Less than comparison operator for TimeInterval
bool operator< (const TimeInterval &) const;
/// Greater than comparison operator for TimeInterval
bool operator> (const TimeInterval &) const;
/// Less than or equal comparison operator for TimeInterval
bool operator<=(const TimeInterval &) const;
/// Greater than or equal comparison operator for TimeInterval
bool operator>=(const TimeInterval &) const;
/// Addition operator for TimeInterval
TimeInterval  operator+ (const TimeInterval &) const;
/// Subtraction operator for TimeInterval
TimeInterval  operator- (const TimeInterval &) const;
/// Increment operator for TimeInterval
TimeInterval& operator+=(const TimeInterval &);
/// Decrement operator for TimeInterval
TimeInterval& operator-=(const TimeInterval &);
TimeInterval  operator* (const int multiplier) const;
/// This is primarily meant for non-calendar intervals, but
/// for calendar intervals, it will multiply the year, month or
/// day interval and convert to the nearest integer.
TimeInterval  operator* (const double multiplier) const;
TimeInterval& operator*=(const int multiplier);
TimeInterval& operator*=(const double multiplier);
/// Divide interval by integer scalar
TimeInterval  operator/ (const int divisor) const;
/// Divide interval in place by integer scalar
TimeInterval& operator/=(const int divisor);

/// Absolute value
static TimeInterval absValue(const TimeInterval &);
/// Negative absolute value
static TimeInterval negAbsValue(const TimeInterval &);

/// Check whether a time interval is positive
bool isPositive(void);

/// Give the Time Instant access to Time Interval so
/// that incrementing/decrementing time is easier.
friend class TimeInstant;

}

The alarm class allows the creation of both one-time and
periodic alarms. There are also methods for querying the
state of an alarm and for resetting.

/// Constructs a one-time alarm using the input ring time.
Alarm(const std::string inName,   ///< [in] Name of alarm
      const TimeInstant alarmTime ///< [in] Time at/after which alarm rings
     );

/// Constructs a periodic/interval alarm based on an input periodic
/// time interval and a start time for the interval period.
Alarm(const std::string  inName,        ///< [in] Name of alarm
      const TimeInterval alarmInterval, ///< [in] interval at which alarm rings
      const TimeInstant  intervalStart  ///< [in] start time of first interval
     );

/// Destructor for alarm
~Alarm(void);

/// Check whether an alarm is ringing
/// \return true if alarm is ringing, false otherwise
bool isRinging(void);

/// Checks whether the alarm should ring based on the current
/// (or supplied) time instant (returns error code)
int updateStatus(const TimeInstant currentTime ///< [in] current time
                );

/// Stops a ringing alarm and sets next a new ring time. If the alarm
/// is a periodic/interval alarm, the next ring time is set to be the
/// next interval boundary after the input time.  If the alarm is a
/// single instance, the input time is used as the next alarm time.
/// (returns error code)
int reset(const TimeInstant inTime ///< [in] new basis for alarm time
         );

/// Stops a ringing alarm (returns error code).
int stop(void);

/// Rename an alarm (returns error code)
int rename(const std::string newName ///< [in] new name for alarm
          );

/// Get alarm name
std::string getName(void) const;

The Clock class defines the model clock and is created based
on a start time and time step. There are a number of retrieval
functions to get the current, past, and next times. A number
of alarms can be attached to a clock. Finally, there is
a method to advance the clock one time and update the state
of all attached alarms.

/// Construct a clock from start time and time step
Clock(const TimeInstant  startTime, ///< [in] Start time for clock
      const TimeInterval timeStep   ///< [in] Time step to advance clock
     );

/// Destructor for clocks
~Clock(void);

// Accessor Methods
/// Set the current time (returns error code)
int setCurrentTime(
    const TimeInstant currTime ///< [in] new value for current time
    );

/// Changes the time step for this clock (returns error code)
int changeTimeStep(
    const TimeInterval newTimeStep ///< [in] new value for time step
    );

/// Retrieves current time of this clock
TimeInstant getCurrentTime(void) const;

/// Retrieves time at the previous time step
TimeInstant getPreviousTime(void) const;

/// Retrieves time at the next time step
TimeInstant getNextTime(void) const;

/// Retrieves start time for this clock
TimeInstant getStartTime(void) const;

/// Retrieves time step for clock
TimeInterval getTimeStep(void) const;

/// Attaches an alarm to this clock. The clock simply stores a
/// pointer to this alarm. (returns error code)
int attachAlarm(Alarm* inAlarm ///< [in] pointer to alarm to attach
               );

/// Advance a clock one timestep and update status of any attached
/// alarms. (returns error code)
int advance(void);

We will test the base TimeFrac class by creating some reference
times, one of which will be a fractional time that cannot be
exactly represented by a float (eg 1/3 seconds). Constructors and
set functions will be paired with get functions to make sure both
set/get are working. All operators (equivalence, relational,
algebraic) operators will be tested with known values. We will
try to anticipate values that could fail.

Similar to TimeFrac, create reference times in each of the
supported calendars using constructor and set functions. Test
retrievals (get) to verify they return the reference value.
Test all the supported TimeInstant operators with known values
in each of the calendars.

For each of the supported calendars (including a custom
calendar), create an instance of the calendar and use
the retrieval (get) functions to verify that various
calendar properties are as expected (eg days per year,
days per month, leap years/days, etc.). For some calendars,
known reference dates exist from the references above and
we will verify that these result in correct elapsed time
or Julian day. Test the few operators (equivalence, copy
constructors).

As in the others, test constructors and set functions by
pairing with retrieval (get) functions for a given
reference time interval. Test assignment constructors
by creating a time interval for a given time step in
various time units (seconds, hours, days, etc.). Test
equivalence/non-equivalence by comparing known time intervals.
Test algebraic operators (addition, subtraction, increment,
decrement, multiply, divide, etc.) using reference values.
Test creation of time interval by subtraction of time instants.

Test both one-time and periodic alarms at various supported
intervals, including annual, monthly/nmonths, daily/ndays,
hourly/nhours, nminutes, seconds. This is accomplished by
choosing a start time in a calendar and integrating over a
time period that covers the alarm. Use the Gregorian calendar
(as most complex) and choose an interval that includes leap
years/days. Also verify that alarms are not ringing at arbitrary
selected times in between. Check the reset function and verify
alarms are reset appropriately.

This is the primary test of putting it all together and
verifying that we can advance a model clock. Construct a
clock with a given calendar and time step. Test various
retrieval functions to get known quantities - also test
the ability to change time step with set/get pair. Set a
number of alarms at various times and periodic intervals.
Integrate forward in time for N years and verify all alarms
are triggered at proper times. Also verify that the final
time is exact with no roundoff error accumulated. Create
another clock with large time steps (years) that extend over a
potential paleoclimate interval to test large number
representations are supported correctly.

The solution methods for a partial differential equation come in two categories: the spatial discretizations and the temporal discretizations. Here we present the methods of solving the time derivative, $\partial / \partial t$, in the Omega-0 shallow water equations. The spatial discretizations are presented in the {ref}`omega-design-shallow-water-omega0` design document.

The solution methods for a partial differential equation come in two categories: the spatial discretizations and the temporal discretizations. Here we present the methods of solving the time derivative, 

, in the Omega-0 shallow water equations. The spatial discretizations are presented in the 

The time discretization scheme must be able to advance the shallow water equations with at least second-order accuracy for standard shallow water test cases.

The scheme must be stable for reasonably long time steps compared to other schemes.

The time stepping scheme must follow the scaling and performance criteria outlined in the {ref}`omega-design-shallow-water-omega0` design document. It will be tested with metrics for:

The time stepping scheme must follow the scaling and performance criteria outlined in the 

The first two items will be compared against MPAS-Ocean with the same test case and resolution.

The time stepping code must be written in such a way that other time stepping schemes can be chosen, and the code is organized in a clear and readable fashion.

The total volume (area integrated layer thickness $h$) and total tracer (area integrated $\phi$) must be conserved. This means that the totals should remain constant to machine precision in time.

) must be conserved. This means that the totals should remain constant to machine precision in time.

For the manufactured solution and time-dependant forcing (winds, tides), the right-hand side functions must accept a specific time argument, which is the time that the RHS is associated with.

The primary time stepping scheme will be forward-backward. This scheme is commonly used in shallow water models and the barotropic mode of layered models. It suits the design requirements because it is second order, is stable for reasonably long time steps compared to competing schemes, and may be implemented in a performant manner.

Given the shallow water equations introduced in the {ref}`omega-design-shallow-water-omega0` design document, we may consolidate the right-hand-side tendency terms to express them as

 design document, we may consolidate the right-hand-side tendency terms to express them as

\frac{\partial \boldsymbol{u}}{\partial t} = \mathcal{RHS}_u \left( \boldsymbol{u},h,t \right),

\frac{\partial h}{\partial t} = \mathcal{RHS}_h \left( \boldsymbol{u},h,t \right),

\frac{\partial h \phi}{\partial t} = \mathcal{RHS}_\phi \left( \boldsymbol{u},h,\phi,t \right).

The forward-backward scheme is simply a single forward time step, but always using the most recent information available:

h^{n+1} = h^n + \mathcal{RHS}_h \left( \boldsymbol{u}^n,h^n,t^n \right) \Delta t

\phi^{n+1} = \frac{1}{h^{n+1}} \left( \phi^n h^n + \mathcal{RHS}_\phi \left( \boldsymbol{u}^n,h^n,\phi^n, t^n \right) \Delta t \right)

\boldsymbol{u}^{n+1} = \boldsymbol{u}^n + \mathcal{RHS}_u \left( \boldsymbol{u}^n,h^{n+1}, t^{n+1} \right) \Delta t

Here the variables are discretized in time only. Spatial discretization can be shown with a subscript if desired, using $u^n_e, h^n_i, \phi^n_i$ etc.
The time domain is discretized into steps $t_0, t_1, ... t_n, t_{n+1}$ where $t_{n+1} = t_n+\Delta t$. The superscript indicates the time on all variables. For example, $h^n = h\left( t_n \right)$.

Here the variables are discretized in time only. Spatial discretization can be shown with a subscript if desired, using 

. The superscript indicates the time on all variables. For example, 

Note that the tracer equation is thickness-weighted, but we divide by the thickness in order to obtain the tracer variable $\phi$, which has units of concentration.
One might argue that the tracer equation could use $h^{n+1}$ in the right-hand-side calculation, because that quantity is available from the previous computation. However, that would violate the consistency of the solution method of the thickness and tracer equations, and cause the globally-integrated tracer to not be conserved. Like the continuous equations, when we set $\phi=1$ the numerical method for $\phi h$ must reduce to the numerical method for $h$.

Note that the tracer equation is thickness-weighted, but we divide by the thickness in order to obtain the tracer variable 

 in the right-hand-side calculation, because that quantity is available from the previous computation. However, that would violate the consistency of the solution method of the thickness and tracer equations, and cause the globally-integrated tracer to not be conserved. Like the continuous equations, when we set 

If time permits, other schemes will be added and tested. This includes the four-stage Runge-Kutta (RK4) for direct comparison to MPAS-Ocean. Other possibilities are the second-order Adams-Bashforth (AB2) which was implemented in MPAS-Ocean and the Heun's method (RK2). Including multiple time-stepping schemes allows developers to test the modularity requirement above.

If time permits, other schemes will be added and tested. This includes the four-stage Runge-Kutta (RK4) for direct comparison to MPAS-Ocean. Other possibilities are the second-order Adams-Bashforth (AB2) which was implemented in MPAS-Ocean and the Heun’s method (RK2). Including multiple time-stepping schemes allows developers to test the modularity requirement above.

The timestepping will be tested with a time-only convergence test on the equations
$$
\frac{\partial \boldsymbol{u}}{\partial t} =
 -Ra \, \boldsymbol{u}
$$

\frac{\partial h \phi}{\partial t} =
- \frac{h}{\tau} \left( \phi - \phi_0 \right)

Where the initial conditions for $\boldsymbol{u}$ and $\phi$ are uniformly one, the tracer restoring value in this case is $\phi_0=0$, and the solution is simply exponential decay to zero.

The timestepping is an intrinsic part of the Omega-0 model, so the verification and testing are identical to those in the {ref}`omega-design-shallow-water-omega0` design document. The timestepping will be tested with convergence tests in time using the manufactured solution test case.

The timestepping is an intrinsic part of the Omega-0 model, so the verification and testing are identical to those in the 

 design document. The timestepping will be tested with convergence tests in time using the manufactured solution test case.

Within OMEGA, we will need a method for profiling aspects
of code performance. In the most simple form, this requires
the ability to report the time spent in various sections of
code. It may also be desirable to track other aspects of
performance or to access hardware timers, though there may be
external tools more appropriate for more detailed profiling.

We require the ability to track the time spent in a selected
code section. This requires the ability to start, stop and
accumulate time in a timer with a given name.

For tracking load balance, we require the timers to report
minimum, maximum and avg time across MPI tasks in a parallel
simulation.

Related to the statistics requirement, timers may be called
from a code region that is operating within a subset of the
default MPI environment so must track the context in which
they are called.

Because some utility functions and related timers may be
called from multiple sections of code, we require the timers
to track where in the call sequence the timer occurs and to
report the time separately when called from a different
location in the code. When reporting, it is desirable to
format timer output with indentation reflecting the level of
the call tree where the timer is called.

We require timer output to be compatible with existing
E3SM performance tools. Output in other common formats
for performance tools is also desired. Output in
human-readable form is also required and it is often useful
for at least a summary profile in readable form be
included in the Log file output.  For consistency with
other E3SM components, we may wish to support the General
Purpose Timing Library (GPTL) as a timer option.

Timers must cover all code within Omega to provide a
complete profile. Granularity should be at least at the
function/routine level, though finer granularity may
be desired for larger routines. It may be desireable
to provide a preprocessor option to support very fine
granularity and more detail if desired.

Because OMEGA allows portions of code to run on a subset
of processors or alternative parallel decompositions, the
timers must be able to distinguish the context so that
statistics across MPI tasks are computed correctly.

It is useful to track performance aspects of hybrid
computing nodes, including kernel launch times and
device timers. However, it is likely that that external
profiling tools are a better way to access this information.

It may be desirable in the future to access other hardware
counters in addition to simple timers (eg flops, memory ops,
cache misses, etc.) to provide additional detail in diagnosing
performance bottlenecks. There are interfaces like PAPI that
may provide a portable means of accessing counters when
available, but this profiling may be better performed with
other tools.

We do not anticipate having large on-node CPU threaded
regions (threads are currently loop-level), so we anticipate
all timers will be called outside of threaded regions.
However, such a capability might be needed in the future.

It would be nice to have the capability of tracking
memory use (eg high-water mark) to diagnose memory leaks
or overall memory use by the model. Other memory metrics
are also useful but may fall under 2.9 above or be tracked
via external profiling tools.

We will likely make use of external profiling tools for
additional diagnostics. While some tools use sampling, others
may require inserting code to assist profiling. It may be
beneficial to have generic interfaces that could support
profiling calls and be optionally activated.

There are no specific algorithms associated with this capability.

The initial implementation will include only a timer class.
We will support both a reference implementation using C++
stdlib timers as well as a GPTL (General Purpose Timing Library)
option that can be selected at compile time.

At compile time, the user can optionally select the GPTL
timers as a pre-processor option (USE_GPTL). In the future,
a TIMER_LEVEL pre-processor option might be provided to
identify the granularity of timers, but this is not currently
implemented.

Currently the only timer option set in the run time configuration
is the filename (or filename template) where the timers will
be printed - a blank entry, None or Log will send the output
to the standard Logging output.  Other options may be added
in the future.

The timers are implemented with a single timer class. For
the reference implementation, this will contain:

class Timer

 private:
   /// Timer name
   std::string TimerName;

   /// Context in which this timer is called
   std::shared_ptr<MachEnv> Context;

   /// Flag to determine when the timer is running
   bool IsRunning;

   /// Number of times the timer is started/called
   /// Used for timer statistics
   I8 NumCalls;

   /// Start time for the current timer interval
   R8 StartTime;

   /// Accumulated time for this timer
   R8 AccumTime;

   /// Track child and parent timers
   std::shared_ptr<Timer> ParentTimer;
   std::vector<std::shared_ptr<Timer>> ChildTimers;

   /// Current active timer
   /// Used with the above to keep track of call sequence
   static std::shared_ptr<Timer> CurTimer;

   /// Level in the timer call hierarchy
   I4 CallLevel;

   /// Timers will be output with this filename or the
   /// constructed filename based on this template
   static std::string TimerFile;

 public:
    // see methods described later

When using external libraries (eg GPTL), those libraries
typically manage the timers internally and the above class will
be largely empty of data but will retain the interfaces
and some of the static members that remain relevant
(eg the output filename).

All of the methods below are static functions.
The user will be responsible for inserting any needed or desired
synchronization like MPI barriers or Kokkos fences before
timer calls.

The start method is used to either create a new timer with a
given name or start an existing timer of that name:

This assumes the timer is called from the default machine
environment. If the timer is started in a different context
than the default machine environment (eg from a defined subset
of tasks), then the relevant MachEnv name must also be passed:

Timer::start(const std::string &TimerName,
             const std::string &MachEnvName);

The stop method stops a running timer and accumulates the
time in a running sum.

The timer print method stops all timers, computes timer
statistics and prints the results of all timers either to a
file or to stdout. The output will be formatted based
on the call sequence with child timers indented
from their parents. Statistics will include the number of
calls to the timer, the min, max and mean across MPI tasks,
the percentage of total run time and the percentage of the
parent timers' run time.

An initialize function will set any run-time options from
the configuration and start a top-level timer named Total.
If an external timing library is used, it will initialize
that library and its options.

A finalize method will stop all timers (including Total),
call the print method to output the timing data, and
clear all defined timers.

A unit test with routines and loops of varying run
times will be used to test multiple timers and timer
statistics across MPI tasks. Visual inspection of the
timer output will be used to determine compliance with
formatting.

The above test code will be repeated after building
and linking with the GPTL library

Tracers refer to either heat (temperature) or material carried by a fluid
parcel (e.g., salt, chemical, or biological constituents). This document
describes a tracer module that manages the definition, description, storage,
and access of tracers throughout OMEGA. For reasons outlined in the following
requirements, specific methods or algorithms for transporting or mixing
tracers are not addressed here. These algorithms will be covered in other
design documents.

The tracer module must provide a centralized location where all supported
tracers are defined. This definition includes all metadata associated with
the tracers—see related documents for metadata and I/O. Where possible,
CF-compliant metadata must be used.

The tracer module must provide a simple way to identify a tracer, either
by name or by index. The latter may be preferred for performance reasons.

It is often convenient to select or define properties for tracers as
a group. It must be possible to assign tracers to particular groups.
Properties of the group should apply to all tracers assigned to that group.
Tracers can belong to multiple groups, as long as properties of each group
do not conflict. A group called "All" will include all selected tracers.
For performance reasons, it may be desirable for some non-intersecting
groups to be assigned contiguous memory or contiguous index space.

do not conflict. A group called “All” will include all selected tracers.

At the start of a simulation, users must be able to select which of the
supported tracers are enabled for the experiment via the configuration
input file. To simplify the configuration, tracers should be selectable
by group as well as individually.

With few exceptions, tracers will generally be included in all restarts and
often in other outputs for post-processing and analysis. Thus, it is
essential for tracers to be registered as available I/O fields (see related
I/O designs) so that they can be read from or written to a file.

Time integration schemes often require the storage of multiple time levels.
Tracer storage must support multiple time levels, and a mechanism for
specifying which time level is needed for a given calculation must be
provided. Ideally, changing time levels at the end of a time step should
avoid a memory copy and instead involve a simple change of time index or
pointer.

Some tracers or tracer groups can be evolved using a longer time step. For
performance reasons, it should be possible to advance these tracers with a
longer time step and avoid computing tendencies at every dynamical time step.

Different tracers may require different methods for their evolution to
preserve desired properties. For example, temperature and salinity should
maintain consistency with the continuity equation. Many tracers will require
monotonic or positive-definite advection schemes to prevent negative
concentrations. We may even use a different time-stepping scheme for
supercycled tracers.

The algorithms used to advect, mix, and evolve tracers in time are described
elsewhere in the respective modules responsible for computing these tendencies.

We will store all tracers in a vector of 3-dimensional device arrays.

  std::vector<Array3DReal> TracerArrays; // multiple time levels

Each device array in the vector has dimensions corresponding to the number
of tracers, the number of all cells in the rank, and the vertical length.
This allows parallelization over tracer and cell dimensions, with the vertical
dimension either parallel or vectorized, depending on the architecture and
computation involved. Many of the requirements for grouping, selecting, and
swapping time levels can be achieved through the collection and manipulation
of array indices. Individual tracers can still be extracted as arrays with
contiguous memory.

All tracers will be defined in a set of code blocks (described below) in
a file named `TracerDefs.inc`, which will be directly included in the
initialization routine to define all supported tracers and their metadata.
Tracer groups are defined as sets of tracer indices in the OMEGA YAML
configuration file.

All supported tracers are defined in a file called TracerDefs.inc, which
includes the code blocks needed to define each tracer, its metadata, and
tracer index variables. Tracer index variables, such as IndxTemp and
IndxSalt shown below, keep the index value for each tracer, allowing users
to easily access the corresponding tracer data using the variable. This
file must include all tracers that might potentially be used in the model.
The file will look something like this:

// At the top of the file, list all available tracers in a comment
// to provide a sort of table of contents.
// Tracers:
//   Temp
//   Salt
//   [Others...]

// Index defined for each tracer. The value of these tracer indices are still
// set in Tracers::init()  and tracers that have not been selected would have
// standard invalid value: Tracers::IndxInvalid = -1
inline static I4 IndxTemp             = Tracers::IndxInvalid;
inline static I4 IndxSalt             = Tracers::IndxInvalid;
inline static I4 IndxMyBGCTracer      = Tracers::IndxInvalid;

// Tracer definitions packaged in a defineAllTracers function
static void defineAllTracers() {

   define("Temp",                            ///< [in] Name of tracer
          "Potential Temperature",           ///< [in] Long name or description
          "degree_C",                        ///< [in] Units
          "sea_water_potential_temperature", ///< [in] CF standard Name
          -273.15,                           ///< [in] min valid field value
          100.0,                             ///< [in] max valid field value
          1.e33,                             ///< [in] value for undef entries
          IndxTemp);                         ///< [out] (optional) static index

   define("Salt", "Salinity", "psu", "sea_water_salinity", 0.0, 50.0, 1.e33,
          IndxSalt);
   define("Debug1", "Debug Tracer 1", "none", "none", 0.0, 100.0, 1.e33);
   define("Debug2", "Debug Tracer 2", "none", "none", 0.0, 100.0, 1.e33);
   define("Debug3", "Debug Tracer 3", "none", "none", 0.0, 100.0, 1.e33);
}

Users will select desired tracers either individually or by group. If a group
is listed, it implies the entire group, but users can choose a subset of
a group by specifying individual tracer names and omitting the group name.

omega:
  Tracers:
    Base: [Temp, Salt]
    Debug: [Debug1, Debug2, Debug3]
    [other individual tracers or groups as needed]

For requirement 2.7 (per-tracer algorithm choice), the configuration might
look something like this:

omega:
  TracerAdv:
    # Start with default values for all tracers
    - All:
         Method: standard
         SomeParameter: 0.5
    # Specialized options - override default by group or tracer name
    - Ecosys:
         Method: MyMonotoneMethod
    - [Other tracers or tracer groups, only if needed]:
         Method: MyAdditionalOption
         SomeParameter: 1.0

There will be a single class that contains the static array for storing
all tracers, along with several containers for managing tracer groups and
indices. Methods are described in Section 4.2 below.

/// The Tracers class provides a container for tracer arrays and methods
class Tracers {

 private:

   // Total number of tracers defined at initialization
   static I4 NumTracers;

   // Static storage of the tracer arrays for device and host memory spaces
   // The arrays are 3-dimensional, with dimensions for Tracer, Cell, and
   // Vertical layers
   static std::vector<Array3DReal>
       TracerArrays; ///< Time levels -> [Tracer, Cell, Vertical]
   static std::vector<HostArray3DReal>
       TracerArraysH; ///< Time levels -> [Tracer, Cell, Vertical]

   // Maps for managing tracer groups
   // The key of this map is a group name and the value is a pair of
   // GroupStartIndex and GroupLength
   static std::map<std::string, std::pair<I4, I4>> TracerGroups;

   // Maps for matching tracer names with indices (in both directions)
   static std::map<std::string, I4> TracerIndexes;
   static std::map<I4, std::string> TracerNames;

   // Halo exchange
   static Halo *MeshHalo;

   // Tracer dimension names
   static std::vector<std::string> TracerDimNames;

   // Current time index
   // This index is circular, so it returns to index 0 when it exceeds the max
   // index
   static I4 CurTimeIndex; ///< Time dimension array index for the current level

   // Pack tracer field name
   static std::string packTracerFieldName(const std::string &TracerName);

   // locally defines all tracers but do not allocates memory
   static I4
   define(const std::string &Name,        ///< [in] Name of tracer
          const std::string &Description, ///< [in] Long name or description
          const std::string &Units,       ///< [in] Units
          const std::string &StdName,     ///< [in] CF standard Name
          const Real ValidMin,            ///< [in] min valid field value
          const Real ValidMax,            ///< [in] max valid field value
          const Real FillValue,           ///< [in] value for undef entries
          I4 &Index = IndxInvalid         ///< [out] (optional) index value
   );

  public:
    [methods defined below]
};

Here is the corrected version with improved grammar and clarity:

The initialization method defines all the tracers by including the
`TracerDefs.inc` file. Once all fields are defined, it allocates the full
tracer array based on the input configuration of selected tracers. The
interface is as follows:

Note that this function does not initialize the tracer values. The values
will be filled either by input I/O streams or by individual modules
associated with specific tracers and tracer groups.

All tracers must be defined. This routine takes the name of the tracer and
some required metadata as input. It uses this information to create
a `MetaData` entry and an `IOField` entry, but only if the tracer is
selected in the input configuration file. It also increments the number of
tracers and assigns a tracer index if the tracer is selected. The function
returns a non-zero error code if any of several possible error conditions
are encountered.

static I4
define(const std::string &Name,        ///< [in] Name of tracer
       const std::string &Description, ///< [in] Long name or description
       const std::string &Units,       ///< [in] Units
       const std::string &StdName,     ///< [in] CF standard Name
       const Real ValidMin,            ///< [in] min valid field value
       const Real ValidMax,            ///< [in] max valid field value
       const Real FillValue,           ///< [in] value for undef entries
       I4 &Index = IndxInvalid         ///< [out] (optional) index value
);

This function retrieves the total number of tracers activated for the current

This function retrieves the tracer index based on the tracer name.

// Retrieves tracer index from tracer name
static I4 getIndex(I4 &TracerIndex,              ///< [out] Tracer index
                   const std::string &TracerName ///< [in] Tracer name

);

This function retrieves the full array of all tracers at a specified time
level. The current time level is specified by an integer value of 0, the
previous time level is -1, and so on. If the requested time level does not
exist, the function returns a negative integer.

// get a device array for all tracers
static I4 getAll(Array3DReal &TracerArray, ///< [out] tracer device array
                 const I4 TimeLevel        ///< [in] time level index
);

This function retrieves a single tracer array by index and time level.
Similar to `getAll`, the current time level is specified by an integer value
of 0, and the previous time level is -1. If the requested time level does
not exist, the function returns a negative integer.

// get a device array by tracer index
static I4 getByIndex(Array2DReal &TracerArray, ///< [out] tracer device array
                     const I4 TimeLevel,       ///< [in] time level index
                     const I4 TracerIndex      ///< [in] global tracer index
);

This function retrieves a pair of the group's start index and group length.

This function retrieves a pair of the group’s start index and group length.

// Retrieves a pair of (group start index, group length)
static I4
getGroupRange(std::pair<I4, I4> &GroupRange, ///< [out] Group range
              const std::string &GroupName   ///< [in] Group name

);

// Get group range
std::pair<I4, I4> GroupRange;
I4 Err = Tracers::getGroupRange(GroupRange, GroupName);

// Unpack group range
auto [StartIndex, GroupLength] = GroupRange;

// Get all tracers at the current time level (0)
Array3DReal TracerArray;
Err = OMEGA::Tracers::getAll(TracerArray, 0);
if (Err != 0)
   LOG_ERROR("getAll returns an error code: {}.", Err);

OMEGA::parallelFor(
   "ComputeGroupTendency",
   {GroupLength, NCells, NVertLayers},
   KOKKOS_LAMBDA(Int iIndex, Int iCell, Int iVert) {
      int iTracer = TracerArray[iIndex + StartIndex];
      // Perform operations on TracerArray(iTracer, iCell, iVert);
   });

When looping over tracers, this function returns `true` if the current
tracer index belongs to a group with a given name.

// Checks if a tracer is a member of a group by tracer index
static bool
isGroupMemberByIndex(const I4 TracerIndex,      ///< [in] Tracer index
                     const std::string GroupName ///< [in] Group name
);

The exact process for managing time integration is not yet clear. We
assume a time integration module will assign and swap time indices (e.g.,
CurTime, PrevTime) to point to the appropriate array index in all
prognostic arrays like tracers. The current time is represented by
an integer value of "0", while the previous time is represented as "-1",
and so on.

an integer value of “0”, while the previous time is represented as “-1”,

This function clears all defined tracers and cleans up memory. It returns
an error code if it is unable to deallocate or destroy all memory.

// Clears all defined tracers and deallocates memory.
static int clear();

A sample `TracerDefs.inc` file will be located in the source directory,
containing the base tracers and three tracers in a Debug group. We will
assume two time levels (Cur: 0, Prev: -1). A unit test driver will read an
input configuration that requests these two tracer groups. The tracers will
be initialized with artificial data, and basic arithmetic will be performed
on the device. In one loop, we will perform arithmetic on one group using
the vector of group tracer indices. In a second loop, we will perform
operations on the second group using the `isMembers` function to select tracers.

All other requirements will need to be tested in later system tests once
I/O streams and time integration schemes have been implemented.

In geophysical models, implicit time integration of vertical terms often requires solution to tridiagonal systems of equations.
Typically, a separate system needs to be solved in each vertical column, which requires an efficient batched tridiagonal solver.
One common situation where a tridiagonal system arises is implicit treatment of vertical diffusion/mixing.
In principle, this problem results in a symmetric diagonally-dominant tridiagonal system.
However, in ocean modeling, the coefficients of vertical mixing can vary by orders of magnitude and
can become very large in some layers.
This requires specialized algorithms that can handle this situation stably.

There should be one module that provides tridiagonal solvers that are sufficiently general to handle every need in Omega.

A specialized solver that is stable for vertical mixing problems with large variations in mixing coefficients must be
provided.

When applied to the implicit vertical mixing of momentum, the specialized diffusion solver should be
sufficiently general to be able to incorporate various bottom drag and wind stress formulations.

All solvers must be performant on CPU and GPU architectures. On CPUs this means supporting vectorization.

Implicit vertical mixing will require some pre-processing (e.g. setup of the system) and post-processing work.
It is desirable to handle all of that in one computational kernel. This requires the ability to call the
solvers inside a `parallelFor`.

A general tridiagonal system has the form:
$$
a_i x_{i - 1} + b_i x_i + c_i x_{i + 1} = y_i.
$$
The standard second-order discretization of a diffusion problem leads to a system of the form
$$
-g_{i - 1} x_{i - 1} + (g_{i - 1} + g_i + h_i) x_i - g_{i} x_{i + 1} = y_i,
$$
where $g_i$, $h_i$ are positive and $g_i$ is proportional to the mixing coefficient
and can be much greater than $h_i$, which is the layer thickness.

The Thomas algorithm is a simplified form of Gaussian elimination for tridiagonal systems of equations.
In its typical implementation, the forward elimination phase proceeds as follows
$$
c'_i &= \frac{c_i}{b_i - a_i c'_{i - 1}}, \\
y'_i &= \frac{y_i - a_i y'_{i - 1}}{b_i - a_i c'_{i - 1}}.
$$
When applied to the diffusion system this leads to
$$
c'_i &= \frac{-g_i}{h_i + g_{i - 1} (1 + c'_{i - 1}) + g_{i}}, \\
y'_i &= \frac{y_i + g_{i - 1} y'_{i - 1}}{h_i + g_{i - 1} (1 + c'_{i - 1}) + g_i}.
$$
Let's consider what happens when the mixing coefficient, and hence $g_i$, abruptly changes.
Suppose that $h_i$ is small, $g_{i - 1}$ is small, but $g_i$ is very large.
Then $c'_i \approx -1$ and in the next iteration the term $g_i (1 + c'_i)$ in
the denominator of both expression will multiply a very small number by a very large number.

Let’s consider what happens when the mixing coefficient, and hence 

Following [Appendix E in Schopf and Loughe](https://journals.ametsoc.org/view/journals/mwre/123/9/1520-0493_1995_123_2839_argiom_2_0_co_2.xml), to remedy that we can introduce
$$
\alpha'_i = g_i (1 + c'_i),
$$
which satisfies the following recursion relation
$$
\alpha'_i = \frac{g_i (h_i + \alpha'_{i - 1})}{h_i + \alpha'_{i - 1} + g_i}.
$$
The above equation together with
$$
c'_i &= \frac{-g_i}{h_i + \alpha'_{i - 1} + g_{i}}, \\
y'_i &= \frac{y_i + g_{i - 1} y'_{i - 1}}{h_i + \alpha'_{i - 1} + g_i},
$$
forms the modifed stable algorithm.

The Thomas algorithm is work-efficient, but inherently serial.
While systems in different columns can
be solved in parallel, this might not expose enough parallelism on modern GPUs.
There are parallel tridiagonal algorithms that perform better on modern GPUs,
see [Zhang, Cohen, and Owens](https://doi.org/10.1145/1837853.1693472).
The two algorithms best suited for small systems are cyclic reduction and parallel cyclic reduction.

The basic idea of both cyclic reduction algorithms is as follows. Let's consider three consecutive equations corresponding to $y_{i- 1}$, $y_i$, and $y_{i+1}$
$$
a_{i - 1} x_{i - 2} + b_{i - 1} x_{i - 1} + c_{i - 1} x_i &= y_{i - 1}, \\
a_i x_{i - 1} + b_i x_i + c_i x_{i + 1} &= y_i, \\
a_{i + 1} x_{i} + b_{i + 1} x_{i + 1} + c_{i + 1} x_{i + 2} &= y_{i + 1}.
$$
We can eliminate $x_{i - 1}$ and $x_{i + 1}$ to obtain a system of the form
$$
\hat{a}_i x_{i - 2} + \hat{b}_i x_i + \hat{c}_i x_{i + 2} = \hat{y}_i,
$$
where the modified coefficients $\hat{a}_i$, $\hat{b}_i$, $\hat{c}_i$ and the modified rhs $\hat{y}_i$ are
$$
\hat{a}_i &= -\frac{a_{i - 1} a_i}{b_{i - 1}}, \\
\hat{b}_i &= b_i - \frac{c_{i - 1} a_i}{b_{i - 1}} - \frac{c_{i} a_{i + 1}}{b_{i + 1}}, \\
\hat{c}_i &= -\frac{c_i c_{i + 1}}{b_{i + 1}}, \\
\hat{y}_i &= y_i - \frac{a_i y_{i - 1}}{b_{i - 1}} - \frac{c_i y_{i + 1}}{b_{i + 1}}.
$$
The resulting system of equations for $x_{2j}$ is still tridiagonal and has roughly half the size of the original.

The basic idea of both cyclic reduction algorithms is as follows. Let’s consider three consecutive equations corresponding to 

 is still tridiagonal and has roughly half the size of the original.

The cyclic reduction algorithm has two phases.
In the first phase, the above elimination step is iterated until the system is reduced to either one or two equations, which can then be directly solved.
In each iteration the computation of modified coefficients can be done in parallel.
The second phase involves finding the rest of the solution by using the final coefficients.
The second phase is also iterative, where at each iteration the number of know solution values increases by a factor of two. A drawback of this algorithm is that the amount of parallel computations available at each iteration is not constant.

The parallel cyclic reduction is based on the same idea, but has only one phase. In the first iteration it reduces the
original system to two systems of half the size. The second iteration reduces it to four systems of quarter the size
and so on. In the final iteration systems of size one or two are solved to obtain the whole solution at once.
In contrast to the cyclic reduction, this algorithm has constant amount of parallelism available at the cost of performing
more redundant work.

A naive application of the cyclic reduction to the diffusion system would result
in the following equation for the modified main diagonal
$$
\hat{b}_i = h_i
             + g_{i - 1} - \frac{g_{i-1}^2}{h_{i - 1} + g_{i - 2} + g_{i - 1}}
             + g_i - \frac{g_i^2}{h_{i + 1} + g_{i} + g_{i + 1}}.
$$
Using this expression can potentially result in catastrophic cancellation errors and overflows if
$g_{i - 1}$ or $g_{i + 1}$ are very large.
To improve its stability, this expression can be rewritten as
$$
             + h_{i - 1} \frac{g_{i - 1}}{h_{i - 1} + g_{i - 2} + g_{i - 1}}
             + h_{i + 1} \frac{g_i}{h_{i + 1} + g_i + g_{i + 1}}
             + g_{i - 1} \frac{g_{i - 2}}{h_{i - 1} + g_{i - 2} + g_{i - 1}}
             + g_{i + 1} \frac{g_i}{h_{i + 1} + g_i + g_{i + 1}}.
$$
This equation can be shown to be in the form
$$
\hat{b}_i = \hat{h}_i + \hat{g}_{i - 2} + \hat{g}_i,
$$
where
$$
\hat{h}_i &= h_i
             + h_{i + 1} \frac{g_i}{h_{i + 1} + g_i + g_{i + 1}}, \\
\hat{g}_i &= g_{i + 1} \frac{g_i}{h_{i + 1} + g_i + g_{i + 1}}.
$$
These two equations form the basis of stable (parallel) cyclic reduction for diffusion problems.

Thomas algorithm for diffusion systems with improved stability properties on CPUs

PCR algorithm for diffusion systems with improved stability properties on GPUs

The algorithms will be designed to work within Kokkos team policies, with each team of threads
solving one column system on GPUs and `VecLength` column systems on CPUs.
The user interface for CPU and GPU solvers will be the same. There will be
two type aliases `TriDiagSolver` and `TriDiagDiffSolver` that will resolve to the
optimal solver class based on the chosen architecture.

There will be a solver struct for each algorithm. The solvers inputs and outputs will be encapsulated in
two scratch data structs, one for the general solver and one for the diffusion solver.

To facilitate constructing the systems on the fly and for performance reasons the solver inputs and outputs will be using the Kokkos scratch memory space. The scratch data for the general tridiagonal solver will be encapsulated in a struct called `TriDiagScratch`

To facilitate constructing the systems on the fly and for performance reasons the solver inputs and outputs will be using the Kokkos scratch memory space. The scratch data for the general tridiagonal solver will be encapsulated in a struct called 

struct TriDiagScratch {
   ScratchArray2DReal DL;
   ScratchArray2DReal D;
   ScratchArray2DReal DU;
   ScratchArray2DReal X;
};

where `DL`, `D`, `DU`, `X` are views of size (`NRow`, `VecLength`) in the scratch memory space. The views
`DL`, `D`, `DU` are inputs denoting the lower, main, and upper diagonal, respectively. The view `X` should contain
the rhs at input and will be overwritten with the solution after `solve` is called.

 are inputs denoting the lower, main, and upper diagonal, respectively. The view 

the rhs at input and will be overwritten with the solution after 

The scratch data for the specialized diffusion tridiagonal solver will be encapsulated in a struct called `TriDiagDiffScratch`

The scratch data for the specialized diffusion tridiagonal solver will be encapsulated in a struct called 

struct TriDiagDiffScratch {
   ScratchArray2DReal G;
   ScratchArray2DReal H;
   ScratchArray2DReal Alpha;
};

where `G`, `H`, `X`, `Alpha` are views of size (`NRow`, `VecLength`) in the scratch memory space. The views
`G` and `H` are inputs corresponding to the variables `g` and `h` introduced in [Section 3](#3-algorithmic-formulation).
The view `X` has the same meaning as in the general case.
The view `Alpha` is an internal workspace used by the algorithm.

The four solver algorithms will be implemented as four structs `ThomasSolver`, `PCRSolver`, `ThomasDiffusionSolver`, and `PCRDiffusionSolver`. Currently, there is no plan for those structs to have any data members and they will only provide static methods, acting essentially as namespaces.

. Currently, there is no plan for those structs to have any data members and they will only provide static methods, acting essentially as namespaces.

The constructors of scratch spaces take a team member and system size (`NRow`)

The constructors of scratch spaces take a team member and system size (

    TriDiagScratch(const TeamMember &Member, int NRow);
    TriDiagDiffScratch(const TeamMember &Member, int NRow);

that creates an appropriate team policy for solving `NBatch` systems of size `NRow`.

The general solvers `ThomasSolver` and `PCRSolver` will have a static solve method

   static void solve(const TeamMember &Member, const TriDiagScratch &Scratch);

that takes a team member and an initialized general scratch space. After calling this method `Scratch.X`
will contain the solution. There will also be a convenience method

that takes a team member and an initialized general scratch space. After calling this method 

static void solve(const TeamMember &Member,
                  const Array2DReal &DL, const Array2DReal &D, const Array2DReal &DU, const Array2DReal &X);

The diffusion solvers `ThomasDiffusionSolver` and `PCRDiffusionSolver` will provide a similar method

   static void solve(const TeamMember &Member, const TriDiagDiffScratch &Scratch);

differing only in the type of scratch space. Similarly, there will be a convenience method

   static void solve(const TeamMember &Member,
                     const Array2DReal &G, const Array2DReal &H, const Array2DReal &X);

Given analytically prescribed matrix `A` and vector `y` the solution to the problem `A x = z` with `z = A y` will be
checked to see if the resulting `x` is equal to the prescribed vector `y`. This will be done for all solvers for a variety of (batch size, system size) combinations.

. This will be done for all solvers for a variety of (batch size, system size) combinations.

5.2 Test diffusion solvers convergence using manufactured solution

The convergence of diffusion solvers will be tested using a manufactured solution.

5.2 test diffusion solvers convergence using manufactured solution

The diffusion solvers stability will be tested on an idealized vertical mixing problem with abrupt changes in
the diffusion coefficient.

The vertical coordinate module will be responsible for computing and storing information related to the vertical mesh in Omega.
Since Omega will be a non-Boussinesq model, the vertical independent variable will be pressure-based (expressed in terms of the pseudo-height) as opposed to $z$-based.
Similar to MPAS-Ocean's support for tilted height ($z^\star$) coordinates, Omega V1 will support a general vertical coordinate, where pseudo thickness $\tilde{h}$ varies in proportion to the total pseudo thickness of the water column.
The prognostic variable in the layered continuity equation is pseudo thickness, which can be used to calculate the total pressure at a given vertical layer interface.
The geometric height of a given layer interface is still necessary in the model to compute sea level, the geopotential, and derivatives with respect to $z$, all of which affect the dynamics.
The geometric thickness of a layer ($\Delta z$) can be found using the pseudo thickness, density, reference density, and known bottom elevation (positive up) relative to the reference geoid.
The vertical coordinate module will also serve as a container for information related to the variable number of active vertical layers for a given ocean column (due to variations in bottom elevation and ice shelf cavities).

Since Omega will be a non-Boussinesq model, the vertical independent variable will be pressure-based (expressed in terms of the pseudo-height) as opposed to 

) coordinates, Omega V1 will support a general vertical coordinate, where pseudo thickness 

 varies in proportion to the total pseudo thickness of the water column.

The geometric height of a given layer interface is still necessary in the model to compute sea level, the geopotential, and derivatives with respect to 

) can be found using the pseudo thickness, density, reference density, and known bottom elevation (positive up) relative to the reference geoid.

2.1 Requirement: Compute and store the total pressure at a given layer interface

The total pressure at each layer interface will be computed and stored within the vertical coordinate module.
This involves a surface-down summation of the pseudo thickness variable times the gravitational acceleration, $g$, and the reference density, $\rho_0$, starting with the surface pressure.
The pressure variable will be needed in the computation of the TEOS-10 equation of state and the pressure gradient.
The variable will be registered with the `IOStreams` framework to enable it to be output during runtime.

This involves a surface-down summation of the pseudo thickness variable times the gravitational acceleration, 

2.1 requirement: compute and store the total pressure at a given layer interface

2.2 Requirement: Compute and store the $z$ location of a given layer interface

The $z$ location at each layer interface will be computed and stored within the vertical coordinate module.
In the model, $z$ is defined as being positive up from the geoid.
The $z$ location is calculated in a bottom-up summation of the pseudo thickness times the specific volume and reference density, starting with the known bottom elevation.
The $z$ location of the interfaces is needed to compute the geopotential gradient and the sea level.

 location at each layer interface will be computed and stored within the vertical coordinate module.

 location is calculated in a bottom-up summation of the pseudo thickness times the specific volume and reference density, starting with the known bottom elevation.

 location of the interfaces is needed to compute the geopotential gradient and the sea level.

2.2 requirement: compute and store the z location of a given layer interface

2.3 Requirement: Compute and store the vertical layer bounds for each horizontal cell

All tendency terms require a variable vertical loop range to account for variation in bottom elevation and ice shelf cavities.
These bounds will be computed on construction in the vertical coordinate module and used for setting the loop bounds for active layers in calculations over the vertical.

2.3 requirement: compute and store the vertical layer bounds for each horizontal cell

The geopotential will be computed and stored within the vertical coordinate module.
Initially this will be calculated by summing together the $z$ location times $g$.
In the future, tidal potential and self attraction and loading will also have optional (default off) contributions to the geopotential.

2.5 Requirement: Compute and store the desired thickness used in the $p^\star$ vertical coordinate

2.5 Requirement: Compute and store the desired thickness used in the 

The $p^\star$ vertical coordinate specifies a desired layer thickness that stretches the initial thicknesses by the bottom pressure anomaly.
This involves adding a vertically-weighted bottom pressure perturbation to the reference (initial) thickness.
This desired thickness will be computed and stored within the vertical coordinate module.
The vertical advection algorithm will use this desired thickness to compute the vertical transport.

 vertical coordinate specifies a desired layer thickness that stretches the initial thicknesses by the bottom pressure anomaly.

2.5 requirement: compute and store the desired thickness used in the p^\star vertical coordinate

2.6 Desired: General Arbitrary Lagrangian Eulerian (ALE) coordinate support

In future versions of Omega, the $p^\star$ coordinate will be extended to a more general ALE coordinate

2.6 desired: general arbitrary lagrangian eulerian (ale) coordinate support

The vertical coordinate module should retain flexibility to support VLR in future versions of Omega.

In the layered non-Boussinesq equations solved in Omega (see [V0 governing equation document](omega-design-governing-eqns-omega1) for details), the prognostic variable for cell $i$ and layer $k$ is the pseudo thickness, $\tilde{h}_{i,k}$, so that the geometric thickness (in meters) is a diagnostic variable defined as:

, so that the geometric thickness (in meters) is a diagnostic variable defined as:

The pressure at vertical cell interfaces is the found by summing the pseudo thicknesses:

  p_{i,k+1/2} = p_{i}^{surf} + g\rho_0 \sum_{k^\prime=1}^k \tilde{h}_{i,k^\prime}. 

 p_{i,k} = p_{i}^{surf} + g\rho_0 \sum_{k'=1}^{k-1} \tilde{h}_{i,k'} + \frac{1}{2} g\rho_0 \tilde{h}_{i,k} 

The $z$ location of cell interfaces is found by summing the pseudo thicknesses from the bottom multiplied by the specific volume:

 location of cell interfaces is found by summing the pseudo thicknesses from the bottom multiplied by the specific volume:

 z_{i,k+1/2} = z_i^{floor} + \rho_0 \sum_{k^\prime=k}^{K_{max}} \alpha_{i,k^\prime} \tilde{h}_{i,k^\prime}, 

where $z_i^{floor}$ is the (positive-up) bottom elevation.
The $z$ location of a layer midpoint is given by:

 z_{i,k} = z_i^{floor} + \frac{1}{2} \rho_0\alpha_{i,k} \tilde{h}_{i,k} + \rho_0\sum_{k^\prime= k+1}^{K_{max}} \alpha_{i,k^\prime} \tilde{h}_{i,k^\prime}. 

Initially, the geopotential is the sum of the $z$ height times $g$.
In the future, it will include contributions from the tidal potential ($\Phi_{tp}$) and self attraction and loading ($\Phi_{SAL}$):

In the future, it will include contributions from the tidal potential (

 \Phi_{i,k} = \left( gz_{i,k} + \Phi_{tp} + \Phi_{SAL} \right). 

The desired pseudo thickness used for the $p^\star$ coordinate is:

\tilde{h}_k^{p^\star} = \tilde{h}_k^{ref} + \left(\frac{p_B-p_{surf}}{g\rho_0} - \sum_{k^\prime=1}^K \tilde{h}_{k^\prime}^{ref} \right)\frac{W_k\tilde{h}_k^{ref}}{\sum_{k^\prime=1}^K W_{k^\prime}\tilde{h}_{k^\prime}^{ref}}, 

where $\tilde{h}_k^{ref}$ is the initial reference pseudo thickness and the weights $W_k$ determine how pressure perturbations are distributed amongst the layers.
Setting all $W_k$ values to a constant, corresponds to uniform stretching of the layers.
Different weight values can be chosen such that perturbations are distributed unequally.

 determine how pressure perturbations are distributed amongst the layers.

 values to a constant, corresponds to uniform stretching of the layers.

The `VertCoord` class will be used to compute pressure, $z$ height, and geopotential at each layer.
It will also contain the vertical loop bounds for cells, edges, and vertices.

class VertCoord {
    public:
        I4 NVertLevels;
        I4 NVertLevelsP1;

        // Variables computed
        Array2DReal PressureInterface;
        Array2DReal PressureMid;
        Array2DReal ZInterface;
        Array2DReal ZMid;
        Array2DReal GeopotentialMid;
        Array2DReal LayerThicknessTarget;

        // Vertical loop bounds (computed on construction)
        Array1DI4 MinLevelCell;
        Array1DI4 MaxLevelCell;
        Array1DI4 MinLevelEdgeTop;
        Array1DI4 MaxLevelEdgeTop;
        Array1DI4 MinLevelEdgeBot;
        Array1DI4 MaxLevelEdgeBot;
        Array1DI4 MinLevelVertexTop;
        Array1DI4 MaxLevelVertexTop;
        Array1DI4 MinLevelVertexBot;
        Array1DI4 MaxLevelVertexBot;

        // Variables read in from vert coord stream
        /// p star coordinate variables
        Array2DReal VertCoordMovementWeights;
        Array2DReal RefLayerThickness;

        /// Variables read in from mesh file
        Array2DReal BottomDepth;
    private:

        // Variables from HorzMesh
        I4 NCells;
        I4 NEdges;
        I4 NVertices;
        Array2DReal CellsOnEdge;
        Array2DReal CellsOnVertex;

        static VertCoord *DefaultVertCoord;
        static std::map<std::string, std::unique_ptr<VertCoord>> AllVertCoords;
}

class VertCoord{
    public:
        static VertCoord *create();
        void computePressure();
        void computeZHeight();
        void computeGeopotential();
        void computeTargetThickness();
    private:
        void minMaxLevelEdge();
        void minMaxLevelVertex();

}

The constructor will be responsible for storing any static mesh information as private variables and handling the selection any user-specified options.
It will also compute the vertical loop bounds on edges and vertices.

VertCoord::VertCoord(const HorzMesh *Mesh,
                     int NVertLevels,
                     Config *Options);

The create method will take the same arguments as the constructor plus a name.
It calls the constructor to create a new vertical coordinate instance, and put it in the static map of all vertical coordinate objects.

VertCoord *VertCoord::create(const std::string &Name,
                             const HorzMesh *Mesh,
                             int NVertLevels,
                             Config *Options);

A vertical coordinate `IOStream` will be defined to read in the `BottomDepth`, `VertCoordMovementWeights` and `RefLayerThickness` variables.

The init method will create the default vertical coordinate and return an error code:

There will be methods for getting the default and non-default vertical coordinate instances:

VertCoord *VertCoord::getDefault();
VertCoord *VertCoord::get(const std::string &Name);

The public `computePressure` will sum the pseudo thicknesses times $g$ from the top layer down, starting with the surface pressure.
This will be done with a `parallel_scan` inside a `parallel_for` over the mesh cells using hierarchical parallelism.

void VertCoord::computePressure(const Array2DReal &PressureInterface,
                                const Array2DReal &PressureMid,
                                const Array2DReal &LayerThickness,
                                const Array2DReal &SurfacePressure) {

}

The public `computeZHeight` will sum the pseudo thicknesses times $\alpha$ from the bottom later up, starting with the bottom elevation.

void VertCoord::computeZHeight(const Array2DReal &ZInterface,
                               const Array2DReal &ZMid,
                               const Array2DReal &LayerThickness,
                               const Array2DReal &SpecVol,
                               const Array2DReal &BottomDepth) {

}

The public `computeGeopotential` will sum together the $z$ height times $g$, the tidal potential, and self attraction and loading:

void VertCoord::computeGeopotential(const Array2DReal &GeopotentialMid,
                                    const Array2DReal &ZMid,
                                    const Array2DReal &TidalPotential
                                    const Array2DReal &SelfAttractionLoading) {

}

Tidal potential forcing and self attraction and loading will be default-off features.
The will be added to (or excluded from) the geopotential based on config flags.

The public `computeTargetThickness` will determine the desired pseudo thickness used for the $p^\star$ vertical coordinate:

void VertCoord::computeTargetThickness(const Array2DReal &LayerThicknessPStar,
                                       const Array2DReal &VertCoordMovementWeights,
                                       const Array2DReal &RefLayerThickness) {

}

The private `minMaxLevel` will determine the various vertical loop bounds on edges and vertices.
It will compute the class member variables: `MinLevelEdgeTop`, `MaxLevelEdgeTop`, `MinLevelEdgeBot`, `MaxLevelEdgeBot`, `MinLevelVertexTop`, `MaxLevelVertexTop`, `MinLevelVertexBot`, and `MaxLevelVertexBot` from `MinLevelCell`, `MaxLevelCell`, `CellsOnEdge`, and `CellsOnVertex`.

 will determine the various vertical loop bounds on edges and vertices.

No operations are needed in the destructor.
The erase method will remove a named vertical coordinate instance, whereas the clear method will remove all of
them.
Both will call the destructor in the process.

void VertCoord::erase(const std::string &Name);
void VertCoord::clear();

Unit tests will be used to test each of the computations (computePressure, computeZHeight, computeGeopotential, computePStarThickness) for a given pseudo thickness array. Comparison will be made to a ''truth'' vertical mesh that includes spatially varying `minLevelCell` and `maxLevelCell`.

Unit tests will be used to test each of the computations (computePressure, computeZHeight, computeGeopotential, computePStarThickness) for a given pseudo thickness array. Comparison will be made to a ‘’truth’’ vertical mesh that includes spatially varying 

2.2 Requirement: Compute and store the z location of a given layer interface

2.5 Requirement: Compute and store the desired thickness used in the p^\star vertical coordinate

Representation of unresolved vertical fluxes of momentum, heat, salt, and biogeochemical tracers in ocean models is essential to simulations fidelity. Models of turbulent fluxes spans a wide range of complexity, but the models generally fall into a few categories: simply polynomial relationships, equilibrium turbulence models, and prognostic turbulence models.

2.1 Requirement: Vertical mixing coefficient interface should be modular

To prepare for additional mixing closures and later improvements, the interface to vertical mixing must allow for easy connection of closures. It is assumed that the strength of vertical mixing from each closure is additive, such that the final vertical diffusion coefficient is the sum of the coefficients and potential non local terms from each closure.

2.1 requirement: vertical mixing coefficient interface should be modular

2.2 Requirement: Vertical mixing models should be have local and/or gradient free terms

For optimal performance, initial models of vertical turbulent fluxes for Omega should be cast as an implicit, down gradient mixing and an explicit, gradient free, component, e.g.,

\overline{w' \phi'}\approx \kappa(\overline{\rho},\overline{u},\overline{v}) \left(\frac{\partial \overline{\phi}}{\partial z} + \gamma(\overline{\rho},\overline{u},\overline{v}) \right)\,.

Here, $\phi$ is a generic tracer, $\overline{w'\phi'}$ is the vertical turbulent flux of that tracer, $\kappa$ is the vertical diffusivity, and $\gamma$ is the gradient free portion of the flux (often referred to as 'non-local'). The vertical diffusivity $\kappa$ can be as simple as a constant value, to complex functions of quantities like shear and stratification. A similar equation can be written for turbulent momentum fluxes and vertical viscosity ($\nu$).

 is the gradient free portion of the flux (often referred to as ‘non-local’). The vertical diffusivity 

 can be as simple as a constant value, to complex functions of quantities like shear and stratification. A similar equation can be written for turbulent momentum fluxes and vertical viscosity (

Invocation of the gradient diffusion hypothesis is a simplifying assumption for turbulence closures that casts turbulence as a purely diffusive process and allows the mixing problem to be cast implicitly in a tridagonal solve.  For more physical mixing closures, other methods can be explored, e.g., direct and iterative implicit solvers or subcycling.

The initial design and implementation of these vertical mixing coefficients will include only the local, down gradient portion of the flux (first right-hand-side term in Eq. (1)). Implementation of the non-local, gradient free portion of the vertical flux will come later. However, it is important to note here that $\gamma$ is only non-zero within the boundary layer, and when non-local flux formulations (such as [KPP (Large et al., 1994)](https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/94rg01872)) are being used, $\kappa$ value contributions such as the convective mixing flux detailed in Section 3.2 are only computed outside of the boundary layer and remain zero within the boundary layer.

The initial design and implementation of these vertical mixing coefficients will include only the local, down gradient portion of the flux (first right-hand-side term in Eq. (1)). Implementation of the non-local, gradient free portion of the vertical flux will come later. However, it is important to note here that 

 is only non-zero within the boundary layer, and when non-local flux formulations (such as 

 value contributions such as the convective mixing flux detailed in Section 3.2 are only computed outside of the boundary layer and remain zero within the boundary layer.

2.2 requirement: vertical mixing models should be have local and/or gradient free terms

2.3 Requirement: Vertical mixing models cannot ingest single columns at a time

Many standard vertical mixing libraries (e.g. CVMix) receive and operate on one column at a time. This results in poor performance, especially on accelerators. Any utilized model of vertical mixing must ingest and operate on multiple columns concurrently.

2.3 requirement: vertical mixing models cannot ingest single columns at a time

For Omega-1 a few simple vertical mixing algorithms will be used. When computing $\kappa$ and $\nu$, linear superposition when combining the various parameterizations is assumed.

For Omega-1 a few simple vertical mixing algorithms will be used. When computing 

, linear superposition when combining the various parameterizations is assumed.

One of the simplest class of models of vertical turbulence are polynomial functions of the gradient Richardson number. As in MPAS-Ocean, we choose to define the Richardson number as

Ri = \frac{N^2}{\left|\frac{\partial \mathbf{U}}{\partial z}\right|^2}\,,

where $N^2$ is the Brunt Vaisala Frequency, which for non Boussinesq flows is defined as

 is the Brunt Vaisala Frequency, which for non Boussinesq flows is defined as

If we assume this term is defined at the top of the cell, it is discretized as

N^2(k) = g \rho_0 \frac{\rho_{DD}(k) - \rho(k)}{z_m(k-1) - z_m(k)}\,.

Here $\rho_{DD}$ is the density of the fluid of layer k-1 displaced to layer k adiabatically.

 is the density of the fluid of layer k-1 displaced to layer k adiabatically.

The shear in the denominator, also defined at the top of the cell, is discretized as

\left| \frac{\partial \mathbf{U}}{\partial z}\right|^2 = \left(\frac{U_n(k-1)-U_n(k)}{z_m(k-1) - z_m(k)}\right)^2 + \left(\frac{U_t(k-1)-U_t(k)}{z_m(k-1) - z_m(k)}\right)^2\,.

Where the subscripts *n* and *t* are the normal and tangential velocities respectively.

With the definition of the gradient richardson number, the viscosity and diffusivity are defined according to [Pacanowski and Philander (1981)](https://journals.ametsoc.org/view/journals/phoc/11/11/1520-0485_1981_011_1443_povmin_2_0_co_2.xml?tab_body=pdf) as

With the definition of the gradient richardson number, the viscosity and diffusivity are defined according to 

$\nu$ and $\kappa$ are defined at the top of the cell, similarly to $Ri$ and $N^2$. In these formulae, $\alpha$ and *n* are tunable parameters, most often chosen as 5 and 2 respectively. $\nu_b$ and $\kappa_b$ are the background viscosity and diffusivity respectively. MPAS-Ocean has an additional $\kappa_{b,passive}$ for applying to just the passive tracers, leaving the active tracers with $\kappa_b$.

 are tunable parameters, most often chosen as 5 and 2 respectively. 

 are the background viscosity and diffusivity respectively. MPAS-Ocean has an additional 

 for applying to just the passive tracers, leaving the active tracers with 

Commonly (and how this is currently handled by CVMix in MPAS-O), mixing due to convective instability is treated as a step function for the diffusivity and viscosity. This is often represented as

\kappa =
\begin{cases}
\kappa_{b} + \kappa_{conv} \quad \text{ if } N^2 \leq N^2_{crit}\\
\kappa_{b} \quad \text{ if } N^2 > N^2_{crit}
\end{cases}

A similar expression is utilized for viscosity. The effect of this formula is to homogenize T, S, passive tracers, and normal velocity, for unstable stratification, but also in neutral stratification, as $N^2_{crit}$ is typically chosen to be 0. The behavior in unstable stratification is likely correct, but for neutral stratification, homogenization of momentum is questionable. In Omega, we will slightly modify the algorithm above to,

A similar expression is utilized for viscosity. The effect of this formula is to homogenize T, S, passive tracers, and normal velocity, for unstable stratification, but also in neutral stratification, as 

 is typically chosen to be 0. The behavior in unstable stratification is likely correct, but for neutral stratification, homogenization of momentum is questionable. In Omega, we will slightly modify the algorithm above to,

\kappa =
\kappa_{b} + \kappa_{conv} \quad \text{ if } N^2 < N^2_{crit}\\
\kappa_{b} \quad \text{ if } N^2 \geq N^2_{crit}

So that homogenization only occurs for unstable stratification (assuming $N^2_{crit} = 0$). Values for $\kappa_{conv}$ are typically large ($\sim 1 \, \mathrm{m^2/s}$). Again, when non-local, gradient free flux formulations are included in Eq. (1) in future developments, this convective instability contribution to the mixing will not be calculated within the boundary layer where $\gamma$ is active.

So that homogenization only occurs for unstable stratification (assuming 

). Again, when non-local, gradient free flux formulations are included in Eq. (1) in future developments, this convective instability contribution to the mixing will not be calculated within the boundary layer where 

Vertical chunking, as is, does not work well for vertical derivatives, so a first design of the vertical mixing coefficients computation does not do vertical chunking and just computes with the full-depth column.

Additionally, to start, only the down gradient contribution to vertical mixing will be added, with contributions to the vertical viscosity and diffusivity coming from the shear (Richardson number mixing), convective, and background mixing models detailed in the prior section. However, in future developments, the K Profile Parameterization [(KPP; Large et al., 1994)](https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/94rg01872) and other non-local and/or higher-order mixing models will be added. Some of these parameterizations require vertical derivatives, full-depth integrals, and/or formulate $\kappa$ and $\gamma$ in Eq. (1) Section 2.1 as functions of surface forcing, which require either full-depth column or surface forcing information at depth, thus we design the mixing coefficients routine with this in mind. A solution that allows chunking with vertical operations such as derivatives and integrals and/or surface forcing values would be advantageous, but is outside of the scope of this design document.

Additionally, to start, only the down gradient contribution to vertical mixing will be added, with contributions to the vertical viscosity and diffusivity coming from the shear (Richardson number mixing), convective, and background mixing models detailed in the prior section. However, in future developments, the K Profile Parameterization 

 and other non-local and/or higher-order mixing models will be added. Some of these parameterizations require vertical derivatives, full-depth integrals, and/or formulate 

 in Eq. (1) Section 2.1 as functions of surface forcing, which require either full-depth column or surface forcing information at depth, thus we design the mixing coefficients routine with this in mind. A solution that allows chunking with vertical operations such as derivatives and integrals and/or surface forcing values would be advantageous, but is outside of the scope of this design document.

Several parameters will be needed for the three different contributions to the local, down gradient vertical mixing coefficients. These parameters should be run-time configurable through the YAML config. A new section in the YAML config should be made for these parameters, as more run-time configurable vertical mixing parameters will be added to this section when the non-local formulations are added.

The following config options should be included for the Richardson number dependent -- Pacanowski and Philander (1981) based -- vertical mixing:

The following config options should be included for the Richardson number dependent – Pacanowski and Philander (1981) based – vertical mixing:

`EnableShearMix`. If true, shear-based mixing is computed based upon Pacanowski and Philander (1981) and applied to velocity components and tracer quantities. \[.true./.false.\]

. If true, shear-based mixing is computed based upon Pacanowski and Philander (1981) and applied to velocity components and tracer quantities. [.true./.false.]

`ShearNuZero`. Numerator of Pacanowski and Philander (1981) Eq (1). \[0.005 $m^2 s^{-1}$\]

`ShearAlpha`. Alpha value used in Pacanowski and Philander (1981) Eqs (1) and (2). \[5\]

. Alpha value used in Pacanowski and Philander (1981) Eqs (1) and (2). [5]

`ShearExponent`. Exponent used in denominator of Pacanowski and Philander (1981) Eqs (1). \[2\]

. Exponent used in denominator of Pacanowski and Philander (1981) Eqs (1). [2]

The following config options should be included for convective vertical mixing:

`EnableConvectiveMix`. If true, convective mixing is computed and applied to velocity components and tracer quantities. \[.true./.false.\]

. If true, convective mixing is computed and applied to velocity components and tracer quantities. [.true./.false.]

`ConvectiveDiffusivity`. Convective vertical viscosity and diffusivity applied to velocity components and tracer quantities, respectively. \[1.0 $m^2 s^{-1}$\]

. Convective vertical viscosity and diffusivity applied to velocity components and tracer quantities, respectively. [1.0 

`ConvectiveTriggerBVF`. Value of Brunt Viasala frequency squared below which convective mixing is triggered. \[0.0\]

. Value of Brunt Viasala frequency squared below which convective mixing is triggered. [0.0]

The following config options should be included for background vertical mixing:

`BackgroundViscosity`. Background vertical viscosity applied to velocity components. \[1.0e-4 $m^2 s^{-1}$\]

. Background vertical viscosity applied to velocity components. [1.0e-4 

`BackgroundDiffusivity`. Background vertical diffusivity applied to all tracer quantities. \[1.0e-5 $m^2 s^{-1}$\]

. Background vertical diffusivity applied to all tracer quantities. [1.0e-5 

<!--- List and describe all public methods and their interfaces (actual code for
interface that would be in header file). Describe typical use cases. -->

It is assumed that viscosity and diffusivity will be stored at cell centers. Additionally, since they will be used elsewhere, it is assumed that zMid and N2 have been computed elsewhere (N2 in the density routine) and stored. Quantities such as squared shear and Richardson number are not needed outside of the vertical mixing routines, so are computed locally. Note: how topography will be dealt with has not been finalized yet, but will be adopted into the following once finalized.

parallelFor(
    {NCellsAll, NVertLayers}, KOKKOS_LAMBDA(int ICell, int K) {

        // Add background contribution to viscosity and diffusivity
        VertViscosity(ICell, K) = BackgroundViscosity;
        VertDiffusivity(ICell, K) = BackgroundDiffusivity;

        // If shear mixing true, add shear contribution to viscosity and diffusivity
        if (EnableShearMix) {
            const Real InvAreaCell = 1._Real / AreaCell(ICell);
            // Calculate the square of the shear
            for (int J = 0; J < NEdgesOnCell(ICell); ++J) {
                const I4 JEdge = EdgesOnCell(ICell, J);
                const Real Factor = 0.5_Real * DcEdge(JEdge) * DvEdge(JEdge) * InvAreaCell;
                const Real DelU2 = pow(NormalVelocity(JEdge,K-1) - NormalVelocity(JEdge,K), 2._Real) + pow(TangentialVelocity(JEdge,K-1) - TangentialVelocity(JEdge,K), 2._Real);
                ShearSquared(ICell, K) = ShearSquared(ICell, K) + Factor * DelU2;
            }
            ShearSquared(ICell, K) = ShearSquared(ICell, K) / pow(zMid(ICell, K-1) - zMid(ICell, K), 2._Real);

            // Calculate Richardson number
            RichardsonNum(ICell, K) = BruntVaisalaFreq(ICell, K) / max(ShearSquared(ICell, K), 1.0e-12_Real);

            // Add in shear contribution to vertical mixing
            VertViscosity(ICell, K) = VertViscosity(ICell, K) + ShearNuZero / pow(1._Real + ShearAlpha * RichardsonNum(ICell, K), ShearExponent);
            VertDiffusivity(ICell, K) = VertDiffusivity(ICell, K) + VertViscosity(ICell, K) / (1 + ShearAlpha * RichardsonNum(ICell, K));
        }

        // If conv mixing true, add conv contribution to viscosity and diffusivity
        if (EnableConvectiveMix) {
            if (BruntVaisalaFreq(ICell, K) < ConvectiveTriggerBVF) {
                VertViscosity(ICell, K) = VertViscosity(ICell, K) + ConvectiveDiffusivity;
                VertDiffusivity(ICell, K) = VertDiffusivity(ICell, K) + ConvectiveDiffusivity;
            }
        }

    });

A no-flux condition should be applied to both the vertical mixing of momentum and tracers through setting the vertical viscosity and diffusivity to zero at the boundaries. From here, the vertical viscosity and vertical diffusivity will enter into the [tridiagonal solver](./TridiagonalSolver.md) to compute the vertical flux of momentum and tracers.

A no-flux condition should be applied to both the vertical mixing of momentum and tracers through setting the vertical viscosity and diffusivity to zero at the boundaries. From here, the vertical viscosity and vertical diffusivity will enter into the 

Unit tests can be initialized with linear-with-depth initial conditions (velocity and density) and use a linear equation of state. Expected values of the Richardson number, vertical viscosity, and vertical diffusion coefficients can be computed and compared to. Tests for both the shear, convective, and combined mixing contributions should be made to test requirement (2.1).

This assumes that the displaced density has already been tested. Vertical fluxes of momentum and tracers will be tested separately with the [tridiagonal solver](./TridiagonalSolver.md).

This assumes that the displaced density has already been tested. Vertical fluxes of momentum and tracers will be tested separately with the 

The `AuxiliaryState` class groups together all of Omega [auxiliary
variables](#omega-dev-aux-vars). It is responsible for managing their lifetime,
registering them with IOStreams, and providing functions that compute them,
to be used in the tendency evaluation. It is possible
to have multiple `AuxiliaryState` instances in Omega. Every instance has a name
and is tracked in a static C++ map called `AllAuxStates`.

An instance of the `AuxiliaryState` class requires a [`HorzMesh`](#omega-dev-horz-mesh), so
the mesh class and all of its dependencies need to be initialized before the `AuxiliaryState` class
can be. The static method:

the mesh class and all of its dependencies need to be initialized before the 

initializes the default `AuxiliaryState`. A pointer to it can be retrieved at any time using:

OMEGA::AuxiliaryState* DefAuxState = OMEGA::AuxiliaryState::getDefault();

A non-default auxiliary state can be created from a string `Name`, horizontal mesh `Mesh`, halo layer `MeshHalo`, vertical coordinate `VCoord`, and a number of Tracers `NTracers`:

OMEGA::AuxiliaryState*  NewAuxState = OMEGA::AuxiliaryState::create(Name, Mesh, MeshHalo, VCoord, NTracers);

For conveniece, this returns a pointer to the newly created state. Given its name, a pointer to a named auxiliary state
can be obtained at any time by calling the static `get` method:

OMEGA::AuxiliaryState* NewAuxState = OMEGA::AuxiliaryState::get(Name);

To compute all auxiliary variables stored in an auxiliary state `AuxState`,
given ocean state `State`, an array of tracers `TracerArray`, and time level `TimeLevel` do:

To compute all auxiliary variables stored in an auxiliary state 

Auxiliary variables are variables that need to be computed at every model time
step in order to advance the model state. Similar auxiliary variables are
grouped together in auxiliary variable groups. An auxiliary variable group is
implemented as a C++ class.

Each auxiliary group class has the same basic structure. It is a light-weight
class, since it needs to be copied to the GPU to perform computations. Each
class contains:

Additionally, there may be enum members that store user-configurable options.

As an example, the class `KineticAuxVars` that groups kinetic energy and
velocity divergence has the following structure:

class KineticAuxVars {
 public:
   Array2DReal KineticEnergyCell;
   Array2DReal VelocityDivCell;

   KineticAuxVars(const HorzMesh *mesh, int NVertLayers);
   void addMetaData() const;
   void defineIOFields() const;

   computeVarsOnCell(int ICell, int KChunk,
                     const Array2DReal &NormalVelEdge) const;

 private:
   Array1DI4 NEdgesOnCell;
   Array2DI4 EdgesOnCell;
   Array2DR8 EdgeSignOnCell;
   Array1DR8 DcEdge;
   Array1DR8 DvEdge;
   Array1DR8 AreaCell;

   // names used in defining MetaData and IOFields
   static const std::string KineticEnergyCellName;
   static const std::string VelocityDivCellName;
};

Some auxiliary variables have user-configurable options. In that case, for each
option an enum is defined in the auxiliary group header file. For example, the
flux thickness choice is represented using

calls the `addMetaData()` and `defineIOFields()` helper functions

The compute functions operate on a vertical chunk of mesh elements and take in
the element index, the chunk index, and input arrays, which may contain state
variables or other auxiliary variables. There is no output argument, they write
their results directly to the member arrays. There might be more than one
compute function, if variables located on different mesh elements are grouped
together. For example, the `VorticityAuxVars` class provides both

void computeVarsOnVertex(int IVertex, int KChunk, const Array2DReal &LayerThickCell, const Array2DReal &NormalVelEdge);
void computeVarsOnEdge(int IEdge, int KChunk);

because the vorticity variables are first computed on vertices and then
averaged to edges. Note that the edge compute function doesn't take any array
arguments, because it operates only on the class member arrays.

averaged to edges. Note that the edge compute function doesn’t take any array

The following auxiliary variable groups are currently implemented:

In a parallel environment, it is sometimes necessary to broadcast values
from a single task to all remaining tasks. Omega provides broadcast
functions for scalars and vectors of all supported data types with the
lone exception of a vector of strings. Scalar strings are supported and
can be used to broadcast a string vector element-wise. All functions are
simple wrappers around the relevant `MPI_Bcast` function.

To broadcast a scalar of any type, there are several supported interfaces:

Broadcast(MyScalar);
Broadcast(MyScalar, FromTask);
Broadcast(MyScalar, AltMachEnv);
Broadcast(MyScalar, AltMachEnv, FromTask);

The first broadcasts a scalar from the master task to all other tasks in
the default machine environment (set of MPI tasks). The second allows the
user to broadcast from a task other than the master task, but still in the
default environment.  The final two interfaces perform the same functions
but operate only in the set of tasks belonging to a different MachEnv than
the default. There are type-specific implementations for each, but all are
aliased to these same generic interfaces and the combinations above are
implemented using a combination of optional arguments and specific interfaces.

Omega also provides for the broadcasts of a `std::vector` of any supported
data type except strings. The interfaces are identical to those for scalars:

Broadcast(MyVector);
Broadcast(MyVector, FromTask);
Broadcast(MyVector, AltMachEnv);
Broadcast(MyVector, AltMachEnv, FromTask);

The implementation of a boolean vector is slightly different from the others
because a `std::vector<bool>` uses an optimal packing that does not allow
certain assumptions of storage for the MPI call. Instead, the boolean vector
is converted to/from an integer vector within the broadcast routine and
uses an MPI integer broadcast. This is hidden from the user. There is no
interface for a vector of strings due to the complexity of extracting
variable-length strings into a character vector for the MPI call. In the
unlikely event such a broadcast is needed, users should simply loop through
the vector and broadcast each scalar string individually.

All of the interfaces above use the blocking MPI broadcast and include an
implicit synchronization. For most use cases, this is adequate. However,
non-blocking broadcasts may be added in the future should the need arise.

As long as you have followed the procedure in {ref}`omega-dev-conda-env` for
setting up your conda environment, you will already have the packages available
that you need to build the documentation.

From the root of an Omega branch, run the following script to build the docs:

You can view the documentation by opening `_build/html/index.html` in your
choice of browsers or copy it to a web portal.

If you run into errors or warnings related to changes you have made when you
build the docs, check with the development team for help on fixing them.

The Omega build system utilizes CMake, a widely-used build tool,
to facilitate the build process.

The build process is defined in the CMakeLists.txt file located
in the top-level directory of Omega. It consists of four consecutive
steps: Setup, Update, Build, and Output.

The build step consists of adding three subdirectories that drive builds
for external libraries, the Omega model, and optional tests.

Python is required to use this build system.
The version of CMake should be 3.21 or later for supporting HIP.

During this step, the build-controlling variables are configured.
The Omega build system supports two modes: standalone and E3SM
component. The build mode is automatically detected, and any
mode-specific differences are resolved to ensure a consistent
build process.

By default, Omega assumes it is part of the E3SM code distribution.
In both build modes, the build system collects build-controlling
parameters, such as compiler paths and flags, from the E3SM build system.

There are three types of Omega build-controlling variables: Omega,
E3SM, and CMake. The names of these variables start with "OMEGA_",
"E3SM_", and "CMAKE", respectively. Temporary variables are prefixed
with an underscore ("_").

E3SM, and CMake. The names of these variables start with “OMEGA_”,

“E3SM_”, and “CMAKE”, respectively. Temporary variables are prefixed

The following is a list of Omega-specific variables available in
this version:

OMEGA_PROJECT_NAME: Name of the project ("OmegaOceanModel")
OMEGA_EXE_NAME: Name of the executable ("omega.exe")
OMEGA_LIB_NAME: Name of the library ("OmegaLib")
OMEGA_BUILD_MODES: List of build modes ("E3SM", "STANDALONE", "NOT_DEFINED")
OMEGA_BUILD_MODE: Selected build mode
OMEGA_BUILD_DIR: Omega top-level build directory
OMEGA_SOURCE_DIR: Directory where the top-level Omega CMakeLists.txt is located
OMEGA_DEFAULT_BUILD_TYPE: Default build type ("Release")
OMEGA_INSTALL_PREFIX: User-defined output directory for the library and executable
OMEGA_ARCH: User-defined programming framework (e.g., "CUDA", "HIP", "OPENMP", "SYCL", "")
OMEGA_CXX_COMPILER: C++ compiler
OMEGA_C_COMPILER: C compiler
OMEGA_Fortran_COMPILER: Fortran compiler
OMEGA_CIME_COMPILER: E3SM compiler name defined in config_machines.xml
OMEGA_CIME_MACHINE: E3SM machine name defined in config_machines.xml
OMEGA_CIME_PROJECT: Slurm account passed to CIME during Omega build
OMEGA_CXX_FLAGS: a list for C++ compiler flags
OMEGA_LINK_OPTIONS: a list for linker flags
OMEGA_BUILD_EXECUTABLE: Enable building the Omega executable
OMEGA_BUILD_TEST: Enable building Omega tests
OMEGA_PARMETIS_ROOT: Parmetis installtion directory
OMEGA_METIS_ROOT: Metis installtion directory
OMEGA_GKLIB_ROOT: GKlib installtion directory
OMEGA_HIP_COMPILER: HIP compiler (e.g., hipcc)
OMEGA_HIP_FLAGS: HIP compiler flags
OMEGA_MEMORY_LAYOUT: Kokkos memory layout ("LEFT" or "RIGHT"). "RIGHT" is a default value.
OMEGA_TILE_LENGTH: a length of one "side" of a Kokkos tile. 64 is a default value.
OMEGA_LOG_LEVEL: a default logging level. "OMEGA_LOG_INFO" is a default value.
OMEGA_LOG_FLUSH: turn on the unbuffered logging. "OFF" is a default value.
OMEGA_LOG_TASKS: set the tasks that generate log file. "0" is a default value.
OMEGA_VECTOR_LENGTH: Vector length used for blocking inner loops for vectorization. "1" is a default value.

E3SM_SOURCE_DIR: E3SM component directory (${E3SM_ROOT}/components)
E3SM_CIME_ROOT: CIME root directory
E3SM_CIMECONFIG_ROOT: E3SM CIME config directory
E3SM_EXTERNALS_ROOT: E3SM externals directory
E3SM_DEFAULT_BUILD_TYPE: E3SM build type (Release or Debug)

CMAKE_CURRENT_SOURCE_DIR
CMAKE_CURRENT_BINARY_DIR
CMAKE_CURRENT_LIST_DIR
CMAKE_CXX_STANDARD
CMAKE_CXX_COMPILER
CMAKE_CXX_FLAGS
CMAKE_BUILD_TYPE
CMAKE_INSTALL_PREFIX
CMAKE_VERSION

In this step, CMake is configured, and external library variables,
such as Kokkos, MPI, NetCDF, and PNetCDF, are set based on the settings
defined in the Setup step. The integrity of the build setup is verified
at the end of this step.

During this step, the build process is configured. It includes building
external libraries, followed by building the Omega main model from source
files. Optionally, tests can also be built.

The final step, which is optional, involves copying a subset of the build
artifacts to designated locations or generating dynamic outputs as needed.

Note: Until the Omega build is integrated into the E3SM build, specific
modifications are required to trigger the Omega build within the E3SM
build process. Please see [Omega Build User Guide](../userGuide/OmegaBuild.md)
for details.

As a developer, if you want to build the documentation locally or lint the
code, you will need your own
[conda](https://conda.io/projects/conda/en/latest/index.html) environment with
the necessary packages.

If you have not already done so, you will install
[Miniforge3](https://github.com/conda-forge/miniforge#miniforge3):

curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh

If you have the space to do so, we recommend installing Miniforge3 somewhere in
our home directory.  Typically, this file system will be faster than on a
scratch space spaces.  Also, a conda environment will be rendered useless if
its contents get purged.  However, conda environments each require several GB
of space, which may quickly use up your disk quota so you will need to weigh
these trade-offs for yourself.

In what follows, we assume you have Miniforge3 installed in
`${HOME}/miniforge3`.

You may wish to skip the step in the Miniforge3 installation where it
wants to modify your `.bashrc`.  If you allow it to, it will add activation of
the `conda` command and the `base` conda environment, and this may not be
something you want.

If you opt not to allow Miniforge3 to update your `.bashrc`, it may be helpful
to add an alias like the following to your `.bashrc`:

alias conda_base='
source ${HOME}/miniforge3/etc/profile.d/conda.sh
conda activate'

Anytime you need `conda` command, you can first run `conda_base`to get it.

With Miniforge3 installed and the `conda` command available, you can create the
Omega development environment, starting from the root
of an Omega branch, with:

cd components/omega/
conda create -n omega_dev --file dev-conda.txt
conda activate omega_dev

If you need to update the `omega_dev` conda environment, the easiest way to
do that is just to recreate it by following the same instructions as above.
You will be prompted about whether you want to delete the old environment
unless you include the `-y` flag (to say "yes" to all prompts).

Hyperlink target "omega-dev-install-miniforge3" is not referenced.

Hyperlink target "omega-dev-create-conda-env" is not referenced.

Hyperlink target "omega-dev-update-conda-env" is not referenced.

Typically, this is read as a single input file that can serve as
a description of the configuration for provenance as well. The input YAML
file and it's requirements are described in the
[User Guide](#omega-user-config).

Within the model, the paradigm should be that a given module will extract
its configuration from the full configuration as part of the module's
initialization and store it in private class or module variables to be used
later. This paradigm keeps variables close to the code that uses them and
will allow the deletion of the configuration at the end of initialization
to free memory.

its configuration from the full configuration as part of the module’s

The configuration is built around the yaml-cpp library. Each configuration
contains a ``YAML::Node`` and a name for the configuration.
Most of the functions below are wrappers around yaml-cpp templated functions.
Rather than using templated forms in omega, we alias the function interfaces
so that the syntax is cleaner.

Within Omega, the full configuration file is read just after the MachEnv
initialization using:

The full Omega configuration is stored in a static variable for later
retrievals.

Each module in Omega will extract its own configuration variables by
first retrieving the stored Omega configuration, then retrieving the module
configuration (multiple times if the module is nested under a parent)
and finally retrieving the variable by name. The sequence might look like
this for the example shown in the User Guide:

// Get pointer to full configuration
Config *OmegaConfig = OMEGA::Config::getOmegaConfig;

Config HmixConfig; // creates an empty config for Hmix

OmegaConfig->get("Hmix",HmixConfig); // extracts all Hmix config vars from omega

// Get individual variables
Error Err;
Err += HmixConfig.get("HmixScaleWithMesh", HmixScaleWithMesh);
Err += HmixConfig.get("MaxMeshDensity"   , MaxMeshDensity);
Err += HmixConfig.get("HmixUseRefWidth"  , HmixUseRefWidth);
Err += HmixConfig.get("HmixRefWidth"     , HmixRefWidth);

All Config get/set/add functions support variables of all Omega data
types (I4, I8, R4, R8, Real, bool, std::string) and also
std::vector vectors of any of those supported types.
If the module is deeper in the hierarchy (eg an HmixDel2 module
under the Hmix module), you would need an additional call to extract
the subconfiguration, like:

OMEGA::Config HmixDel2Config; // creates and empty config for Hmix Del2
Err += HmixConfig.get(HmixDel2Config);

If the variable cannot be found in the input configuration, a failure
error code is returned.  Existence can also be checked with the bool
functions existsGroup or existsVar, where the variable name is provided.
The developer will need to decide the action to take based on that
condition. For example, the code could return a critical error and terminate
if it is required. In some cases, it may be sufficient to issue a warning
and set the value of the variable to an internal default.
In this latter case, it is a good idea to add the variable and its value to
the configuration using the add function. For example:

Then it will be included in any future writing of the configuration in
order to save provenance. Similarly, an existing value can be changed using
the ``set`` function that has an analogous form. There is also a remove
function that can remove a variable or subconfiguration from any Config -
only the name needs to be supplied. Finally, any configuration can be
written using the write function with a filename supplied as in

In some cases, the variable or group name is not known prior to reading.
For example, users can define an arbitrary number of IO streams in the
Config file. In these cases, a module must loop through the entries,
retrieving the name of each entry. Then the variable or sub-configuration
can be retrieved by name for further processing. The streams example might
then look like:

// extract streams sub-configuration from full Omega config
Config StreamsConfig; // creates an empty config for streams
OmegaConfig->get("Streams",StreamsConfig);

// The iterator here is a Config::Iter (we use auto to save space)
for (auto It=StreamsConfig->begin(); It != StreamsConfig->end(); ++It) {
   std::string NodeName;
   Err += Config::getName(It, NodeName);

   // Now we can get the specific configuration of each stream by name
   // and use it to process each stream
   Config ThisStreamConfig;
   Err += StreamsConfig->get(NodeName, ThisStreamConfig);

   // Extract info from ThisStreamConfig as needed
}

The Config iterator is identical to a `YAML::const_iterator` which in
turn behaves like the `std::map` iterator and they can be used in a
similar manner. For example, you can retrive the key and value of a
YAML map node entry using `Iter->first.as<std::string>()` (for the key) and
`Iter->second.as<DataType>()` (for the value). Generally, the other
interfaces described in this section are preferred after retrieving the
name with the getName interface as shown above, but this iterator functionality
is available if needed when iterating through Configuration entries that
are not known in advance.

While the focus is on reading a configuration, a new configuration can be
built up from scratch by creating an empty Config instance using the
constructors and adding variables and sub-configurations through the add
functions. In this process, the configuration must be built up from the
bottom of the hierarchy and added to each higher layer until reaching the
top. For an example of this process, see the ConfigTest unit test that
builds a sample config for testing.

In order to support the automatic generation of a default Config file,
we request all modules include the default configuration as comments in
the header file, using a backslash ConfigInput to delimit the lines to
extract into a default Config input. For example:

The automated tool (not yet developed) can extract these into a default
input config file and optionally remove the comment lines for a more
compact input. User's can then modify this default file to configure their
specific simulation.

compact input. User’s can then modify this default file to configure their

Omega supports all standard data types and uses some additional defined
types to guarantee a specific level of precision. In particular, we
define I4, I8, R4 and R8 types for 4-byte (32-bit) and 8-byte (64-bit)
integer and floating point variables.
In addition, we define a Real data type that is, by default,
double precision (8 bytes/64-bit) but if the code is built with a
`-DOMEGA_SINGLE_PRECISION` preprocessor flag
(see [build system documentation](#omega-dev-cmake-build)),
the default Real becomes single precision (4-byte/32-bit). For floating
point variables, developers should use this Real type instead of the
specific R4 or R8 forms unless the specific form is required
(eg converting to single precision before output or if a particular
algorithm is known to require double precision). This allows us to
easily convert all reals to single precision to explore performance or
accuracy in single precision mode. In some cases creating literal values
of type Real is necessary to avoid unwanted promotions. For that purpose
a user-defined literal `_Real` is provided. As an example, we can compute
the inverse area of a cell using only the Real type as follows:

The C++ language does not have native support for multi-dimensional
arrays as part of the language standard, though there are a number
of implementations as part of the Standard Template Library and
elsewhere. Omega uses the [Kokkos](https://github.com/kokkos)
framework for defining and allocating arrays on both the CPU host and
any accelerator device that may be present. Because the syntax for
defining such arrays is somewhat long, we instead define a number of
alias array types of the form `ArrayNDTT` where N is the dimension of
the array and TT is the data type (I4, I8, R4, R8 or Real) corresponding
to the types described above. The dimension refers to the number of
ranks in the array and not the physical dimension. Although Kokkos
supports Fortran ordering, we use C ordering for array indices.
Within Omega, the default location for an Array should be on the device
with a similar type HostArrayNDTT defined for arrays needed on the host.
As an example, we can define and allocate a device and host array using:

   Array3dReal Temperature("Temperature",nTimeLevels, nCells, nVertLayers);
   HostArray3dReal TemperatureHost("Temperature",nTimeLevels, nCells, nVertLayers);

Alternatively, you can use the copy functions to create a host copy
from the device or vice versa.

   auto TemperatureHost = OMEGA::createHostMirrorCopy(Temperature);

Note that MirrorCopy will perform a deep copy on creation, but will not
automatically synchronize the copies during later computations. The
synchronization must be performed manually after any changes to host or
device arrays using a deepCopy call:

Finally, arrays can be deallocated explicity using the class
deallocate method, eg `Temperature.deallocate();` or if they are local
to a routine, they will be automatically deallocated when they fall out
of scope on exit. More details on Kokkos arrays are available in the Kokkos
documentation.

In order to run across nodes in a parallel computer, Omega subdivides the
horizontal domain into subdomains that are distributed across the machine
and communicate using message passing via the Message Passing Interface (MPI).
To decompose the domain, we utilize the
[METIS/ParMETIS](http://glaros.dtc.umn.edu/gkhome/metis/metis/overview)
libraries.  An Omega mesh is fully described by the
[MPAS Mesh Specification](https://mpas-dev.github.io/files/documents/MPAS-MeshSpec.pdf)
which we will reproduce here eventually. The Decomp class decomposes
the domain based on the index space and number of MPI tasks (currently
one subdomain per task). Once decomposed, the Decomp class holds all
of the index space information as described below.

Within Omega, a default decomposition of the mesh is first created
with the call:

This must be called very early in the init process, just after initializing
the MachEnv, Config and IO.  Mesh information is first read using parallel
IO into an equally-spaced linear decomposition, then partitioned by METIS or
ParMETIS into a more optimal decomposition. The parallel ParMETIS
implementation should be used to reduce the memory footprint for
high-resolution configurations. Although the algorithmic method is similar,
ParMETIS leads to a somewhat different partitioning than the serial METIS
version. The serial version should give the same partition as the serial
offline gpmetis tool used by MPAS when the same version of the METIS library
is used.

METIS requires information about the connectivity in the mesh. In particular,
it needs the total number of cells and edges in the mesh and the connectivity
given by the CellsOnCell array that stores the indices of neighboring cells
for each cell. Once the cells have been partitioned, we determine multiple
layers of halo cells that will be needed to avoid excessive communcation with
remote neighboring subdomains. The CellsOnCell, EdgesOnCell
and VerticesOnCell arrays are then distributed according to this cell
partitioning. Edges and Vertices are then partitioned. Ownership of each
edge/vertex is determined by the first (valid) cell index in the CellsOnEdge
or CellsOnVertex for that edge/vertex. The remaining connectivity arrays
(CellsOnEdge, EdgesOnEdge, VerticesOnEdge, CellsOnVertex, EdgesOnVertex)
are the redistributed to match the edge and vertex distributions. Halos
are filled to ensure all necessary edge and vertex information for the
cell decomposition (and cell halos) are present in the subdomain.

After the call to the Decomp initialization routine, a Decomp named
Default has been created and can be retrieved with

Once retrieved all Decomp members are public and can be accessed using

OMEGA::I4 NCells = DefDecomp->NCells;
OMEGA::HostArray1DI4 CellIDH = DefDecomp->CellIDH;

Decomp is a container for all mesh index and connectivity arrays as
described in the mesh specification above. In particular, it contains

NCellsSize: the length of the Cell array axis (typicall NCellsAll+1)

NCellsHalo(i): the number of owned+halo cells for each halo layer

CellLoc(NCellsSize,2): the task and local index for every local cell
(redundant for owned cells but gives the remote location for halo cells)

CellsOnCell(NCellsSize,MaxEdges): the local index for neighbor cells
across each cell edge

EdgesOnCell(NCellsSize,MaxEdges): the local edge index for each cell edge

VerticesOnCell(NCellsSize,MaxEdges): the local index for each cell vertex

CellsOnEdge(NEdgesSize,2): the index of the 2 cells sharing an edge

VerticesOnEdge(NEdgesSize,2): the vertex index at each edge endpoint

EdgesOnEdge(NEdgesSize,MaxEdges x 2): the index for all edges contributing
to an edge (all edges of the 2 cells sharing an edge)

CellsOnVertex(NVerticesSize,VertexDegree): indices for all cells meeting
at a vertex

EdgesOnVertex(NVerticesSize,VertexDegree): indices for all edges meeting
at a vertex

NEdgesOnCell(NCellsSize): the number of actual edges on each cell

NEdgesOnEdge(NEdgesSize): the number of actual edges on each edge

For each of the arrays above, there is a copy of the array on the host and
device (GPU) with the host array named with an extra H on the end
(eg CellsOnCellH). All are Kokkos arrays so are accessed with (index) rather
than [index] and for some of the arrays noted above are multi-dimensional.
A typical host loop might then look something like:

for (int Cell = 0; Cell < NCellsOwned; ++Cell) {
   for (int Edge = 0; Edge < NEdgesOnCell(Cell); ++Edge) {
      VarH(Cell) = VarH(Cell) + FluxH(Cell,Edge);
   }
}

And on the device, we use `OMEGA::parallelFor` in place of `Kokkos::parallel_for`;

OMEGA::parallelFor( {NCellsOwned,MaxEdges},
                       KOKKOS_LAMBDA (int Cell, int Edge) {
  if (Edge < NEdgesOnCell(Cell)) {
      Var(Cell) = Var(Cell) + Flux(Cell,Edge);
  }
});

and all decompositions *must* be removed before the Kokkos finalize call using

which destroys all host and device arrays before Kokkos finalizes and removes
the memory pool in which all the arrays are allocated. Failure to call clear
before `Kokkos::finalize()` will result in an error.

When performing IO for fields in Omega, some metadata associated with each
dimension of multi-dimensional arrays are required. The Dimension class is
a container that stores this dimension information and a Dimension instance
must be created for each dimension in Omega. Most of these will be created
by the Mesh or Decomposition classes, but if developers introduce a new
dimension, the dimension must be created.

For dimensions that are distributed across the parallel machine, the creation
must provide length and an offset array that describes the global index offset
for each local point along the dimension. Typically this is the zero-based
ID of the location in the global mesh. A negative value is used for any halo
regions that will not participate in IO. For example, the NCells dimension
is created using:

   Decomp *DefDecomp = Decomp::getDefault();

   I4 NCellsSize   = DefDecomp->NCellsSize; // Local dim length
   I4 NCellsGlobal = DefDecomp->NCellsGlobal; // Global dim length
   I4 NCellsOwned  = DefDecomp->NCellsOwned;

   HostArray1DI4 CellOffset("NCellsOffset", NCellsSize);

   for (int N = 0; N < NCellsSize; ++N) {
      if (N < NCellsOwned) {
         CellOffset(N) = DefDecomp->CellIDH(N) - 1; // Offset is zero-based
      } else {
         CellOffset(N) = -1; // Denotes halo cells not to be used in IO
      }
   }

   std::shared_ptr<Dimension> CellDim =
      Dimension::create("NCells", NCellsGlobal, NCellsSize, CellOffset);

For dimensions that are local (eg the vertical dimension or MaxEdgeOnCell),
only the name and length are needed for creation using:

   std::shared_ptr<Dimension> LocalDim =
      Dimension::create("MyDimName", LengthOfDim);

The offset array is computed internally for these dimensions and is simply
the array index along the dimension.

Each Dimension stores the global and local length of the dimension as well
as the offset array. It also stores a flag to denote whether the dimension
is distributed or not. This information is used primarily by the IOStreams
class to write dimension metadata and define offsets for parallel IO. It is
not expected to be used elsewhere in Omega.

Dimension information is available through retrieval functions. These can be
retrieved by dimension name as in:

   I4 LengthGlob  = Dimension::getDimLengthGlobal("MyDimName");
   I4 LengthLoc   = Dimension::getDimLengthLocal("MyDimName");
   bool DistrbDim = Dimension::isDistributedDim("MyDimName");
   HostArray1DI4 MyOffset = Dimension::getDimOffset("MyDimName");

or it can be retrieved from an instance directly using member retrieval
functions:

   std::shared_ptr<Dimension> MyDim = Dimension::get("MyDimName");
   I4 LengthGlob = MyDim->getLengthGlobal();
   I4 LengthLoc  = MyDim->getLengthLocal();
   bool DistrbDim = MyDim->isDistributed();
   HostArray1DI4 MyOffset = MyDim->getOffset();

An iterator is also provided to enable looping over all defined dimensions:

   for (auto Iter = Dimension::begin(); Iter != Dimension::end(); ++Iter) {
      std::string MyDimName = Iter->first;
      std::shared_ptr<Dimension> ThisDim = Iter->second;
      I4 LengthLoc = ThisDim->getLengthLocal();
      // other retrievals, etc
   }

and all dimensions should be removed during Omega finalization before exiting
the Kokkos environment using:

Examples of the use of Dimensions can be found in the Dimension unit test.

The Omega documentation is generated using the
[Sphinx](https://www.sphinx-doc.org/en/master/) package and is written in
[MyST](https://myst-parser.readthedocs.io/en/latest/syntax/syntax.html)
format.  We recommend these
[basic tips](https://myst-parser.readthedocs.io/en/latest/syntax/roles-and-directives.html#roles-directives)
on using MyST in Sphinx.

Another easy way to get started is by taking a look at the existing source
code for the documentation:

Each time you add new features to Omega, the corresponding documentation must
be included with the pull request to add the code.  This includes documentation
for both the User's Guide and the Developer's Guide. We will add some examples
as the documentation gets fleshed out.

for both the User’s Guide and the Developer’s Guide. We will add some examples

Please add anchors for internal linking to the top of every page before
headings within the page whenever you think internal linking might be useful
(now or in the future).  In the User's Guide, please start each anchor with
`omega-`:

(now or in the future).  In the User’s Guide, please start each anchor with

This is in anticipation of a future time when the Omega documentation might
be combined with other components of E3SM.

In the Developer's Guide, anchors should start with `omega-dev-`:

Documentation for an Omega feature in the User's Guide should contain
information that is needed for users who set up and run Omega, including:

Documentation for an Omega feature in the User’s Guide should contain

Background information on the feature (though not as much as in Developer's
Guide and not referencing code) that gives the user an understanding of
the configurable parameters.

Background information on the feature (though not as much as in Developer’s

Config options related to the feature that users can modify in a YAML file.

Flags related to the feature that a user can (or must) set when building
Omega.

The Developer's Guide should also serve as a reference manual.  Among other
things, the documentation in the Developer's Guide needs to provide an easy way
for other developers or reviewers to verify the code against the intended
implementation. Therefore, documentation for a new feature in the Developer's
Guide should:

The Developer’s Guide should also serve as a reference manual.  Among other

things, the documentation in the Developer’s Guide needs to provide an easy way

implementation. Therefore, documentation for a new feature in the Developer’s

Describe the actual mathematical terms (or a reference to them).

Describe the discretized form with variables close to what are used in the
code.

Provide details on unit tests that cover that feature, if applicable.

Describe any polaris regression tests that cover that feature, if applicable.

The Omega Driver consists of methods to run the Omega ocean model in either
standalone mode or as a component of E3SM. In both cases, the execution of Omega
is divided into three phases, carried out by the `ocnInit`, `ocnRun`, and
`ocnFinalize` methods. An interface layer to translate data types of the parent
model to and from internal Omega data types will also be needed for running
Omega as a component. For standalone mode, a `main` function is defined to
serve as the top-level driver. The driver methods will evolve as more modules
are developed and as Omega is integrated with E3SM.

 methods. An interface layer to translate data types of the parent

The `ocnInit` method initializes everything needed to run the Omega ocean model,
and has the following interface:

 method initializes everything needed to run the Omega ocean model,

int ocnInit(MPI_Comm Comm, Calendar &OmegaCal, TimeInstant &StartTime, Alarm &EndAlarm);

It requires an MPI communicator as input, as well as a yaml configuration file
named `omega.yml` in the directory Omega is launched from. Default configuration
files are currently kept in the `configs` directory, eventually an automated
tool will extract default configuration options from module header files and
build the default config file. The `ocnInit` method will read the configuration
options into the static `ConfigAll` object. The time management options that
determine the duration of the simulation are read from the Config object and
stored in `OmegaCal`, `StartTime`, and `EndAlarm`, which are output by the
method. The `init()` methods for each Omega module necessary to run the ocean
model are called. An integer error code is returned.

The `ocnRun` method advances the model over the interval defined by time
management options, integrating from the start time until the `EndAlarm` is
ringing. The interface is:

The `TimeStep` is retrieved from the default `TimeStepper` object initialized
during `ocnInit` and used with the value of `CurrTime` on input to construct
a `Clock` object which manages the time loop. The `EndAlarm` is attached to
the `Clock` to signal the end of the time loop. Eventually, there will be
periodic alarms also attached to the `Clock` to trigger forcings, writing
output, restarts, etc. As the time loop advances, the `doStep` method of the
`TimeStepper` object is called to update the ocean model until the `EndAlarm`
is triggered. The updated `CurrTime` and status of `EndAlarm` are returned,
along with an integer error code.

The `ocnFinalize` method is needed to clean up all the objects allocated by the
model. The interface is:

The `CurrTime` is input in case a restart file needs to be written. An integer
error code is returned.

 is input in case a restart file needs to be written. An integer

Omega includes an `Eos` class that provides functions that compute `SpecVol`, `SpecVolDisplaced`,
and `BruntVaisalaFreqSq`. Current EOS options are a linear EOS or an EOS computed using the TEOS-10
75 term expansion from [Roquet et al. 2015](https://www.sciencedirect.com/science/article/pii/S1463500315000566).
If `SpecVolDisplaced` is calculated with the linear EOS option, it will be equal to `SpecVol` as there
is no pressure/depth dependence for the linear EOS. `SpecVolDisplaced` computes specific volume
adiabatically displaced to `K + KDisp` (where `K` counted positive downward, ie `K+1` is one layer below `K`). Note: `SpecVol` must be calculated before `BruntVaisalaFreqSq`, as
`SpecVol` is an input for the `BruntVaisalaFreqSq` calculation. If the linear EOS option is used, then the `BruntVaisalaFreqSq`
is calculated using linear coefficients. If the TEOS-10 option is used, the `BruntVaisalaFreqSq` is calculated with non-linear
coefficients according to the [TEOS-10 toolbox](https://www.teos-10.org/software.htm). Note: two assumption for ease of computation and efficiency have been made
for the `BruntVaisalaFreqSq` TEOS-10 option that differ from how it is calculated in the TEOS-10 toolbox:
(1) gravity is assumed to be constant and not a function of depth and latitude, and (2) the interface value of the specific volume is
calculated as the average between two layer values, rather than being recalculated using the interface values of temperature,
salinity, and pressure. Both of these assumptions incur less than a 1% error.

. Current EOS options are a linear EOS or an EOS computed using the TEOS-10

is calculated using linear coefficients. If the TEOS-10 option is used, the 

. Note: two assumption for ease of computation and efficiency have been made

 TEOS-10 option that differ from how it is calculated in the TEOS-10 toolbox:

An enumeration listing all implemented schemes is provided. It needs to be extended every time an
EOS is added. It is used to identify which EOS method is to be used at run time.

An instance of the `Eos` class requires a [`HorzMesh`](#omega-dev-horz-mesh), so the mesh class
and all of its dependencies need to be initialized before the `Eos` class can be. The static method:

initializes the default `Eos`. A pointer to it can be retrieved at any time using:

To compute `SpecVol` for a particular set of temperature, salinity, and pressure arrays, do

 for a particular set of temperature, salinity, and pressure arrays, do

`SpecVolDisplaced` is calculated using local temperature and salinity values, but a pressure
value at `K + KDisp`. To compute `SpecVolDisplaced` for a particular set of temperature, salinity,
and pressure arrays and displaced vertical index level, do

 is calculated using local temperature and salinity values, but a pressure

Eos.computeSpecVolDisp(ConsrvTemp, AbsSalinity, Pressure, KDisp);

where `KDisp` is the number of `k` layers you want to displace each specific volume layer to.
For example, to displace each level to one below, set `KDisp = 1`.

To compute `BruntVaisalaFreqSq` for a particular set of temperature, salinity, pressure, and specific
volume arrays, do

 for a particular set of temperature, salinity, pressure, and specific

Eos.computeBruntVaisalaFreqSq(ConservTemp, AbsSalinity, Pressure, SpecVol);

The error handling facility in Omega provides a number of macros and an
error class to determine behavior when encountering errors and to provide a
standard error return code that routines can use. Macros are used to provide
a concise single-line interface when checking errors and to ensure the error
message reports the correct line number.  It builds on the
[Logging](#omega-dev-logging) facility for writing the error messages and
for some interfaces will generate a stack trace using the
[cpptrace](https://github.com/jeremy-rifkin/cpptrace) library. All interfaces
can be used after including the `Error.h` and after initializing the Logging
facility (typically done as early as possible in Omega initialization).

The most common use case is to check for critical errors using the
`ABORT_ERROR` macro. For example:

  if (condition) ABORT_ERROR("Bad value for variable: {}", Value);

The macro takes a string argument for the error message and allows values
to be inserted where `{}` placeholders exist, similar to the Logging macros.
This macro writes the error message and aborts the simulation. If Omega has
been built with debug information available (either a debug build or the
-g option has been passed to the compiler), the macro also creates and writes
a stack trace to provide the calling sequence that generated the error.

Closely related to this critical error macro are the assert and require
macros:

OMEGA_ASSERT(Condition, Message, addition args for message);
OMEGA_REQUIRE(Condition, Message, addition args for message);

These emulate the behavior of the C++ assert function in which the condition
is checked and a critical error is thrown if the condition is not met (false).
The `ASSERT` macro is only evaluated in debug builds while the `REQUIRE`
macro is always evaluated. These have identical behavior to the critical error
macro, but take the additional boolean argument of the condition to be checked.

The remaining macros operate on or with an Error class that contains an error
code and an error message. These can be used as a return code, as in:

The error code within the class is an ErrorCode enum that defines the severity
of the error:

enum class ErrorCode {
   Success = 0, /// Standard success code
   Warn,        /// Standard warning code
   Fail,        /// Standard failure code
   Critical     /// Standard critical error code
}

Errors can be accumulated using either the `+` or `+=` operators. This is
useful especially in test drivers where we may not want to exit when an
error is encountered or if the error is only a warning.

Error MyError; // defaults to success
MyError += myRoutine1();
MyError += myRoutine2();

When adding or accumulating errors, the resulting error code is the max (most
severe) error code and the error messages are concatenated (separated by a
newline character) so that all messages are retained.

For routines that return an error, we provide an `RETURN_ERROR` macro. For
example,

Error ReturnVal; // initializes to success
[do stuff]
if (Condition) RETURN_ERROR(ReturnVal, ErrorCode::Fail, ErrMsg);

This macro accumulates the new error (Code, ErrMsg) into the return code
and returns control and the error to the calling routine. The error message
is _not_ printed by this macro; the calling routine must decide what action
to take with the returned error code and message.

 printed by this macro; the calling routine must decide what action

To check these returned error codes or any other accumulated errors in a code,
the following macros can be used:

CHECK_ERROR(ErrorToBeChecked, NewMessage, MsgArgs...);
CHECK_ERROR_WARN(ErrorToBeChecked, NewMessage, MsgArgs...);
CHECK_ERROR_ABORT(ErrorToBeChecked, NewMessage, MsgArgs...);

These macros check the error and if it is not Success, prints the new message
to the log file along with any other accumulated messages associated with the
error. In the first two cases, the error is then reset (back to success
with an empty message) and the simulation continues. In the third case,
the new message is treated as a critical error, so after printing all the
messages, it prints a stack trace and aborts.

While the macros above cover most use cases, it is possible to use the
Error class functions directly to customize the treatment of particular errors.
For checking error codes, there are two useful member functions:

that check whether the code is `ErrorCode::Success` or `ErrorCode::Fail`.
These functions check for equality so it is sometimes more useful to check
their negation, as in

This example will pick up all non-successful conditions (warn, fail, critical).

The default constructor is often used (eg in the examples above) to create
and initialize an Error instance for use as a return code.  Three other
constructors in addition to this default constructor are used by the
macros and examples above:

Error MyError; // default constructor with success code and empty message
Error MyError(ErrorCode); // constructor with error code only and empty msg
Error MyError(ErrorCode, ErrorMessage); // constructor with both code and msg
Error MyError(ErrorCode, Line, File, ErrorMessage); // as above but adds prefix

In the last two cases, the constructor initializes the error message either
with or without a prefix. In cases where the message is expected to be
accumulated without an immediate print (eg the `RETURN_ERROR` macro) the prefix
case is used and the prefix includes the message severity and the routine name
and line number where the error occurs (the cpp `__LINE__` and `__FILE__` can
be used). In the other constructor with a message, the prefix is left off,
typically for cases where the relevant LOG function will be called underneath
which adds the approporiate prefix as part of Logging.

In the `CHECK_ERROR` macros, the Error is reset to Success and a blank message
using the function

Finally, the Error class includes the abort function `Error::abort()` which
aborts the simulation. Currently, the function simply prints a generic aborting
message and calls the `MPI_abort()` function to kill the simulation, but
additional functionality may be added later to clean up before shutting down.
The abort function assumes it is being called from one of the Error functions
or macros so does not print any previous accumulated errors and does not
print a stack trace. It is best to use one of the ABORT macros above rather
than calling the abort member function directly.

Omega includes a Field class that registers available fields and associated
metadata for use in IO (or any other part of Omega that may require the field
metadata). The module owning the field will define the Field together
with some required metadata for compliance with the Climate and Forecast
[(CF) metadata conventions.](http://cfconventions.org/). Groups of Fields
can also be defined to provide a short cut to groups that are commonly used
together, like the model state and tracer groups. Two special Fields, CodeMeta
and SimMeta (with default names "code" and "simulation", respectively), are
defined on initialization to store global metadata or scalars needed for
provenance or for restarting. The arrays containing Field data can be attached,
retrieved or updated as needed, typically during IO. The Field class is not
meant to be used during computations - the native Omega array types without
metadata are more appropriate for that context.

and SimMeta (with default names “code” and “simulation”, respectively), are

To use the Field class, the Field header file must be included and as part of
the application initialization, there should be a call to the init method:
Fields initialized with the init method:

which primarily defines the CodeMeta and SimMeta fields for later use.
For array fields, the appropriate Dimensions must be defined. The default
dimensions (eg NCells, NEdges, NVertLayers) will be defined by the relevant
Mesh initialization and should be done before any Fields are defined.
See {ref}`omega-dev-dimension`.

   std::shared_ptr<Field> MyField =
   Field::create(FieldName,   ///< [in] Name of variable/field (string)
                 Description, ///< [in] long Name or description (string)
                 Units,       ///< [in] units (string)
                 StdName,     ///< [in] CF standard Name (string)
                 ValidMin,    ///< [in] min valid field value (same type as
                 ValidMax,    ///< [in] max valid field value  field data)
                 FillValue,   ///< [in] scalar used for undefined entries
                 NumDims,     ///< [in] number of dimensions (int)
                 Dimensions,  ///< [in] dim names (vector of strings)
                 TimeDependent   ///< [in] (opt, default true) if time varying
                 RetainPrecision ///< [in] (opt, false) retain full prec in IO
   );

This interface enforces a list of required metadata. If a CF standard name does
not exist, an empty string can be provided. This is uncommon for most fields
since the CF conventions maintain a fairly complete list, but can be the case
for some intermediate calculations or unique analyses. If there is no
restriction on valid range, an appropriately large range should be provided for
the data type. Similarly, if a FillValue is not being used, a very unique
number should be supplied to prevent accidentally treating valid data as a
FillValue. The optional TimeDependent argument can be omitted and is assumed
to be true by default. Fields with this attribute will be output with the
unlimited time dimension added. Time should not be added explicitly in the
dimension list since it will be added during I/O. Fields that do not change
with time should include this argument with the value false so that the time
dimension is not added. The argument RetainPrecision is also optional and should
be used if the full precision of the field needs to be retained during IO even
if it is a part of an IO stream that requests reduced precision. An example is
the time field that can require full precision to represent the seconds from a
reference time far in the past. The default is false and the argument does not
need to be supplied in most cases. However, if it is needed, then the
TimeDependent argument must also be explicitly included due to the language
rules on argument ordering. Actual field data stored in an array is attached in a
separate call as described below. Scalar fields can be added by setting the
NumDims to zero (the Dimensions vector is ignored but an empty vector
must still be supplied in the argument list). Scalar data is attached using
a 1D array with size 1. Fields without a data array can be created with:

   Field::create(FieldName ///< [in] Name of field
   );

but this interface should generally be not be used and global metadata
should be added to the existing CodeMeta or SimMeta fields for this
purpose.

Additional metadata in the form of a name-value pair can be added using:

   MyField->addMetadata(MetaName, // [in] Name of new metadata (string)
                        Value     // [in] Value of new metadata
   );

where Value can be any supported data type (I4, I8, R4, R8, bool, string).
Multiple pairs can be added in a single call using:

   SimField->addMetadata(
         {std::make_pair("Name1", Val1),
          std::make_pair("Name2", Val2),
          std::make_pair("Name3", Val3),
          std::make_pair("Name4", Val4)});

where any number of pairs can be included and the values can be any supported
type.

As mentioned above, the actual data array is attached in a separate call
using a templated form. If the pointer to the Field is available, use the
member function:

   MyField->attachData<ArrayType>(InDataArray);
   // for example:
   MyField->attachData<HostArray1DI4>(CellID);
   MyField->attachData<Array2DR8>(NormalVelocity);

where ArrayType is one of the supported array types (Array1DI4, etc. or
HostArray1DI4, etc.). If the Field pointer has not been retrieved, an interface
is provided using the field name:

   Field::attachFieldData<ArrayType>(
                    FieldName,   // [in] Name of Field (string)
                    InDataArray  // [in] Array with data to attach
                    );

Note that the data is assumed to reside in only one location so if a mirror
array exists (eg if replicated on host and device), a separate Field may be
needed. However, it is is better to define only one location and allow the
IO or other modules to determine whether a transfer of data or mirror is needed.
If the data resides in an array whose location does not change (ie the
pointer always points to a fixed location), the attach can be performed when
the Field and array have been created and any updates to the data will be
captured correctly. If the location of the data changes (eg the time
level changes and the pointer points to a different time slice), the data must
be updated by calling the attach routine to replace the pointer to the new
location. It is up to the developer to insert the appropriate call to reattach
the data. As mentioned previously, scalar data should be attached using the
appropriate 1D HostArray with a size of 1. The attach function primarily sets
the pointer to the data location but it also sets the data type of the variable
and its memory location using two enum classes:

enum class FieldType {Unknown, I4, I8, R4, R8};
enum class FieldMemLoc {Unknown, Device, Host, Both};

These provide a simple way to query these properties when needed in some cases
(eg IO) where the type or location are unknown and need to be determined before
the field data is retrieved. The "Both" memory location is used in CPU-only
configurations where the host and device are the same or in future cases where
both may share a memory space. These are determined based on the data type of
the attached data array.

the field data is retrieved. The “Both” memory location is used in CPU-only

Once a field is defined, various query functions are available. The existence
of a field with a given name can be determined with:

Other queries have two forms. If the Field pointer has already been retrieved,
a member function can be used. If not, a query by field name can be used
instead. The data type and memory location can be determined using:

   FieldType MyType1 = MyField->getType(); // member function
   FieldType MyType2 = Field::getFieldType(FieldName); // name version

   FieldMemLoc MyLoc1 = MyField->getMemoryLocation(); // member function
   FieldMemLoc MyLoc2 = Field::getFieldMemoryLocation(FieldName); // name vers

Sometimes it is only necessary to query whether the data exists on the
host rather than retrieving the full memory location.

   if (MyField->isOnHost()) { do stuff; } // member function
   if (Field::isFieldOnHost(FieldName)) { do stuff }; // name version

When looping through all defined fields, it is handy to have a query for the
field name, so we provide:

   int NDims = MyField->getNumDims();
   std::vector<std::string> MyDimNames(NDims);
   MyField->getDimNames(MyDimNames);

Once the dimension names have been retrieved, the Dimension class API can be
used to extract further dimension information. Two other field quantities
can be retrieved, but are used only by the IOStream capability:

   bool IsTimeDependent = MyField->isTimeDependent();
   bool IsDistributed   = MyField->isDistributed();

The first determines whether the unlimited time dimension should be added
during IO operations. The second determines whether any of the dimensions
are distributed across MPI tasks so that parallel IO is required.

The data and metadata stored in a field can be retrieved using several
functions.  To retrieve a pointer to the full Field, use:

With this pointer all the member functions above can be used.
The Metadata associated with a field can be retrieved individually using:

where the MetaValue can be a scalar of any supported data type (I4, I8, R4, R8,
bool, std::string). If the value of a metadata entry needs to be changed,
an update function is provided:

The entire group of metadata is stored in a Metadata type which is simply an
alias for a ``std::map<std::string, std::any>``. This collection of metadata
can be retrieved using:

   std::shared_ptr<Metadata> ThisMeta = ThisField->getAllMetadata();
   // or
   std::shared_ptr<Metadata> ThisMeta = Field::getFieldMeta(FieldName);

This can be useful if we need to extract all the metadata at once (eg during
the IO write phase). However, because the value is stored as a ``std::any``,
it must be coerced to the proper data type using the ``std::any`` type query
and the ``std::any_cast`` coercion function.

To retrieve the field data arrays, there are a few methods available. If the
array forms are needed, there are templated retrievals by either a member
function or a by-name interface:

   HostArray1DI4 MyData1 = MyField->getDataArray<HostArray1DI4>();
   Array2DR8 MyData2 = Field::getFieldDataArray<Array2DR8>(FieldName);

where all of the array and host array types are supported. If the array type
is not known in advance, the field can be queried for both type and memory
location as described previously.

   MyField->removeMetadata(MetaName);
   MyField->removeAllMetadata();

depending on whether a single metadata entry or all metadata entries need to
be deleted. Entire fields can be removed using:

As mentioned above, Fields can be assigned to groups to provide an easy way
to reference fields that commonly appear together, especially when listing
contents of fields in IO files. Internally, a field group is implemented as
a simple set of field names stored as a ``std::set<std::string>``. A group
is created by first creating an empty group with the desired name:

   std::shared_ptr<FieldGroup> MyGroup = FieldGroup::create(GroupName);

Fields can then be added either through a member function if the group
pointer is available, or by group name:

   MyGroup->addField(FieldName);
   FieldGroup::addFieldToGroup(FieldName, GroupName);

The latter is useful especially if the group was created elsewhere. If the
field has already been added to the group, no additional entries are created.
To determine whether a group exists or whether a field is in a group, several
forms of these queries are available:

   bool MyGroupExists = FieldGroup::exists(GroupName);
   bool FieldIsInGroup = MyGroup->hasField(FieldName);
   bool FieldIsInGroup = FieldGroup::isFieldInGroup(FieldName,GroupName);

In addition, the entire list of fields current assigned to a group can be
retrieved using:

   std::set<std::string> MyFieldList = MyGroup->getFieldList();
   // or
   std::set<std::string> MyFieldList =
                   FieldGroup::getFieldListFromGroup(GroupName);

While this list can be used with the Field interfaces above to retrieve the
Field, we also provide a shortcut to retrieve a full field pointer from a
FieldGroup:

   std::shared_ptr<Field> MyField = MyGroup->getField(FieldName);
   // or
         FieldGroup::getFieldFromGroup(FieldName, GroupName);

   std::shared_ptr<FieldGroup> MyGroup = FieldGroup::get(GroupName);

   MyGroup->removeField(FieldName);
   // or
   FieldGroup::removeFieldFromGroup(FieldName, GroupName);

The removal of a field from a group does not remove the field itself, it only
removes the field name from the list of fields assigned to the group.
The entire group can be removed with:

and the usual ``FieldGroup::clear();`` should be used to remove all field
groups before exiting.

Omega utilizes domain-based parallelism, where the domain is divided into
partitions that run in parallel as separate tasks distributed across the
resources of the machine. Arrays representing physical quantities that are
defined throughout the domain are divided into overlapping chunks and
distributed among the tasks. As the model evolves, the values stored in these
arrays are updated. Array elements that are needed by a task to evolve the
model forward in time, but are defined on and updated by a neighboring task
are halo elements. It is necessary for each parallel task to regularly send
locally defined array elements that belong to the halos of neighboring tasks
and to receive local halo elements owned by neighboring tasks. In Omega, these
halo exchanges are perfromed using the Halo class which, once constructed,
contains lists of indices to send to and receive from neighboring tasks, and
methods to conduct the exchanges. The halo exchanges are carried out using
non-blocking MPI (Message Passing Interface) communication.

An instance of the Halo class is dependent on a parallel domain decomposition
and the associated parallel machine environment, so those objects need to be
constructed first. The default Halo is created with the call:

This must be done after the default MachEnv and Decomp have been initialized.
Once initialized, a pointer to the default Halo can be retrieved with:

Additional Halo objects can be constructed for defined MachEnv and Decomp
objects by calling the Halo create method with a supplied `std::string` Name:

A pointer to each constructed Halo is stored in a `std::map` container, which
can be retrieved by supplying the Name to `OMEGA::Halo::get(Name)`.

The Halo constructor will save the MPI communicator handle and the MPI task ID
of the local task from the MachEnv object, and will extract and save from the
Decomp object indices defining the local and neighboring halo elements for the
cell, edge, and vertex index spaces. This information is organized into objects
of two nested classes of the Halo class: the ExchList class and the Neighbor
class. The ExchList class contains the indices of Halo elements to either send
to or receive from a single neighboring task for a particular index space, and
the Neighbor class contains all the ExchList objects needed to carry out a halo
exchange with a single neighboring task in any index space, as well as the
buffer memory space to communicate with that neighbor. All member variables
and objects and most member methods of the Halo class are declared private
as they are only needed by the Halo class methods to execute an exchange.

The main methods of the Halo class which execute an exchange are

packBuffer: packs halo elements into a buffer to send to a neighbor

startSend: wrapper to call MPI_Isend to send a buffer to each neighbor

unpackBuffer: unpacks halo elements from a received buffer into an array

The packBuffer and unpackBuffer functions are overloaded to support different
array types.

The exchangeFullArrayHalo function is the main interface to conduct a halo
exchange for any supported array defined in the DataTypes module. The
exchangeFullArrayHalo function is a template which passes the array to the
Halo methods above which perform the exchange. The index space on which
the array is defined must also be passed to this function. This is currently
done using an enum defined in the Halo header file.

MyHalo.exchangeFullArrayHalo(SomeCellBasedArray, OMEGA::OnCell);

for a constructed Halo named MyHalo and any supported array type in the cell
index space.

The Omega horizontal mesh uses the [MPAS Mesh
Specification](https://mpas-dev.github.io/files/documents/MPAS-MeshSpec.pdf).
A mesh object is created by the `init` method, which assumes that `Decomp` has
already been initialized.

The constructor replicates the subdomain mesh cell/edge/vertex counts and
connectivity information from Decomp so this information can be passed among the
computational routines, alongside the other local mesh information.  It then
creates several parallel I/O decompositions and reads in the remaining subdomain
mesh information.  Finally, any mesh information needed on the device is copied
from the host to a device Kokkos array. Arrays such as the coordinate variables,
which are not involved in tendency calculations, are not transferred to the
device. These tasks are organized into several private methods. Eventually,
dependent mesh variables will be computed from the minimum set of required mesh
information.

After initialization, the default mesh object can be retrieved via:

Once this retrieval has been performed, public member variables can be accessed
using:

OMEGA::I4 NCellsOwned = HMesh->NCellsOwned;
OMEGA::Array1DR8 AreaCell = HMesh->AreaCell;

The HorzMesh is meant to be a container that allows the mesh information to be
passed to the PDE solver routines:

void computeFluxTendency(OMEGA::HorzMesh *HMesh, ...) {
OMEGA::parallelFor({HMesh->NCellsOwned,HMesh->MaxEdges},
                      KOKKOS_LAMBDA (int Cell, int Edge) {
  if (Edge < HMesh->NEdgesOnCell(Cell)) {
  }
}

For member variables that are host arrays, variable names are appended with an
`H`.  Array variable names not ending in `H` are device arrays.  The copy from
host to device array is performed in the constructor via:

The device arrays are deallocated by the `HorzMesh::clear()` method, which is
necessary before calling `Kokkos::finalize`.

The TRiSK scheme horizontal operators are implemented in C++ as functors, which
are classes that can be called with input arguments in the same way as functions
can. However, in contrast to plain functions, functors can contain internal
state. In the case of Omega operators they contain shallow copies of various
`HorzMesh` arrays that are needed to compute the operator, but are not strictly
inputs of the operator.

 arrays that are needed to compute the operator, but are not strictly

Operators are constructed from an instance of the `HorzMesh` class, for example

    auto mesh = OMEGA::HorzMesh::getDefault();
    DivergenceOnCell DivOnCell(mesh);

Each Omega operator provides a C++ call operator, which computes its value on a
vertical chunk of mesh elements given the element index, the chunk index,
and operator-specific input arrays. The first argument to an operator is
an array that gets updated with the computed values. Typically, operators are
created outside of a parallel region and are used inside a parallel loop
over mesh elements, for example

    parallelFor({mesh->NCellsOwned, NVertLayers / VecLength}, KOKKOS_LAMBDA(int ICell, int KChunk) {
        // computes divergence of Vec for cells with indices (ICell, KChunk:KChunk+VecLength-1)
        // stores the result in DivVec
        DivOnCell(DivVec, ICell, KChunk, Vec);
    });

Some tendency terms in the Omega PDE solver could in principle be constructed
using these operators as building blocks. However, very often tendency terms
require evaluation of slightly modified operators. Moreover, there is a
potential performance cost of nesting operators within classes. Hence, the
primary motivation for introducing these classes is to provide reference
implementation of the basic TRiSK operators and for diagnostic and debugging
purposes.

For input and output of data needed for Omega, we use the Software for
Caching Output and Reads for Parallel I/O
([SCORPIO](https://github.com/E3SM-Project/scorpio)) library. This
library supports parallel reading and writing of distributed arrays in various
self-describing
formats like [NetCDF](https://docs.unidata.ucar.edu/netcdf/),
[HDF5](https://www.hdfgroup.org/solutions/hdf5/),
and [ADIOS](https://csmd.ornl.gov/adios).
SCORPIO is responsible for rearranging data from the data decomposition used
for computation to a different rearrangement for parallel IO. In particular,
for optimal IO, the user would specify a different number of IO tasks than
for computation. For example, the user could specify the number of IO tasks to
match underlying hardware like the network interfaces (NICs) on a node.
Users and developers will generally access IO via
[IOStreams](#omega-user-iostreams)
and other interfaces. The base IO layer described here only provides
an Omega-aware wrapper around SCORPIO calls.

The base interfaces provide functions for file operations (open/close),
reading and writing of metadata, and reading and writing of data arrays.
Interfaces at this level utilize raw pointers to data and assume
contiguous storage for arrays. SCORPIO utilizes integer handles
to various files and data types so these must often be defined or
retrieved for many operations. Although there is no IO class, we encapsulate
the IO routines within the Omega and IO namespaces.

Before using any IO functions, the parallel IO system must be initialized
using:

where Comm is an MPI communicator and should in most cases be the communicator
from the Omega default MachEnv (see [MachEnv](#omega-dev-mach-env)). This
function also extracts the user-defined variables from the model configuration,
include the number of IO tasks, the IO task stride, the default data
rearranger method, and the default file format
(see [User Guide](#omega-user-IO)).

As mentioned above, most I/O operations will take place within the IOStreams
module, but the base IO functions can be accessed directly. To open and close
files for reading/writing, use:

   IO::openFile(FileID, Filename, Mode);
   IO::closeFile(FileID);

   IO::openFile(FileID, Filename, Mode, Format, IfExists);

In both case, an integer FileID is assigned to the open file which is then
used by all subsequent operations. The Filename is a ``std::string`` that
should include the full path and name of the file. The mode can be either
``OMEGA::IO::ModeRead`` or ``OMEGA::IO::ModeWrite``. The shorter interface
uses default values for the file format, overwrite behavior and precision.
In the longer interface, the user must supply these values using the enums:

/// Supported file formats
enum FileFmt {
   FmtNetCDF3  = PIO_IOTYPE_NETCDF,   ///< NetCDF3 classic format
   FmtPnetCDF  = PIO_IOTYPE_PNETCDF,  ///< Parallel NetCDF3
   FmtNetCDF4c = PIO_IOTYPE_NETCDF4C, ///< NetCDF4 (HDF5-compatible) cmpressed
   FmtNetCDF4p = PIO_IOTYPE_NETCDF4P, ///< NetCDF4 (HDF5-compatible) parallel
   FmtNetCDF4  = PIO_IOTYPE_NETCDF4P, ///< NetCDF4 (HDF5-compatible) parallel
   FmtHDF5     = PIO_IOTYPE_HDF5,     ///< native HDF5 format
   FmtADIOS    = PIO_IOTYPE_ADIOS,    ///< ADIOS format
   FmtUnknown  = -1,                  ///< Unknown or undefined
   FmtDefault  = PIO_IOTYPE_NETCDF4C, ///< NetCDF4 is default
};

/// Behavior (for output files) when a file already exists
enum class IfExists {
   Fail,    /// Fail with an error
   Replace, /// Replace the file
   Append,  /// Append to the existing file
};

with the defaults for each being ``IO::FmtDefault``, and
``IO::IfExists::Fail``. The E3SM standard format is currently
NetCDF4. Earlier NetCDF formats should be avoided, but are provided in
case an input file is in an earlier format.

Once the file is open, data is read/written using one of two interfaces,
depending on whether the array is decomposed across MPI tasks or not. For
large decomposed arrays, the interface is:

Error Err = IO::readArray (&Array, Size, VariableName, FileID, DecompID, VarID);
IO::writeArray(&Array, Size, &FillValue,   FileID, DecompID, VarID);

where the pointer to the data array is passed and data is assumed to be
contiguous with the full local size of the array passed as Size. The FileID is
the integer ID for the open file. The DecompID is a defined data decomposition
as described below. For reading, the variable name (as a ``std::string``) is
supplied and the variable ID (VarID) is returned in case it is needed for
reading of variable metadata. For writing, a FillValue is supplied to fill
undefined locations in an array and the variable ID must have been assigned
in a prior defineVar call prior to the write as described below. The readArray
returns an error code so that the calling routine can re-try the read. This
is used in Omega to manage name changes between Omega and the earlier MPAS
names.

Writing or reading multiple time slices (where there in an unlimited time
dimension) is also possible and an additional optional Frame argument
specifies the time index along that dimension that should be read/written:

Error Err = IO::readArray (&Array, Size, VariableName, FileID, DecompID,
                           VarID, Frame);
IO::writeArray(&Array, Size, &FillValue,   FileID, DecompID, VarID, Frame);

For arrays or scalars that are not distributed, the non-distributed variable
interface must be used:

Error Err = IO::readNDVar(&Array, VariableName, FileID, VarID);
IO::writeNDVar(&Array, FileID, VarID);

with arguments similar to the distributed array calls above. Note that
when defining dimensions for these fields, the dimensions must be
non-distributed. For scalars, the number of dimensions should be zero.
Multiple time slices can be also be read/written for non-distributed fields,
but require two additional arguments. As in the distributed array, the
Frame (index of the time slice) must be provided. In addition, a vector
``std::vector<int> DimLengths`` containing the length of the non-time
dimensions must be provided:

Error Err = IO::readNDVar(&Array, VariableName, FileID, VarID, Frame, DimLengths);
IO::writeNDVar(&Array, FileID, VarID, Frame, DimLengths);

Note that the full arrays in this case are written so if any masking or
pruning of points is required, it should be performed before the call.

The IO subsystem must know how the data is laid out in the parallel
decomposition. Both the dimensions of the array and the decomposition
across tasks must be defined. For each dimension, a dimension must be
defined using:

where FileID is the ID of an open file, the DimName is a ``std::string``
with the dimension name (eg NCells, NEdges, NVertices, NVertLayers or
NTracers), length is the length of the full global array and DimID is
the ID assigned to this dimension. Note that for reading a file, we
supply the function:

   Error Err = IO::getDimFromFile(FileID, DimName, DimID, DimLength);

that can be used to inquire about the dimension length and retrieve the
dimension ID from the file.

Once the dimensions are defined, the decomposition of an array must
be defined using:

   int DecompID = createDecomp(IODataType, NDims, DimLengths,
                               Size, GlobalIndx, Rearr);

where DecompID is the ID assigned to the newly created decomposition,
NDims is the number of dimensions for the array, DimLengths is an
integer ``std::vector`` of the NDims local dimension lengths, Size is the
full size of the local array, and Rearr is the data rearranger method
to use to map to the number of IO tasks. The Rearr can be set to
``OMEGA::IO::DefaultRearr`` to make use of the overall default defined
when the IO system was initialized, but can also be set explicitly to
``OMEGA::IO::RearrBox`` or ``OMEGA::IO::RearrSubset``. The box rearranger
is generally preferred (see [UserGuide](#omega-user-IO)). The GlobalIndx
array describes the global location (as a zero-based offset) of each
local array entry. This can be computed from the Omega Default Decomp
arrays. For example, an array dimensioned (NCellsAll,NVertLayers) would
have an offset computed using:

   std::vector<int> OffsetCell(NCellsAll*NVertLayers,-1);
   int Add = 0;
   for (int Cell; Cell = 0; Cell < NCellsOwned){
      for (int k; k = 0; k < NVertLayers){
         OffsetCell[Add] = (CellIDH(Cell)-1)*NVertLayers + k;
         Add++;
      }
   }

Note that we exclude Halo layers by assigning an offset of -1. Finally,
the data type of the array must be supplied. To map the standard Omega
data types to the data types used in the IO subsystem, we define:

enum IODataType {
   IOTypeI4      = PIO_INT,    /// 32-bit integer
   IOTypeI8      = PIO_INT64,  /// 64-bit integer
   IOTypeR4      = PIO_REAL,   /// 32-bit real
   IOTypeR8      = PIO_DOUBLE, /// 64-bit real
   IOTypeChar    = PIO_CHAR,   /// Character/string
   IOTypeLogical = PIO_INT     /// Logicals are converted to ints for IO
};

so that in the above interface, we would supply for example ``IO::IOTypeI4``
for an Omega I4 data type.

Now that dimensions and decompositions have been defined, a variable can
be defined (this is required for writing only) using:

   int VarID = IO::defineVar(FileID, VarName, IODataType, NDims, DimIDs);

where VarID is the ID assigned to the variable, FileID is the usual ID of
the data file, VarName is a ``std::string`` holding the variable name,
IODataType is the data type described above, NDims are the number of dimensions,
and DimIDs are an integer ``std::vector`` holding the dimension IDs for each
dimension. For scalar variables, NDims should be set to zero and a null pointer
should be used in place of the DimID argument. Once defined, the variable ID
is used in all IO calls related to this variable.

In addition to data in a file, we can also read and write metadata. As with
the data itself, metadata is typically managed by the IOStreams and Metadata
interfaces, but the base IO module contains interfaces for reading and
writing metadata associated with either an array or the file or simulation
itself. To read/write metadata, use:

   IO::writeMeta(MetaName, MetaValue, FileID, VarID);
   Error Err = IO::readMeta (MetaName, MetaValue, FileID, VarID);

where MetaName is a ``std::string`` holding the name of the metadata and
the MetaValue is the value of the MetaData. All supported Omega data types are
allowed except boolean which must be converted to an integer type. The FileID
is once again the ID of the open data file and VarID is the variable to which
this metadata is attached. For global file and simulation metadata not attached
to a variable, the ID ``IO::GlobalID`` is used to denote global metadata. If
data is being appended to a file (eg for multiple time slices), writeMeta will
check first to see if metadata already exists with the same value. If the
entry is absent, new metadata will be written. If it exists but with a
different value, the writeMeta function will replace the current entry with
the new value.

For an example of the full read/write process, the IO unit test contains
the full reading and writing of a data file and associated metadata. We
will repeat again that users and developers are not expected to learn and
use the interfaces above, but should access the IO system from the IOStreams
interfaces.

Most input and output for Omega occurs through IOStreams. Each stream
defines a file, the contents to be read/written and the time frequency
for reading and writing. Defining streams via the input configuration
file is described in the [User Guide](#omega-user-iostreams). IOStreams
are built on top of the parallel IO infrastructure described in the
[IO Section](#omega-dev-IO) and the Field and Metadata described in the
[Field Section](#omega-dev-Field). Here we describe the classes and functions
used to implement IOStreams. Any module accessing an IOStream instance
or related functions must include the ``IOStream.h`` header file.

All IOStreams are initialized in a two-step process. A call to the
init routine should take place early in the Omega initialization after
the ModelClock has been initialized using:

This routine extracts all the stream definitions from the input configuration
file and creates all the Streams. This initialization also defines the
contents of each Stream but does not yet validate those contents against all
the defined Fields. The contents of all streams should be validated at the
end of initialization (when all Fields have been defined) using the call:

However, if a stream is needed (eg a read stream) during initialization
before the validateAll call, a single stream can be validated using

All streams must be validated before use to make sure the Fields have
been defined and the relevant data arrays have been attached to Fields and
are available to access.  At the end of a simulation, IOStreams must be
finalized using

so that any final writes can take place for the OnShutdown streams and to
deallocate all defined streams and arrays. If a stream needs to be removed
before that time, an erase function is provided:

For most output streams, we provide a writeAll interface that should be placed
at an appropriate time during the time step loop:

This function checks each write stream and writes the file if it is time, based
on a time manager alarm that is defined during initialization for each stream
based on the time frequency in the streams configuration. After writing the
file, the alarm is reset for the next write time. If a file must be written
outside of this routine, a single-stream write can take place using:

Reading files (eg for initialization, restart or forcing) does not often
take place all at once, so no readAll interface is provided. Instead, each
input stream is read using:

   Error Err = IOStream::read(StreamName, ModelClock, ReqMetadata);

The returned error code typically means that a field in the stream could
not be found in the input file - most other errors abort immediately. The
calling routine is then responsible for deciding what action to take.
The ReqMetadata argument is a variable of type Metadata (defined in Field but
essentially a ``std::map<std::string, std::any>`` for the name/value pair).
This variable should include the names of global metadata that are desired
from the input file. For example, if a time string is needed to verify the
input file corresponds to a desired time, the required metadata can be
initialized with

The Metadata corresponding to ForcingTime will then be read from the file
and inserted as the Metadata value. If no metadata is to be read from the
file, then an empty ReqMetadata variable can be passed.

As described in the [User Guide](#omega-user-iostreams), all streams are
defined in the input configuration file and most other IOStream functions
are associated either with that initialization or to support the read/write
functions above.

First, please follow the procedure in {ref}`omega-dev-conda-env` to set up a
conda environment for linting the code and building the docs.

Once on each machine you are developing on, before committing code, please run:

After this, your code will be linted with all the tools discussed below as part
of each call to `git commit`. Any modified files will also be linted when CI
(GitHub Actions) runs on each pull request and merge to the `develop` branch.

If you wish to commit without first checking the code for lint (e.g. you plan
to fix the code in a later commit), run:

with the same arguments you would normally use for `git commit`.

The tools used to lint C++ code are from
[cmake-pre-commit-hooks](https://github.com/Takishima/cmake-pre-commit-hooks)
and include [clang-format](https://clang.llvm.org/docs/ClangFormat.html),
[clang-tidy](https://clang.llvm.org/extra/clang-tidy/),
[cppcheck](https://cppcheck.sourceforge.io/), and
[include-what-you-use](https://github.com/include-what-you-use/include-what-you-use).
(Currently `clang-tidy`, `cppcheck` and `include-what-you-use` are disabled
but they will be enabled shortly.)

pre-commit run clang-format --all-files
pre-commit run clang-tidy --all-files
pre-commit run cppcheck --all-files
pre-commit run include-what-you-use --all-files

The tools used to lint python code include
[isort](https://pycqa.github.io/isort/) for sorting imports,
[flynt](https://github.com/ikamensh/flynt) for enforcing string formatting
with f-strings,
[flake8](https://flake8.pycqa.org/en/latest/) for enforcing the
[PEP8](https://peps.python.org/pep-0008/) style guide for python, and
[mypy](https://mypy-lang.org/) for performing variable type checking.

pre-commit run isort --all-files
pre-commit run flynt --all-files
pre-commit run flake8 --all-files
pre-commit run mypy --all-files

To update the linting packages, you just need to recreate the development
conda environment.  See {ref}`omega-dev-update-conda-env`.

Omega's logging system is built upon the
[spdlog](https://github.com/gabime/spdlog) logging tool.

Logging macros and custom formatters for Kokkos data types have been
integrated into the Omega logging system via the Logging.h header file
located in the src/infra directory. Users who wish to utilize Omega's
logging capabilities must include this header file in their code.

located in the src/infra directory. Users who wish to utilize Omega’s

The `src/infra/Logging.cpp` file encompasses various functions related
to logging. As of the current version, this file contains the
`OMEGA::initLogging` function, which initialize the logging process.
It is recommended to invoke the `OMEGA::initLogging` function at the beginning
of an Omega application to properly initialize the logging system.

The `OMEGA::initLogging` function, located within the `src/infra/Logging.cpp`
file, serves as a pivotal component for initializing the Omega logging system.

The function establishes the default logger configuration, ensuring that
logging messages are effectively saved to a designated file. The path to
this file is determined by utilizing the `OMEGA_LOG_FILEPATH` macro, which
allows users to specify the desired file location for logging purposes.

The Omega logging macros, denoted by the prefix `LOG_`, are defined within
the `src/infra/Logging.h` file. These macros constitute a main part of
the logging infrastructure, enabling users to seamlessly incorporate
logging functionality into their Omega applications.

Furthermore, the logging framework includes a distinct set of macros that
commence with the prefix `LOGGER_`. These macros offer enhanced
versatility by accommodating the utilization of the spdlog logger as
their first argument. This approach facilitats the integration of various
logger types.

Within the same header file, you will encounter specialized spdlog formatter
structs designed to accommodate Kokkos data types.

For further information on customizing the spdlog formatter, refer to
[Custom formatting](https://github.com/gabime/spdlog/wiki/3.-Custom-formatting).

The MachEnv class maintains a number of parameters associated with
the parallel machine environment. These include message-passing parameters
like MPI communicators and task information, the number of threads
if threaded, vector length for CPUs, and GPU and node information as
needed. Multiple environments can be defined to support running portions
of the model on subsets of tasks.

On model initiation, the initialization routine `MachEnv::init()`
is called to set up the default MachEnv, which can be retrieved at
any time using:

that returns a pointer to the default environment.
Once an environment has been retrieved, the individual data members
are retrieved using various get functions:

  MPI_Comm Comm   = DefEnv->getComm();
  int MyTask      = DefEnv->getMyTask();
  int NumTasks    = DefEnv->getNumTasks();
  int MasterTask  = DefEnv->getMasterTask();

There is also a logical function `isMasterTask` that can be used
for work that should only be done on the master task. By default,
the master task is defined as task 0 in the environment, but if
the master task is overloaded, there is a `setMasterTask(TaskID)`
that can redefine any other task in the group as the master.

If Omega has been built with OpenMP threading, a `getNumThreads`
function is available; it returns 1 if threading is not on.
The MachEnv also has a public parameter `VecLength` that can
be used to tune the vector length for CPU architectures. For
GPU builds, this VecLength is set to 1.

As noted previously, additional environments can be defined for
subsets of a parent environment. There are three constructor
interfaces for creating an environment:

  MachEnv MyNewEnv(Name, ParentEnv, NewSize);
  MachEnv MyNewEnv(Name, ParentEnv, NewSize, Begin, Stride);
  MachEnv MyNewEnv(Name, ParentEnv, NewSize, Tasks);

An optional additional argument exists for all three that can supply an
alternative task to use as the Master Task for message passing. If not
provided, the master task defaults to 0. The first interface above
creates a new environment from the first `NewSize` contiguous
tasks in the parent machine environment. The second creates a new
environment from a strided set of tasks starting at the `Begin` Task
(eg all odd tasks would have Begin=1 and Stride=2). The final
interface creates a subset containing the selected tasks stored in
a vector `Tasks`. Each new environment is given a `Name` and can be
retrieved by name using `MachEnv::get(Name)`. Once retrieved,
the other get functions listed above are used to get each data member.
Because the new environments are subsets of the default environment, one
additional function `MachEnv::isMember()` is provided so that
non-member tasks can be excluded from calculations. The retrieval functions
call from non-member tasks will return invalid values. Finally, there is
a `removeEnv(Name)` function that can delete any individual defined
environment by name and a `removeAll()` function that cleans up by
removing all defined environments.

As a class that is basically a container for the environment parameters,
the implementation is a simple class with several scalar data members and
the retrieval/creation functions noted above. To track all defined
environments, a c++ map container is used to pair a name with each
defined environment.

The Omega `OceanState` class
A state object is created by the `init` method, which assumes that `Decomp` and `HorzMesh` have

Err = OMEGA::Decomp::init();
Err = OMEGA::HorzMesh::init();
Err = OMEGA::OceanState::init();

OceanState::create(const std::string &Name, ///< [in] Name for mesh
                   HorzMesh *Mesh,          ///< [in] Horizontal mesh
                   Decomp *MeshDecomp,      ///< [in] Decomp for Mesh
                   Halo *MeshHalo_,         ///< [in] Halo for Mesh
                   const int NVertLayers_,  ///< [in] Number of vertical layers
                   const int NTimeLevels_   ///< [in] Number of time levels
);

allocates the `NormalVelocity` and `LayerThickness` arrays for a given number of time levels.
The current time level is then registered with the IO infrastructure.

After initialization, the default state object can be retrieved via:

The `OceanState` is meant to be a container that allows the non-tracer prognostic variables to be
passed to the PDE solver routines for tendency and auxiliary variable calculation. For example

 is meant to be a container that allows the non-tracer prognostic variables to be

void AuxiliaryState::compute(OceanState *State) const {
  ...
}

For now, member variables that are host arrays have variable names are appended with an
`H`.  Array variable names not ending in `H` are device arrays.  For a given time level,
host to device array is performed via:

Eventually, the host arrays will be eliminated when `IO` and `Halo` are extended to
handle host <-> device transfers.

A time level update to advance the solution at the end of a model timestep is done by:

This shifts the time level indices within the `State` instance. A halo exchange is
also performed on these arrays and the IOFields data pointer is attached
to the current time level.

The arrays associated with a given time level can be accessed with the functions:

Array2DReal LayerThick;
Err = State->getLayerThickness(LayerThick, TimeLevel);
Array2DReal NormVel;
Err = State->getNormalVelocity(NormVel, TimeLevel);

HostArray2DReal LayerThickH;
Err = State->getLayerThicknessH(LayerThickH, TimeLevel);
HostArray2DReal NormVelH;
Err = State->getNormalVelocityH(NormVelH, TimeLevel);

For a `State` with `NTimeLevels = 1`, only the current time level, `TimeLevel=0` is
availiable.

The state arrays are deallocated by the `OceanState::clear()` method, which is

Omega adopts the Kokkos programming model to express on-node parallelism. To provide
simplified syntax for the most frequently used computational patterns, Omega provides
wrappers funtions that internally handle creating and setting-up Kokkos policies.

To perform parallel iteration over a multi-dimensional index range Omega provides the
`parallelFor` wrapper. For example, the following code shows how to set every element of
a 3D array in parallel.

 wrapper. For example, the following code shows how to set every element of

   Array3DReal A("A", N1, N2, N3);
   parallelFor(
       {N1, N2, N3},
       KOKKOS_LAMBDA(int J1, int J2, int J3) {
          A(J1, J2, J3) = J1 + J2 + J3;
       });

Ranges with up to five dimensions are supported.
Optionally, a label can be provided as the first argument of `parallelFor`.

   parallelFor("Set A",
       });

Adding labels can result in more informative messages when
Kokkos debug variables are defined.

To perform parallel reductions over a multi-dimensional index range the
`parallelReduce` wrapper is available. The following code sums
every element of `A`.

   Real SumA;
   parallelReduce(
       KOKKOS_LAMBDA(int J1, int J2, int J3, Real &Accum) {
          Accum += A(J1, J2, J3);
       },
       SumA);

Note the presence of an accumulator variable `Accum` in the `KOKKOS_LAMBDA` arguments.
You can use `parallelReduce` to perform other types of reductions.
As an example, the following snippet finds the maximum of `A`.

   Real MaxA;
          Accum = Kokkos::max(Accum, A(J1, J2, J3));
       },
       Kokkos::Max<Real>(MaxA));

To perform reductions that are not sums, in addition to modifying the lambda body,
the final reduction variable needs to be cast to the appropriate type. In the above example,
`MaxA` is cast to `Kokkos::Max<Real>` to perform a max reduction.
The `parallelReduce` wrapper supports performing multiple reductions at the same time.
You can compute `SumA` and `MaxA` in one pass over the data:

 wrapper supports performing multiple reductions at the same time.

       KOKKOS_LAMBDA(int J1, int J2, int J3, Real &AccumSum, Real &AccumMax) {
          AccumSum += A(J1, J2, J3);
          AccumMax = Kokkos::max(AccumMax, A(J1, J2, J3));
       },
       SumA, Kokkos::Max<Real>(MaxA));

There is no limit to how many reductions can be done at the same time.
It is usually beneficial for performance to group a small number (2-8) of simple
reductions together.
Similarly to `parallelFor`, `parallelReduce` supports labels and up to five dimensions.

A more flexible alternative to flat parallelism is hierarchical parallelism.
In general, Kokkos supports a three-level parallelism hierarchy of

In Omega, we simplify this design to a two-level hierarchy (outer and inner) that naturally
corresponds to the  model parallelism in the horizontal and vertical dimensions.
The outer level of parallelism in Omega corresponds to parallelism over Kokkos teams,
whereas the inner level may be a combination of parallelism over team threads and thread vector lanes.
The simplest example of a hierarchical parallel loop using Omega wrappers looks as follows

   parallelForOuter(
       {NCells},
       KOKKOS_LAMBDA(int ICell, const TeamMember &Team) {
           parallelForInner(Team, NVertLayers, INNER_LAMBDA (int KLayer) {
               // Inner body
           });
   });

Note the appearance of a variable named `Team` of type `TeamMember` in the outer lambda
argument list. This variable handles launching of inner loops and team synchronization.
The flexibility of hierarchical parallelism comes from:

the possibility of the inner loop range to depend on the outer index

the option to use different patterns at the outer and inner levels

To provide even more flexibility, the outer loops support iterating over a multi-dimensional range.
Currently, the inner loops are limited to one dimension.

When using hierarchical parallelism the outer loop lambda has to be annotated with `KOKKOS_LAMBDA` to
be device accessible. In the inner lambda it must be possible to obtain correct results using capture by value (`[=]`).
However, in some cases, better performance might be possible by using
capture by reference (`[&]`). For that reason, Omega provides the
`INNER_LAMBDA` macro that expends to the best performing capture clause depending on the chosen
architecture.

When using hierarchical parallelism the outer loop lambda has to be annotated with 

be device accessible. In the inner lambda it must be possible to obtain correct results using capture by value (

 macro that expends to the best performing capture clause depending on the chosen

When multiple parallel inner loops are present within one outer loop, synchronization within
a thread team might be necessary. Omega provides the `teamBarrier` function that
synchronizes the threads in a team, which can be called as follows.

This function may only be called at the outer level and needs to be called by every thread in a team.
Failure to follow these rules might result in deadlocks.

When using hierarchical parallelism, it might be necessary to restrict execution to just one thread of a team.
To do that Kokkos provides the `single` function. To execute a statement once per team with a single thread do

 function. To execute a statement once per team with a single thread do

  Kokkos::single(PerTeam(Team), [=](){
      // executed once per team (by one of the team threads)
  });

To start outer iterations over a multidimensional index range the `parallelForOuter` wrapper is available.
A call to `parallelForOuter` might look as follows.

To start outer iterations over a multidimensional index range the 

       {N1},
       KOKKOS_LAMBDA(int J1, const TeamMember &Team) {
           // launch inner loops here
       });

Labels and multi-dimensional iteration ranges of up to five dimensions are supported by `parallelForOuter`.

Labels and multi-dimensional iteration ranges of up to five dimensions are supported by 

To start outer reductions over a multi-dimensional index range the `parallelReduceOuter` wrapper is available.
A call to `parallelReduceOuter` might look as follows.

To start outer reductions over a multi-dimensional index range the 

   Real OuterSum;
   parallelReduceOuter(
       {N1},
       KOKKOS_LAMBDA(int J1, const TeamMember &Team, Real &Accum) {
           Real InnerContribution;
           // compute InnerContribution
           Kokkos::single(PerTeam(Team), [&](){
               Accum += InnerContribution;
           });
       }, OuterSum);

Note how in this example the addition of `InnerContribution` is done inside `Kokkos::single`
with just one thread from a team. This shows how to add one contribution per team to the total sum.
The reduction accumulator comes after the team member variable in the lambda argument list.
Labels and multi-dimensional iteration ranges of up to five dimensions are supported in `parallelReduceOuter`.
Different types of reductions (min, max) and multiple reducers are also supported.

Labels and multi-dimensional iteration ranges of up to five dimensions are supported in 

To launch inner parallel iterations Omega provides the
`parallelForInner` wrapper. For example, the following code shows how to set every element of
a 3D array in parallel using hierarchical parallelism.

       {N1, N2}, KOKKOS_LAMBDA(int J1, int J2, const TeamMember &Team) {
        parallelForInner(Team, N3, INNER_LAMBDA(Int J3) {
        });
    });

Labels are not supported by `parallelForInner` and only one-dimensional index range can be used.
The inner loop range can depend on the outer loop index. For example, to set the elements below the main
diagonal of a square matrix one can do:

   Array2DReal M("M", N, N);
       {N}, KOKKOS_LAMBDA(int J1, const TeamMember &Team) {
        parallelForInner(Team, J1, INNER_LAMBDA(Int J2) {
          M(J1, J2) = J1 + J2;
        });
    });

To launch inner parallel reductions Omega provides the `parallelReduceInner` wrapper.
For example, computing sums along the third dimension of a 3D array in parallel and storing them
in a 2D array might be done as follows.

   Array2DReal B("B", N1, N2);
        Real SumD3;
        parallelReduceInner(Team, N3, INNER_LAMBDA(Int J3, Real &Accum) {
            Accum += A(J1, J2, J3);
        }, SumD3);
        B(J1, J2) = SumD3;
    });

Labels are not supported by `parallelReduceInner` and only one-dimensional index range can be used.
Different types of reductions (min, max) and multiple reducers are supported.
For example, to additionally compute and store maxima along the third dimension of A do

   Array2DReal C("C", N1, N2);
        Real SumD3, MaxD3;
        parallelReduceInner(Team, N3, INNER_LAMBDA(Int J3, Real &AccumSum, Real &AccumMax) {
            AccumSum += A(J1, J2, J3);
            AccumMax = Kokkos::Max(AccumMax, A(J1, J2, J3));
        }, SumN3, MaxN3);
        C(J1, J2) = MaxD3;
    });

Inner reductions are collective operations within a team. This means that the reduction results
are available to every thread of a team, without any need for synchronization

To launch inner parallel scans Omega provides the `parallelScanInner` wrapper.
For example, computing prefix sums (partial sums) along the third dimension of a 3D array might
be done as follows.

   Array3DReal D("D", N1, N2, N3);
       parallelScanInner(Team, N1, INNER_LAMBDA(Int J3, Real &Accum, bool IsFinal) {
            if (IsFinal) {
              D(J1, J2, J3) = Accum;
            }
        });
       });

This example computes partial sums up to and including `A(J1, J2, J3)` because the accumulator is updated
before the `if` statement. That is, it performs an inclusive scan. To compute an exclusive scan
simply move the addition after the `if` statement.

 statement. That is, it performs an inclusive scan. To compute an exclusive scan

  Real FinalScanValue;
  parallelScanInner(Team, N1, INNER_LAMBDA(Int J3, Real &Accum, bool IsFinal) {
       if (IsFinal) {
         D(J1, J2, J3) = Accum;
       }
       Accum += A(J1, J2, J3);
  }, FinalScanValue);

Moreover, this example illustrates that the final scan value can be obtained by providing
an additional argument `FinalScanValue`. Labels are not supported by `parallelScanInner`
and only one-dimensional index range can be used. In contrast to `parallelReduceInner`,
`parallelScanInner` supports only sum-based scans and only one scan variable.

and only one-dimensional index range can be used. In contrast to 

We ask developers to make their own fork of the E3SM to use for development
branches.  To do this, go to the page for the
[E3SM-Project/E3SM](https://github.com/E3SM-Project/E3SM) repository and
click on the `fork` button near the upper right corner.  Set "owner" to your
GitHub username (this should be the default) and click "Create fork".

GitHub username (this should be the default) and click “Create fork”.

There is no need to have a separate fork for Omega, since the Omega repository
is also a fork of E3SM.  Your E3SM fork will serve for both E3SM and Omega
development.  In fact, GitHub will not let you make an Omega fork if you
already have an E3SM fork (or vice versa).

Make sure you have added the SSH key for the machine you are using to your
GitHub account, see
[Adding a new SSH key to your GitHub account](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account).

You may wish to rename `origin` to `E3SM-Project/Omega` for clarity:

git remote add E3SM-Project/E3SM git@github.com:E3SM-Project/E3SM.git
git remote add <github_username>/E3SM git@github.com:<github_username>/E3SM.git

where `<github_username>` is your username or that of a collaborator.

If you do not have conda set up in your own space yet, follow
{ref}`omega-dev-install-miniforge3` for instructions on how to install it.

Activate the `base` conda environment.  Go to the base of the Omega branch you
plan to develop and create a conda environment for
linting your code and building the documentation:

(You can reuse `omega_dev` for other branches as long as `dev-conda.txt` has
not changed between branches.)

The first time you set up the environment, you need to initiate `pre-commit`:

The first time you set up the environment, you need to initiate 

This needs to be done once per environment, so re-run it as necessary if you
re-create the development environment.

Please activate this environment each time you commit code.  This will ensure
that `pre-commit` and the associated linting utilities are available and that
they check your code as it is committed (rather than requiring fix-up commits
later on).

If you are using Polaris, you may wish to use its
[Omega CTest utility](https://github.com/E3SM-Project/polaris/tree/main/utils/omega/ctest)
to build and test Omega. The utility automates many of the steps below.

In the Omega branch you would like to build, first update the submodules that
Omega requires:

git submodule update --init --recursive \
    externals/YAKL \
    externals/ekat \
    externals/scorpio \
    externals/cpptrace \
    components/omega/external \
    cime

Since some systems require tests to be run in a scratch space, it is a good
idea to build the code somewhere in your scratch space.  We will simply refer
to the build directory as `$BUILD_DIR` and leave it up to you to decide where
it is best to put it.  If you have previously built in `$BUILD_DIR`, you
very likely need to remove it first and start fresh:

Set `$PARMETIS_ROOT` to the appropriate location for Metis and Parmetis
libraries built for your machine and compiler (see
{ref}`omega-dev-parmetis-libs` below for shared locations on supported
machines):

cmake \
   -DOMEGA_BUILD_TYPE=<build_type> \
   -DOMEGA_CIME_COMPILER=<compiler> \
   -DOMEGA_CIME_MACHINE=<machine> \
   -DOMEGA_PARMETIS_ROOT=${PARMETIS_ROOT} \
   -DOMEGA_BUILD_TEST=ON \
   -Wno-dev \
   -S <omega_branch>/components/omega \
   -B .

where `<build_type>` is either `Debug` or `Release`, `<omega_branch>` is the
path to the base of the Omega branch you want to build, and where `<machine>`
and `<compiler>` are the current machine and a compiler supported by E3SM on
that machine (see
[config_machines.xml](https://github.com/E3SM-Project/Omega/blob/develop/cime_config/machines/config_machines.xml)).

path to the base of the Omega branch you want to build, and where 

The command above will configure Omega to build CTests
(`-DOMEGA_BUILD_TEST=ON`), which is recommended.

If CMake configuration runs correctly, you should have an `omega_build.sh`
script that you can run to build Omega:

Some tests require a valid Omega mesh file. Different tests require different
meshes. At the moment, mesh files need to be copied or linked to specifically
named files under the `test` directory. Appropriate mesh files can be
downloaded from:

wget -O ocean_test_mesh.nc https://web.lcrc.anl.gov/public/e3sm/inputdata/ocn/mpas-o/oQU240/ocean.QU.240km.151209.nc
wget -O global_test_mesh.nc https://web.lcrc.anl.gov/public/e3sm/polaris/ocean/polaris_cache/global_convergence/icos/cosine_bell/Icos480/init/initial_state.230220.nc
wget -O planar_test_mesh.nc https://gist.github.com/mwarusz/f8caf260398dbe140d2102ec46a41268/raw/e3c29afbadc835797604369114321d93fd69886d/PlanarPeriodic48x48.nc
cd test
ln -sf  ../ocean_test_mesh.nc OmegaMesh.nc
ln -sf  ../global_test_mesh.nc OmegaSphereMesh.nc
ln -sf  ../planar_test_mesh.nc OmegaPlanarMesh.nc
cd ..

Omega includes several unit tests that run through CTest. The unit tests need
to be run on a compute node.

Test project /gpfs/fs1/home/ac.xylar/e3sm_work/polaris/add-omega-ctest-util/build_omega/build_chrysalis_intel
    Start 1: DATA_TYPES_TEST
1/9 Test #1: DATA_TYPES_TEST ..................   Passed    0.38 sec
    Start 2: MACHINE_ENV_TEST
2/9 Test #2: MACHINE_ENV_TEST .................   Passed    0.98 sec
    Start 3: BROADCAST_TEST
3/9 Test #3: BROADCAST_TEST ...................   Passed    1.13 sec
    Start 4: LOGGING_TEST
4/9 Test #4: LOGGING_TEST .....................   Passed    0.03 sec
    Start 5: DECOMP_TEST
5/9 Test #5: DECOMP_TEST ......................   Passed    1.20 sec
    Start 6: HALO_TEST
6/9 Test #6: HALO_TEST ........................   Passed    1.08 sec
    Start 7: IO_TEST
7/9 Test #7: IO_TEST ..........................   Passed    2.94 sec
    Start 8: CONFIG_TEST
8/9 Test #8: CONFIG_TEST ......................   Passed    1.01 sec
    Start 9: KOKKOS_TEST
9/9 Test #9: KOKKOS_TEST ........................   Passed    0.03 sec

100% tests passed, 0 tests failed out of 9

Total Test time (real) =   8.91 sec

If Omega CTests are failing or simulations are crashing, setting
`OMEGA_BUILD_TYPE` to `Debug` can be helpful for debugging purposes. If you
need to identify which test has failed, it may be useful to examine the CTest
log file located at `$BUILD_DIR/Testing/Temporary/LastTest.log`.

The following table shows locations for Metis and Parmetis libraries on
supported E3SM machines. The pattern is:

We require that the C++ code in Omega adhere to the
[LLVM code style](https://llvm.org/docs/CodingStandards.html). These
conventions are enforced using the linting tools described in
{ref}`omega-dev-linting`.  This style may take new developers some time to get
used to but the hope is that it leads to a coherent code style in Omega.

You may wish to consider using an integrated development environment (IDE) to
develop your code.  A convenient option for developing on HPC is
[Visual Studio Code (VS Code)](https://code.visualstudio.com/).  It has plugins
for [C++](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools),
[CMake](https://marketplace.visualstudio.com/items?itemName=ms-vscode.cmake-tools),
[Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python),
and so on.  If configured correctly, it should help to enforce the
[code formatting style](https://code.visualstudio.com/docs/cpp/cpp-ide#_code-formatting)
by recognizing Omega's `.clang-format` file.

A convenient feature is the ability to connect and edit code directly on HPC
systems from your laptop or desktop
[using an SSH connection](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh).

VS Code also provides a convenient way to preview the Markdown files used in
the Omega documentation as you are writing them.

Hyperlink target "omega-dev-quick-start-build-test" is not referenced.

Hyperlink target "omega-dev-quick-start-ctest-util" is not referenced.

Hyperlink target "omega-dev-quick-start-build" is not referenced.

Hyperlink target "omega-dev-quick-start-getting-meshes" is not referenced.

Global reductions are global collective operations to reduce distributed
data to a scalar value using either sum, min or max operators.

For scalars, `globalSum` computes a reproducible sum of all scalars
on MPI processors in a given MPI communicator:

where `TT` is the data type I4, I8, R4 or R8. Computed sum is stored
in `Result`. Any errors from the MPI collective call are passed back
in the function `int` return code.

For arrays, `globalSum` first computes a local sum of the array before
the global sum. The local sum is either between 0 and max indexes of
the array or between indexes defined in the specified `indexRange`:

int globalSum(const ArrayTTDD array,
              const MPI_Comm Comm,
              TT *Result,
              const std::vector<I4> *indexRange = nullptr)

`ArrayTTDD` is either a host or device Omega array of type I4, I8, R4 or R8
and dimension DD ranging from 1D to 5D. The `indexRange` vector is of length
2x the number of array dimensions: e.g. for a 2D array the min and
max indexes for a local sum might be
`indexRange{0, nCellsOwned-1, 0, nVertLayers-1}`. The index range vector
is an optional parameter and, when absent, local sum defaults to
min,max indexes of the array. Computed global sum is stored

 is either a host or device Omega array of type I4, I8, R4 or R8

int globalSum(const ArrayTTDD array1,
              const ArrayTTDD array2,

performs a sum of element-by-element product `array1*array2`. The second
array must be as big as the first array; if `indexRange` is provided, the
indexes must be valid for both arrays. This signature can be used to
mask the local sum of `array1` with values of masking `array2`.

To sum multiple fields at once, this function signature accepts a vector
of scalars:

int globalSum(const std::vector<TT> scalars,
              std::vector<TT> *Result)

int globalSum(const std::vector<ArrayTTDD> arrays,
              std::vector<TT> *Result,

int globalSum(const std::vector<ArrayTTDD> arrays1,
              const std::vector<ArrayTTDD> arrays2,
              std::vector<TT> *Result,,

Functions `globalMinVal` and `globalMaxVal` provide interfaces similar
to `globalSum` but reduce data with MIN and MAX operators:

int globalMinVal(const ArrayTTDD array,
                 const MPI_Comm Comm,
                 TT *Result,
                 const std::vector<I4> *indexRange = nullptr)

int globalMinVal(const ArrayTTDD array1,
                 const ArrayTTDD array2,

int globalMinVal(const std::vector<ArrayTTDD> arrays,
                 std::vector<TT> *Result,

These functions are utility functions to compute global MIN and MAX
across all MPI processors for a given scalar or array:

int globalMin(const TT *scalar,
              const MPI_Comm Comm)

int globalMin(const ArrayTTDD array,
              ArrayTTDD Result,

The `Tendencies` class groups together all of the OMEGA [tendency terms](#omega-dev-tend-terms).
It is responsible for managing their lifetime and providing functions that compute the
tendency terms and the [auxiliary state](#omega-dev-aux-state).
It is possible to have multiple `Tendencies` instances in OMEGA. Every instance has a name
and is tracked in a static C++ map called `AllTendencies`.

An instance of the `Tendencies` class requires a [`HorzMesh`](#omega-dev-horz-mesh), so
the mesh class and all of its dependencies need to be initialized before the `Tendencies` class

initializes the default `Tendencies`. A pointer to it can be retrieved at any time using:

OMEGA::Tendencies* DefTendencies = OMEGA::Tendencies::getDefault();

The constructor of the `Tendencies` class initializes member instances of each of the
tendency terms, which each store constant mesh information as private member variables.

A non-default tendency group can be created with or without custom tendencies.
Without custom tendencies, it is created from a string `Name`, horizontal mesh `Mesh`, vertical coordinate `VCoord`, number of tracers `NTracers`, and a configuration `Options`:

OMEGA::Tendencies*  NewTendencies = OMEGA::Tendencies::create(Name, Mesh, VCoord, NTracers, Options);

For convenience, this returns a pointer to the newly created instance.
To allow the user to provide custom tendencies, the `create` function can take two additional arguments
`CustomThicknessTend` and `CustomVelocityTend`

OMEGA::Tendencies*  NewTendencies = OMEGA::Tendencies::create(Name, Mesh, VCoord, NTracers, Options, CustomThicknessTend, CustomVelocityTend);

The two custom tendency arguments need to be callable objects that take a Kokkos array `Tend`, ocean state `State`,
auxiliary state `AuxState`, two integers: `ThickTimeLevel` and `VelTimeLevel`, and time instant `Time`.

The two custom tendency arguments need to be callable objects that take a Kokkos array 

Given its name, a pointer to a named tendency group

OMEGA::Tendencies* NewTendencies = OMEGA::Tendencies::get(Name);

To compute all tendencies for layer thickness, normal velocity, and tracer equations,
given ocean state, `State`, a group of auxiliary variables, `AuxState`, an array of tracers,
`TracerArray`, thickness time level `ThickTimeLevel`, velocity time level `VelTimeLevel`,
and time instant `Time` do:

Tendencies.computeAllTendencies(State, AuxState, TracerArray, ThickTimeLevel, VelTimeLevel, Time);

Tendencies.computeThicknessTendencies(State, AuxState, ThickTimeLevel, VelTimeLevel, Time);

Tendencies.computeVelocityTendencies(State, AuxState, ThickTimeLevel, VelTimeLevel, Time);

To call only the tracer tendency terms:
Tendencies.computeTracerTendencies(State, AuxState, TracerArray, ThickTimeLevel, VelTimeLevel);

The tendency terms are each implemented as a separate functor. They contain
private member variables to store any constant data needed for computation.
This data is initialized during construction and comes from either an instance
of the `HorzMesh` class or a `Config` object, which are passed as arguments to
the constructor:

   auto Mesh = OMEGA::HorzMesh::getDefault();
   auto Options = OMEGA::Config::getOmegaConfig();
   OMEGA::ThicknessFluxDivOnCell ThickFluxDivOnC(Mesh, Options));

Each functor is called like a function to compute tendency values for a range
of vertical layers at a given mesh element. So, the functors all take as
input the mesh element index and the index for the vertical chunk, along with
arrays computed by [Auxillary Variables](#omega-dev-aux-vars) classes. In each
case, the first argument is a tendency array that gets updated by the
contributon to the tendency for the given term. The functors are designed to be
used inside a Kokkos parallel loop:

   OMEGA::parallelFor(
       {mesh->NCellsOwned, NVertLayers}, KOKKOS_LAMBDA(int ICell, int KChunk) {
          ThickFluxDixOnC(ThicknessTend, ICell, KChunk, ThickFluxEdge);
       });

The following tendency terms for the shallow water equations are currently
implemented:

For instructions on how to build Omega and run CTest unit tests, see
{ref}`omega-dev-quick-start-build-test`.

The CTest unit tests on their own are not sufficient to ensure that changes
to Omega behave as expected.  Developers also need to run the regression tests
in the form of the
[omega_pr](https://docs.e3sm.org/polaris/main/users_guide/ocean/suites.html#omega-pr-suite)
test suite from Polaris.  The tests in the suite are defined in
[omega_pr.txt](https://github.com/E3SM-Project/polaris/blob/main/polaris/suites/ocean/omega_pr.txt)
in the Polaris repository.

You will run the suite twice: once using the Omega version vendored in Polaris
under `e3sm_submodules/Omega` (the official baseline) and once using your
development branch. This means you will need to
{ref}`build Omega <omega-dev-quick-start-build>` from both sources before
running the suite. You may wish to use the {ref}`omega-dev-quick-start-ctest-util`
to automate the build and run the CTests for each.

While the Omega `develop` branch often matches the Polaris submodule
bit-for-bit (b4b), the submodule at `e3sm_submodules/Omega` is the
authoritative baseline for regression comparison.

Before setting up and running the `omega_pr` suite, you must build Omega twice:

Build the baseline from the Omega sources vendored by Polaris at
`<path-to-polaris-repo>/e3sm_submodules/Omega`. Record the CMake build
directory path as `BASELINE_BUILD` (used below with `polaris suite -p`).

Build from your development (PR) branch. Record the CMake build directory
path as `PR_BUILD` (also used with `-p`).

The `-p` option passed to `polaris suite` should point to the CMake build
directory for the corresponding branch (the directory where you ran CMake and
invoked `./omega_build.sh`).

Run both baseline and PR suite on the same machine, CPU/GPU partition,
compiler, and Omega build type (Debug/Release) to avoid false positive/negative
diffs.

If you have not yet set up Polaris, you will need to follow its
[quick start](https://docs.e3sm.org/polaris/main/developers_guide/quick_start.html)
including [cloning the repo](https://docs.e3sm.org/polaris/main/developers_guide/quick_start.html#set-up-a-polaris-repository-for-beginners),
[configuring the software environment](https://docs.e3sm.org/polaris/main/developers_guide/quick_start.html#supported-machines),
and [activating the environment](https://docs.e3sm.org/polaris/main/developers_guide/quick_start.html#activating-the-environment).

Then, with the Polaris environment active, you can set up the baseline
`omega_pr` suite as follows:

BASELINE_BUILD=/path/to/build/from/polaris/e3sm_submodules/Omega
POLARIS_SCRATCH=/path/to/scratch/polaris_testing/
polaris suite -c ocean -t omega_pr --model omega \
  -w $POLARIS_SCRATCH/baseline_omega_pr \
  -p $BASELINE_BUILD

where `$POLARIS_SCRATCH` is a directory within your scratch space that is
useful for setting up and running the Polaris test suites and
`$BASELINE_BUILD` is the directory where you built Omega from the Omega
sources located at `<polaris-repo>/e3sm_submodules/Omega`, or where the
Polaris CTest utility built it for you (e.g. `build_omega/build_chrysalis_intel`).

Next, you can set up the `omega_pr` for your development branch, using the
baseline set up above:

PR_BUILD=/path/to/omega/pr/branch/build
    -b $POLARIS_SCRATCH/baseline_omega_pr \
    -w $POLARIS_SCRATCH/my_branch_omega_pr \
  -p $PR_BUILD

Here, `$POLARIS_SCRATCH` is the same scratch directory as above,
`$PR_BUILD` is the directory where you build Omega from your
development branch (or where the CTest utility built for you), and
`my_branch_omega_pr` can be any name that helps you keep track of what you
were testing.

cd $POLARIS_SCRATCH/baseline_omega_pr
sbatch job_script.omega_pr.sh

Note the job ID for the next step.  Next, run the suite for the test branch,
waiting until the baseline is done:

cd $POLARIS_SCRATCH/my_branch_omega_pr
sbatch --kill-on-invalid-dep=yes --dependency=afterok:$JOBID job_script.omega_pr.sh

Monitor the jobs (e.g. `squeue -j $JOBID`) and take a look at combined
stdout/stderr log in `polaris_omega_pr.o$JOBID` in each work directory to see
whether the tests passed.  Report the results as part of any (non-trivial)
Omega PR.

For other schedulers (e.g. PBS on Aurora), take a look at the machine's
documentation for equivalent commands.

For other schedulers (e.g. PBS on Aurora), take a look at the machine’s

Test execution errors for baseline (Polaris `e3sm_submodules/Omega`)

If you see test execution failures in the baseline run from Polaris
`e3sm_submodules/Omega`,
these need to be reported as [issues](https://github.com/E3SM-Project/Omega/issues)
on the Omega repo, preferably mentioning a relevant Omega developer who may
be able to investigate the cause of the failure.  Please provide the contents
of the relevant log file for the failing test(s) and optionally the location of
the tests (and log files) if other Omega developers will have access.

test execution errors for baseline (polaris e3sm_submodules/omega)

If you see execution failures on the tests for your development branch, please
take a look at the log files and attempt to debug the issues yourself first,
then reach out to the Omega team if you need help.  In many cases, the test
failures may related to changes that are required in Polaris, see
{ref}`omega-dev-testing-update-polaris` below.

Update `e3sm_submodules/Omega` in Polaris (after merge of Omega PR)

If you see errors related to comparisons with the baseline, you will need to
determine if these are expected based on the code changes or not.  If they
are expected, you will need to document (in a PR comment) how large the
differences are in the tests that show differences.  In addition, you will need
to make a pull request to the Polaris repository to update the Omega submodule
in `e3sm_submodules/Omega` once your branch has been merged so that the changes
become the new baseline in Polaris.

We do not update `e3sm_submodules/Omega` after every branch is merged into
`develop` on Omega to reduce maintenance burden.

If you see unexpected differences between the results for your development
branch and those from the baseline, you should attempt to debug the cause yourself
and then reach out to the Omega team if you need help.

You may want to look at Polaris'
[bisect](https://github.com/E3SM-Project/polaris/blob/main/utils/bisect/README.md)
utility to help debug which commit introduced the unexpected differences.

Please add a PR comment titled "Testing" that summarizes what you ran and the
outcomes. Include the following so reviewers can validate apples-to-apples
runs.

Please add a PR comment titled “Testing” that summarizes what you ran and the

At a minimum, you need to run both CTests and the `omega_pr` suite on one
machine and compiler.

Result: either "All tests passed" or a short list of failing tests with
a one-line note each

Result: either “All tests passed” or a short list of failing tests with

Machine/partition, compiler, and build type (should match between baseline and PR)

Result: "All tests passed" or a concise list of diffs/failures with a brief note

Result: “All tests passed” or a concise list of diffs/failures with a brief note

If there are issues, add the path to the relevant log(s) (e.g.,
polaris_omega_pr.o$JOBID) and a short excerpt if helpful

Briefly list any new or updated CTest or Polaris tests and what they cover

Link to the relevant PACE experiment and summarize before/after metrics

The [Omega CTest Utility](https://github.com/E3SM-Project/polaris/blob/main/utils/omega/ctest/README.md)
and the `omega_pr` suite will produce output in a format you can copy/paste into the PR
description. These outputs follow roughly the template below.  If you want to add the
information manually, you can copy-paste template you can use in your PR comment:

 suite will produce output in a format you can copy/paste into the PR

## Testing

CTest unit tests
- Machine: <machine>[, <partition>]
- Compiler: <compiler>
- Build type: <Debug|Release>
- Result: All tests passed | Failures (X of Y): <name> (<one-line note>), ...

Polaris omega_pr regression suite
- Baseline build (-p): <path/to/build of Polaris e3sm_submodules/Omega>
- Baseline workdir (-w): <path/to/baseline_omega_pr>
- PR build (-p): <path/to/pr/build>
- PR workdir (-w): <path/to/my_branch_omega_pr>
- Machine/partition: <machine>[, <partition>]
- Compiler/build type: <compiler>, <Debug|Release>
- Result: All tests passed | Diffs/Failures: <brief summary>
- Logs (if applicable): <path/to/polaris_omega_pr.o$JOBID>

Tests added/modified/impacted
- CTest: <name> — <one-line purpose>
- Polaris: <path or name> — <one-line purpose>

Performance (if applicable)
- PACE: <link>
- Before: <metric(s)>
- After: <metric(s)>
- Delta: <summary>

Many Omega changes are likely to also require Polaris changes.  In such cases,
it makes sense to co-develop your Omega branch and a Polaris branch.  The
Omega branch will need to be merged first, then the `e3sm_submodules/Omega`
submodule will need to be updated to include the Omega changes as part of
the Polaris pull request.

Here are some examples of Omega changes that will require corresponding
Polaris changes.

If you add new Omega variables that have a corresponding MPAS-Ocean name,
you need to add the mapping between the names to
[mpaso_to_omega.yaml](https://github.com/E3SM-Project/polaris/blob/main/polaris/ocean/model/mpaso_to_omega.yaml).
For more details, see the Polaris
[Ocean Framework documentation](https://docs.e3sm.org/polaris/main/developers_guide/ocean/framework.html).

dimensions:
  Time: time
  nVertLevels: NVertLayers

variables:
  temperature: Temperature
  salinity: Salinity
  ssh: SshCell

In each case, the keys are the MPAS-Ocean names and the values are the
corresponding Omega names.

If you add or remove sections, config options, streams, etc. in
[Default.yml](https://github.com/E3SM-Project/Omega/blob/develop/components/omega/configs/Default.yml),
you may need to make corresponding changes to the YAML files for Polaris
test cases for them to continue to work as expected.

Whenever possible, we try to use a mapping between MPAS-Ocean namelist options
and Omega config options within
For example:

config:
- section:
    time_management: TimeIntegration
  options:
    config_start_time: StartTime
    config_stop_time: StopTime
    config_run_duration: RunDuration
    config_calendar_type: CalendarType

If you have added new Omega config sections or options that have corresponding
MPAS-Ocean namelist sections or options, please add them to the map. In each
case, the keys are the MPAS-Ocean names and the values are the corresponding
Omega names.

Frequently, there is no MPAS-Ocean equivalent to an Omega config option, and
we currently do not try to map streams from MPAS-Ocean to Omega (their
syntax is too different).  In such cases, we have do define Omega YAML config
options and streams in each test case, and you may need to update each one
with your changes.  Here is an example exerpt from the
[manufactured_solution](https://github.com/E3SM-Project/polaris/blob/main/polaris/tasks/ocean/manufactured_solution/forward.yaml)
test:

Omega:
  Tendencies:
    VelDiffTendencyEnable: false
    VelHyperDiffTendencyEnable: false
    UseCustomTendency: true
    ManufacturedSolutionTendency: true
  IOStreams:
    InitialVertCoord:
      Filename: init.nc
    InitialState:
      UsePointerFile: false
      Contents:
      - NormalVelocity
      - LayerThickness
    RestartRead: {}
    History:
      Filename: output.nc
      Freq: {{ output_freq }}
      FreqUnits: Seconds
      IfExists: append
      # effectively never
      FileFreq: 9999
      FileFreqUnits: years
      Contents:
      - SshCell

For more details on updating either the map or the YAML files for individual
tests, see the Polaris

Hyperlink target "omega-dev-testing-update-polaris" is not referenced.

The Omega Time Manager module tracks simulation time during a run and manages
alarms for triggering events at specified times This is handled by six
interacting defined classes, each of which have a number of methods defined for
accessing, comparing, and manipulating the class members.

The TimeFrac class is the fundamental base time representation. To avoid
potential rounding errors that can accumulate over millions of time steps, time
is tracked in seconds as a fractional representation using three 8-byte integers
representing the whole number, numerator, and denominator. Most of the other
classes in the Time Manager are based on the TimeFrac class and generally
convert time representations from other units and store them as a TimeFrac
object.

The Calendar class is mostly an immutable class that stores all information for
supported calendars that can be utilized in a simulation. The supported
calendars are: Gregorian, No Leap, Julian, Julian Day, Modified Julian Day,
360 Day, Custom, and No Calendar. These supported calendars are specified via an
enum defined in the Time Manager header file:

   CalendarGregorian = 1, ///< usual Gregorian calendar
   CalendarUnknown        ///< uninitialized or invalid
};

During a simulation, only one calendar can be defined and is set with the call:

that must be set early in the model initialization before other time quantities
are defined. CalendarName is a string associated with the defined calendars
above and found in the list CalendarKindName in ``TimeMgr.h``. Typically, they
are the same as the kinds above with a space for multi-word names
(eg "No Leap"). If a custom calendar is desired rather than the supported
calendars, a custom calendar can be defined with a different init
interface in which all the relevant quantities are provided:

(eg “No Leap”). If a custom calendar is desired rather than the supported

Calendar::init(
   std::vector<I4> &InDaysPerMonth, ///< [in] array of days per month
   I4 InSecondsPerDay,              ///< [in] seconds per day
   I4 InSecondsPerYear,             ///< [in] seconds per year
   I4 InDaysPerYear                 ///< [in] days per year (dpy)
);

Time is generally tracked as the amount of time elapsed from a reference date
for a given calendar. A number of public functions exist to retrieve Calendar
information, convert between dates and elapsed time, check for leap years and
other checks. However, these are typically used by other TimeMgr classes and
not directly by the developer. Examples of their use can be found in the
TimeMgr unit test.

The TimeInstant class represents a moment in time with the particular calendar
that has been defined for the simulation. It consists of a TimeFrac that
represents the time since a reference time associated with the defined calendar.
There are three constructors for initializing a TimeInstant. One method is with
five 8-byte integers, and an 8-byte real value:

OMEGA::TimeInstant TI1(Year, Month, Day, Hour, Minute, RealSecond);

Alternatively, the TimeInstant can be initialized with eight 8-byte integers,
with the seconds value represented by three 8-byte integers:

A final constructor creates a time instant based on a time string that
conforms roughly to the ISO standard `"YYYYYY-MM-DD_HH:MM:SS.SSSS"` though
the constructor allows for any single-character non-numeric separator between
each of the numeric fields and the width of the YY and SS fields can be up
to the 8-byte standards for integer and floats, respectively.

Among the methods defined in the TimeInstant class is `getString` which will
produce a `std::string` representation of the time conforming to ISO standards,
 e.g. `"YYYYYY-MM-DD HH:MM:SS.SSSSS"`. The `getString` method takes two 4-byte
integers which represent the number of digits to use for the year and the
number of digits for the decimal portion of seconds, as well as a `std::string`
for the seprator between the Y/M/D calendar date and the H/M/S clock time. A
string formatted as above is returned with the following call:

number of digits for the decimal portion of seconds, as well as a 

A range of accessor functions can also get/set the TimeInstant in various
forms.

The TimeInterval class represents a length of time between two moments in time.
A TimeInterval can represent the time step, but is also utilized by other
classes for computing differences between TimeInstant objects, such as when
checking to trigger an Alarm. A TimeInterval can be initialized with a
fractional second representation, by supplying three 8-byte integers:

Alternatively, a TimeInterval can also be initialized with either an integer
or floating point value and a specified unit of time:

The supported time units are defined via an enum class in the Time Manager
header file:

enum class TimeUnits {
   None = 0, ///< value for undefined units
   Seconds,  ///< time units in seconds (typical)
   Minutes,  ///< time units in minutes
   Hours,    ///< time units in hours
   Days,     ///< time units in days
   Months,   ///< time units in months
   Years,    ///< time units in years
};

Finally, a time interval can be defined as the time between two time instants:

OMEGA::TimeInterval MyDeltaTime = MyTimeInstant2 - MyTimeInstant1;

The Alarm class is designed to trigger events at specified times. Alarms can be
set to ring at a single time or to ring periodically. A one-time Alarm is
initialized with a `std::string` name and a TimeInstant:

A periodic Alarm is initialized with a `std::string` name, a TimeInterval, and a
TimeInstant representing the start of the first interval:

OMEGA::Alarm PeriodicAlarm("Periodic Alarm", AlarmInterval, AlarmStart);

The status of an Alarm can be checked using the `isRinging` method, which
returns true or false depending on the Alarm status:

The `reset` method takes a TimeInstant as input and will switch a ringing Alarm
off, as well as set a new ring time. If it is a one-time Alarm, the input time
is set as the new ring time:

 method takes a TimeInstant as input and will switch a ringing Alarm

If the Alarm is periodic, the ringing will be switched off as above and the
new ring time will be set to the next interval boundary after the input time.
The interval boundary is an integer number of intervals after the start time
provided on Alarm creation.

It is sometimes useful to retrieve other aspects of a periodic Alarm, namely
the Alarm interval and the last time the Alarm was ringing. Two retrieval
functions are provided that return const pointers to these values:

const TimeInterval *AlarmInterval = PeriodicAlarm.getInterval();
const TimeInstant *PriorRingTime = PeriodicAlarm.getRingTimePrev();

The Clock class is the primary object for managing time during a simulation.
A Clock is initialized with a TimeInstant representing the start time and a
TimeInterval representing the time step the Clock advances with each integration
step:

The class contains a `std::vector` of pointers to Alarms. Any number of Alarms
can be attached to the Clock using the `attachAlarm` method:

The Clock is advanced during each time step with the `advance` method:

The Clock will automatically trigger any attached Alarms as the Clock marches
forward. The time step for a Clock can be changed by passing a TimeInterval
to the `changeTimeStep` method:

During a simulation, the `getCurrentTime` method will return the current time
stored in the Clock as a TimeInstant:

In order to utilize the Time Manager, the first step is to initialize a
Calendar object:

OMEGA::Calendar CalGreg("Gregorian", OMEGA::CalendarGregorian);

A start time and a time step are needed in order to initialize the Clock. The
following lines will initialize a Clock that starts Jan 1st, 2000 at midnight
with a 20 minute time step:

OMEGA::TimeInstant StartTime(&CalGreg, 2000, 1, 1, 0, 0, 0.0);
OMEGA::TimeInterval TimeStep(20, OMEGA::TimeUnits::Minutes);

OMEGA::Clock ModelClock(StartTime, TimeStep);

Now, Alarms can be attached to the Clock. For example, if some output
is desired on the first of each month, the following lines will attach a
monthly Alarm:

OMEGA::TimeInterval IntervalMonthly(1, OMEGA::TimeUnits::Months);
OMEGA::Alarm AlarmMonthly("Every Month", IntervalMontly, StartTime);

ModelClock.attachAlarm(AlarmMonthly);

The Clock can be used to control the duration of a simulation by using a
TimeInstant that represents the stop time:

OMEGA::TimeInstant StopTime(&CalGreg, 2025, 1, 1, 0, 0, 0.0);
OMEGA::TimeInstant CurrentTime = StartTime;

while (CurrentTime < StopTime) {

   ModelClock.advance();
   CurrentTime = ModelClock.getCurrentTime();

  ...

}

To utilize any attached Alarms, the ringing status needs to be checked each
time step:

if (AlarmMonthly.isRinging()) {
   AlarmMonthly.reset(CurrentTime);

   ...
}

Omega supports choice of multiple time stepping schemes. The time stepping
module provides implementations of different time stepping schemes and general
infrastructure for managing time steppers. To achieve modularity, each time
stepper is implemented as a separate class. To ensure common interface and
enable sharing of common code and data, every time stepper inherits from the
`TimeStepper` base class.

An enumeration listing all implemented schemes is provided. It needs to be extended every time a new time stepper
is added. It is used to identify a time stepper at run time.

enum class TimeStepperType { ForwardBackward, RungeKutta4, RungeKutta2 };

The `TimeStepper` class is a base class for all Omega time steppers. It stores
as members common data needed by every time stepper. These include a name and
time stepping method as well as some time tracking variables, including
StartTime, StopTime, TimeStep, a Clock and an EndAlarm. Its `doStep` method
defines the interface that every time stepper needs to implement. In addition to this
method, it provides a number of other methods that can divided into two groups.
The first group is for general time stepper management. The second group is
utility functions that provide common functionality that can be re-used in
implementation of different time steppers.

The `doStep` method is the main method of every time stepper class. It exists
in the base class as a pure virtual method, meaning it defines an interface but
has no implementation. Its signature is

 method is the main method of every time stepper class. It exists

    void TimeStepper::doStep(OceanState *State, TimeInstant &SimTime) const;

It is meant to advance `State` by one time step, from `Time` to `Time +
TimeStep`. Classes implementing concrete time stepping schemes need to provide
implementations of this method. As part of the time step, this routine advances
the Clock associated with the time stepper and returns the current simulation
time. For the default time stepper, this is advances the primary model clock.

. Classes implementing concrete time stepping schemes need to provide

initialize the default time stepper in two phases. The first phase must be
called early in the initialization process. It creates and time stepper
instance, fills it with various time quantities (StartTime, EndTime, TimeStep)
and creates a Clock (the Model Clock for the simulation) and an EndAlarm to
ring when this portion of the simulation should stop. The second phase is
called after the default tendency, auxiliary state, horizontal mesh, and halo
objects have been initialized. It attaches pointers to these objects for later
use during the time stop phase. A pointer to the default time stepper can be
retrieved at any time using:

A non-default time stepper can be created from a string `Name`, time stepper
type `Type`, `TimeStep`, `StartTime`, `EndTime`, tendencies `Tend`,
auxiliary state `AuxState`, horizontal mesh `Mesh`, and halo layer `MyHalo`

TimeStepper*  NewTimeStepper = TimeStepper::create(Name, Type, StartTime,
         EndTime, TimeStep, Tend, AuxState, Mesh, MyHalo);

For convenience, this returns a pointer to the newly created time stepper.
Given its name, a pointer to a named time stepper can be obtained at any time
by calling the static `get` method:

The time step is set on creation, but if the time step must change, the
function

can be used where `TimeStep` is an instance of `TimeInterval` class.

Given a pointer to a `TimeStepper` you can obtain various class members.
Pointers to the associated Clock and EndAlarm can also be retrieved.

TimeStepperType Type = Stepper->getType();
int NTimeLevels = Stepper->getNTimeLevels();
std::string Name = Stepper->getName();
TimeInterval TimeStep = Stepper->getTimeStep();
TimeInstant StartTime = Stepper->getStartTime();
TimeInstant StopTime = Stepper->getStopTime();
Clock *ModelClock = Stepper->getClock();
Alarm *EndAlarm = Stepper->getEndAlarm();

In addition to the above management methods, the base class provides common
functions that are useful in implementing different time steppers. Given two
pointers to `OceanState` objects `State1` and `State2`, two integers
representing two time levels `TimeLevel1` and `TimeLevel2`, and coefficient
`Coeff` of type `TimeInterval` calling

updateStateByTend(State1, TimeLevel1, State2, TimeLevel2, Coeff);

conceptually performs the following update
`State1(TimeLevel1) = State2(TimeLevel2) + Coeff * Tend`
element-wise for every state variable. To perform this update only for
layer thickness do

updateThicknessByTend(State1, TimeLevel1, *State2, TimeLevel2, Coeff);

updateVelocityByTend(State1, TimeLevel1, State2, TimeLevel2, Coeff);

updateTracersByTend(NextTracers, CurTracers, State1, TimeLevel1, State2, TimeLevel2, Coeff);

Additionally, for the RK4 method, there are methods for initializing, accumulating, and
finalizing the tracer integration for a given time step

weightTracers(NextTracers, CurTracers, CurState, TimeLevel1);
accumulateTracersUpdate(AccumTracer, Coeff);
finalizeTracersUpdate(NextTracers, State, TimeLevel);

The timing infrastructure builds upon the E3SM Pacer library for wall clock timing.
Pacer integrates, whenever possible, platform-specific marker APIs.

where `FilePrefix` is the prefix for all output files and an optional argument `PrintAllRanks` determines
if all MPI ranks should print their data.

To time a region of code enclose it with calls to `Pacer::start` and `Pacer::stop` functions, like so

Pacer::start(Name, Level);
// region of code to be timed
Pacer::stop(Name, Level);

These functions take a string `Name` and a non-negative integer `Level`.
The added timer will be active only if the timing level set in the config file is greater or equal to `Level`.

The added timer will be active only if the timing level set in the config file is greater or equal to 

Properly timing MPI communication might require inserting MPI barriers. It might be desirable
to remove those barriers in production runs. Pacer provides a function

which adds an MPI barrier and puts a timer around it using the communicator `Comm`.
Whether barriers added by this function are actually called can be controlled by the following functions

which adds an MPI barrier and puts a timer around it using the communicator 

  Pacer::enableTimingBarriers();
  Pacer::disableTimingBarriers();

It might be desirable to add a prefix to a group of timers based on their parent timer.
To enable this Pacer provides the `addParentPrefix()` and `removeParentPrefix()` functions.
For example, the following call sequence

Pacer::start("Parent", 0);
Pacer::addParentPrefix();

Pacer::start("Child", 0);
Pacer::stop("Child", 0);

Pacer::removeParentPrefix();

results in output where the "Child" timer shows up as "Parent:Child" in the output files.
This is useful when timers are added inside general purpose routines, that are called
from many places in the code, such as halo exchange.

results in output where the “Child” timer shows up as “Parent:Child” in the output files.

It might be desirable programmatically disable or enable timing. To allow that, Pacer provides
the `disableTimers()` and `enableTimers()` functions. In the following call sequence

Pacer::disableTiming();

Pacer::start("Timer1", 0);
Pacer::stop("Timer1", 0);

Pacer::enableTiming();

Pacer::start("Timer2", 0);
Pacer::stop("Timer2", 0);

`Timer1` is not timed while `Timer2` is. This is useful mainly when done conditionally.
For example, the first call to some function takes much longer than subsequent calls,
and having a detailed timing breakdown of the first call is not important. In that case,
it might be desirable to have a separate timer for the first call with its child timers disabled.

parcel (e.g., salt, chemical, or biological constituents).

To manage tracer definitions, users will update the `TracerDefs.inc` file
located in the `omega/src/ocn` directory of E3SM. The file contains
Tracer index variables and `defineAllTracers` C++ function defintion that
contains the calls to `define` function per each tracer as shown below:

}

To add a new tracer, simply call the `define` function with the appropriate
arguments. Index argument is optional one that allows to access the tracer
data using the given tracer index variable.

The following sections explain the concept and implementation of OMEGA tracers.

All tracers are stored in a vector of 3-dimensional device and host arrays.

  std::vector<Array3DReal> TracerArrays;   // Device: multiple time levels
  std::vector<HostArray3DReal> TracerArraysH; // Host: multiple time levels

Each device and host array in the vector has dimensions corresponding to the
number of tracers, the number of cells in the rank, and the vertical layers.

The host tracer array (`TracerArraysH`) is internally managed, so users
should not directly modify it in most cases.

Each tracer is assigned to a tracer group defined in the OMEGA YAML
configuration file, such as `Default.yml`. A tracer should be assigned to only one group.

To access the member tracers of a group in the code, users can use the
`getGroupRange` function as shown below:

static I4
);

   });

During the initialization of Tracers, the number of time levels is determined,
and the length of the tracer vectors (`TracerArrays` and `TracerArraysH`)
is set accordingly. The Tracers class internally manages the index of the
current time level in the arrays. To access a specific time level in the
arrays, users will use the integer "0" or negative integers. For example,
the following code retrieves the device tracer arrays for the current and
previous time levels:

arrays, users will use the integer “0” or negative integers. For example,

Array3DReal CurrentTracerArray;
I4 Err1 = OMEGA::Tracers::getAll(CurrentTracerArray, 0);

// Get all tracers at the previous time level (-1)
Array3DReal PreviousTracerArray;
I4 Err2 = OMEGA::Tracers::getAll(PreviousTracerArray, -1);

To use Tracers, the class should first be initialized by calling the
`init()` function and finalized by calling `finalize()`. These function
calls are typically handled within the initialization and finalization of
OMEGA itself, so users may not need to call them separately.

These functions return all device and host tracer arrays, respectively. If
the specified `TimeLevel` does not exist, they return a negative integer.

static HostArray3DReal getAllHost(
   HostArray3DReal &TracerArrayH, ///< [out] tracer host array
   const I4 TimeLevel ///< [in] Time level index
);

static Array3DReal getAll(
   Array3DReal &TracerArray, ///< [out] tracer device array
);

`getGroupRange` returns a pair of two `I4` values representing the group
start index and the group length. If the group is not found, it returns
negative integers.

static I4 getGroupRange(
   std::pair<I4, I4> &GroupRange, ///< [out] Group range
   const std::string &GroupName   ///< [in] Group name
);

These functions return device and host tracer arrays, respectively, based
on the `TracerIndex`. If the specified `TimeLevel` and/or `TracerIndex`
does not exist, they return a negative integer.

static Array2DReal getByIndex(
   Array2DReal &TracerArray, ///< [out] tracer device array
   const I4 TimeLevel,  ///< [in] Time level index
   const I4 TracerIndex ///< [in] Global tracer index
);

static HostArray2DReal getHostByIndex(
   HostArray2DReal &TracerArrayH, ///< [out] tracer host array
);

`getIndex` returns the index of the tracer specified by the `TracerName`
argument. If the tracer is not found, it returns a negative integer.

static I4 getIndex(
   I4 &TracerIndex,              ///< [out] Tracer index
   const std::string &TracerName ///< [in] Tracer name
);

`getFieldByIndex` returns the `Field` object associated with the tracer
specified by the `TracerIndex`. If not found, it returns `nullptr`.

// Returns a field by tracer index. If it does not exist, returns nullptr
static std::shared_ptr<Field>
getFieldByIndex(const I4 TracerIndex ///< [in] Global tracer index
);

`updateTimeLevels` increments the current time level by one. If the current
time level exceeds the number of time levels, it returns to `0`. It also
exchanges the halo cells of all tracers at the current time level and updates
the associated field with the new tracer arrays.

`getNumTracers` returns the total number of tracers used in the simulation.

Omega provides batched solvers for tridiagonal systems of equations.
Here, batched means that many systems of the same size need to be solved at the same time.
Typically, this is because one system needs to be solved per ocean column.
In addition to providing solvers for general tridiagonal systems,
specialized solvers for diffusion-type systems with better stability
properties are available.
To obtain portable performance, different solution algorithms are utilized
on CPUs and GPUs.
The solvers implement the same user interface for CPUs and GPUs, so users
should be able to ignore which algorithm is used.
Two type aliases are provided that resolve to the optimal solver for a given
architecture:

A general tridiagonal system is represented by three vectors
`DL`, `D`, `DU`, representing the lower, main, and upper diagonal, respectively, and a right-hand-side vector `X`
$$
(DL)_i Y_{i - 1} + D_i Y_i + (DU)_i Y_{i + 1} = X_i.
$$

, representing the lower, main, and upper diagonal, respectively, and a right-hand-side vector 

A diffusion-type system is represented by two vectors `G` and `H`, and a right-hand-side vector `X`. It has the form
$$
-G_{i - 1} Y_{i - 1} + (G_{i - 1} + G_i + H_i) Y_i - G_{i} Y_{i + 1} = X_i.
$$

There are two main ways to use Omega batched tridiagonal solvers:

Create global arrays storing the coefficients of all systems and call the top-level `solve` method

Create global arrays storing the coefficients of all systems and call the top-level 

Use a team-level solver inside a parallel loop using Kokkos team policy

While the first method is simpler, for computational performance it is better to use
the second method. This is because the second method allows one to define a system, solve it,
and do some post-processing of the solution in one parallel loop.
With the first method this would require multiple parallel loops, and allocation of storage
for the coefficients of all systems, which are typically no longer needed after the solution is obtained.

To solve `NBatch` systems of size `NRow` using the general solver
four arrays `DL`, `D`, `DU`, and `X` need to be defined.
All arrays are of size (`NBatch`, `NRow`) and contain the
coefficients for each of the systems.

Array2DReal DL("DL", NBatch, NRow); // DL(n, :) = lower diagonal of system n
Array2DReal D("D", NBatch, NRow);   // D(n, :)  = main diagonal of system n
Array2DReal DU("DU", NBatch, NRow); // DU(n, :) = upper diagonal of system n
Array2DReal X("X", NBatch, NRow);   // X(n, :)  = rhs of system n

Once these arrays are filled with the coefficients, to solve the systems the
static `TriDiagSolver:solve` method is called

Solving diffusion-type systems is similar, except the input arrays contain coefficients in the
form presented [above](#diffusion-type-tridiagonal-system)
and the static `TriDiagDiffSolver::solve` method is called

Array2DReal G("G", NBatch, NRow); // G(n, :) = coefficients G_i of system n
Array2DReal H("H", NBatch, NRow); // H(n, :) = coefficients H_i of system n
Array2DReal X("X", NBatch, NRow); // X(n, :) = rhs of system n

// fill G, H, X

TriDiagDiffSolver::solve(G, H, X);

Note: since the steps to use the team-level general and specialized solvers are similar,
this subsection first shows how to use the general solver.
At the end, an example of using the specialized solver is presented.

The team-level solvers need to be used inside a parallel loop using Kokkos team
policy. To create a team policy for solving `NBatch` systems of size `NRow`
the static member function `makeTeamPolicy` is used

TeamPolicy Policy = TridDiagSolver::makeTeamPolicy(NBatch, NRow);

Every team of threads except the last is responsible for solving `VecLength` systems.
Since the total number of systems likely doesn't evenly divide `VecLength`, the last
team solves fewer than `VecLength` systems. Hence, it is necessary to handle that with
if statements.

Every team of threads except the last is responsible for solving 

At the beginning of the loop, temporary scratch storage for the coefficients of `VecLength` systems
needs to be allocated.
This is done by creating a `TriDiagScratch` struct, which contains
four member arrays `DL`, `D`, `DU`, and `X` of size (`NRow`, `VecLength`).
Note that, contrary to the global arrays approach, the system dimension comes first.

At the beginning of the loop, temporary scratch storage for the coefficients of 

These arrays need to be filled with the coefficients of `VecLength` systems solved by the team.
This is done in a parallel loop over system size (using `TeamThreadRange`)
and a serial loop over `VecLength`.
Overall, the code that defines the systems coefficients looks typically like this:

parallel_for(Policy, KOKKOS_LAMBDA (TeamMember &member) {
   // create scratch data
   TriDiagScratch Scratch(Member, NRow);

   int IStart = Member.league_rank() * VecLength;

   // define the systems coefficients
   parallel_for(TeamThreadRange(Member, NRow), [=] (int K) {
      for (int IVec = 0; IVec < VecLength; ++IVec) {
         const int I = IStart + IVec;
         // handle last team having fewer systems
         if (I < NBatch) {
            Scratch.DL(K, IVec) = ...;
            Scratch.D(K, IVec)  = ...;
            Scratch.DU(K, IVec) = ...;
            Scratch.X(K, IVec)  = ...;
         }
      }
      ...
   });

To perform a team-level solve a different overload of the static member function `TridDiagSolver::solve`
needs to be called.
This overload takes the team member `Member` and the filled scratch struct `Scratch`.
This call needs to be surrounded by barriers to ensure that the
parallel loop setting the coefficients has finished and to make the computed solution available to all threads.

To perform a team-level solve a different overload of the static member function 

   // previous code

   // solve the system
   Member.team_barrier();
   TridDiagSolver::solve(Member, Scratch);

   ...
});

After the call to `solve` the solution is available in `Scratch.X`. It can be manipulated further or simply stored.

   // multiply the solution by 2 (just an example)
        Scratch.X(K, IVec) *= 2;
      }
   });

   // store the solution
            X(I, K) = Scratch.X(K, IVec);
         }
      }
   });
});

The team-level specialized diffusion solver can be used similarly, the only substantial difference is in the form
of the coefficients. A minimal complete example of using it is shown below.

// create team policy
TeamPolicy Policy = TridDiagDiffSolver::makeTeamPolicy(NBatch, NRow);

   TriDiagDiffScratch Scratch(Member, NRow);

            Scratch.G(K, IVec) = ...;
            Scratch.H(K, IVec)  = ...;
         }
      }

   TridDiagDiffSolver::solve(Member, Scratch);

         }
      }
   });
});

The `VertAdv` class contains methods and arrays for calculating vertical
velocities and the tendencies of thickness, horizontal velocity, and tracers due
to vertical advection.

The default `VertAdv` instance is created by calling `VertAdv::init()` method.
The default `HorzMesh`, `VertCoord`, and `Tracers` objects must be initialized
first. The default instance is retrieved by:

Additional instances can be created with the `create` method, which call the
constructor and places the new instance in a map identified with the label
`Name`:

VertAdv *VertAdv::create(const std::string &Name, ///< [in] name for new VertAdv
                         const HorzMesh *Mesh,    ///< [in] associated HorzMesh
                         const VertCoord *VCoord, ///< [in] associated VertCoord
                         Config *Options          ///< [in] configuration options
)

The constructor reads configuration options, stores some info from the
`HorzMesh`, `VertCoord`, and `Tracers` objects, allocates needed arrays,
and defines `Field` metadata.

The following arrays are stored as members of the `VertAdv` class.

The `VerticalVelocity` represents the raw vertical pseudovelocity through the
top interfaces of cell layers, computed from the divergence of the horizontal
velocity. The `TotalVerticalVelocity` is the transport velocity and includes
corrections applied to the `VerticalVelocity`. The `VertFlux` and
`LowOrderVertFlux` store tracer fluxes at layer interfaces. The specific
numerical algorithms to compute these fluxes are chosen by the user through
configuration options.

The `VerticalVelocity` and `TotalVerticalVelocity` arrays are computed by
calling the `computeVerticalVelocity` method:

VertAdv::computeVerticalVelocity(NormalVelocity, FluxLayerThickEdge);

This method takes as input the `NormalVelocity` field from the `OceanState`
object, and the `FluxLayerThickEdge` field from the `AuxiliaryState`. At
present, `TotalVerticalVelocity` is equivalent to `VerticalVelocity`;
additional corrections will be added in subsequent updates. The
`TotalVerticalVelocity` array is used by each of the tendency methods,
therefore `computeVerticalVelocity` must be called before any tendency
computations.

There are three methods for computing vertical advection tendencies of
thickness, horizontal velocity and tracers that are called from the time
stepper: `computeThicknessVAdvTend`, `computeVelocityVAdvTend`, and
`computeTracerVAdvTend`. Each method updates the corresponding tendency array
in place (inout).

Only the thickness tendency array is passed to the `computeThicknessVAdvTend`
method:

The `computeVelocityVAdvTend` method requires the `NormalVelocity` and
`FluxLayerThickEdge` fields, in addition to the velocity tendency array:

VertAdv::computeVelocityVAdvTend(VelTend, NormalVelocity, FluxLayerThickEdge);

The tracer vertical advection tendency depends on the configured settings.
The `computeTracerVAdvTend` method takes as arguments the full 3D array of
tracers, a thickness array (selected based on the configured advection
algorithm) and a `TimeInterval` representing the time step, along with the
tracer tendency array:

VertAdv::computeTracerVAdvTend(TracerTend, TracerArray, Thickness, TimeStep);

For the standard advection algorithm, `LayerThickness` from the `OceanState` is
passed as the thickness argument; for the FCT algorithm, the `ProvThickness`
from the `AuxiliaryState` is used instead.

This method first calls `computeVerticalFluxes` to compute the tracer fluxes at
layer interfaces, using the order of accuracy specified in the configuration
file. It then dispatches to the selected advection algorithm (standard or FCT)
to compute the tracer tendencies.

The default `VertCoord` instance is created by the `init` method, which assumes that `Decomp` has already been initialized.

The `init` method accepts two optional arguments with default values:

VertCoord::init(const bool ReadStream = true, const int NVertLayers = 0)

These arguments provide flexibility, particularly for unit testing. When `init(false)` is used, the method skips reading
the `InitialVertCoord` stream. This is useful for unit tests that rely on meshes lacking the fields required by that stream.
In this case, the min/max layer arrays are initialized with default values based on the number of vertical layers, and
`bottomDepth` remains uninitialized. Some tests may utilize a specific number of vertical layers that differs from what is
defined in the mesh file. To handle this, the `VertCoord` can be initialized with `init(false, LocNVertLayers)` to explicitly
set the number of vertical layers. An argument for `ReadStream` must be provided in order to explicitly set `NVertLayers`.
For example, `init(42)` is invalid, `init(false, 42)` must be called instead.

These arguments provide flexibility, particularly for unit testing. When 

 stream. This is useful for unit tests that rely on meshes lacking the fields required by that stream.

 remains uninitialized. Some tests may utilize a specific number of vertical layers that differs from what is

Additional instances can be created by calling the `create` method.

VertCoord *VertCoord::create(const std::string &Name,      ///< [in] Name for vertical coordinate
                             const Decomp *MeshDecomp,     ///< [in] Decomp for mesh
                             Config *Options,              ///< [in] Confiuration options
                             const bool ReadStream = true, ///< [in] optional logical to read stream
                             const int NVertLayers = 0     ///< [in] optional int to set vertical dim
)

This calls the constructor and places the instance in a map that can be used to retrieve the instance by name:

The constructor stores some information from `Decomp`, allocates the primary member variables that are computed by methods of the `VertCoord` class during a model timestep.
Host mirror copies are also created, with variables appended with `H`.
It also reads in some additional mesh information and computes the information describing the min/max location of the active vertical layers.
Finally, it registers variables with `IOStreams` so they can be requested as output.

, allocates the primary member variables that are computed by methods of the 

Host mirror copies are also created, with variables appended with 

A list of member variables along with their types and dimension sizes is below:

The methods `computePressure` and `computeZHeight` are similar in that they use hierarchical parallelism to split the work for horizontal cells over teams of threads, with a `parallel_for`.
This is done with a `TeamPolicy`:

 are similar in that they use hierarchical parallelism to split the work for horizontal cells over teams of threads, with a 

Kokkos::parallel_for("loopName", Policy, KOKKOS_LAMBDA(const TeamMember &Member) {
       const I4 ICell = Member.league_rank();
     ...
}

The cumulative sum in the vertical is computed among threads (in parallel) within in the `parallel_for` using  a `parallel_scan`.
The `parallel_scan` is called inside the `parallel_for`:

The cumulative sum in the vertical is computed among threads (in parallel) within in the 

Range = KMax - KMin + 1;
Kokkos::parallel_scan(
    TeamThreadRange(Member, Range),
    [=](int K, Real &Accum, bool IsFinal) {
    ...
}

where `KMax` and `KMin` are the maximum and minimum active vertical layer indices.
In `computeTargetThickness` an outer loop divides the horizontal cells over teams of threads as above, however, there is a nested `parallel_reduce` that computes the column sum of the reference layer thicknesses times the vertical coordinate movement weights among threads in a team:

 an outer loop divides the horizontal cells over teams of threads as above, however, there is a nested 

 that computes the column sum of the reference layer thicknesses times the vertical coordinate movement weights among threads in a team:

Real SumWh 0= 0;
Kokkos::parallel_reduce(
    Kokkos::TeamThreadRange(Member, KMin, KMax + 1),
    [=](const int K, Real &LocalWh) {
       LocalWh += VertCoordMovementWeights(ICell, K) *
                  RefLayerThickness(ICell, K);
    },
    SumWh);

also, the vertical computation of the target thicknesses are computed in a nested `parallel_for`:

also, the vertical computation of the target thicknesses are computed in a nested 

Kokkos::parallel_for(
    Kokkos::TeamThreadRange(Member, NChunks), [=](const int KChunk) {
   ...
}

This `parallel_for` iterates over vertical chunks to facilitate vectorization on CPUs within an inner `for` loop over the vector length.
The vector length on GPUs is set to 1 to maximize parallelism.
The `computeGeopotential` method uses hierarchical parallelism in a very similar way to `computeTargetThickness`, except that it doesn't require a column sum.
It has an outer `parallel_for` that splits horizontal cells into teams and an inner `parallel_for` that does vertical computations in chunks.

 iterates over vertical chunks to facilitate vectorization on CPUs within an inner 

Omega includes a `VertMix` class that provides functions that compute `VertDiff` and `VertVisc`, the
vertical diffusivity and viscosity, where both are defined at the center of the cell and top of the layer.
Currently the values of `VertDiff` and `VertVisc` are calculated using the linear combination of three options: (1) a
constant background mixing value, (2) a convective instability mixing value, and (3) a Richardson
number dependent shear mixing value from the [Pacanowski and Philander (1981)](https://journals.ametsoc.org/view/journals/phoc/11/11/1520-0485_1981_011_1443_povmin_2_0_co_2.xml) parameterization. These options are linearly additive. In the future, additional additive options will be implemented, such as the K Profile Parameterization [(KPP; Large et al., 1994)](https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/94rg01872). For both the convective and shear mixing values `BruntVaisalaFreqSq` is needed, which
is calculated by the `EOS` class.

 are calculated using the linear combination of three options: (1) a

 parameterization. These options are linearly additive. In the future, additional additive options will be implemented, such as the K Profile Parameterization 

The primary class `VertMix` is implemented using the Singleton pattern to ensure a single instance manages all vertical mixing calculations.

 is implemented using the Singleton pattern to ensure a single instance manages all vertical mixing calculations.

// Initialize VertMix
VertMix* VMix = VertMix::getInstance();

// Compute mixing coefficients
VMix->computeVertMix(NormalVelocity, TangentialVelocity, BruntVaisalaFreqSq);

The initialization process reads parameters from the yaml configuration file with the following structure and
default values:

VertMix:
  Background:
    Viscosity: 1e-4
    Diffusivity: 1e-5
  Convective:
    Enable: true
    Diffusivity: 1.0
    TriggerBVF: 0.0
  Shear:
    NuZero: 0.005
    Alpha: 5.0
    Exponent: 2.0

`VertDiff`: 2D array storing vertical diffusivity coefficients (m²/s)

`VertVisc`: 2D array storing vertical viscosity coefficients (m²/s)

`BackDiff`: Background vertical diffusivity (m²/s; Default: 1e-5)

`EnableConvMix`: Flag to enable/disable convective mixing (Default: True)

`ConvTriggerBVF`: Trigger threshold for convective mixing (Default: 0.0)

`EnableShearMix`: Flag to enable/disable shear mixing (Default: True)

`ShearNuZero`: Base coefficient for Pacanowski-Philander scheme (Default: 0.005)

: Base coefficient for Pacanowski-Philander scheme (Default: 0.005)

`ShearExponent`: Exponent parameter for P-P scheme (Default: 2.0)

void computeVertMix(const Array2DReal &NormalVelocity,
                   const Array2DReal &TangentialVelocity,
                   const Array2DReal &BruntVaisalaFreqSq);

Shear instability driven mixing (Pacanowski-Philander scheme; to be changed to [Large et al., 1994](https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/94rg01872) shear mixing scheme in a later development)

Shear instability driven mixing (Pacanowski-Philander scheme; to be changed to 

The Ocean Model for E3SM Global Applications (Omega) initially is an eddy-resolving,
global ocean model in the early stages of development by the
[E3SM](https://e3sm.org/) ocean team.  The first release is planned for
Summer 2026.  A non-eddying configuration will be released in early 2027.

The model is written in c++ using the [Kokkos](https://github.com/kokkos)
framework for performance portability.  Omega is based on the
[TRSK formulation](https://doi.org/10.1016/j.jcp.2009.08.006) for geophysical
models on unstructured meshes. The first version of Omega will primarily be a direct port
of the [MPAS-Ocean](https://e3sm.org/model/e3sm-model-description/v1-description/v1-ocean-sea-ice-land-ice/)
component of E3SM for comparison purposes.

Development is taking place at https://github.com/E3SM-Project/Omega.

The `AuxiliaryState` class provides a container for the [auxiliary variables](#omega-user-aux-vars) in Omega.
Upon creation of an `AuxiliaryState` instance, these variables are allocated and registered with the IO infrastructure.
There are no user-configurable options.

 instance, these variables are allocated and registered with the IO infrastructure.

Omega needs to compute a couple of auxiliary variables at every model time step
in order to advance the model state. Some auxiliary variables can be computed in
different ways, and the user can specify that in the input configuration file.
For example, the value of thickness used in advective flux can be centered or upwind.

layer thickness used for fluxes through edges. May be centered, upwinded, or a combination of the two

earth's rotational rate (Coriolis parameter, f) divided by layer thickness

earth’s rotational rate (Coriolis parameter, f) divided by layer thickness

curl of horizontal velocity divided by layer thickness, averaged from vertices to edges

earth's rotational rate (Coriolis parameter, f) divided by layer thickness, averaged from vertices to edges

earth’s rotational rate (Coriolis parameter, f) divided by layer thickness, averaged from vertices to edges

thickness-weighted tracers used for fluxes through edges. May be centered, upwinded or a combination of the two

In any parallel code, there is sometimes a need to broadcast values from
one processor to all other processors. This is implemented in Omega as a
Broadcast. Functions are available to broadcast scalars or vectors of any
supported data type. There are no user-configurable options for this
functionality. More details can be found in the
[Developer Guide](#omega-dev-broadcast).

In Omega, the input configuration file is in [YAML](https://yaml.org) format
and must be a file named ``omega.yml`` in the same directory as the executable.
However, Unix soft links can be used to point a link with that name to a
specific configuration file stored elsewhere.
In YAML format, most of the configuration variables are typically represented
as key-value pairs called maps, where the name and the value are separated
by a colon. However, lists of variables (eg contents of an IO stream) can
be used and are called sequences. Sequences can be in the form of a list
with an entry on each line starting with a dash and space. They can also
be formatted as a vector with a comma-delimited list within square brackets.
Each of these can be nested to build up the full configuration with indentation
used to separate the nests.
Each map or sequence is called a node in YAML.  A typical omega configration
file will start with the main omega map node with sub-maps associated with
various modules in omega. For example, a file might look like this:

omega:
   TimeManagement:
      RestartOn: false
      RestartTimestampName: restartTimestamp
      StartTime: 0001-01-01_00:00:00
      StopTime: none
      RunDuration: 0010_00:00:00
      CalendarType: noleap

   Hmix:
      HmixScaleWithMesh: false
      MaxMeshDensity: -1.0
      HmixUseRefWidth: false
      HmixRefWidth: 30.0e3

   MyVector: [1, 2, 3, 4, 5]

   Streams:

      Mesh:
         Type: input
         FilenameTemplate: mesh.nc
         InputInterval: initial_only

      Output:
         Type: output
         FilenameTemplate: output/output.$Y-$M-$D_$h.$m.$s.nc
         FilenameInterval: 01-00-00_00:00:00
         ReferenceTime: 0001-01-01_00:00:00
         Precision: single
         OutputInterval: 0001_00:00:00
         Contents:
         - tracers
         - ssh

The variable names in the example above may not be actual omega variables and
are shown only as an example.
We intend to create a script that will generate a default input file that
users can modify for standalone ocean experiments but this capability is not
currently available. Similarly, standard configurations for E3SM compsets will
be provided as part of E3SM releases. The configuration supports all Omega
data types (bool, string, I4, I8, R4, R8). Because YAML internally represents
all data as strings, configuration variables are converted by omega into the
correct type and this conversion may slightly change floating point results
in the last digits, but in a consistent manner based on C++ coercion rules.
This can be minimized by supplying all digits for the precision desired.

Users will configure the model primarily by editing or modifying this
input YAML-formatted file.  Details of the implementation within omega
can be found in the [Developer's Guide](#omega-dev-config) and the actual
interfaces for extracting configuration variables into the modules that
own them are described there as well.

Omega supports all standard data types and uses some specific defined
types to guarantee a specific level of precision. There is only one
user-configurable option for precision. When a specific floating point
precision is not required, we use a Real data type that is, by default,
`-DOMEGA_SINGLE_PRECISION` (see insert link to build system) preprocessor flag,
the default Real becomes single precision (4-byte/32-bit). Users are
encouraged to use the default double precision unless exploring the
performance or accuracy characteristics of single precision.

Omega is designed to be run in parallel across multiple nodes and cores
in a clustered architecture. As in many Earth System Models, we decompose
the horizontal domain into subdomains that are distributed across the
system. Communication between the domains is accomplished using the
Message Passing Interface standard. To reduce the communication between
subdomains, a halo of points is filled with information from remote neighbor
cells, edges and vertices so that a time step can largely be completed
without the need to communicate.

The partitioning itself is performed using the
libraries that can partition unstructured domains in a way that also optimizes
communication. More details on the mesh, connectivity and partitioning can be
found in the [Developer's Guide](#omega-dev-decomp).

There are two parameters that are set by the user in the input configuration
file. These are:

The HaloWidth is set to be able to compute all of the baroclinic terms in a
timestep without communication and for higher-order tracer advection terms,
this currently must be at least 3.

METIS and ParMETIS support a number of partitioning schemes, but the MetisKWay
and its parallel equivalent ParMetisKWay are the only currently supported
decomposition methods for Omega. The serial form (MetisKWay) is typically
faster for small meshes and decompositions. The ParMetisKWay option is a
parallel form that is faster and requires less memory than the serial option.
Note that the two methods will not result in the same partition.

the total number of cells, edges and vertices (NCells, NEdges, NVertices)

the mesh connectivity contained in the arrays CellsOnCell, EdgesOnCell
VerticesOnCell, CellsOnEdge, EdgesOnEdge, CellsOnVertex, EdgesOnVertex.
Again, a full description of the mesh is given in the
[Developer's Guide](#omega-dev-decomp).
The mesh information is read via parallel IO into an initial linear domain
decomposition and then is partitioned by METIS and rearranged into the
final METIS parallel decomposition.

Once the mesh is decomposed, all of the mesh index arrays are stored in
a Decomp named Default which can be retrieved as described in the
Developer guide. In the future, additional decompositions associated
with processor subsets (as described in MachEnv) should be possible, but
this capability is not yet supported.

When performing IO, some metadata associated with each dimension in Omega
must be defined. We provide a Dimension class to hold this metadata for use by
the IO and Streams classes. This metadata is defined internally in Omega and
there are no user-configurable options. More detail is proved in
{ref}`omega-dev-dimension`.

The Omega Driver manages the execution of the Omega ocean model. Currently,
the only user-configurable parameters unique to the Driver relate to the
time management of the simulation.

      RunDuration: 0000_02:00:00.000
      CalendarType: No Leap

The `StartTime` and `StopTime` should be formatted strings in the form
`YYYY-MM-DD_HH:MM:SS.SSSS` and the `RunDuration` should be a formatted string in
the form `DDDD_HH:MM:SS.SSSS` (the actual width of each unit can be arbitrary
and the separators can be any single non-numeric character). Either the
`StopTime` or `RunDuration` can be set to `none` in order to use the other to
determine the duration of the run. If both are set and `StopTime - StartTime`
is incosistent with `RunDuration`, then `RunDuration` is used for the
simulation. The supported calendar types are:

The equation of state (EOS) for the ocean describes the relationship between specific volume of seawater (in $\textrm{m}^3/\textrm{kg}$; the reciprocal of density) and temperature (in $^{\circ}\textrm{C}$), salinity (in $\textrm{g/kg}$), and pressure (in $\textrm{dbar}$). Through the hydrostatic balance (which relates density/specific volume gradients to pressure gradients), the equation of state provides a connection between active tracers (temperature and salinity) and the fluid dynamics.

The equation of state (EOS) for the ocean describes the relationship between specific volume of seawater (in 

). Through the hydrostatic balance (which relates density/specific volume gradients to pressure gradients), the equation of state provides a connection between active tracers (temperature and salinity) and the fluid dynamics.

Two choices of EOS are provided by Omega: a linear EOS and a TEOS-10 EOS. The linear EOS simplifies the relationship by excluding the influence of pressure and using constant expansion/contraction coefficients, making the specific volume a simple linear function of temperature and salinity. However, this option should only be used for simpler idealized test cases as it is not appropriate for realistic ocean simulations. The TEOS-10 EOS is a 75-term polynomial expression from [Roquet et al. 2015](https://www.sciencedirect.com/science/article/pii/S1463500315000566) that approximates the [Thermodynamic Equation of Seawater 2010](https://www.teos-10.org/pubs/TEOS-10_Manual.pdf), but in a less complex and more computationally efficient manner, and is the preferred EOS for real ocean simulations in Omega.

Two choices of EOS are provided by Omega: a linear EOS and a TEOS-10 EOS. The linear EOS simplifies the relationship by excluding the influence of pressure and using constant expansion/contraction coefficients, making the specific volume a simple linear function of temperature and salinity. However, this option should only be used for simpler idealized test cases as it is not appropriate for realistic ocean simulations. The TEOS-10 EOS is a 75-term polynomial expression from 

, but in a less complex and more computationally efficient manner, and is the preferred EOS for real ocean simulations in Omega.

The user-configurable options are: `EosType` (choose either `Linear` or `Teos-10`), as well as the parameters needed for the linear EOS.

Eos:
   EosType : teos10
   Linear:
      DRhoDT: -0.2
      DRhoDS: 0.8
      RhoT0S0: 1000.0

where `DRhoDT` is the thermal expansion coefficient ($\textrm{kg}/(\textrm{m}^3 \cdot ^{\circ}\textrm{C})$), `DRhoDS` is the saline contraction coefficient ($\textrm{kg}/\textrm{m}^3$), and `RhoT0S0` is the reference density at (T,S)=(0,0) (in $\textrm{kg}/\textrm{m}^3$).

In addition to `SpecVol`, the displaced specific volume `SpecVolDisplaced` and `BruntVaisalaFreqSq` are also calculated by the EOS.

The `Eos` class calculates the density of a parcel of fluid that is adiabatically displaced by a relative `k` levels (`k` counted positive downward), capturing the effects of pressure/depth changes. This is primarily used to calculate quantities for determining the water column stability (i.e. the stratification) and the vertical mixing coefficients (viscosity and diffusivity). Note: when using the `Linear` EOS option, `SpecVolDisplaced` will be the same as `SpecVol` since the specific volume calculation is independent of pressure/depth.

 class calculates the density of a parcel of fluid that is adiabatically displaced by a relative 

 counted positive downward), capturing the effects of pressure/depth changes. This is primarily used to calculate quantities for determining the water column stability (i.e. the stratification) and the vertical mixing coefficients (viscosity and diffusivity). Note: when using the 

 since the specific volume calculation is independent of pressure/depth.

The `Eos` class also calculates the squared Brunt Vaisala Frequency, which is used by the `VertMix` class to determine water column stability for both the convective adjustment and shear-instability driven mixing. When using the `Linear` EOS option, the `BruntVaisalaFreqSq` is calculated using linear coefficients. When using the `Teos-10` EOS option, the `BruntVaisalaFreqSq` is calculated with non-linear coefficients according to the TOES-10. For both options, `SpecVol` must be calculated prior to calculating `BruntVaisalaFreqSq`, as it is an input to the `BruntVaisalaFreqSq` calculation.

 class also calculates the squared Brunt Vaisala Frequency, which is used by the 

 class to determine water column stability for both the convective adjustment and shear-instability driven mixing. When using the 

 is calculated with non-linear coefficients according to the TOES-10. For both options, 

Omega includes an error handling facility that checks for potential errors
and responds to those errors when encountered.
It uses the [Logging](#omega-dev-logging) facility for writing the error
messages so its behavior follows that of Logging. In particular, the error
messages will be output to a single log file (omega.log) unless the option
to write a log file for each MPI task is enabled at build time. Each message
will have the format

where level is the severity (warn, error, critical) and file and line number
refer to the source code line where the error occurs. The error facility
sometimes accumulates multiple error messages before aborting. In
these cases, the most recent error in these accumulated errors will be printed
first and the other error messages in the chain will be printed in reverse
order and indented.  For critical errors, a full stack trace is generated
if debug information is available, either with the `-g` compiler
option or a DEBUG build using `-DOMEGA_BUILD_TYPE=Debug` during the Cmake
configuration.

Within OMEGA, each module registers fields that it owns together with the
metadata associated with that field. Metadata must be compliant with the
Climate and Forecast [(CF) metadata conventions.](http://cfconventions.org/)
This information can then be used by IOStreams or other modules that may
require metadata information. Two special fields are defined internally as
CodeMeta and FieldMeta to hold any global metadata not associated with a
data field, but may be required for provenance. Required metadata are enforced
by the Field create method but additional metadata can be added after field
creation. The data array holding field values can be attached/retrieved and
updated whenever needed.

Groups of fields can also be defined to provide a shortcut to a list of fields
commonly used together, like the model state or tracer groups. This reduces the
size of the input YAML configuration. The user can select which fields or field
groups they wish to include via the streams section of the input YAML
configuration file. This is described further in {ref} `omega-user-iostreams`.
There are no user-configurable options for the Field class itself since all
fields are defined and updated internally, but further details on the methods
and implementation can be found in the {ref}`omega-dev-field` section of the
Developer's Guide.

When Omega is run in a parallel machine environment, the computaional domain
is broken into subdomains that are distrubted among the computational nodes
of the machine. It is necessary to regularly exchange data that is located
at or near the interfaces of adjacent subdomains, these regions are called
halos. The Halo class accomplishes these halo exchanges using the MPI
(Message Passing Interface) standard.

The Halo class depends on the MachEnv class and the Decomp class. Currently,
the only configuration parameters that directly impact Halo are those set
to configure the [Decomp](#omega-user-decomp) class. During the build
process, the pre-processing parameter `-DOMEGA_MPI_ON_DEVICE` toggles
whether or not buffer arrays in device memory space can be passed to
MPI send and receive functions during halo exchanges. By default,
`OMEGA_MPI_ON_DEVICE=ON`, but can be turned off during build.

Once a Halo object is constructed, all the information needed to perform
a halo exchange on any supported array type defined in any index space
is contained in the Halo object. Halo exchanges are executed via the
exchangeFullArrayHalo function.

Omega uses the MPAS mesh specification found
[here](https://mpas-dev.github.io/files/documents/MPAS-MeshSpec.pdf). The names
of the mesh variables have been retained, with the caveat that they now begin
with a capital letter.

The Mesh class is meant to be a container for all mesh variables local to a
decomposed sub-domain that can be easily passed among the dycore routines. It
depends on a given [Decomp](#omega-user-decomp) and reproduces the
cell/edge/vertex totals and connectivity information from that class. The Mesh
class also creates parallel I/O decompositions that are used to read in the
additional mesh variables, which are not required for Decomp.

Currently, the Mesh class reads in all variables from the MPAS mesh
specification except those read by [Decomp](#omega-dev-decomp).
This includes the following variables:

Area of the portions of each dual cell that are part of each cellsOnVertex

Length of each edge, computed as the distance between verticesOnEdge

Length of each edge, computed as the distance between CellsOnEdge

Value of density function used to generate a particular mesh at cell centers

In the future, the Mesh class will optionally compute the mesh variables that
are dependent on the Cartesian mesh coordinates internally.
This includes the various areas, lengths, angles, and weights needed for the
TRiSK discretization (e.g. rows 5-11 in the table above).

The horizontal discretization in Omega is a staggered numerical scheme known as
TRiSK. It defines discrete versions of basic differential operators (divergence,
gradient, and curl) as well as other operators that are needed by the scheme,
such as reconstruction of tangential velocity from its normal components. Omega
provides reference implementations of these operators, each in a separate C++
class. The class name describes the operator and the mesh element associated
with its result.

SCORPIO is responsible for rearranging data from data decomposition used
Users will generally access IO via the [IOStreams](#omega-user-iostreams)
and other interfaces and the base IO layer described here only provides

There are some general parameters that must be set for IO performance and
formatting via the input configuration file. These are:

IO:
    IOTasks:  1
    IOStride: 1
    IOBaseTask: 0
    IORearranger: box
    IODefaultFormat: NetCDF4

where ``IOTasks`` is the total number of IOTasks to assign to reading
and writing. The default is 1 (serial IO) for safety but this number
should be set appropriately for the underlying hardware. A simple
starting point might be one IOTask per node or socket. The ``IOStride``
is set to spread the IOTasks across the total number of MPI tasks so
that every IOStride task (starting with the root task IOBaseTask) is an
IOTask. The product of IOTasks and IOStride should equal the total number
of MPI Tasks.

When using parallel IO, the data must be rearranged to match the IO task
decomposition. There are two algorithms for rearranging data available
in SCORPIO: box and subset. Box is the default and preferred for most
configurations. It ensures each IO task has a contiguous chunk of data to
read/write but may draw from the full range of MPI tasks. In the subset
rearranger, each IO task is assigned a subset of the MPI tasks and only
communicates with those tasks. It no longer is guaranteed to have a
contiguous chunk of data to read/write. This option can sometimes result
in more optimal communication patterns but at the cost of less efficient
I/O as the underlying library must handle the non-contiguous data. The
subset option is available for exploring the most efficient approach.
See SCORPIO documentation for details.

Finally, the user can specify the file format. The SCORPIO library
supports all the various NetCDF formats, though the use of older
NetCDF formats is strongly discouraged. NetCDF-4 is currently the E3SM
default and is the default for Omega. In addition, SCORPIO supports
native HDF5 files (note that the latest NetCDF formats are all implemented
with HDF5 and are mostly compatible with HDF5) The ADIOS format is different
since ADIOS writes each chunk of data to a separate file and these files must
be combined later. E3SM provides the necessary tools to combine/convert
ADIOS files to netcdf as part of pre/post-processing. For standalone Omega,
users would need to become familiar with these tools if they wish to use
ADIOS.

IO Streams are the primary mechanism for users to specify input and output
for Omega. An IOStream can be defined for any number of fields and at desired
time frequencies (including one-time or at startup/shutdown). IOStreams are
defined in the Omega input configuration file in an IOStreams section:

Omega:
  # other config options removed for brevity
      Filename: OmegaMesh.nc
      Mode: read
      Precision: double
      Freq: 1
      FreqUnits: OnStartup
      UseStartEnd: false
      Contents:
        - Restart
    RestartWrite:
      UsePointerFile: true
      PointerFilename: ocn.pointer
      Filename: ocn.restart.$Y-$M-$D_$h.$m.$s
      Mode: write
      IfExists: replace
      Freq: 6
      FreqUnits: months
      Contents:
        - Restart
    History:
      Filename: ocn.hist.$SimTime
      Freq: 1
      Contents:
        - Tracers
    Highfreq:
      Filename: ocn.hifreq.$Y-$M
      Precision: single
      Freq: 10
      FreqUnits: days
      FileFreq: 1
      FileFreqUnits: months
      UseStartEnd: true
      StartTime: 0001-06-01_00.00.00
      EndTime: 0001-06-30_00.00.00
      Contents:
        - Tracers

Each stream has a number of required and optional parameters for customizing
input and output. These options are indented below the stream name as shown
in the sample YAML entries above. They include:

**UsePointerFile:** A required flag that is either true or false. A pointer
file is used for cases like restart files where the last file written can
be stored for the next job submission so that the configuration file does
not need to be edited between job submissions.

**PointerFilename:** Only required if UsePointerFile is true and should
be set to the full filename (with path) for the pointer file. Each stream
using a pointer file must define a unique pointer file name.

**Filename:** Required in all cases except input streams using a pointer
file. This is the complete name (with path) of the file to be read or written.
A filename template is also supported in which simulation (or real) time
can be used in the file name. As the examples above show, accepted keys for
a template can be:

$SimTime for the current simulation time in a standard time string (note
 that this time string may include colon separators that can be a problem
 for filenames so using the individual keys below is preferred).

that this time string may include colon separators that can be a problem

$WallTime for the time IRL for use when you might need the actual time for
a debug time stamp

**Mode:** A required field that is either read or write. There is no
 readwrite option (eg for restarts) so a separate stream should be
 defined for such cases as in the examples above.

readwrite option (eg for restarts) so a separate stream should be

**IfExists:** A required field for write streams that determines behavior
 if the file already exists. Acceptable options are:

Replace if you want to replace the existing file with the new file

Append if you want to append (eg multiple time slices) to the existing
file. When re-running an interval, this will also over-write existing
slices corresponding to the simulation time if they exist in the file.

**Precision:** A field that determines whether floating point numbers are
 written in full (double) precision or reduced (single). Acceptable values
 are double or single. If not present, double is assumed, but a warning
 message will be generated so it is best to explicitly include it.

written in full (double) precision or reduced (single). Acceptable values

are double or single. If not present, double is assumed, but a warning

message will be generated so it is best to explicitly include it.

**Freq:** A required integer field that determines the frequency of
 input/output in units determined by the next FreqUnits entry.

**FreqUnits:** A required field that, combined with the integer frequency,
 determines the frequency of input/output. Acceptable values include:

determines the frequency of input/output. Acceptable values include:

AtTime or OnTime or Time or TimeInstant for a one-time read or write
at the time specified in the StartTime entry

Never if the stream should not be used but you wish to retain the
entry in the config file (a warning will be output to log file)

Years for a frequency every Freq years (*not* Freq times per year)

Months for a frequency every Freq months (*not* Freq times per month)

Hours for a frequency every Freq hours (*not* Freq times per hour)

Minutes for a frequency every Freq minutes (*not* Freq times per minute)

Seconds for a frequency every Freq seconds (*not* Freq times per seconds)

**FileFreq:** An optional entry for including multiple time slices in a
 single file (only supported for output presently)

**FileFreqUnits:** A field that, combined with the integer frequency,
 determines the frequency of new files when multiple time slices are
 included in a file. The file frequency must be less than or equal to
 the data frequency. Acceptable values include all of the frequency values
 listed above except the one-time values (eg OnStartup/Shutdown or AtTime).
 The filename template should reflect the frequency and the time used for
 the file will correspond to the template at the time the file is first
 opened.

determines the frequency of new files when multiple time slices are

included in a file. The file frequency must be less than or equal to

the data frequency. Acceptable values include all of the frequency values

listed above except the one-time values (eg OnStartup/Shutdown or AtTime).

The filename template should reflect the frequency and the time used for

the file will correspond to the template at the time the file is first

**UseStartEnd:** A required entry that is true or false and is used if the
 I/O is desired only within a certain time interval. An example might be
 for specifying high-frequency output within a certain period of a simulation.

I/O is desired only within a certain time interval. An example might be

for specifying high-frequency output within a certain period of a simulation.

**StartTime:** A field only required when UseStartEnd is true or if
 the FreqUnits request a one-time read/write. The StartTime must be a time
 string of the format YYYY-MM-DD_hh.mm.ss (though the delimiters can be
 any non-numeric character). The year entry is the integer year and can be
 four or more digits. The StartTime is inclusive - the I/O will occur at or
 after that date/time.

the FreqUnits request a one-time read/write. The StartTime must be a time

string of the format YYYY-MM-DD_hh.mm.ss (though the delimiters can be

any non-numeric character). The year entry is the integer year and can be

four or more digits. The StartTime is inclusive - the I/O will occur at or

**EndTime:** A field that is only required when UseStartEnd is true. It
 requires the same format as StartTime but unlike StartTime, the EndTime
 is not inclusive and I/O only occurs for times before the EndTime. If a
 file is desired at the EndTime, the user should specify an EndTime slightly
 later (less than a time step) than the desired end time.

requires the same format as StartTime but unlike StartTime, the EndTime

is not inclusive and I/O only occurs for times before the EndTime. If a

file is desired at the EndTime, the user should specify an EndTime slightly

**Contents:** This is a required field that contains an itemized list of
 each Field or FieldGroup that is desired in the output. The name must
 match a name of a defined Field or Group within Omega. Group names are
 preferred to keep the list of fields short so Omega will define convenient
 FieldGroups like Restart, State, Tracers that will include all members
 of the group. If only a subset of Fields from a Group is desired, the
 individual Field names should be specified and not the Group name.

each Field or FieldGroup that is desired in the output. The name must

match a name of a defined Field or Group within Omega. Group names are

preferred to keep the list of fields short so Omega will define convenient

FieldGroups like Restart, State, Tracers that will include all members

of the group. If only a subset of Fields from a Group is desired, the

individual Field names should be specified and not the Group name.

This streams configuration should be sufficient to define all input and output
from the model and provides a relatively simple interface for a typical user.
However, if necessary (eg before streams have been defined), the specific
interfaces in the lower level [IO](#omega-user-IO) module can be used.

Omega's logging system is built upon the [spdlog](https://github.com/gabime/spdlog)
logging tool. As of this version, It only supports Standalone build mode.

To use Omega's logging system in Standalone build mode, you need to include
the Logging.h header file in your code.

To use Omega’s logging system in Standalone build mode, you need to include

You can then use the following macros to write logs of different severity
levels:

LOG_TRACE   (logmsg);
LOG_DEBUG   (logmsg);
LOG_INFO    (logmsg);
LOG_WARN    (logmsg);
LOG_ERROR   (logmsg);
LOG_CRITICAL(logmsg);

The "logmsg" argument in these macros can be any string that follows the syntax
of C++'s std::format function. For example, the following code writes a
"Hello, World!" message as a TRACE level log:

The “logmsg” argument in these macros can be any string that follows the syntax

of C++’s std::format function. For example, the following code writes a

For more information about the syntax of logmsg, please see the spdlog documentation.

Here is a table that summarizes the different log severities and their meanings:

You can replace the string "World" with any variable whose type is
a C++ basic data type, such as an int, a float, or a string. You can
also use a limited set of Kokkos variables. For example, the following code
writes the value of the variable `MyInt` as a TRACE level log:

You can replace the string “World” with any variable whose type is

Within Omega, many aspects of the machine environment are stored
in a class called MachEnv. These include message-passing parameters
like MPI communicators and task information, number of threads
if threaded, vector length for CPUs, GPU and node information as
needed. Multiple environments are supported in case portions of the
code need to run on subsets of tasks or in different contexts. A
default environment is defined early in the initialization of the
model and can be retrieved as described in the
[Developer's Guide](#omega-dev-mach-env).

The user is not expected to set any parameters in MachEnv. All
quantities are derived from either the job launch command
(eg mpirun or srun) that defines the number of MPI tasks and tasks
layouts or from machine parameters enforced during the build based
on supported machine xml configurations.
The latter include the pre-processing parameters
`-DOMEGA_VECTOR_LENGTH=xx` and `-DOMEGA_THREADED` that define an
optimal vector length for CPU code and turn on OpenMP threading
if desired.

The `OceanState` class provides a container for the non-tracer prognostic variables in Omega, namely `normalVelocity` and `layerThickness`.
Upon creation of a `OceanState` instance, these variables are allocated and registered with the IO infrastructure.
The class contains a method to update the time levels for the state variables between timesteps.
This involves a halo update, time level index update, and updating the `IOFields` data references.

 class provides a container for the non-tracer prognostic variables in Omega, namely 

This involves a halo update, time level index update, and updating the 

Omega's build system is constructed using the CMake build tool,
offering a strong foundation for managing the building process.

The Omega build system has two modes: standalone and E3SM component.

For standalone build, the Omega build creates a generic E3SM case and
reads build configurations from the case. In E3SM component build, the
Omega build reads build configrations from the E3SM case created by
CIME build system. Following this, it configures CMake variables and
environment variables based on the computing system where the build is
taking place, as well as user input from the CMake command-line.

For the Omega build system to function, a Python interpreter is necessary.
The minimum version of CMake is 3.21.

CMake and Omega prefer an out-of-source build. This enables a user to build
and maintain multiple executables from the same source directory.
The standard practice is for a user to create a separate directory where
the build should take place and the commands below should be launched from
that directory.

To utilize a compiler name that's defined in the E3SM machine configuration
file (`config_machines.xml`), employ the `OMEGA_CIME_COMPILER` CMake variable,
as illustrated below. In some cases, you may also want to add `OMEGA_CIME_MACHINE`
to specify which system you intend to use. On systems where there is not a
default `PROJECT` defined, you can use `OMEGA_CIME_PROJECT` to specify an
account to use as a placeholder during the Omega build.
The values of `OMEGA_CIME_COMPILER` and `OMEGA_CIME_MACHINE` are defined in
"${E3SM}/cime\_config/machines/config\_machines.xml".
Omega requires some external libraries. Many of these are built automatically
from the E3SM distribution. However, the METIS, ParMETIS, and, optionally,
GKlib libraries must be built separately and the path must be supplied during
the cmake invocation as shown below. If a `OMEGA_METIS_ROOT` is not supplied,
it is assumed that both METIS and ParMETIS are installed in the same
`OMEGA_PARMETIS_ROOT` location. If a `OMEGA_GKLIB_ROOT` is not supplied, it is
assumed that both GKLIB and METIS are installed in the same `OMEGA_METIS_ROOT`
location. All three libraries should be static libraries.

To utilize a compiler name that’s defined in the E3SM machine configuration

>> cmake \
  -DOMEGA_CIME_COMPILER=nvidiagpu \
  -DOMEGA_CIME_MACHINE=pm-gpu \
  -DOMEGA_PARMETIS_ROOT=/path/to/parmetis \
  ${E3SM_HOME}/components/omega

Once the command completes successfully, several scripts will be created
in the build directory.

omega\_env.sh   : load specific modules and set env. variables read from CIME

omega_env.sh   : load specific modules and set env. variables read from CIME

omega\_build.sh : run `make` command after sourcing `omega_env.sh`

omega\_run.sh   : run `./src/omega.exe` after sourcing `omega_env.sh`

To employ a specific compiler, users can utilize the `OMEGA_CXX_COMPILER`
CMake variable, providing CMake with the compiler's path.

>> cmake \
  -DOMEGA_CXX_COMPILER=CC \

To build Omega for GPU, add OMEGA\_ARCH to one of "CUDA" or "HIP".

To build Omega for GPU, add OMEGA_ARCH to one of “CUDA” or “HIP”.

>> cmake \
  -DOMEGA_ARCH=CUDA \

To enable the ctest-based unittest option, include the `OMEGA_BUILD_TEST`
option as illustrated below.

>> cmake \
  -DOMEGA_BUILD_TEST=ON \

Once the `cmake` command succeeds, the directory where the command was
executed will contain the following files and directories.

>> ls -1 ${WORKDIR}
CMakeCache.txt
CMakeFiles
cmake_install.cmake
CTestTestfile.cmake
external
Makefile
omega_build.sh
omega_ctest.sh
omega_env.sh
omega_run.sh
src
test

To build the Omega, execute the `./omega_build.sh` command in the build directory.

If the build succeeds, the Omega library and executable are created in the
`src` sub-directory of the build directory.

To specify the output directory for saving the generated output files,
a user can include the `OMEGA_INSTALL_PREFIX` variable in the `cmake` command,
as shown below:

cmake \
  ... \
  -DOMEGA_INSTALL_PREFIX="/path/to/output/directory" \
  ...

Afterwards, a user can use the `make install` command to copy the Omega library
and executable to the `lib` and `bin` sub-directories under the directory
specified by `DOMEGA_INSTALL_PREFIX`.

The `./omega_ctest.sh` command runs Omega unit tests. To run the tests, MPI
parallel job launcher such as SLURM srun should be available. You may first
get allocation of an interactive computing node or use batch system.
In addition, you must either copy (or soft link) a mesh file into the same
directory as the unit test executables or specify in a configuration file
(not implemented yet) the path to that mesh input file. For detailed instructions
on how to obtain appropiate test meshes see {ref}`omega-dev-quick-start-getting-meshes`.

For example, if you are on an interactive computing node, you can run
Omega unit test by running `omega_ctest.sh` as shown below.

>> ./omega_ctest.sh
Test project <cmake working directory>
1/1 Test #1: DATA_TYPES_TEST ..................   Passed    0.03 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.04 sec

In the E3SM component build, the compilation is triggered by
the E3SM build system.

Although all the build configurations in the E3SM component build
should ideally be managed through the E3SM build configuration,
the current Omega build is not yet integrated into the E3SM build.
Therefore, users need to make specific modifications to enable
the Omega build within the E3SM build process.

Add the lines indicated as "added" in the `CMakeLists.txt` file.

include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/cmake_util.cmake)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/build_mpas_model.cmake)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/build_omega.cmake) # <= added
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/build_eamxx.cmake)
include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/build_model.cmake)
...
set(BUILDCONF ${CASEROOT}/Buildconf)

build_mpas_models()
build_omega() # <= added

Step 2: Create `${E3SM_ROOT}/components/cmake/build_omega.cmake`

function(build_omega)

  # Set CIME source path relative to components
  set(CIMESRC_PATH "../cime/src")

  add_subdirectory("omega")

endfunction(build_omega)

At this stage, a user can create any E3SM case as usual without
requiring any specific configuration for Omega.

NOTE: In this version, the compiled library file for Omega,
"libOmegaLib.a", is not linked to the E3SM executable. You can
find the Omega library file at `${E3SM_BLD_DIRECTORY}/cmake-bld/omega/src`.

Omega is still in the early stages of development, so we are not yet ready for
outside users.  For now, please refer to the {ref}`omega-dev-quick-start`.

The details and various interfaces to global reduction operations
are described in the [Reductions](#omega-dev-reductions)
section of the Developer's Guide.

The `Tendencies` class provides a container for the [tendency terms](#omega-user-tend-terms) in OMEGA.
Upon creation of an `Tendencies` instance, these functors are initialized and arrays for the
accumulated tendencies are allocated.
There are no user-configurable options beyond those for the tendency term functors.

Tendencies for updating state variables are computed using functors, which are
class objects that can be called like functions. These functors are used within
Kokkos parallel loops to compute the full tendency arrays. The following
tendency terms are currently implemented:

gradient of sea-surface height, multiplied by gravitational acceleration, defined on edges

Among the internal data stored by each functor is a `bool` which can enable or
disable the contribution of that particular term to the tendency. These flags
need to be provided in the configuration file:

Some functors have member variables that can be set by the user by the
configuration file. The following are all the user-configurable parameters for
the currently available tendency terms:

The time manager manages various time quantities during a simulation, including
time steps, time instants, the calendar, clocks and alarms. The user can
specify a few options through the ``TimeIntegration`` portion of the Omega
configuration file. In particular, the user can specify the Calendar used
for tracking simulation time. Supported calendars include:

Julian: earlier calendar still used by Eastern Orthodox traditions

Julian Day: a calendar that simply tracks the number of days since
a reference day

Modified Julian Day: a modified version of Julian Day that changes the
reference time

Custom: A calendar in which a developer can specify the number of
days per month and days per year. This currently cannot be configured
entirely via the configuration and would require additional code to
specify those variables.

No Calendar: This is supplied for test cases in which the calendar
is meaningless and time is simply tracked from time zero.

A number of other time quantities are specified in the ``TimeIntegration``
section (eg time step, start time) and are described in the
[TimeStepping](#omega-user-time-stepping) section.

Time stepper refers to the means by which a simulation is advanced in time.
The configuration of a simulation is set by the ``TimeIntegration`` section
of the Omega configuration file:

  TimeIntegration:
    CalendarType: No Leap
    TimeStepper: Forward-Backward
    TimeStep: 0000_00:10:00
    StartTime: 0001-01-01_00:00:00
    StopTime: 0001-01-01_02:00:00
    RunDuration: none

This configuration refers to the default time stepping used for the model
dynamics (momentum and continuity equations). Additional time steppers can
be used for other portions of the model (eg the barotropic mode or tracer
transport) that may have different time steps and use different algorithms.

The Calendar choice is describe in the
[Time Management](#omega-user-time-manager) section.

The TimeStepper option refers to the numerical scheme used to advance the
model in time. Omega implements a number of time-stepping schemes. The user
can select the scheme they want in the configuration file.
The following time steppers are currently available:

The time step refers to the main model time step used to advance the solution
forward. The format is in ``dddd_hh:mm:ss`` for days, hours, minutes and
seconds.

The StartTime refers to the starting time for the simulation. It is in the
format ``yyyy-mm-day_hh:mm:ss`` for year, month, day, hour, minute, second.
This refers to the initial start time; for a longer simulation, the current
time will be modified by the restart file to update to the present time for
the current segment of the simulation.

The simulation will be stopped either at a fixed StopTime if provided or
by the RunDuration. For shorter simulations, the StopTime can be used to
specify a specific time to stop. For longer simulations with multiple
segments that are restarted, the RunDuration should used and should be set
to fit within the queue time. The format for StopTime is the same as StartTime.
The format for RunDuration is the same as the TimeStep.

Only one of the StopTime or RunDuration should be specified with the other
set to either an empty string or "none". If both are specified, the
RunDuration is used instead of the StopTime.

set to either an empty string or “none”. If both are specified, the

Omega uses the Pacer library for timing the code and incorporates timers around
various parts of the code.

By default, the timing output is written to two files: `omega.summary` and `omega.timing0`.
The `omega.summary` file presents accumulated timing statistics across all MPI ranks.
The `omega.timing.0` show timing result only from the first rank.

 file presents accumulated timing statistics across all MPI ranks.

There are four parameters that are set by the user in the input configuration
file that control the timing behavior. These are:

Timing:
   Level: 2
   AutoFence: true
   TimingBarriers: false
   PrintAllRanks: false

The `Level` parameter is a non-negative integer that determines the granularity of timers.
Increasing it will turn on more timers.
Having more timers provides more detailed information, but it also comes with increased overhead,
and may be counter-productive if a high-level look at model performance is sufficient.

 parameter is a non-negative integer that determines the granularity of timers.

The `Autofence` Boolean option determines if Kokkos fences are automatically added before every timer call.
This option **needs** to be true for accurate timing using Omega timers on GPU-based systems.
However, there are circumstances when turning off automatic fences is useful.
The main use case is using external profiling tools.
Another one is measuring the overhead of automatic synchronization for very high timing levels.

 Boolean option determines if Kokkos fences are automatically added before every timer call.

 to be true for accurate timing using Omega timers on GPU-based systems.

The `TimingLevel` Boolean option determines if MPI barriers are added before or after certain timers.
Adding barriers may be necessary to properly measure communication time, but it can add top much overhead in
production runs.

 Boolean option determines if MPI barriers are added before or after certain timers.

The `PrintAllRanks` Boolean option determines if all ranks should print their timing information. If this
option is set to `true` the output will include additional files `omega.timing.i` with the
timing data from rank `i`.

 Boolean option determines if all ranks should print their timing information. If this

External profilers often include APIs to mark regions of code for detailed profiling.
On some platforms, Omega timers automatically add these annotations.
Currently, this is only implemented on systems with NVIDIA GPUs using NVTX.

This allows, for example, to use the Nsight Compute kernel profiler to obtain
detailed kernel information for all kernels enclosed in the `Tend:computeVelocityTendencies`
Omega timer.

mpirun -np 1 ncu --nvtx --nvtx-include "Tend:computeVelocityTendencies/" omega.exe

Note that not all tracers defined in `TracerDefs.inc` will be used during
a simulation. To select tracers and groups of tracers, users will configure
YAML files in OMEGA, such as `Default.yml` in the `omega/configs` directory,

In the above example, two tracer groups (Base and Debug) are selected. The
Base group includes the `Temp` and `Salt` tracers, while the Debug group
includes `Debug1`, `Debug2`, and `Debug3`.

OMEGA provides two types of solvers for solving many tridiagonal systems of equations
in one go:

Different solver algorithms are used on CPU and GPU platforms.
The optimal solver algorithm for a given platform is chosen automatically.
There are no user-configurable parameters.
The [dev guide](#omega-dev-tridiagonal) provides instructions on how to use the solvers.

In Omega, vertical advection represents the transport of mass, momentum, and
tracer quantities along the vertical coordinate due to vertical motion within a
water column. In the context of Omega's
[governing equations](omega-design-governing-eqns-omega1), vertical advection
contributes to the tendencies of pseudo-thickness, horizontal velocity, and
tracer fields. Vertical motion is inferred from the divergence of horizontal
flow through the continuity equation and is necessary for accurately
representing vertical transport in three-dimensional ocean simulations.

The `VertAdv` class contains variables and functions for computing the vertical
velocity and the tendencies of `LayerThickness`, `NormalVelocity`, and
`Tracers`. The algorithms used are determined by options specified in the

 class contains variables and functions for computing the vertical

. The algorithms used are determined by options specified in the

Within the `Tendencies` group of the configuration file, flags are used to
enable or disable the contribution of vertical advection to the tendencies of
thickness, velocity, and tracers:

Within the `Advection` group, options are provided to configure the algorithms
used to compute the tracer tendencies:

coefficient used for blending 3rd- and 4th-order fluxes when `VerticalTracerFluxOrder == 3`

For `VerticalTracerFluxLimiterEnable`, `true` enables flux-corrected transport,
and `false` selects the standard algorithm. For `Coef3rdOrder`, `1` corresponds
to purely 3rd-order, `0` purely 4th-order.

The `VertAdv` class provides four fields that can be written to output:
`VerticalVelocity`, `TotalVerticalVelocity`, `VertFlux`, and `LowOrderVertFlux`.
These fields collectively make up the `VertAdv` output group.

Omega uses pseudo height, $\tilde{z} = \frac{p}{\rho_0 g}$,  as the vertical coordinate ([V1 governing equation document](omega-design-governing-eqns-omega1)).
In practice, $\tilde{z}$ is not computed directly.
Instead, the model tracks and evolves the pseudo-thickness, $\tilde{h} = \Delta \tilde{z}$, defined as the difference between adjacent layer interfaces.
The pseudo height is essentially a normalized pressure coordinate, with the advantage that it has units of meters.
The `VertCoord` class contains variables and functions relevant to keeping track of:

 class contains variables and functions relevant to keeping track of:

the maximum possible number of layers in a cell, i.e. the extent of the vertical dimension (read in from the mesh file)

the location of active vertical layers (used to set extents of vertical loop bounds):

the max and min indices of active layers in each cell (read in from the mesh file)

the max and min indices of active layers for edges and vertices at the bottom and top of the water column (computed from min/max cell layers)

pressure (computed from the layer thickness and surface pressure)

$z$ height (computed from the bottom depth, specific volume, and layer thickness)

 height (computed from the bottom depth, specific volume, and layer thickness)

desired vertical interface locations (computed from pressure, reference layer pseudo thickness, and user-specified weights)

Multiple instances of the vertical coordinate class can be created and accessed by a unique name.

desired layer thickness based on total perturbation from the reference thickness

weights to specify how total column thickness changes are distributed across layers

reference layer thickness used to distribute total column thickness changes

The vertical coordinate movement can be specified by the `MovementWeightType` option in the configuration file.

The option `Uniform` specifies that perturbations to the total pseudo-thickness of the water column is distributed evenly to all vertical layers.
This is similar to the standard "z-star" coordinate in Boussinesq models.
The option `Fixed` means that total pseudo-thickness perturbations are confined to the top layer, while all others remain constant.
This is similar to the traditional "z" coordinate in Boussinesq models.

 specifies that perturbations to the total pseudo-thickness of the water column is distributed evenly to all vertical layers.

This is similar to the standard “z-star” coordinate in Boussinesq models.

 means that total pseudo-thickness perturbations are confined to the top layer, while all others remain constant.

This is similar to the traditional “z” coordinate in Boussinesq models.

The vertical mixing module in Omega handles the parameterization of unresolved vertical mixing
processes in the ocean. It calculates vertical diffusivity and viscosity coefficients that
determine how properties (like momentum, heat, salt, and biogeochemical tracers) mix vertically
in the ocean model. Currently, Omega offers three different mixing processes within the water column: (1) a constant
background mixing value, (2) a convective instability mixing value, and (3) a Richardson number
dependent shear instability driven mixing value from the [Pacanowski and Philander (1981)](https://journals.ametsoc.org/view/journals/phoc/11/11/1520-0485_1981_011_1443_povmin_2_0_co_2.xml) parameterization. These are linearly additive and are describe a bit
more in detail below. Other mixing processes and parameterizations, such as the the K Profile Parameterization [(KPP; Large et al., 1994)](https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/94rg01872) will be added in the future.

 parameterization. These are linearly additive and are describe a bit

more in detail below. Other mixing processes and parameterizations, such as the the K Profile Parameterization 

The user-configurable options are the following parameters in the yaml configuration file:

VertMix:
    Viscosity: 1.0e-4    # Background vertical viscosity (m²/s)
    Diffusivity: 1.0e-5  # Background vertical diffusivity (m²/s)
    Enable: true         # Enables the convective-induced mixing option
    Diffusivity: 1.0     # Convective mixing coefficient (m²/s)
    TriggerBVF: 0.0      # Squared Brunt-Vaisala frequency threshold
  Shear:
    Enable: true         # Enables the shear-instability driven mixing option
    NuZero: 1.0e-2       # Base viscosity coefficient
    Alpha: 5.0           # Stability parameter
    Exponent: 2          # Richardson number exponent

A constant background mixing value that represents small-scale mixing processes not explicitly resolved by the model. Typically, this is chosen to represent low values of vertical mixing
happening in the ocean's stratified interior.

Enhanced convective adjustment mixing that occurs in statically unstable regions of the water column to parameterize convection and homogenize properties. In Omega this is mixing is defaulted to occur when the squared Brunt Vaisala Frequency is less than 0.0 (unstable), and is off when equal to (neutral) or greater than (stable) 0.0.

\kappa =
+\kappa_b + \kappa_{conv} \quad \text{ if } N^2 < N^2_{crit}\\
+\kappa_b \quad \text{ if } N^2 \geq N^2_{crit}

This is different than some current implementations (i.e. in MPAS-Ocean and the CVMix package), where convective adjustment occurs both with unstable and neutral conditions ($N^2 \leq N^2_{crit}$). $\kappa_{conv}$ and $N^2_{crit}$ are constant parameters set in the `VertMix` section of the yaml file (`Diffusivity` and `TriggerBVF` under the `Convective` header).

This is different than some current implementations (i.e. in MPAS-Ocean and the CVMix package), where convective adjustment occurs both with unstable and neutral conditions (

Mixing induced by vertical velocity shear, implemented using the Pacanowski-Philander scheme, through the gradient Richardson number (ratio of buoyancy to shear).

where $\nu_o$, $\alpha$, $n$, $\nu_b$, $\kappa_b$ are constant parameters set in the `VertMix` section of the yaml file (`NuZero`, `Alpha`, `Exponent` under the `Shear` header and `Viscosity`, `Diffusivity` under the `Background` header). $N^2$ is calculated by the EOS based on the ocean state and $\mathbf{U}$ is the magnitude of the horizontal velocity. $N^2$, $\partial \mathbf{U}}{\partial z}\right|^2$ and $Ri$ of `K` are all defined at the cell center, top interface of layer `K`.  $N^2$, $\nu_{shear}$ and $\kappa_{shear}$ are set to zero for the surface layer. In a later development, the shear mixing option will be changed to the interior shear mixing formulation in [Large et al., 1994](https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/94rg01872).

 are set to zero for the surface layer. In a later development, the shear mixing option will be changed to the interior shear mixing formulation in 

sphinx.ext.apidoc – Generate API documentation from Python packages

Extension internationalization (i18n) and localization (l10n) using i18n API

sphinx.ext.imgconverter – A reference image converter using Imagemagick

sphinx.ext.autosectionlabel – Allow referencing sections by their title

sphinx.ext.napoleon – Support for NumPy and Google style docstrings

polaris conda environment, spack environment, compilers and system modules

f-strings support = for self-documenting expressions and debugging

email.message.Message: Representing an email message using the compat32 API

What is the most efficient way to concatenate many strings together?

Why does a_tuple[i] += [‘item’] raise an exception when the addition works?

What does the slash(/) in the parameter list of a function mean?

Why am I getting an UnboundLocalError when the variable has a value?

PEP 3118: New memoryview implementation and buffer protocol documentation

PEP 476: Enabling certificate verification by default for stdlib http clients

PEP 471 - os.scandir() function – a better and faster directory iterator

PEP 655: Marking individual TypedDict items as required or not-required

Why must ‘self’ be used explicitly in method definitions and calls?

argparse — Parser for command-line options, arguments and subcommands

compression.zstd — Compression compatible with the Zstandard format

concurrent.interpreters — Multiple interpreters in the same process

functools — Higher-order functions and operations on callable objects

importlib.resources – Package resource reading, opening and access

multiprocessing.shared_memory — Shared memory for direct access across processes

sysconfig — Provide access to Python’s configuration information

Creating artists specifically for adding to the legend (aka. Proxy artists)

boxplot and bxp vert parameter, and rcParams["boxplot.vertical"]

MEP15: Fix axis autoscaling when limits are specified for one axis only

Gumbel (LogWeibull, Fisher-Tippetts, Type I Extreme Value) Distribution

Laplace (Double Exponential, Bilateral Exponential) Distribution

Chi-square test of independence of variables in a contingency table

Multivariate data interpolation on a regular grid  (RegularGridInterpolator)

How do xarray data structures differ from those found in pandas?