Zarr output

What is Zarr format?

"Zarr is a relatively new cloud-based data format specifically for improving access to N-Dimensional arrays. It is an effective way to store large N-dimensional data in the cloud and access the data in predefined chunks. Zarr can be viewed as the cloud based version of HDF5/NetCDF files as it follows a similar data model. Zarr does not come in a single file as NetCDF or HDF5 does but as a directory with chunks of data in compressed binary files and metadata describing the binary content in external JSON files." (EARTHDATA ESCO Emerging Standards)

Netcdf vs. Zarr


References

Tutorials

Xarray and Zarr (from xarray website)

"In implementing support for the Zarr storage format, Xarray developers made some ad hoc choices about how to store NetCDF data in Zarr. ..."

"First, Xarray can only read and write Zarr groups. There is currently no support for reading / writing individual Zarr arrays. Zarr groups are mapped to Xarray Dataset objects.

Second, from Xarray’s point of view, the key difference between NetCDF and Zarr is that all NetCDF arrays have dimension names while Zarr arrays do not. Therefore, in order to store NetCDF data in Zarr, Xarray must somehow encode and decode the name of each array’s dimensions. ..."

"... Because of these choices, Xarray cannot read arbitrary array data, but only Zarr data with valid _ARRAY_DIMENSIONS or NCZarr attributes on each array (NCZarr dimension names are defined in the .zarray file). ... "

Zarr Encoding Specification

PINACLES Variables in zarr files

The scalar, velocity, and diagnostic variables are saved in zarr format at the frequency specified in the input json file as

 "fields": {
        "frequency": 600.0,
        "io_type": "zarr"
    },

The current RCE configuration saves the following variables (listed in the .zattrs json file under the fields.zarr output directory):

{
     "ScalarState_long_names": {
        "qc": "cloud water mixing ratio",
        "qi1": "total ice mixing ratio",
        "qib1": "ice rime volume mixing ratio",
        "qir1": "rime ice mixing ratio",
        "qnc": "cloud number concentration",
        "qni1": "ice number concentration",
        "qnr": "rain number concentration",
        "qr": "rain water mixing ratio",
        "qv": "water vapor mixing ratio",
        "s": "static energy"
    },
     "VelocityState_long_names": {
        "u": "u velocity component",
        "v": "v velocity component",
        "w": "w velocity component"
    },
     "DiagnosticState_long_names": {
        "Q_criterion": "Q criterion",
        "T": "Temperature",
        "alpha": "Specific Volume",
        "buoyancy": "buoyancy",
        "buoyancy_gradient_mag": "buoyancy",
        "bvf": "Brunt\u2013V\u00e4is\u00e4l\u00e4 frequency squared",
        "dTdt_rad": "None",
        "diag_effc_3d": "cloud droplet effective radius",
        "diag_effi_3d": "cloud ice effective radius",
        "dynamic pressure": "Dynamic Pressure",
        "eddy_diffusivity_horizontal": "Horizontal Eddy Diffusivity",
        "eddy_diffusivity_vertical": "Vertical Eddy Diffusivity",
        "eddy_viscosity_horizontal": "Horizontal Eddy Viscosity",
        "eddy_viscosity_vertical": "Vertical Eddy Viscosity",
        "grad_ri": "gradient Richardson",
        "heating_rate_lw": "None",
        "heating_rate_sw": "None",
        "helicity": "helicity",
        "horizontal divergence": "Horizontal Divergence",
        "ice_sed": "None",
        "liq_sed": "None",
        "p3_ncacc": "change in cloud droplet number from accretion by rain",
        "p3_ncautc": "change in cloud droplet number from autoconversion",
        "p3_ncautr": "change in rain number from autoconversion of cloud water",
        "p3_ncnuc": "change in cloud droplet number from activation of CCN",
        "p3_ncslf": "change in cloud droplet number from self-collection",
        "p3_nrevp": "change in rain number from evaporation",
        "p3_nrslf": "change in rain number from self-collection",
        "p3_qcacc": "cloud droplet accretion by rain",
        "p3_qcaut": "cloud droplet autoconversion to rain",
        "p3_qccon": "cloud droplet condensation",
        "p3_qcevp": "cloud droplet evaporation",
        "p3_qcnuc": "activation of cloud droplets from CCN",
        "p3_qrcon": "rain condensation",
        "p3_qrevp": "rain evaporation",
        "p_hydrostatic": "hydro_static_pressure",
        "qt": "Total water specific humidity",
        "reflectivity": "radar reflectivity",
        "rh": "relative humidity",
        "s_dry": "Dry Static Energy",
        "s_tend_ice_sed": "s tend ice water sedimentation",
        "s_tend_liq_sed": "s tend liquid water sedimentation",
        "strain_rate_mag": "Magnitude of strain rate tensor",
        "thetal": "Liquid-Ice Potential Temperatue",
        "thetav": "Virtual Potential Temperature",
        "tke_sgs": "Subgrid-scale turbulence kinetic energy",
        "uw_sgs": "Vertical SGS flux of u velocity",
        "vertical_vorticity": "Vertical Component of Vorticity",
        "vw_sgs": "Vertical SGS flux of v velocity"
    },
}

Dimensions and other file characteristics

Group attribute Values
Data type float64
Dimension (time, variable, x, y, z)
Chunk shape (1, nvar, nx_chun, ny_chunk, nz)
Order C