cdo cmor

Blue beam: Explanation slides

Red beam: Contains instructions and executable code

Entry words

Follow this Presentation live via

http://slides.com/wachsylon/cdo-cmor-handson-principles/live

or find all slides under

http://slides.com/wachsylon/cdo-cmor-handson-principles#/

where you can

scroll with arrow keys through the presentation
post comments and ask questions
(or to wachsmann AT dkrz.de)
find other presentations

Entry words

This presentation focuses on producing CORDEX compliant data since running experiments for PRINCIPLES are using CMIP5 data. If you want to generate CMIP6/CORDEX2 data instead, you will have to

use the correct installation
substitute all CORDEX input with CMIP6 input

Remind me at the corresponding point if you plan to do this.

The operator is not yet fully functional for all axis types, but for the most common ones. To guarantee the correct processing with CMOR, it needs example data for the missing types.

cdo cmor

Introduction

CMIP standard

CDO

Project defines a data standard

Program to create the data standard

Collection of operators to process climate data

Definitions and motivation

"Systematic analysis across models only easy to do if model output is written in

a common format
with files structured similary
and with sufficient metadata uniformly stored

*Coupled Model Intercomparison Project

*Climate Model Output Rewriter

*Climate Data Operators

CMOR

CMIP standard

General requirements on CORDEX compliant data

netCDF4 Format
conform to CF 1.4
Each file must contain only a single output field from a single simulation including coordinates and additional meta data

Detailled requirements for the output design of CORDEX can be found in the document cordex_archive_design

More information about global attributes, controlled vocabulary and filenames can be found in the document

CMIP6_global_attributes_filenames_CVs from Taylor et. al (2017)

CMIP standard

# Get the strucutre of header of a example raw model output:
ncdump -h example_interface.nc 
# results:
netcdf example_interface {
dimensions:
	time = UNLIMITED ; // (12 currently)
	lon = 384 ;
	lat = 192 ;
	height_2 = 1 ;
	height = 1 ;
variables:
	double time(time) ;
		time:standard_name = "time" ;
		time:units = "day as %Y%m%d.%f" ;
		time:calendar = "proleptic_gregorian" ;
		time:axis = "T" ;
	float lon(lon) ;
		lon:standard_name = "longitude" ;
		lon:long_name = "longitude" ;
		lon:units = "degrees_east" ;
		lon:axis = "X" ;
	float lat(lat) ;
		lat:standard_name = "latitude" ;
		lat:long_name = "latitude" ;
		lat:units = "degrees_north" ;
		lat:axis = "Y" ;
	double height_2(height_2) ;
		height_2:standard_name = "height" ;
		height_2:long_name = "height" ;
		height_2:units = "m" ;
		height_2:positive = "up" ;
		height_2:axis = "Z" ;
	double height(height) ;
		height:standard_name = "height" ;
		height:long_name = "height" ;
		height:units = "m" ;
		height:positive = "up" ;
		height:axis = "Z" ;
	float uas(time, height_2, lat, lon) ;
		uas:units = "m s-1" ;
		uas:code = 1 ;
		uas:table = 255 ;
	float tas(time, height, lat, lon) ;
		tas:units = "K" ;
		tas:code = 1 ;
		tas:table = 255 ;

// global attributes:
		:CDI = "Climate Data Interface version 1.9.0 (http://mpimet.mpg.de/cdi)" ;
		:Conventions = "CF-1.6" ;
		:history = "Tue Sep 12 10:30:24 2017: cdo_recent_cmor2 merge testuas.nc testtas.nc test.nc\n",
			"Tue Sep 12 10:30:01 2017: cdo_recent_cmor2 -setunit,m s-1 -selname,uas example_interface.nc testuas.nc\n",
			"Tue Sep 12 10:03:02 2017: cdo_recent_cmor2 merge test.nc test2.nc test3.nc\n",
			"Tue Sep 12 10:02:45 2017: cdo_recent_cmor2 -f nc setname,tas test.grb test.nc" ;
		:institution = "Max-Planck-Institute for Meteorology" ;
		:CDO = "Climate Data Operators version 1.9.1rc1 (http://mpimet.mpg.de/cdo)" ;
}

# Get the strucutre of a header of a CMIP6 compliant file:
ncdump -h .//CMIP6/CMIP/MPI-M/MPI-ESM-1-2-HR/historical/r1i1p1f1/Amon/tas/gn/v20180109/tas_Amon_MPI-ESM-1-2-HR_historical_r1i1p1f1_gn_185001-185012.nc
# results:
netcdf tas_Amon_MPI-ESM-1-2-HR_historical_r1i1p1f1_gn_185001-185012 {
dimensions:
	time = UNLIMITED ; // (12 currently)
	lat = 192 ;
	lon = 384 ;
	bnds = 2 ;
variables:
	double time(time) ;
		time:bounds = "time_bnds" ;
		time:units = "days since 1850-1-1 00:00:00" ;
		time:calendar = "proleptic_gregorian" ;
		time:axis = "T" ;
		time:long_name = "time" ;
		time:standard_name = "time" ;
	double time_bnds(time, bnds) ;
	double lat(lat) ;
		lat:bounds = "lat_bnds" ;
		lat:units = "degrees_north" ;
		lat:axis = "Y" ;
		lat:long_name = "latitude" ;
		lat:standard_name = "latitude" ;
	double lat_bnds(lat, bnds) ;
	double lon(lon) ;
		lon:bounds = "lon_bnds" ;
		lon:units = "degrees_east" ;
		lon:axis = "X" ;
		lon:long_name = "longitude" ;
		lon:standard_name = "longitude" ;
	double lon_bnds(lon, bnds) ;
	double height ;
		height:units = "m" ;
		height:axis = "Z" ;
		height:positive = "up" ;
		height:long_name = "height" ;
		height:standard_name = "height" ;
	float tas(time, lat, lon) ;
		tas:standard_name = "air_temperature" ;
		tas:long_name = "Near-Surface Air Temperature" ;
		tas:comment = "near-surface (usually, 2 meter) air temperature" ;
		tas:units = "K" ;
		tas:cell_methods = "area: time: mean" ;
		tas:cell_measures = "area: areacella" ;
		tas:history = "2018-01-09T13:40:08Z altered by CMOR: Treated scalar dimension: \'height\'. 2018-01-09T13:40:08Z altered by CMOR: replaced missing value flag (-9e+33) with standard missing value (1e+20)." ;
		tas:coordinates = "height" ;
		tas:missing_value = 1.e+20f ;
		tas:_FillValue = 1.e+20f ;

// global attributes:
		:Conventions = "CF-1.7 CMIP-6.2" ;
		:activity_id = "CMIP" ;
		:branch_method = "standard" ;
		:branch_time_in_child = 365. ;
		:branch_time_in_parent = 365. ;
		:contact = "cmip6-mpi-esm@dkrz.de" ;
		:creation_date = "2018-01-09T13:40:08Z" ;
		:data_specs_version = "01.00.16" ;
		:experiment = "all-forcing simulation of the recent past" ;
		:experiment_id = "historical" ;
		:external_variables = "areacella" ;
		:forcing = "GHG Oz SD Sl Vl LU" ;
		:forcing_index = 1 ;
		:frequency = "mon" ;
		:further_info_url = "https://furtherinfo.es-doc.org/coolness" ;
		:grid = "gn" ;
		:grid_label = "gn" ;
		:history = "2018-01-09T13:40:08Z ; CMOR rewrote data to be consistent with CMIP6, CF-1.7 CMIP-6.2 and CF standards.;\n",
			"Model raw output postprocessed with modelling environment (IMDI) at DKRZ: URL: :https://svn.dkrz.de/mad/Model/IMDI/trunk" ;
		:initialization_index = 1 ;
		:institute_id = "MPI-M" ;
		:institution = "Max Planck Institute for Meteorology, Hamburg 20146, Germany" ;
		:institution_id = "MPI-M" ;
		:mip_era = "CMIP6" ;
		:model_id = "MPI-ESM-1-2-HR" ;
		:nominal_resolution = "100 km" ;
		:parent_activity_id = "CMIP" ;
		:parent_experiment_id = "piControl" ;
		:parent_experiment_rip = "N/A" ;
		:parent_mip_era = "CMIP6" ;
		:parent_source_id = "MPI-ESM-1-2-HR" ;
		:parent_time_units = "days since 1850-1-1 00:00:00" ;
		:parent_variant_label = "r1i1p1f1" ;
		:physics_index = 1 ;
		:product = "output" ;
		:project_id = "CMIP6" ;
		:realization_index = 1 ;
		:realm = "atmos" ;
		:references = "ECHAM6: Stevens, B., et al. (2013), Atmospheric component of the MPI-M Earth system model: ECHAM6, J. Adv. Model. Earth Syst., 5, 146–172, doi:10.1002/jame.20015. JSBACH: Reick, C. H., T. Raddatz, V. Brovkin, and V. Gayler (2013), The representation of natural and anthropogenic land cover change in MPIESM, J. Adv. Model. Earth Syst., 5, 1–24, doi:10.1002/jame.20022" ;
		:source = "MPI-ESM1.2-HR (2017): \n",
			"aerosol: none, prescribed MACv2-SP\n",
			"atmos: ECHAM6.3 (spectral T127; 384 x 192 longitude/latitude; 95 levels; top level 0.01 hPa)\n",
			"atmosChem: none\n",
			"land: JSBACH3.20\n",
			"landIce: none/prescribed\n",
			"ocean: MPIOM1.63 (tripolar TP04, approximately 0.4deg; 802 x 404 longitude/latitude; 40 levels; top grid cell 0-12 m)\n",
			"ocnBgchem: HAMOCC\n",
			"seaIce: unnamed (thermodynamic (Semtner zero-layer) dynamic (Hibler 79) sea ice model)" ;
		:source_id = "MPI-ESM-1-2-HR" ;
		:source_type = "AOGCM" ;
		:sub_experiment = "none" ;
		:sub_experiment_id = "none" ;
		:table_id = "Amon" ;
		:table_info = "Creation Date:(08 November 2017) MD5:0e73e044458fa79166d97d5cf546d0be" ;
		:title = "MPI-ESM-1-2-HR output prepared for CMIP6" ;
		:variable_id = "tas" ;
		:variant_label = "r1i1p1f1" ;
		:license = "CMIP6 model data produced by DKRZ is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law." ;
		:cmor_version = "3.2.8" ;
		:tracking_id = "hdl:21.14100/3b2ffff9-68d2-47e8-8c2c-c05711cde5da" ;
}

CMIP6 standard violations in example raw model output:

File (and path) name
Contains more than one variable
No bounds for lats and lons
No attributes for coordinate variables and target variables
No required global attributes
Scalar dimensions

CMIP standard

Differences between CMIP5 and CMIP6:

The Data request!
Filenames and directory structures templates have been modified (see next slides)
A few CMIP5 global attributes have been renamed.
A number of additional global attributes are now required.
The controlled vocabularies relied on in defining some of the attributes have been modified.

#CMIP5
institute_id="MPI-M"
institution="Max Planck Institute for Meteorology"
#member=r1i1p1

#CMIP6
institution_id="MPI-M"
institution="Max Planck Institute for Meteorology, Hamburg 20146, Germany"
variant_label=r1i1p1f1

CMOR

CMOR ensures that ouput is CMIP compliant.
Different CMIP standards can be produced
Use synergies, avoid repeating wrok

Why using CMOR to create CMIP standard?
Why integrate CDO with CMOR?

No user side preparation of CMIP format description

CMOR

CDO is a well known tool with an active support
The CDO's interface allows
- Different infile formats
- enables access to all infile information no matter how structured

Use the power of CDOs...

Why using CMOR to create CMIP standard?
Why integrate CDO with CMOR?

CMOR

... to simplify CMOR usage:

Why using CMOR to create CMIP standard?
Why integrate CDO with CMOR?

cmor_setup();
cmor_dataset_json();
cmor_load_table();
cmor_set_table();
cmor_axis();
cmor_grid();
cmor_set_grid_mapping();
cmor_time_varying_grid_coordinate();
cmor_zfactor();
cmor_variable();
cmor_set_deflate();
cmor_set_variable_attribute();
cmor_create_output_path();
cmor_write();
cmor_close();

are included by one cdo cmor operator

cdo cmor,Amon,\
         i=info_table.txt,\
        gi=grid_info.nc,\
        mt=mapping_table.txt\
                             infile

"variable_entry":{
  "ccb":  {
    "frequency": "mon",
    "modeling_realm":"atmos"
    }
  }

CMIP6_Amon.json

grid_
info.nc

mapping_
table.txt

ncdump -h mrsofc_fx_MPIESM-1-2-HR_historical_r1i1p1f1_gn.nc
#stdout:
netcdf test {
dimensions:
	lat = 192 ;
	lon = 384 ;
	bnds = 2 ;
variables:
        double lat(lat);
        double lat_bnds(lat,bnds);
        double lon(lon);
        double lon_bnds(lon,bnds);

&parameter cmor_name=tasmax      code=201 units="K"          cell_methods="m" project_mip_table=Amon /
&parameter cmor_name=tasmax      code=201 units="K"          cell_methods="m" project_mip_table=day /
&parameter cmor_name=tasmin      code=202 units="K"          cell_methods="m" project_mip_table=day  /
&parameter cmor_name=tasmin      code=202 units="K"          cell_methods="m" project_mip_table=Amon /
&parameter cmor_name=rsds        units="W m-2"      cell_methods="m" positive="d" /
&parameter cmor_name=rlds        units="W m-2"      cell_methods="m" positive="d" /

info_
table.txt

activity_id="CMIP"
experiment="AMIP"
experiment_id="amip"
forcing_index="1"
grid="All grid attributes are set for the native grid and based on information from attribute source."
grid_label="gn"
initialization_index="1"
institution="Max Planck Institute for Meteorology, Hamburg 20146, Germany"
institution_id="MPI-M"
license="CMIP6 model data produced by REQUIRED is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). Consult https://pcmdi.llnl.gov/CMIP6/TermsOfUse for terms of use governing CMIP6 output, including citation requirements and proper acknowledgment. Further information about this data, including some limitations, can be found via the further_info_url (recorded as a global attribute in this file) and. The data producers and data providers make no warranty, either express or implied, including, but not limited to, warranties of merchantability and fitness for a particular purpose. All liabilities arising from the supply of the information (including any liability arising in negligence) are excluded to the fullest extent permitted by law."
mip_era="CMIP6"
nominal_resolution="100 km"
physics_index="1"
realization_index="1"
source="MPIESM1.2-HR (2017): \naerosol: none, prescribed MACv2-SP\natmos: ECHAM6.3 (spectral T127; 384 x 192 longitude/latitude; 95 levels; top level 0.01 hPa)\natmosChem: none\nland: JSBACH3.2\nlandIce: none/prescribed\nocean: MPIOM1.63 (tripolar TP04, approximately 0.4deg; 900 x 450 longitude/latitude; 40 levels; top grid cell 0-12 m)\nocnBgchem: HAMOCC\nseaIce: unnamed (thermodynamic (Semtner zero-layer) dynamic (Hibler 79) sea ice model)"
source_id="MPIESM-1-2-HR"
source_type="AGCM"
sub_experiment="none"
sub_experiment_id="none"
parent_activity_id="no parent"
parent_experiment_id="no parent"
calendar="proleptic_gregorian"
required_time_units="days since REQUIRED"

cdo cmor

Installation

Debian CDO (sudo apt-get install cdo) is installed without CMOR. Therefore, an individual installation (static) is required if 'cdo cmor' shall be used on a local computer.
Great efforts are being made on providing a conda-installation simplifying the installation of CDO with CMOR however it still takes time
We recommend to work on DKRZ system mistral where various CDO versions are available that are installed with different CMOR versions.

Installation

ls -1 /work/bm0021/cdo_incl_cmor/
cdo_01_25_2018_cmor2
cdo_01_25_2018_cmor3
cdo_01_25_2018_cmor3_gcc
cdo_01_25_2018_cmor3_intel
cdo_03-31-2017
cdo_06_14_2017_cmor2
cdo_06_14_2017_cmor3
cdo_08_30_2017_cmor2
cdo_08_30_2017_cmor3
cdo_11_15_2017_cmor2
cdo_11_15_2017_cmor3

CDO 1.9.4 can be installed locally with CMOR (see script on next slide)
The version of the integrated CMOR package is essential and determines the data standard the user will achieve
Therefore, before the installation, the user needs to know which standard shall be generated and which CMOR version shall be integrated accordingly.

CMOR2

is able to create standard of CMIP5 and CORDEX

CMOR3

is able to create standard of CMIP6 and CORDEX2

Uses JSON files internally

Has predefined function cmor_dataset to pass input meta data

Installation

technical issues (not to be mistaken for CMIP standard issues)

#!/bin/sh
# INSTALL SCRIPT FOR CDO WITH SUPPORT FOR NETCDF, GRIB2, HDF5 AND CMOR on a LINUX platform
# based on the instructions by Mithil Shah (http://www.studytrails.com/blog/install-climate-data-operator-cdo-with-netcdf-grib2-and-hdf5-support/)
# Author: Fabian Wachsmann
#
#You might need to install a fortran compiler, C compiler and a macro environment:
#sudo apt-get install gfortran
#sudo apt-get install gcc
#sudo apt-get install m4
#
cd ../libs/
INSTALLDIR=$(pwd)

#Install CDO with support:
#ZLIB, HDF5, EXPAT, UDUNITS, UUID, NETCDF, JASPER, GRIB_API, CMOR
zlib=Y
hdf5=Y
expat=Y
udunits=Y
uuid=Y
netcdf=Y
jasper=Y
gribapi=Y
cmor=Y

#ZLIB 1.2.8:
#for unzip and zipping:
zlib_link="http://www.zlib.net/fossils/zlib-1.2.8.tar.gz"
#HDF5 1.8.12
#data model used by netCDF allowing infinity file sizes
hdf5_link="https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8/hdf5-1.8.13/src/hdf5-1.8.13.tar.gz"
#EXPAT 2.2.2
#xml parser (used by cmor2)
expat_link="https://github.com/libexpat/libexpat/releases/download/R_2_2_2/expat-2.2.2.tar.bz2"
#UDUNITS 2.2.25
#changes units of variables
udunits_link="ftp://ftp.unidata.ucar.edu/pub/udunits/udunits-2.2.25.tar.gz"
#UUID 1.6.2
#creates universal unique identifier
uuid_link="https://launchpad.net/ubuntu/+archive/primary/+files/ossp-uuid_1.6.2.orig.tar.gz"
#NETCDF 4.4.1
#data format of climate data
netcdf_link="https://github.com/Unidata/netcdf-c/archive/v4.4.1.tar.gz"
#JASPER 1.9.1
#jpeg compressor
jasper_link="http://www.ece.uvic.ca/~mdadams/jasper/software/jasper-1.900.1.zip"
#GRIB_API 1.14.4
#data format of climate data
gribapi_link="https://software.ecmwf.int/wiki/download/attachments/3473437/grib_api-1.14.4-Source.tar.gz?api=v2"

if [ "$zlib" = "Y" ]; then
#ZLIB 1.2.8
#wget ${zlib_link} &&
tar xf zlib* || exit 1
cd zlib* || exit 1
./configure --prefix=${INSTALLDIR} || exit 1
make && make check && make install || exit 1
cd ..
fi

if [ "$hdf5" = "Y" ]; then
#HDF5 1.8.12
#wget --no-check-certificate ${hdf5_link} &&
tar xf hdf5* || exit 1
cd hdf5* || exit 1
./configure --with-zlib=${INSTALLDIR} --prefix=${INSTALLDIR} --enable-hl CFLAGS=-fPIC || exit 1
make && make check || exit 1
make install || exit 1
cd ..
fi

if [ "$expat" = "Y" ]; then
#EXPAT 2.2.2
#wget --no-check-certificate ${expat_link} && 
tar xf expat* || exit 1
cd expat* || exit 1
./configure --prefix=${INSTALLDIR} CFLAGS=-fPIC || exit 1
make && make check && make install || exit 1
cd ..
fi

if [ "$udunits" = "Y" ]; then
#UDUNITS 2.2.25
#wget ${udunits_link} && 
tar xf udunits* || exit 1
cd udunits* || exit 1
CPPFLAGS=-I${INSTALLDIR}/include LDFLAGS=-L${INSTALLDIR}/lib ./configure --prefix=${INSTALLDIR} CFLAGS=-fPIC || exit 1
make && make check && make install || exit 1
cd ..
fi

if [ "$uuid" = "Y" ]; then
#UUID 1.6.2
#wget --no-check-certificate ${uuid_link} && 
tar xf ossp-uuid* || exit 1
cd uuid*
./configure --prefix=${INSTALLDIR} CFLAGS=-fPIC || exit 1
make && make check && make install || exit 1
cd ..
fi

if [ "$netcdf" = "Y" ]; then
#NETCDF 4.4.1
#wget --no-check-certificate ${netcdf_link} && 
tar xf v4* || exit 1
cd net*
CPPFLAGS=-I${INSTALLDIR}/include LDFLAGS=-L${INSTALLDIR}/lib ./configure --prefix=${INSTALLDIR} --enable-netcdf-4 CFLAGS=-fPIC || exit 1
make && make check && make install || exit 1
cd ..
fi

if [ "$jasper" = "Y" ]; then
#JASPER 1.9.1
#wget ${jasper_link} && 
unzip jasper* || exit 1
cd jasper*
./configure --prefix=${INSTALLDIR} CFLAGS=-fPIC || exit 1
make && make check && make install || exit 1
cd ..
fi

if [ "$gribapi" = "Y" ]; then
#GRIB_API 1.14.4
#wget --no-check-certificate ${gribapi_link} && 
tar xf grib* || exit 1
cd grib*
./configure --prefix=${INSTALLDIR} CFLAGS=-fPIC --with-netcdf=${INSTALLDIR} --with-jasper=${INSTALLDIR} || exit 1
make && make install || exit 1
cd ..
fi

#if [ "$cmor" = "Y" ]; then
#CMOR 3.3.1.
#cd cmor3_v331/ || exit 1
#CFLAGS=-fPIC CPPFLAGS=-I${INSTALLDIR}/include LDFLAGS=-L${INSTALLDIR}/lib ./configure --prefix=${INSTALLDIR} --with-udunits2=${INSTALLDIR} --with-uuid=${INSTALLDIR} --with-netcdf=${INSTALLDIR} || exit 1
#make && make install || exit 1
#cd ${INSTALLDIR}/../cdo/
#fi

if [ "$cmor" = "Y" ]; then
#CMOR 2.9.2
cd cmor2_v292/ || exit 1
F77=gcc CC=gcc CFLAGS=-fPIC CPPFLAGS=-I${INSTALLDIR}/include LDFLAGS=-L${INSTALLDIR}/lib ./configure --prefix=${INSTALLDIR} --with-uuid=${INSTALLDIR} --with-udunits2=${INSTALLDIR} --with-netcdf=${INSTALLDIR}
make install
cd ..
fi

cd ../cdo
echo "CDO 1.9.5rc1";
#CDO 1.9.5rc1
tar xf cdo-1.9.5rc1.tar.gz || exit 1
cd cdo-1.9.5rc1|| exit 1  
make distclean
CPPFLAGS="-I${INSTALLDIR}/include -I${INSTALLDIR}/local/include -I${INSTALLDIR}/local/include/json-c -I${INSTALLDIR}/local/include/cdTime" OPENMP=0 ./configure --prefix=$(pwd)/ --with-cmor=${INSTALLDIR} --with-hdf5=${INSTALLDIR} --with-netcdf=${INSTALLDIR} --with-udunits2=${INSTALLDIR} --with-grib_api=${INSTALLDIR} --with-ossp-uuid=${INSTALLDIR} || exit 1
                                                                                                                               
make -j8 && make install                                                                                                                                                                                                
cd ..

Installation

cdo cmor

HandsOn preparation
Usage
First steps

#connect to mistral
ssh -X <usernr>@mistral.dkrz.de

#create workdir
mkdir cdocmor_handson
cd cdocmor_handson

#copy examples directory
cdoDir=/work/bm0021/cdo_incl_cmor/
cp -r ${cdoDir}examples ./
cd examples

#link cdo cmor installation
alias cdo=${cdoDir}cdo_handson_cmor2

Preparation

#**General usage:
#cdo cmor,MIP-table[,keyvaluelist] infile

#**The keyvaluelist is of format
#key=value[,value]

#**Some keywords only allow exactly one value.
#**Others allow a comma separated list (CSL) of values.

#**If the input file is of netCDF format and contains all required meta data,
#**it can be sufficient to call:
#cdo cmor,MIP-table infile

#**example:
cdo cmor,cordex_mip_tables/CORDEX_mon example_interface.nc

#**Parameter one MUST be the MIP table
#**It can be specified with a relative or absoulte path

Usage

#example:
cdo cmor,cordex_mip_tables/CORDEX_mon example_interface.nc

Usage

The outfile is generated by CMOR and must not be specified.

Name and path of the outfile are generated according to templates defined by the corresponding project:

	Filename	Filepath
CMOR2
CMOR3

<variable_id>_<table_id>_
<source_id>_<experiment_id>_
<variant_label>_<grid_label>

<variable_id>_<table_id>_
<model_id>_<experiment_id>_
<member>

<activity_id>/<product>/<institute_id>/
<model_id>/<experiment_id>/<frequency>/
<realm>/<variable_id>/<member>

<mip_er>/<activity_id>/<institution_id>/<source_id>/
<experiment_id>/<variant_label>/<table_id>/
<variable_id>/<grid_label>/<version>

First steps

#example:
#select tas from infile
cdo cmor,cordex_mip_tables/CORDEX_mon,cn=tas example_interface.nc
#select tas and uas from infile
cdo cmor,cordex_mip_tables/CORDEX_mon,cn=tas,uas example_interface.nc

Parameter one must be the MIP-table
The outfile is generated by CMOR and must not be specified
Keyword cmor_name selects a subset of variables to be processed
All keys have short forms, e.g. cn for cmor_name

#example:
cdo cmor,cordex_mip_tables/CORDEX_mon example_interface.nc

#example:
cdo cmor,cordex_mip_tables/CORDEX_mon example_interface.nc

First steps

cdo cmor,cordex_mip_tables/CORDEX_mon,cn=tas,uas example_interface.nc 
cdo_handson_cmor2 cmor: 2.2. Start to find a MIP table file.
cdo_handson_cmor2 cmor: 2.2. MIP table file 'cordex_mip_tables/CORDEX_mon' exists in MIP table directory 'cordex_mip_tables/'.
cdo_handson_cmor2 cmor: Function 'attribute check, compare and set':
          Be aware of differences between infile and user specification.
          Attribute 'calendar' in input file: 'proleptic_gregorian' does not agree with user specification 'noleap'.
          Cmor library is called with 'noleap'.
cdo_handson_cmor2 cmor: Function 'attribute check, compare and set':
          Be aware of differences between infile and user specification.
          Attribute 'time_units' in input file: 'days since 1850-1-16 12:00:00' does not agree with user specification 'days since 1949-12-01T00:00:00Z'.
          Cmor library is called with 'days since 1949-12-01T00:00:00Z'.
cdo_handson_cmor2 cmor: Vertical axis is either default and scalar or not available.
cdo_handson_cmor2 cmor: Vertical axis is either default and scalar or not available.
cdo_handson_cmor2 cmor (Warning): Chunk description file 'CHUNK_FILE_uas_mon_historical_r1i1p1.txt' could not be opened.
Switched to replace mode.
cdo_handson_cmor2 cmor (Warning): In checking the last chunk:
          Chunk './/tas_mon_CLMcom-CCLM4-8-17_historical_r1i1p1_185001-185012.nc' configured via chunk description file does not exist.
          Switched to replace mode for this variable.

cdo_handson_cmor2 cmor:      File stored in:  './/CORDEX/output/EUR-11/CLMcom/CCCma-CanESM2/historical/r1i1p1/CLMcom-CCLM4-8-17/v1/mon/.//uas/.//uas_EUR-11_CCCma-CanESM2_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_mon_185001-185012.nc' with cmor!
cdo_handson_cmor2 cmor:      File stored in:  './/CORDEX/output/EUR-11/CLMcom/CCCma-CanESM2/historical/r1i1p1/CLMcom-CCLM4-8-17/v1/mon/.//tas/.//tas_EUR-11_CCCma-CanESM2_historical_r1i1p1_CLMcom-CCLM4-8-17_v1_mon_185001-185012.nc' with cmor!
cdo_handson_cmor2 cmor: Processed 1769472 values from 10 variables over 12 timesteps [0.61s 441MB]

#The usual CDO options are valid for this operator as well:
#cdo cmor can be called in silent (-s) or verbose (-v) mode.
#The verbose mode is recommended if the processing is unclear:

cdo -v cmor,cordex_mip_tables/CORDEX_mon example_interface.nc
cdo -s cmor,cordex_mip_tables/CORDEX_mon example_interface.nc

#In many cases, operator chaining can be helpful:

cdo -cmor,cordex_mip_tables/CORDEX_mon -addc,10 example_interface.nc

First steps

cdo cmor

Configuration and data streams

Infile

CDO

CMOR

Outfile

replace

append

variables (attributes)

coordinates (attributes)

global
(attributes)

mapping
table

grid info
file

info
table

MIP-

table

predefinitons

user setting

data stream

cdo

cmor

mapping
table

grid info
file

info
table

MIP-
table

infile

data

new file

data

old file

data

command

replace

append

line

metadata stream

data stream

cdo cmor

Project standard

variables (attributes)

coordinates (attributes)

global
(attributes)

MIP-

table

predefinitons

user setting

data stream

MIP-

table

A MIP-table is CMOR-readable and contains the
This standard is expressed by header, axis entries and variable entries

cdo cmor,cordex_mip_tables/CORDEX_mon example_interface.nc

CMIP standard

ls cordex_mip_tables
CORDEX_3h
CORDEX_6h
CORDEX_day
CORDEX_fx
CORDEX_grids
CORDEX_mon
CORDEX_sem

table_id: Table mon
modeling_realm: atmos

frequency: mon

cmor_version: 2.6         ! version of CMOR that can read this table
cf_version:   1.4         ! version of CF that output conforms to
project_id:   CORDEX  ! project id
table_date:   Mar 2015 ! date this table was constructed

missing_value: 1.e20      ! value used to indicate a missing value
                          !   in arrays output by netCDF as 32-bit IEEE 
                          !   floating-point numbers (float or real)

product: output
! CMIP5's Table adapted for CORDEX
required_global_attributes: creation_date tracking_id model_id contact institute_id ! space separated required global attribute 

expt_id_ok: 'Evaluation run with reanalysis forcing' 'evaluation'
expt_id_ok: 'historical' 'historical'
expt_id_ok: 'RCP2.6' 'rcp26'
expt_id_ok: 'RCP4.5' 'rcp45'
expt_id_ok: 'RCP8.5' 'rcp85'
expt_id_ok: 'AMIP' 'amip'
expt_id_ok: '10- or 30-year run initialized in year XXXX' 'decadalXXXX'

approx_interval:  30.000000     ! approximate spacing between successive time
                               !   samples (in units of the output time 
                               !   coordinate.

!============
axis_entry: longitude
!============
!----------------------------------
! Axis attributes:
!----------------------------------
standard_name:    longitude
units:            degrees_east
axis:             X             ! X, Y, Z, T (default: undeclared)
long_name:        longitude
!----------------------------------
! Additional axis information:
!----------------------------------
out_name:         lon
stored_direction: increasing
type:             double
must_have_bounds: yes
!----------------------------------
!


!============
axis_entry: latitude
!============
!----------------------------------
! Axis attributes:
!----------------------------------
standard_name:    latitude
units:            degrees_north
axis:             Y             ! X, Y, Z, T (default: undeclared)
long_name:        latitude
!----------------------------------
! Additional axis information:
!----------------------------------
out_name:         lat
valid_min:        -90.0         
valid_max:        90.0 
stored_direction: increasing
type:             double
must_have_bounds: yes
!----------------------------------
!


!
!============
axis_entry: time
!============
!----------------------------------
! Axis attributes:
!----------------------------------
standard_name:    time
units:            days since ?
axis:             T             ! X, Y, Z, T (default: undeclared)
long_name:        time
!----------------------------------
! Additional axis information:
!----------------------------------
out_name:         time
stored_direction: increasing
type:             double
must_have_bounds: yes
!----------------------------------
!


!============
axis_entry: height2m
!============
!----------------------------------
! Axis attributes:
!----------------------------------
standard_name:    height
units:            m
axis:             Z             ! X, Y, Z, T (default: undeclared)
positive:         up         ! up or down (default: undeclared)
long_name:        height
!----------------------------------
! Additional axis information:
!----------------------------------
out_name:         height
valid_min:        1.0         
valid_max:        10.0 
stored_direction: increasing
type:             double
value:            2.            ! of scalar (singleton) dimension 
must_have_bounds: no
!----------------------------------
!


!============
axis_entry: height10m
!============
!----------------------------------
! Axis attributes:
!----------------------------------
standard_name:    height
units:            m
axis:             Z             ! X, Y, Z, T (default: undeclared)
positive:         up         ! up or down (default: undeclared)
long_name:        height
!----------------------------------
! Additional axis information:
!----------------------------------
out_name:         height
valid_min:        1.0         
valid_max:        30.0 
stored_direction: increasing
type:             double
value:            10.            ! of scalar (singleton) dimension 
must_have_bounds: no
!----------------------------------
!


!============
axis_entry: sdepth1 
!============
!----------------------------------
! Axis attributes:
!----------------------------------
standard_name:    depth
units:            m
axis:             Z             ! X, Y, Z, T (default: undeclared)
positive:         down         ! up or down (default: undeclared)
long_name:        depth
!----------------------------------
! Additional axis information:
!----------------------------------
out_name:         depth
valid_min:        0.0         
valid_max:        0.2 
stored_direction: increasing
type:             double
value:            0.05            ! of scalar (singleton) dimension 
bounds_values:    0.0 0.1    ! of scalar (singleton) dimension bounds
must_have_bounds: yes
!----------------------------------
!

!============
axis_entry: plev850
!============
!----------------------------------
! Axis attributes:
!----------------------------------
standard_name:    air_pressure
units:            Pa
axis:             Z             ! X, Y, Z, T (default: undeclared)
positive:         down         ! up or down (default: undeclared)
long_name:        pressure
!----------------------------------
! Additional axis information:
!----------------------------------
out_name:         plev
stored_direction: decreasing
valid_min:        85000.
valid_max:        85000.
type:             double
value:            85000.
must_have_bounds: no
!----------------------------------
!

!============
axis_entry: plev500
!============
!----------------------------------
! Axis attributes:
!----------------------------------
standard_name:    air_pressure
units:            Pa
axis:             Z             ! X, Y, Z, T (default: undeclared)
positive:         down         ! up or down (default: undeclared)
long_name:        pressure
!----------------------------------
! Additional axis information:
!----------------------------------
out_name:         plev
stored_direction: decreasing
valid_min:        50000.
valid_max:        50000.
type:             double
value:            50000.
must_have_bounds: no
!----------------------------------
!

!============
axis_entry: plev200
!============
!----------------------------------
! Axis attributes:
!----------------------------------
standard_name:    air_pressure
units:            Pa
axis:             Z             ! X, Y, Z, T (default: undeclared)
positive:         down         ! up or down (default: undeclared)
long_name:        pressure
!----------------------------------
! Additional axis information:
!----------------------------------
out_name:         plev
stored_direction: decreasing
valid_min:        20000.
valid_max:        20000.
type:             double
value:            20000.
must_have_bounds: no
!----------------------------------
!

!============
variable_entry:    tas
!============
deflate: 1
deflate_level: 1
shuffle: 1
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     air_temperature
units:             K
cell_methods:      time: mean

long_name:         Near-Surface Air Temperature
comment:           daily-mean near-surface (usually, 2 meter) air temperature.
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time height2m
out_name:          tas
type:              real
!----------------------------------
!

!============
variable_entry:    tasmax
!============
deflate: 1
deflate_level: 1
shuffle: 1
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     air_temperature
units:             K
cell_methods:      time: maximum within days time: mean over days

long_name:         Daily Maximum Near-Surface Air Temperature
comment:           daily-maximum near-surface (usually, 2 meter) air temperature.
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time height2m
out_name:          tasmax
type:              real
!----------------------------------
!



!============
variable_entry:    tasmin
!============
deflate: 1
deflate_level: 1
shuffle: 1
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     air_temperature
units:             K
cell_methods:      time: minimum within days time: mean over days

long_name:         Daily Minimum Near-Surface Air Temperature
comment:           daily-minimum near-surface (usually, 2 meter) air temperature.  
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time height2m
out_name:          tasmin
type:              real
!----------------------------------
!

!============
variable_entry:    pr
!============
deflate: 1
deflate_level: 1
shuffle: 1
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     precipitation_flux
units:             kg m-2 s-1
cell_methods:      time: mean

long_name:         Precipitation
comment:           at surface; includes both liquid and solid phases from all types of clouds (both large-scale and convective)
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time
out_name:          pr
type:              real
!----------------------------------
!

!============
variable_entry:    psl
!============
deflate: 1
deflate_level: 1
shuffle: 1
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     air_pressure_at_sea_level
units:             Pa
cell_methods:      time: mean

long_name:         Sea Level Pressure
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time
out_name:          psl
type:              real
!----------------------------------
!

!============
variable_entry:    hurs
!============
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     relative_humidity
units:             %
cell_methods:      time: mean
cell_measures:     area: areacella
long_name:         Near-Surface Relative Humidity
comment:           This is the relative humidity with respect to liquid water for T> 0 C, and with respect to ice for T<0 C.
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time height2m
out_name:          hurs
type:              real
!----------------------------------
!

!============
variable_entry:    huss
!============
deflate: 1
deflate_level: 1
shuffle: 1
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     specific_humidity
units:             1
cell_methods:      time: mean

long_name:         Near-Surface Specific Humidity
comment:           near-surface (usually, 2 meter) specific humidity.
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time height2m
out_name:          huss
type:              real
!----------------------------------
!


!============
variable_entry:    sfcWind
!============
deflate: 1
deflate_level: 1
shuffle: 1
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     wind_speed
units:             m s-1
cell_methods:      time: mean

long_name:         Near-Surface Wind Speed
comment:           near-surface (usually, 10 meters) wind speed.
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time height10m
out_name:          sfcWind
type:              real
!----------------------------------
!

MIP-

table

MIP-tables may be subdivided by frequencies, realm, time cell method, grid and vertical axis types.
The MIP-Table nomenclauture is explained in detail on the offical web site

Prefix	Frequency	Suffix	Qualifier
A, CF, E, I, AER, O, LI	fx, hr, 1hr, 3hr, 6hr, day, mon, yr, clim	Lev, Plev, Ant, Gre	Pt, Z, Off

e.g.: 6hrPlevPt

cdo cmor,cordex_mip_tables/CORDEX_mon example_interface.nc

For CORDEX it was sufficient to only use the frequency for naming MIP tables. There is an ongoing discussion about the approach in CORDEX2. It will depend on the extent of the data request.

MIP-

table

Data request (dreq)

MIP-tables and CV

CMOR

CDO

There are incompatabilities:

MIP-Tables based on older dreq version 01.00.06 cannot be processed with CMOR versions beginning with version 3.2.5.
Recent CMOR >3.2.8 can only be linked to most recent CDO versions >=1.9.2

--> Which versions of CDO and CMOR one has to use depends on the dreq version the user used to configure the model simulation

Workflow in CMIP6

We will use a similar approach for CORDEX2 however necessary efforts and complexity is distinctly reduced compared to CMIP6

MIP-

table

A CMOR-variable is the unique combination of the cmor_name and the corresponding MIP-table which includes the cmor_name

Air temperature can have different requirements when monthly aggregated (CORDEX_mon) compared to when daily aggregated (CORDEX_day)
The outfile variable name does not need to be the cmor_name

cdo cmor,cordex_mip_tables/CORDEX_mon,cn=tas example_interface.nc

A Controlled Vocabulary (CV) is a special MIP-table and defines

required and optional CMIP attributes
allowed values for attributes
restrictions resulting from a setting of attributes (e.g. min. simulation years of an experiment)
whether additional attributes must be specified (e.g. parent attributes)

{
    "CV":{
        "required_global_attributes":[
            "Conventions",
            "activity_id",
            "creation_date",
            "data_specs_version",
            "experiment",
            "experiment_id",
            "forcing_index",
            "frequency",
            "further_info_url",
            "grid",
            "grid_label",
            "initialization_index",
            "institution",
            "institution_id",
            "license",
            "mip_era",
            "nominal_resolution",
            "physics_index",
            "product",
            "realization_index",
            "realm",
            "source",
            "source_id",
            "source_type",
            "sub_experiment",
            "sub_experiment_id",
            "table_id",
            "tracking_id",
            "variable_id",
            "variant_label"
        ],
        "version_metadata":{
            "author":"Paul J. Durack <durack1@llnl.gov>",
            "creation_date":"Wed Jul 5 10:30:00 2017 -0700",
            "institution_id":"PCMDI",
            "latest_tag_point":"3.2.4 (56; gcc9356e)",
            "note":"Revise source_id CNRM-CM6-1",
            "previous_commit":"f042b9339fd1cb3f5ee39bb2b5ccf958812bbd21"
        },
        "activity_id":[
            "AerChemMIP",
            "C4MIP",
            "CFMIP",
            "CMIP",
            "CORDEX",
            "DAMIP",
            "DCPP",
            "DynVarMIP",
            "FAFMIP",
            "GMMIP",
            "GeoMIP",
            "HighResMIP",
            "ISMIP6",
            "LS3MIP",
            "LUMIP",
            "OMIP",
            "PMIP",
            "RFMIP",
            "SIMIP",
            "ScenarioMIP",
            "VIACSAB",
            "VolMIP"
        ],
        "institution_id":{
            "AWI":"Alfred Wegener Institute, Helmholtz Centre for Polar and Marine Research, Am Handelshafen 12, 27570 Bremerhaven, Germany",
            "BNU":"Beijing Normal University, Beijing 100875, China",
            "CAMS":"Chinese Academy of Meteorological Sciences, Beijing 100081, China",
            "CCCR-IITM":"Centre for Climate Change Research, Indian Institute of Tropical Meteorology Pune, Maharashtra 411 008, India",
            "CCCma":"Canadian Centre for Climate Modelling and Analysis, Victoria, BC V8P 5C2, Canada",
            "CMCC":"Fondazione Centro Euro-Mediterraneo sui Cambiamenti Climatici, Lecce 73100, Italy",
            "CNRM-CERFACS":"CNRM (Centre National de Recherches Meteorologiques, Toulouse 31057, France), CERFACS (Centre Europeen de Recherche et de Formation Avancee en Calcul Scientifique, Toulouse 31057, France)",
            "CSIR-CSIRO":"CSIR (Council for Scientific and Industrial Research - Natural Resources and the Environment, Pretoria, 0001, South Africa), CSIRO (Commonwealth Scientific and Industrial Research Organisation and Bureau of Meteorology, Melbourne, Victoria 3208, Australia)",
            "CSIRO-BOM":"Commonwealth Scientific and Industrial Research Organisation and Bureau of Meteorology, Melbourne, Victoria 3208, Australia",
            "EC-Earth-Consortium":"KNMI, The Netherlands; SMHI, Sweden; DMI, Denmark; AEMET, Spain; Met Eireann, Ireland; CNR-ISAC, Italy; Instituto de Meteorologia, Portugal; FMI, Finland; BSC, Spain; Centro de Geofisica, University of Lisbon, Portugal; ENEA, Italy; Geomar, Germany; Geophysical Institute, University of Bergen, Norway; ICHEC, Ireland; ICTP, Italy; IMAU, The Netherlands; IRV, Sweden;  Lund University, Sweden; Meteorologiska Institutionen, Stockholms University, Sweden; Niels Bohr Institute, University of Copenhagen, Denmark; NTNU, Norway; SARA, The Netherlands; Unite ASTR, Belgium; Universiteit Utrecht, The Netherlands; Universiteit Wageningen, The Netherlands; University College Dublin, Ireland; Vrije Universiteit Amsterdam, the Netherlands; University of Helsinki, Finland; KIT, Karlsruhe, Germany; USC, University of Santiago de Compostela, Spain; Uppsala Universitet, Sweden; NLeSC, Netherlands eScience Center, The Netherlands",
            "FIO-RONM":"FIO (First Institute of Oceanography, State Oceanic Administration, Qingdao 266061, China), RONM (Laboratory for Regional Oceanography and Numerical Modeling, Qingdao National Laboratory for Marine Science and Technology, Qingdao 266237, China)",
            "INM":"Institute for Numerical Mathematics, Russian Academy of Science, Moscow 119991, Russia",
            "INPE":"National Institute for Space Research, Cachoeira Paulista, SP 12630-000, Brazil",
            "IPSL":"Institut Pierre Simon Laplace, Paris 75252, France",
            "MESSy-Consortium":"The Modular Earth Submodel System (MESSy) Consortium, represented by the Institute for Physics of the Atmosphere, Deutsches Zentrum fur Luft- und Raumfahrt (DLR), Wessling, Bavaria 82234, Germany",
            "MIROC":"JAMSTEC (Japan Agency for Marine-Earth Science and Technology, Kanagawa 236-0001, Japan), AORI (Atmosphere and Ocean Research Institute, The University of Tokyo, Chiba 277-8564, Japan), NIES (National Institute for Environmental Studies, Ibaraki 305-8506, Japan), and AICS (RIKEN Advanced Institute for Computational Science, Hyogo 650-0047, Japan)",
            "MOHC":"Met Office Hadley Centre, Fitzroy Road, Exeter, Devon, EX1 3PB, UK",
            "MPI-M":"Max Planck Institute for Meteorology, Hamburg 20146, Germany",
            "MRI":"Meteorological Research Institute, Tsukuba, Ibaraki 305-0052, Japan",
            "NASA-GISS":"Goddard Institute for Space Studies, New York, NY 10025, USA",
            "NCAR":"National Center for Atmospheric Research, Boulder, CO 80301, USA",
            "NCC":"NorESM Climate modeling Consortium consisting of CICERO (Center for International Climate and Environmental Research, Oslo 0349), MET-Norway (Norwegian Meteorological Institute, Oslo 0313), NERSC (Nansen Environmental and Remote Sensing Center, Bergen 5006), NILU (Norwegian Institute for Air Research, Kjeller 2027), UiB (University of Bergen, Bergen 5007), UiO (University of Oslo, Oslo 0313) and UNI (Uni Research, Bergen 5008), Norway",
            "NERC":"Natural Environment Research Council, STFC-RAL, Harwell, Oxford, OX11 0QX, UK",
            "NIMS-KMA":"National Institute of Meteorological Sciences/Korea Meteorological Administration, Climate Research Division, Seoho-bukro 33, Seogwipo-si, Jejudo 63568, Republic of Korea",
            "NOAA-GFDL":"National Oceanic and Atmospheric Administration, Geophysical Fluid Dynamics Laboratory, Princeton, NJ 08540, USA",
            "NUIST":"Nanjing University of Information Science and Technology, Nanjing, 210044, China",
            "PCMDI":"Program for Climate Model Diagnosis and Intercomparison, Lawrence Livermore National Laboratory, Livermore, CA 94550, USA",
            "THU":"Department of Earth System Science, Tsinghua University, Beijing 100084, China"
        },
        "source_id":{
            "ACCESS-1-0":{
                "activity_participation":[
                    "CMIP"
                ],
                "cohort":[
                    "CMIP5"
                ],
                "institution_id":[
                    "CSIRO-BOM"
                ],
                "source_id":"ACCESS-1-0",
                "source":"ACCESS 1.0 (2011): \naerosol: CLASSIC (v1.0)\natmos: HadGAM2 (r1.1; N96, 192 x 145 longitude/latitude; 38 levels; top level 39255 m)\natmosChem: none\nland: MOSES2.2\nlandIce: none\nocean: ACCESS-OM (MOM4p1; tripolar primarily 1deg, 360 x 300 longitude/latitude; 50 levels; top grid cell 0-10 m)\nocnBgchem: none\nseaIce: CICE4.1"
            },
            "AWI-CM-1-0-HR":{
                "activity_participation":[
                    "CORDEX",
                    "HighResMIP",
                    "OMIP",
                    "SIMIP",
                    "VIACSAB"
                ],
                "cohort":[
                    "Registered"
                ],
                "institution_id":[
                    "AWI"
                ],
                "source_id":"AWI-CM-1-0-HR",
                "source":"AWI-CM 1.0 HR (2017): \naerosol: none\natmos: ECHAM6.3.02p4 (T127L95 native atmosphere T127 gaussian grid; 384 x 192 longitude/latitude; 95 levels; top level 80 km)\natmosChem: none\nland: JSBACH 3.10\nlandIce: none\nocean: FESOM 1.4 (unstructured grid in the horizontal with 1306775 wet nodes; 46 levels; top grid cell 0-5 m)\nocnBgchem: none\nseaIce: FESOM 1.4"
            },
            "AWI-CM-1-0-LR":{
                "activity_participation":[
                    "CMIP",
                    "CORDEX",
                    "HighResMIP",
                    "OMIP",
                    "PMIP",
                    "SIMIP",
                    "ScenarioMIP",
                    "VIACSAB"
                ],
                "cohort":[
                    "Registered"
                ],
                "institution_id":[
                    "AWI"
                ],
                "source_id":"AWI-CM-1-0-LR",
                "source":"AWI-CM 1.0 LR (2017): \naerosol: none\natmos: ECHAM6.3.02p4 (T63L47 native atmosphere T63 gaussian grid; 192 x 96 longitude/latitude; 47 levels; top level 80 km)\natmosChem: none\nland: JSBACH 3.10\nlandIce: none\nocean: FESOM 1.4 (unstructured grid in the horizontal with 126859 wet nodes; 46 levels; top grid cell 0-5 m)\nocnBgchem: none\nseaIce: FESOM 1.4"
            },

The CORDEX2 CV will be based on the CMIP6 CV so that registered models and institutions for CMIP6 will also be valid for CORDEX2

CMIP6 CV:

MIP-

table

Exercise: Examine the MIP-table CORDEX_mon

What are the three CMOR-Variables having the long_name: Air Temperature?
Your atmosphere model produces a variable "NotInRequest" in monthly frequency which is surprisingly not in any MIP-table. You would like to convert it into a form similar to CMIP standard as well. What do you do?
What is the frequency only CORDEX MIP-tables are named by?

ta200, ta500, ta850
Copy a variable entry in CORDEX_mon and change all attributes according to your will
sem which is seasonal

cdo cmor,cordex_mip_tables/CORDEX_mon,cn=tas example_interface.nc

cdo cmor

info table and global attributes

Infile

CDO

CMOR

Outfile

replace

append

global
(attributes)

info
table

global
(attributes)

Info table format conventions are:

One keyvalue per line
key=value is of format word=string
String which contains commas or blanks must be specified in quotes

info_table.txt

#control keywords:
mip_table_dir="cordex_mip_tables/"
#required global attributes:
# experiment info:
PROJECT_ID=CORDEX
PRODUCT=output
DRIVING_MODEL_ID=CCCma-CanESM2
EXPERIMENT_ID=historical
MEMBER=r1i1p1
CORDEX_DOMAIN="EUR-11"
REQUIRED_TIME_UNITS="days since 1949-12-01T00:00:00Z"
COMMENT="There is no special global comment available for this experiment."
HISTORY="There is no special historyfor this experiment."
PARENT_EXPERIMENT_ID="N/A"
PARENT_EXPERIMENT_RIP="N/A"
#BRANCH_TIME=0,0
FORCING="N/A"
TBNDS_FORCE=y
driving_experiment_name="historical"
driving_experiment="CCCma-CanESM2, historical, r1i1p1"
data_policy="The Deutscher Wetterdienst (DWD) is the producer of the data. The General Terms and Conditions of Business and Delivery apply for services provided by DWD (http://www.dwd.de/EN/service/terms/terms.html)."
driving_model_ensemble_member="r1i1p1"
# model info:
MODEL_ID=CLMcom-CCLM4-8-17
RCM_VERSION_ID=v1
CALENDAR=noleap
REFERENCES="http://www.remo-rcm.de/"
SOURCE=CLMcom-CCLM4-8-17
# institution and contact info:
INSTITUTION="Climate Limited-area Modelling Community (CLM-Community)"
INSTITUTE_ID="CLMcom"
CONTACT="cordex-cclm@dkrz.de" 
required_time_units="days since REQUIRED"

info
table

Info tables

contain values of control keywords and required global attributes
have to be passed to the operator via keyword info

global
(attributes)

info
table

Most important attributes:	project_id	experiment_id	source_id	institution_id
Example values:	CMIP6, CMIP5, CORDEX	historical, piControl, amip	MPIESM-1-2-HR, MPIESM-1-2LR	MPI-M, AWI
Linked required attributes:		experiment, member, required_time_units	source, calendar, grid

It is recommended to combine required attributes which are linked by a higher level topic in one info table

E.g., by using info tables source.txt and experiment.txt, one can easily detect all required attributes which have to be changed when performing another experiment or using another source.

global
(attributes)

Attributes\experiment_id	1pctCO2	amip	ssp585
activity_id	CMIP	CMIP	ScenarioMIP
experiment	1 percent per year increase in CO2	AMIP	update of RCP8.5 based on SSP5
sub_experiment_id	none	none	none
parent_activity_id	CMIP	no parent	CMIP
parent_experiment_id	piControl	no parent	historical

Experiments are registered in the CV with attached predefined attributes:

Attribute\project	CMIP6	CMIP5
MIP:	activity_id	project_id
Model:	source_id	model_id
Institute:	institution_id	institute_id
Ensemble member:	variant_label	member
Grid resolution:	nominal_resolution

Project dependence of attribute nomenclature:

info
table

global
(attributes)

info
table

Keyword	Associated global attributes in CMIP6	Associated global attributes in CMIP5	Format
parent_dates	parent_time_units, branch_time_in_parent, branch_time_in_child	branch_time	Two comma separated integers: <parentReferenceDate>,<parentBranchDate>
member	realization_index, initialization_index, physics_index, forcing_index	realization, initialization_method, physcis_version	r%di%dp%df%d (CMIP6) or r%di%dp%d (CMIP5) where %d is an integer for the respective index
v_date	<version>		<Date>*. If not specified, CMOR uses the recent date in the path template which may lead to confusions
tracking_prefix		tracking_id	'y' or 'n'. Sets a prefix for tracking_id for generating a PID-compliant tracking_id.

*<date> has the format <year><month><day>, e.g.: 18500101

The operator offers additional 'control' keywords, e.g. to specifiy several attributes at once.

Ifile=example_interface.nc
Mtable=cmip6_mip_tables/CMIP6_Amon.json
#since a default file ".cdocmorinfo" exist,
#it is sufficient to call
cdo cmor,${Mtable} ${Ifile}
# which is equivalent to:
cdo cmor,${Mtable},info=.cdocmorinfo ${Ifile}

global
(attributes)

The default info table is

hidden and named „.cdocmorinfo“.
taken from the current working directory (cwd).

# The operator can create the path to the MIP-table
# with the help of keywords mip_table_dir and project_id
# so that it is sufficient to call:
cdo cmor,mon ${Ifile} 
# which is equivalent to:
cdo cmor,cordex_mip_tables/CORDEX_mon ${Ifile}

We can specify "control" keywords like mip_table_dir in info tables to simplify cdo cmor calls.

info
table

global
(attributes)

A tool to create an info table for CMIP6 is provided at

https://c6dreq.dkrz.de/cdocmorinfo

info
table

global
(attributes)

#Exercises:
#1. Create your own cdocmorinfo
#2. Copy that file to CWD/.cdocmorinfo
#3. Repeat
    cdo cmor,Amon example_interface.nc
#4. Have a look at differences between the files with
    ncdump -h

info
table

Infile

CDO

CMOR

Outfile

replace

append

variables (attributes)

predefinitons

user setting

data stream

cdo cmor

variable mapping

mapping
table

Infile

variables (attributes)

mapping
table

If the source variable of a target cmor variable in infile features wrong or no variable attributes at all, its attributes can be provided and overwritten via the command line or via a mapping table.

Variable attributes specifications can be combined. The priority order is:

Command line
Mapping table
Infile

The mapping table can include mapping for all target variables. If a mapping table is provided, cmor names and the MIP-table specified in the command line are used to find a matching mapping table line. Subsequently, a corresponding mapping table line provides the infile variable selector.

Infile

variables (attributes)

Keyword	Short name	Value format
project_mip_table	pmt	substring following the underscore '_' in the MIP-table name
name	n	Word
code	c	Three digits integer. GRIB code.

The project mip table

is only specifiable in the mapping table
is needed to identify a cmor variable.

Name and code are infile variable selectors. If the infile is of netCDF format, name is used; if the infile is a GRIB-file, code is used.

mapping
table

Infile

variables (attributes)

Keyword	Short name	Value format
project_mip_table	pmt	MT name substring
name	n	Word
code	c	Three digits integer. GRIB code.

# ways to configure the
# infile variable selector (ivs)
# and mapping table line selector (mtls)
#
# 1. ivs=cmor_name
cdo cmor,${MIP-table},cmor_name=${cn} ${infile}
# 2. ivs=name
cdo cmor,${MIP-table},cmor_name=${cn},name=${name} ${infile}
#
# with mapping table:
# 3. mtls=ALL,       ivs=what is in EACH mtl
cdo cmor,${MIP-table},mapping_table=${mt} ${infile}
# 4. mtls=cmor_name, ivs=what is in corresponding mtl
cdo cmor,${MIP-table},mapping_table=${mt},cmor_name=${cn} ${infile}
# 5. mtls=cmor_name, ivs=name (overwrites mtl variable selector)
cdo cmor,${MIP-table},mapping_table=${mt},cmor_name=${cn},name=${name} ${infile}

mapping
table

Infile

variables (attributes)

Keyword	Short name	Value format	Default
project_mip_table	pmt	MT name substring
name	n	Word
code	c	Three digits integer. GRIB code.
units	u	String. Must be readable by udunits.
cell_methods	cm	Character (see below)	m
positive	p	u=upward, d=downward
variable_comment	vc	String

Five valid time cell methods can be specified. It is used to choose the correct time axis.

m = mean. Averaged over time.
p = point. Used for measurements and sums.
d = diurnal cycle. Used for means within days (1hr means, e.g. 0:00-1:00 ) over days (1.-31.1.). (CMIP6)
c = climate. Used for means within years (january) over years (30 years).
n = none. Used for fixed fields like topography.

mapping
table

Infile

variables (attributes)

Keyword	Short name	Value format	Default
project_mip_table	pmt	MT name substring
name	n	Word
code	c	Three digits integer. GRIB code.
units	u	String. Must be readable by udunits.
cell_methods	cm	Character (see below)	m
positive	p	u=upward, d=downward
variable_comment	vc	String

A positive flux direction must be specified if the request demands it. This is the case for fluxes, e.g. radiation variables. Otherwise, cmor aborts.

mapping
table

Infile

variables (attributes)

mapping_
table.txt

&parameter cmor_name=tasmax  code=201 units="K"      cell_methods="m" project_mip_table=Amon /
&parameter cmor_name=tasmax  code=201 units="K"      cell_methods="m" project_mip_table=day /
&parameter cmor_name=tasmin  code=202 units="K"      cell_methods="m" project_mip_table=day  /
&parameter cmor_name=tasmin  code=202 units="K"      cell_methods="m" project_mip_table=Amon /
&parameter cmor_name=rsds             units="W m-2"  cell_methods="m" positive="d" /
&parameter cmor_name=rlds             units="W m-2"  cell_methods="m" positive="d" /

A Mapping table

begins with '&parameter'
ends with /

Each line should contain

cn and pmt to identify a cmor variable
an infile variable selector

mapping
table

infile=example_mapping.grb

#1. Commandline mapping:
cdo cmor,mon,cn=tas,c=167,u=K,cm=m ${infile}
#  Variable with code=167 from ${infile}.
#  is used. Its model units is Kelvin (u=K)
#  and its time cell method is mean (cm=m).

#2. Mapping table application:
#   create mapping table:
echo '&parameter cn=tas c=167 u=K cm=M' >>mt.txt
#   apply cdo cmor with mapping table:
cdo cmor,mon,mapping_table=mt.txt ${infile}
#  Without additional information,
#  all variables from ${infile}
#  are taken and converted
cdo cmor,mon,cmor_name=tas,mt=mt.txt ${infile}
#  If a cmor_name is provided, it is used
#  as a mapping table line selector.
#  The variable selector is taken
#  from the corresponding mapping table line.

Infile

variables (attributes)

mapping
table

Exercises:

Convert the variable included in example_T_2M.nc to CMIP standard. Use mt.txt from the last slide and overwrite the infile variable selector.
Convert the variable included in example_celsius.grb to CMIP standard. Use mt.txt from the last slideand adapt the correct model variable unit.

#1.:
cdo cmor,mon,cn=tas,n=T_2M,mt=mapping_table.txt example_T_2M.nc
#2.:
cdo cmor,mon,cn=tas,mt=mapping_table.txt,u=„degC“ example_celsius.nc

#Convert many variables at once with a
#complete mapping table mtPERFECT.txt:
cdo cmor,mon,mt=mtPERFECT.txt example_collect.grb

Infile

variables (attributes)

mapping
table

A tool to create an mappping table is provided at

https://c6dreq.dkrz.de/

Infile

CDO

CMOR

Outfile

replace

append

coordinates (attributes)

cdo cmor

coordinates and grid info file

grid info
file

coordinates (attributes)

grid info
file

Coordinates are also defined in the CMIP standard with axis entries
The target variables can be requested on a specific coordinate variable (e.g. tas on height2m). Since this is not always the case (Requests for dimension "alev" means atmospheric model level; various coordinates are valid), CMOR requires the cmor_name of the axis a variable is submitted on as input.

!============
axis_entry: latitude
!============
!----------------------------------
! Axis attributes:
!----------------------------------
standard_name:    latitude
units:            degrees_north
axis:             Y             ! X, Y, Z, T (default: undeclared)
long_name:        latitude
!----------------------------------
! Additional axis information:
!----------------------------------
out_name:         lat
valid_min:        -90.0         
valid_max:        90.0 
stored_direction: increasing
type:             double
must_have_bounds: yes
!----------------------------------
!

!============
variable_entry:    tas
!============
deflate: 1
deflate_level: 1
shuffle: 1
modeling_realm:    atmos
!----------------------------------
! Variable attributes:
!----------------------------------
standard_name:     air_temperature
units:             K
cell_methods:      time: mean

long_name:         Near-Surface Air Temperature
comment:           daily-mean near-surface (usually, 2 meter) air temperature.
!----------------------------------
! Additional variable information:
!----------------------------------
dimensions:        longitude latitude time height2m
out_name:          tas
type:              real
!----------------------------------

coordinates (attributes)

grid info
file

The interface of CDO is able to distinguish axis types by exploiting infile information: Attributes 'standard_name' and 'units' of coordinate variables are evaluated according to the CF-conventions and some key values for attributes are predefined (GRIB tables). Gained information is used to pass the correct axis names to CMOR.
Sometimes, the retrieved information from infile is insufficient. For that case, the operator provides the option to substitute the infile coordinates by specifying a grid info file. For specific axis types, less extensive configuration is possible.

bool is_lat_axis(const char *units, const char *stdname)
{
  bool status = false;
  char lc_units[16];

  memcpy(lc_units, units, 15);
  lc_units[15] = 0;
  str_tolower(lc_units);

  if ( (str_is_equal(lc_units, "degree") || str_is_equal(lc_units, "radian")) &&
        (str_is_equal(stdname, "grid_latitude") || str_is_equal(stdname, "latitude")) )
    {
      status = true;
    }
  else if ( str_is_equal(lc_units, "degree")
            && !str_is_equal(stdname, "grid_longitude")
            && !str_is_equal(stdname, "longitude") )
    {
      int ioff = 6;
      if ( lc_units[ioff] == 's' ) ioff++;
      if ( lc_units[ioff] == '_' ) ioff++;
      if ( lc_units[ioff] == 'n' || lc_units[ioff] == 's' ) status = true;
    }

  return status;
}

coordinates (attributes)

grid info
file

grid_
info.nc

ncdump -h example_gridinfo_CCLM4-8-17.nc
netcdf example_gridinfo_CCLM4-8-17 {
dimensions:
	rlon = 424 ;
	rlat = 412 ;
	vertices = 4 ;
	time = UNLIMITED ; // (1 currently)
	bnds = 2 ;
variables:
	double lon(rlat, rlon) ;
		lon:standard_name = "longitude" ;
		lon:long_name = "longitude" ;
		lon:units = "degrees_east" ;
		lon:_CoordinateAxisType = "Lon" ;
		lon:bounds = "lon_bnds" ;
	double lon_bnds(rlat, rlon, vertices) ;
	double lat(rlat, rlon) ;
		lat:standard_name = "latitude" ;
		lat:long_name = "latitude" ;
		lat:units = "degrees_north" ;
		lat:_CoordinateAxisType = "Lat" ;
		lat:bounds = "lat_bnds" ;
	double lat_bnds(rlat, rlon, vertices) ;
	double rlon(rlon) ;
		rlon:standard_name = "grid_longitude" ;
		rlon:long_name = "longitude in rotated pole grid" ;
		rlon:units = "degrees" ;
		rlon:axis = "X" ;
	double rlat(rlat) ;
		rlat:standard_name = "grid_latitude" ;
		rlat:long_name = "latitude in rotated pole grid" ;
		rlat:units = "degrees" ;
		rlat:axis = "Y" ;
	int rotated_pole ;
		rotated_pole:grid_north_pole_latitude = 39.25 ;
		rotated_pole:grid_north_pole_longitude = -162. ;
		rotated_pole:grid_mapping_name = "rotated_latitude_longitude" ;
	double height ;
		height:standard_name = "height" ;
		height:long_name = "height" ;
		height:units = "m" ;
		height:positive = "up" ;
		height:axis = "Z" ;
	double time(time) ;
		time:standard_name = "time" ;
		time:long_name = "time" ;
		time:bounds = "time_bnds" ;
		time:units = "days since 1949-12-01 00:00:00" ;
		time:calendar = "365_day" ;
		time:axis = "T" ;
	double time_bnds(time, bnds) ;
	float tas(time, rlat, rlon) ;
		tas:standard_name = "air_temperature" ;
		tas:long_name = "Near-Surface Air Temperature" ;
		tas:units = "K" ;
		tas:grid_mapping = "rotated_pole" ;
		tas:coordinates = "height lat lon" ;
		tas:_FillValue = 1.e+20f ;
		tas:missing_value = 1.e+20f ;
		tas:cell_methods = "time: mean" ;
}

A grid info netCDF file contains CF-conform
- complete grid description (bounds, grid mapping parameter)
- vertical axis (level)
- A 'dummy' variable wich fits to the target variable in dimension sizes
A grid info netCDF file substitutes coordinate information gained from infile

coordinates (attributes)

grid info
file

time axis

time values

time bounds

The CMIP6 standard requires a relative axis and time bounds. Relatives means, time values and time bounds are the temporal distance to a reference time.
This reference time needs to be provided, either via a relative axis in the infile or via keyword 'required_time_units' in the format "days since <year><month><date> <hour>:<minute><second>". Each experiment usually predefines a reference time.
The operator can pass correct time values to CMOR based on an absolute time axis in infile as well with the help of the 'required_time_units'

1970-01-02

1970-01-03

1970-01-04

1970-01-05

1970-01-06

Absolute:

Relative:

days since 1970-01-01 00:00:00

coordinates (attributes)

grid info
file

# MIP-Table request:
cell_methods:      time: mean
dimensions: time
# Mapping attribute:
cm=m

# MIP-Table request:
cell_methods: time: point
dimensions: time1
# Mapping attribute:
cm=p

# MIP-Table request:
#  Climate:
"cell_methods": "time: mean within years time: mean over years"
"dimensions": "time2"
# Mapping attribute:
cm=c

#  Diurnal:
"cell_methods": "time: mean within days time: mean over days"
"dimensions": "time3"
# Mapping attribute:
cm=d

The time cell method of a CMOR variable indicates the requirements on the time axis and time bounds.
CMOR distinguishes five time axes (one of them is 'none' for fixed fields) which can be identified via the name specified in the dimensions of CMOR variables.
In a conclusion, each of these can be built with the operator based on time values with a specific precision, however, each type is discussed in the following.

time values

time bounds

time axis

coordinates (attributes)

grid info
file

# MIP-Table request:
cell_methods: time: mean
dimensions: time
# Mapping attribute:
cm=m

tbound_u =

tbound_u =

tbound_l =

tbound_l =

For at least daily frequency:

E.g., for monthly requested variables, each time value must match the month the corresponding record is valid for. CMOR generates time bounds covering the complete frequency unit. If the first record is valid for month January 2018, CMOR generates:

For subdaily frequency:

E.g., for 6hourly requested variables, each time value must match the mid time of the averaging interval. If a time value "2018-01-01 03:00:00" is provided, CMOR adds(substracts) the half of the frequency to the mid for the upper time bound (lower time bound):

tbound_u =

tbound_u =

tbound_l =

tbound_l =

"2018-01-01 00:00:00"

"2018-02-01 00:00:00"

"2018-01-01 00:00:00"

"2018-01-01 06:00:00"

Provide correct time bounds. If no time bounds are available, the operator is able to build them on the basis of time values. If infile time bounds are wrong, specify tbnds_ignore.

time values

time bounds

time axis

coordinates (attributes)

grid info
file

# MIP-Table request:
"cell_methods": "time: point"
"dimensions": "time1"
# Mapping attribute:
cm=p

Provide exact time values. Time bounds are not required for 'point' variables.

time values

time bounds

time axis

coordinates (attributes)

grid info
file

# MIP-Table request:
#  Climate:
"cell_methods": "time: mean within years time: mean over years"
"dimensions": "time2"
# Mapping attribute:
cm=c

#  Diurnal:
"cell_methods": "time: mean within days time: mean over days"
"dimensions": "time3"
# Mapping attribute:
cm=d

time values

time bounds

The time bounds requested for 'climate' and 'diurnal cycle' axes differ from 'mean' axis requests because of double averaging. The bounds generation is exemplified for both axis types:

tbound_u =

tbound_u =

tbound_l =

tbound_l =

"1971-01-01 00:00:00"

"2000-02-01 00:00:00"

Climate:
A januarly mean averaged for 30 years period 1971-2000 has the following bounds:

tbound_u =

tbound_u =

tbound_l =

tbound_l =

"1971-01-01 00:00:00"

"1971-02-01 01:00:00"

Diurnal cycle:
The mean of 00:00:00-01:00:00 averaged for all januar days of 1971 has the following bounds:

time axis

coordinates (attributes)

grid info
file

# MIP-Table request:
#  Climate:
"cell_methods": "time: mean within years time: mean over years"
"dimensions": "time2"
# Mapping attribute:
cm=c

#  Diurnal:
"cell_methods": "time: mean within days time: mean over days"
"dimensions": "time3"
# Mapping attribute:
cm=d

time values

time bounds

The bounds variable for a 'climate' axis is named "climatology_bnds" in outfile. For both axis types, these bounds can be provided as time_bnds variable similar to the procedure for 'mean' types. If not applicable, you can do this:

Climate:
Time values must match the corresponding month (see 'mean' type). Specify the start and end year of the averaging interval as comma separated integers for keyword climatology_interval

Diurnal cycle:
Time values must be within the firtst average interval (correct hour) and within the correct month.

time axis

coordinates (attributes)

grid info
file

time values

time bounds

time axis

Name	Short name	Format	Default
required_time_units	rtu	’<Frequency> since <Year>-<Month>-<Day> <Hours>:<Minutes>:<Seconds>’, example: ’days since 1979-1-1 00:00:00’.	No default
calendar		Value is one of ’standard’ (’gre- gorian’), ’proleptic gregorian’, ’360 day’, ’noleap’ and ’all leap’. The calendar depends on the model configuration	A calendar is usually available in infile
climatology_interval		Two comma separated years as integers. E.g.: 2001,2010	No default
tbnds_ign		’y’ for ignoring infile time bounds variable, ’n’ for not.	'n'

coordinates (attributes)

grid info
file

scalar axes

character axes

# Get the strucutre of header of a example raw model output:
ncdump -h example_interface.nc 
# results:
netcdf example_interface {
dimensions:
	time = UNLIMITED ; // (12 currently)
	lon = 384 ;
	lat = 192 ;
	height = 1 ;

A dimension with size 1 is named 'scalar'. The CMIP standard provides that a coordinate variable associated with attribute 'coordinates' of the requested variable is used to represent this axis instead of a dimension:

variables:
	double height ;
		height:units = "m" ;
		height:axis = "Z" ;
		height:positive = "up" ;
		height:long_name = "height" ;
		height:standard_name = "height" ;
	float tas(time, lat, lon) ;
		tas:coordinates = "height" ;

coordinates (attributes)

grid info
file

scalar axes

character axes

The operator automatically generates the scalar coordinate variable with the value requested by CMIP if not otherwise indicated.
The name of the coordinate indicates the requested level, e.g. the requested value of height2m is 2m.
However, an interval for valid values for this variable is defined by CMIP and spanned by valid_min and valid_max values.

"height2m": {
  "valid_max": "10.0", 
  "valid_min": "1.0", 
  "value": "2.",

If the model level deviates from the requested level,

the name of the scalar axis needs to be associated with the requested variable during mapping via z_axis (since all scalar axes are vertical). See below for an example.
<z_axis_name>=<value> needs to be specified in a grid info table file. E.g. height2m=1.5

mapping table

&parameter cn=tas c=201 u=K cm=m pmt=Amon z_axis=height2m /

coordinates (attributes)

grid info
file

scalar axes

character axes

"basin": { 
  "standard_name": "region",
  "requested": [
    "atlantic_arctic_ocean", 
    "indian_pacific_ocean", 
    "global_ocean"
   ], 
  "type": "character",

mapping table

&parameter cn=hfbasinpadv c=004 pmt=Omon character_axis=basin /

"hfbasinpadv": {
  "dimensions": "latitude basin time",

Some variables are requested on character axes (recognizable by the dimension names) representing regions or areas (see above).
The corresponding coordinate variable is of type "character".
It can contain "requested" values. If specified, these are the minimum values which needs to be provided.

If only an index dimension is available in infile, the procedure similar to the scalar axes method can be an option:

Specify a value for keyword character_axis within the target variable mapping
Specify <character_axis_name>=<comma_separated_values> in a grid info table file. The sequence of strings must match the index sequence of the dimension.

coordinates (attributes)

grid info
file

vertical parametric axis

The CMIP standard defines different vertical parametric axes which can be used if a variable is requestedon ’alev’s.
These axes are built on the basis of a formula which contains constant parameters. E.g., the hybrid sigma pressure coordinate named ’alternate hybrid sigma’ is built by the formula:

p=a_p+b*p_s

p=a_p+b*p_s

Right now, only this ’alternate hybrid sigma’ axis is enabled and test data for other axes built by a formula is required to guarantee a correct processing

The operator is able to process this type when passed in an already CF-conform (Appendix D) format

ps must also be available in infile or in a substitution grid info netCDF file. The operator identifies the ps via variable name ’ps’. If a mapping table is specified, the name which is in the line with cmor name=ps is used to find the correct infile variable.

coordinates (attributes)

grid info
file

vertical parametric axis

p=a_p+b*p_s

p=a_p+b*p_s

Right now, only this ’alternate hybrid sigma’ axis is enabled and test data for other axes built by a formula is required to guarantee a correct processing

For a grib formatted infile :
- If the key ’indicatorOfTypeOfLevel’ (octet 10) has either the value 109 or 110.
- The corresponding parameter ap and b should be saved for gribapi key ”pv”.
For a netCDF formatted infile:
- if variables named ’hyai’ and ’hybi’ are available, they will be associated with ap and b.
- ps must also be available in the infile.

coordinates (attributes)

grid info
file

vertical axis

Type	Default units
ZAXIS_PRESSURE	"Pa"
ZAXIS_HYBRID	"Pa" for all parameters
ZAXIS_DEPTH_BELOW_SEA	"m"
ZAXIS_DEPTH_BELOW_LAND	"cm"
name="rho"	"kg m^-3"
Scalar	"m"

If no z-axis units are provided in infile via the units attribute of the coordinate variable, the operator uses the following default values

Infile

CDO

CMOR

Outfile

replace

append

predefinitons

user setting

data stream

cdo cmor

output control

1851

1852

1850

CMOR

1850_post

1851_post

1852_post

tas_
185001_185012.nc

tas_
185001_185112.nc

tas_
185001_185212.nc

model simulation year

Postprocessing skript

CMIP6 compliant example variable

Workflow

Name	Short name	Format	Default
output_mode	om	Character	r
last_chunk	lc	Strings	Read from chunk description file
max_size	ms	Integer	2

The output mode can be switched from default replace mode to append mode by specifying output_mode=a.
In append mode, the last_chunk keyword specifies a corresponding chunk file to which data is appended.
max_size defines the upper limit size of a chunk_file in [gb] before switching to replace mode.

CMOR

Outfile

replace

append

CMOR

Outfile

replace

append

#From earlier examples, you should have created a tas-file 
#for year 2001. Now you want to append year 2002:
#
#Save the last chunk in a variable:
DRS=CMIP6/CMIP/MPI-M/MPIESM-1-2-HR/historical/ri1ip1f1/Amon/tas/gn/v20171010/
lc=${DRS}tas_Amon_MPIESM-1-2-HR_historical_r1i1p1f1_gn_200101-200112.nc

cdo cmor,Amon,om=a,mt=mapping_table.txt,last_chunk=${lc} example_append2002.grb


#The outfile path and filename is saved in
#„CHUNK_FILE_tas_Amon_MPI-ESM_amip_r1i1p1.txt“.
#This is the description for the default chunk in append mode.
#Now you can append the next year without defining lc:
cdo cmor,Amon,om=a,mt=mapping_table.txt example_append2003.grb

Function	Name	Short name	Necessary information (M: mandatory, R: If required, D: Default, W: Works without)	Format and description. CV: Value must be in Controlled Vocabulary. CV restricted: Format of value is restricted by Controlled Vocabulary	Default
CMOR variable selector in mapping table	project_mip_table	pmt	W	Substring of the MIP-table name
Coordinate keyword	calendar		D	Word, 5 options.	standard'
Coordinate keyword	climatology_interval		R	Two integers: “%d,%d“
Coordinate keyword	leap_year		R
Coordinate keyword	leap_month		R
Coordinate keyword	req_time_units		M	"$units since $year-$month-$day $hours:$minutes:$seconds"
Coordinate keyword	time_bounds		D	Either „c“ for create or „i“ for infile	i
Filename keyword	grid_info	gi	W	Filename. Grid of a netCDF file must fit to infile variable dimensions
Filename keyword	grid_info_dir		W	Filepath
Filename keyword	info	i	D	CSV of filenames.	".cdocmorinfo" in the current working directory
Filename keyword	mapping_table	mt	W	Filename
Filename keyword	mapping_table_dir		W	Filepath
Filename keyword	mip_table_dir		W	Filepath
Mapping keyword	cell_methods	cm	D	Character, 5 options	m
Mapping keyword	character_axis	ca	R	Cmor axis label
Mapping keyword	positive	p	R	d for downward, u for upward or blank for undirected
Mapping keyword	units	u	M	String, must be readable by udunits
Mapping keyword	variable_comment	vc	W	String
Mapping keyword	z_axis	za	R	Cmor axis label
Output control keyword	drs	d	D	y' for building a DRS and 'n' for not building a DRS	y
Output control keyword	drs_root	dr	D	Filepath to where DRS will be built	./
Output control keyword	last_chunk	lc	W	CSV of filenames
Output control keyword	max_size	ms	D	Unit: Gb	2
Output control keyword	output_mode	om	D	r for replace or a for append	a
Variable selector keyword	cmor_name	cn	M	CSV. Value must be in MIP-table.
Variable selector keyword	code	c	W	Three digits Intege. GRIB-cod.
Variable selector keyword	name	n	W	Word

cdo cmor handson for PRINCIPLES

By Fabian Wachsmann

cdo cmor handson for PRINCIPLES

Get to know the CMIP6 data standard. Explore the advantages of CDO, CMOR and cdo cmor. Learn how to convert variables into the CMIP6 format. Learn how to provide a complete metadata set. Integrate the app into the operational workflow.

2,354

Fabian Wachsmann

Scientific programmer in the joint research project DICAD at DKRZ Developer of Climate Data Operators (CDO)

cdo cmor

cdo cmor

cdo cmor

cdo cmor

cdo cmor

cdo cmor

cdo cmor

cdo cmor

cdo cmor

cdo cmor

cdo cmor handson for PRINCIPLES

More from Fabian Wachsmann