cdo cmor

state of affairs and examples

cdo cmor

Make data easy to analyze

Convention (1 and 2) about climate data accepted in Coupled Model Inter-comparison Project

Collection of operators to process climate data

Climate Model Output Rewrite to ensure correct conversion

• It guarantees the correct conversion to the CMIP standard
• Use synergies instead of produce twice the same

'cdo cmor'

• because of CDO's large community
• to simplify the usage of CMOR

cdo cmor

Motivation

Definitions

introduction

MIP-

table

• A MIP-table is CMOR-readable and contains the CMIP standard
• Consider the components workflow:

cdo cmor,cmip6_mip_tables_325/CMIP6_Amon.json example_interface.nc

cdo cmor,cmip6_mip_tables_325/CMIP6_Amon.json,cn=tas example_interface.nc

cdo cmor

Project standard

cdo cmor

CMOR2

• CMIP5
• CORDEX

CMOR3

• CMIP6
• CORDEX2

Use the CDO installation which uses the CMOR Version suitable to create the data standard of your project

Instructions

Recommendations

• Use preinstalled versions on DKRZ system mistral.
• Use the WebGUI for generating components for the application
  
   

installation

cdo cmor

Recent challenges

• Termination signals are not processed when compiled with intel
• Not all error messages from CMOR are absorped

• A tagged CDO version converts MPIESM-1-2-HR data for historical to CMIP5 (with CMOR2) and to CMIP6 (with CMOR3) successfully
• Documentation extended with slides containing code as well as videos
• Installation simplified (UUID lib). Conda-forge pull request is submitted

Progress

state of affairs

cdo cmor

1. HandsOn preparation
2. Usage
3. First steps

#connect to mistral
ssh -X <usernr>@mistral.dkrz.de

#create workdir
mkdir cdocmor_handson
cd cdocmor_handson

#copy examples directory
cdoDir=/work/bm0021/cdo_incl_cmor/
cp -r ${cdoDir}examples ./
cd examples

#link cdo cmor installation
alias cdo=${cdoDir}cdo_handson_cmor3

Preparation

#**General usage:
#cdo cmor,MIP-table[,keyvaluelist] infile

#**The keyvaluelist is of format
#key=value[,value]

#**Some keywords only allow exactly one value.
#**Others allow a comma separated list (CSL) of values.

#**If the input file is of netCDF format and contains all required meta data,
#**it can be sufficient to call:
#cdo cmor,MIP-table infile

#**example:
cdo cmor,cmip6_mip_tables_325/CMIP6_Amon.json example_interface.nc

#**Parameter one MUST be the MIP table
#**It can be specified with a relative or absoulte path

Usage

#example:
cdo cmor,cmip6_mip_tables_325/CMIP6_Amon.json example_interface.nc

Usage

The outfile is generated by CMOR and must not be specified.

Name and path of the outfile are generated according to templates defined by the corresponding project:

<variable_id>_<table_id>_
<source_id>_<experiment_id>_
<variant_label>_<grid_label>

<variable_id>_<table_id>_
<model_id>_<experiment_id>_
<member>

<activity_id>/<product>/<institute_id>/
<model_id>/<experiment_id>/<frequency>/
<realm>/<variable_id>/<member>

<mip_er>/<activity_id>/<institution_id>/<source_id>/
<experiment_id>/<variant_label>/<table_id>/
<variable_id>/<grid_label>/<version>

#example:
cdo cmor,cmip6_mip_tables_325/CMIP6_Amon.json example_interface.nc

First steps

#example:
#select tas from infile
cdo cmor,cmip6_mip_tables_325/CMIP6_Amon.json,cn=tas example_interface.nc
#select tas and uas from infile
cdo cmor,cmip6_mip_tables_325/CMIP6_Amon.json,cn=tas,uas example_interface.nc

• Parameter one must be the MIP-table
• The outfile is generated by CMOR and must not be specified

• Keyword cmor_name selects a subset of variables to be processed

• All keys have short forms, e.g. cn for cmor_name

#example:
cdo cmor,cmip6_mip_tables_325/CMIP6_Amon.json example_interface.nc

First steps

~/cdo-git/src/cdo cmor,Amon example_interface.nc
cdo cmor: (2.2.) Start to find a MIP table file.
cdo cmor: Is 'Amon' a valid 'MIP table' file?
          Empty file: 'Amon'.
cdo cmor: Try to build a path with additional configuration attributes:
          'mip_table_dir' and 'project_id'
          in order to use 'Amon' as MIP-table.
cdo cmor: (2.2.) MIP table file 'cmip6_mip_tables_328//CMIP6_Amon.json' exists.
cdo cmor: Function 'attribute check, compare and set':
          Be aware of differences between infile and user specification.
          Attribute 'time_units' in input file: 'days since 1850-1-16 12:00:00' does not agree with user specification 'days since 1850-1-1 00:00:00'.
          Cmor library is called with 'days since 1850-1-1 00:00:00'.
cdo cmor: Function 'all axes registration':
          You have not requested a particular variable via 'cmor_name'.
          There are several in infile and all will be processed.
          Notice that attributes specified in the cmdline will be used for all infile variables.
cdo cmor: Vertical axis is either default and scalar or not available.
cdo cmor: Vertical axis is either default and scalar or not available.
cdo cmor:      File stored in:  './/CMIP6/CMIP/MPI-M/MPI-ESM-1-2-HR/historical/r1i1p1f1/Amon/uas/gn/v20180116/uas_Amon_MPI-ESM-1-2-HR_historical_r1i1p1f1_gn_185001-185012.nc' with cmor!
cdo cmor:      File stored in:  './/CMIP6/CMIP/MPI-M/MPI-ESM-1-2-HR/historical/r1i1p1f1/Amon/tas/gn/v20180116/tas_Amon_MPI-ESM-1-2-HR_historical_r1i1p1f1_gn_185001-185012.nc' with cmor!
cdo cmor: Processed 1769472 values from 8 variables over 12 timesteps ( 1.06s 1044MB )

#The usual CDO options are valid for this operator as well:
#cdo cmor can be called in silent (-s) or verbose (-v) mode.
#The verbose mode is recommended if the processing is unclear:

cdo -v cmor,cmip6_mip_tables_325/CMIP6_Amon.json example_interface.nc
cdo -s cmor,cmip6_mip_tables_325/CMIP6_Amon.json example_interface.nc

#In many cases, operator chaining can be helpful:

cdo -cmor,cmip6_mip_tables_325/CMIP6_Amon.json -addc,10 example_interface.nc

First steps

cdo cmor

Configuration and data streams

variables (attributes)

coordinates (attributes)

global
(attributes)

MIP-

table

CV

predefinitons

user setting

data stream

cdo

cmor

mapping
table

grid info
file

info
table

MIP-
table

command

replace

append

line

metadata stream

data stream

cdo cmor

info table and global attributes

Infile

CDO

CMOR

Outfile

replace

append

global
(attributes)

global
(attributes)

info
table

global
(attributes)

info
table

Infile

CDO

CMOR

Outfile

replace

append

variables (attributes)

predefinitons

user setting

data stream

cdo cmor

variable mapping

Infile

variables (attributes)

A Mapping table

• begins with '&parameter'
• ends with /
   

Each line should contain

• cn and pmt to identify a cmor variable
• an infile variable selector

mapping
table

Infile

variables (attributes)

mapping
table

If the source variable of a target cmor variable in infile features wrong or no variable attributes at all, its attributes can be provided and overwritten via the command line or via a mapping table.

Variable attributes specifications can be combined. The priority order is:

1. Command line
2. Mapping table
3. Infile

The mapping table can include mapping for all target variables. If a mapping table is provided, cmor names and the MIP-table specified in the command line are used to find a matching mapping table line. Subsequently, a corresponding mapping table line provides the infile variable selector.

Infile

variables (attributes)

mapping
table

Infile

CDO

CMOR

Outfile

replace

append

coordinates (attributes)

cdo cmor

coordinates and grid info file

coordinates (attributes)

grid info
file

• Coordinates are also defined in the CMIP standard with CMOR variables (in CMIP6_coordinates.json)
   
• The target variables can be requested on a specific coordinate variable (e.g. tas on height2m). Since this is not always the case (Requests for dimension "alev" means atmospheric model level; various coordinates are valid), CMOR requires the cmor_name of the axis a variable is submitted on as input.

"latitude": {
  "standard_name": "latitude", 
  "units": "degrees_north", 
  "axis": "Y", 
  "long_name": "latitude", 
  "climatology": "", 
  "formula": "", 
  "must_have_bounds": "yes", 
  "out_name": "lat", 
  "positive": "", 
  "requested": "", 
  "requested_bounds": "", 
  "stored_direction": "increasing", 
  "tolerance": "", 
  "type": "double", 
  "valid_max": "90.0", 
  "valid_min": "-90.0", 
  "value": "", 
  "z_bounds_factors": "", 
  "z_factors": "", 
  "bounds_values": ""
}

"tas": {
  "frequency": "mon", 
  "modeling_realm": "atmos", 
  "standard_name": "air_temperature", 
  "units": "K", 
  "cell_methods": "area: time: mean", 
  "cell_measures": "area: areacella", 
  "long_name": "Near-Surface Air Temperature", 
  "comment": "near-surface (usually, 2 meter) air temperature", 
  "dimensions": "longitude latitude time height2m", 
  "out_name": "tas", 
  "type": "real", 
  "positive": "", 
  "valid_min": "", 
  "valid_max": "", 
  "ok_min_mean_abs": "", 
  "ok_max_mean_abs": ""
}

coordinates (attributes)

grid info
file

grid_
info.nc

ncdump -h example_gridinfo_CCLM4-8-17.nc
netcdf example_gridinfo_CCLM4-8-17 {
dimensions:
	rlon = 424 ;
	rlat = 412 ;
	vertices = 4 ;
	time = UNLIMITED ; // (1 currently)
	bnds = 2 ;
variables:
	double lon(rlat, rlon) ;
		lon:standard_name = "longitude" ;
		lon:long_name = "longitude" ;
		lon:units = "degrees_east" ;
		lon:_CoordinateAxisType = "Lon" ;
		lon:bounds = "lon_bnds" ;
	double lon_bnds(rlat, rlon, vertices) ;
	double lat(rlat, rlon) ;
		lat:standard_name = "latitude" ;
		lat:long_name = "latitude" ;
		lat:units = "degrees_north" ;
		lat:_CoordinateAxisType = "Lat" ;
		lat:bounds = "lat_bnds" ;
	double lat_bnds(rlat, rlon, vertices) ;
	double rlon(rlon) ;
		rlon:standard_name = "grid_longitude" ;
		rlon:long_name = "longitude in rotated pole grid" ;
		rlon:units = "degrees" ;
		rlon:axis = "X" ;
	double rlat(rlat) ;
		rlat:standard_name = "grid_latitude" ;
		rlat:long_name = "latitude in rotated pole grid" ;
		rlat:units = "degrees" ;
		rlat:axis = "Y" ;
	int rotated_pole ;
		rotated_pole:grid_north_pole_latitude = 39.25 ;
		rotated_pole:grid_north_pole_longitude = -162. ;
		rotated_pole:grid_mapping_name = "rotated_latitude_longitude" ;
	double height ;
		height:standard_name = "height" ;
		height:long_name = "height" ;
		height:units = "m" ;
		height:positive = "up" ;
		height:axis = "Z" ;
	double time(time) ;
		time:standard_name = "time" ;
		time:long_name = "time" ;
		time:bounds = "time_bnds" ;
		time:units = "days since 1949-12-01 00:00:00" ;
		time:calendar = "365_day" ;
		time:axis = "T" ;
	double time_bnds(time, bnds) ;
	float tas(time, rlat, rlon) ;
		tas:standard_name = "air_temperature" ;
		tas:long_name = "Near-Surface Air Temperature" ;
		tas:units = "K" ;
		tas:grid_mapping = "rotated_pole" ;
		tas:coordinates = "height lat lon" ;
		tas:_FillValue = 1.e+20f ;
		tas:missing_value = 1.e+20f ;
		tas:cell_methods = "time: mean" ;
}

• A grid info netCDF file contains CF-conform
  • complete grid description (bounds, grid mapping parameter)
  • vertical axis (level)
  • A 'dummy' variable wich fits to the target variable in dimension sizes
• A grid info netCDF file substitutes coordinate information gained from infile

Infile

CDO

CMOR

Outfile

replace

append

predefinitons

user setting

data stream

cdo cmor

output control

1851

1852

1850

CMOR

1850_post

1851_post

1852_post

tas_
185001_185012.nc

Workflow