Engineering overview of  ESMs

• Very important: the models are of huge and growing importance to society - scientists, policy makers and the public require demonstrably high quality, credible models.                                     

• Very high complexity: millions of lines of code developed over decades. Code complexity on par with Linux kernel, Hubble space telescope, a military drone.

• Low engineering quality: the models are developed in an ad-hoc manner without engineering processes/tools/metrics. e.g. climate models typically have 0 unit tests.

• Very high risk! Just combine the points above and most engineering managers would start to worry. 

Why are we in this situation?

• Scientists are rewarded for doing science, publishing papers etc. not building systems.

• Scientific work practice often focuses on short term and local - this make sense since long-term investment may not pay off.

• For a variety of reasons software is viewed differently to other engineered scientific infrastructure such as telescopes, satellites, supercomputers.

• Limited budgets are typically spent on more tangible or measurable things than engineering quality. e.g. NCI currently spending $$$ on model performance but very little on other (more important) aspects of software quality.

ESM development approach needs to change

• The systems are becoming too complex to use in a time efficient manner

• Scientists and the public are becoming more discerning - quality (and correctness) matter

• Growing pressure from funding bodies and publishers particularly around scientific integrity and process, e.g. accessibility and reproducibility matter

• Scientists in other disciplines are making amazing breakthroughs by embracing engineering, e.g. astronomy, plasma physics

How to improve ESM quality

Idea:

Employ more engineers (and fewer scientists) - hope that scientific output eventually increases. This is a long-term and risky strategy.

How to improve ESM quality

Idea:

Modelling as a service (MaaS) 

Treat geophysical models like any other (engineered) infrastructure - such as a supercomputer or Internet connection.

• Users pay for what they consume, e.g. per CPU hour or per simulation day
• Very low risk for the user since no large investment in HR or machinery required
• The service provider (Double Precision) is highly motivated to:
  • increase the number of, and ease with which simulations are run
  • invest in the quality and performance of the model and related infrastructure for the long term (it's our livelihood)
  • work with scientists to extend the scientific reach of the model

More about Modelling as a Service

Key points about MaaS:

• as it stands model quality is of little interest or concern, under MaaS DP invests in a long-term, holistic approach to software quality. This includes such things as: testing, reliability, documentation.
• restructuring hires to solve this problem within ARCCSS, BOM is expensive and risky, MaaS reduces risk
• MaaS is customer focused and outward facing, this is an improvement on existing approaches model development in Australia.
• the model becomes a tangible, visible asset much like a piece of hardware

MaaS Status at Double Precision

• We have interest from commercial (non-research) organisations who presently buy machines and hire staff to run WRF, using a service represents a saving for them.

• We are building a relationship with Amazon scientific computing and have done preliminary testing of MOM and WRF using their HPC cloud