Evaluation of AI Climate Models
Understanding and improvement of AI models will require a broad set of evaluation metrics. These should include and expand upon the rich set of metrics that have been used in physics-based model development. Existing open-source climate model evaluation tooling will therefore need to be applied to both Machine Learning (ML) and physics-based models. This call invites proposals to deliver capability enhancements to the open-source climate model evaluation tool ESMValTool, with a focus on enabling the comparison of AI and physics-based models, understanding the suitability of physics-based models for training AI models, and enabling this tooling to interface to the next generation of data formats and platforms that will be used for hosting model and observation data.
- Opening date:
- Closing date: (Midday)
Contents
Summary
Access to the documents will provide everything in terms of description of the Grant.
Understanding and improvement of AI models will require a broad set of evaluation metrics. These should include and expand upon the rich set of metrics that have been used in physics-based model development. Existing open-source climate model evaluation tooling will therefore need to be applied to both Machine Learning (ML) and physics-based models.
This call invites proposals to deliver capability enhancements to the open-source climate model evaluation tool ESMValTool, with a focus on enabling the comparison of AI and physics-based models, understanding the suitability of physics-based models for training AI models, and enabling this tooling to interface to the next generation of data formats and platforms that will be used for hosting model and observation data.
Eligibility
The following criteria must be met in order for a Bid to be eligible for a Grant Award:
(a) The Bidder must be an organisation operating and registered in the United Kingdom.
(b) The Bid must demonstrate how it contributes to the Met Office’s funding aim to develop science and innovation partnerships.
(c) The Bid must demonstrate ODA compliance.
(d) The Bid does not cover activities in relation to which the Bidder has received, or will receive, external funding.
(e) There must be an In Country economic and societal benefit to which must be demonstrated.
(f) The proposed Grant Activities in a Bid will last the full duration of the Grant Period.
(g) The Bidder must be willing and able to work with Met Office and other organisations and individuals associated with the National Capability AI Programme, including by attending meetings and other collaborative events.
Multiple Bids can be submitted from a single organisation where they are led by different academic departments.
Objectives
AI prediction and projection models are machine learning (ML) models trained on high-quality physics-based model simulation data. The training and initial verification for development of these models involves a narrow set of metrics that capture relatively simple measures of model performance in simulating present-day climate. To understand the suitability of the AI models for climate applications, a much broader set of measures and diagnostics are required that characterise model behaviour and highlight the strengths and weaknesses of AI models relative to physics-based models.
A rich set of metrics have been used for physics-based model development and evaluation that inform our understanding of the model’s effectiveness at simulating impactful weather, and in the model’s ability to represent the relationships and connections between variables and across space and time that are responsible for correctly projecting where the impacts of climate change will be felt.
Production of these broader metrics require evaluation tools that can compare AI models with physics-based model and diverse climate-quality observation datasets. These tools must have the capability to combine these datasets to calculate a diverse collection of metrics and plots. They must also have the flexibility to be adapted by evaluation scientists to answer specific questions about model behaviour. The tools and metrics must be open source and freely useable, in order to accelerate the advancement of AI climate modelling internationally.
Much of this evaluation capability exists already in open-source evaluation tools that were initially developed for physics-based models. We expect that the comprehensive evaluation tooling for AI models should not start from a blank sheet but should extend existing open-source tools to meet new needs for evaluating AI models, and to ensure that physics-based models meet the new requirement of acting as training data for AI models. The Met Office is adopting the open-source community-developed software ESMValTool (esmvaltool.org) for climate model evaluation and, wherever possible, new capabilities should be developed in conjunction with ESMValTool.
The data volumes associated with the next generation of AI and physics-based models are orders of magnitude larger than have been available to date. These datasets are going to be hosted on a heterogeneous range of formats including cloud object storage. Evaluation tools are going to need to handle these increased data volumes through a range of techniques that include interfacing to formats and platforms that allow intelligent sub-sampling or server-side processing of large datasets.
Dates
Stage
Target Times
The deadline for submitting clarification questions
Wednesday 9th July 2025, 12:00 noon
Call Return Date
Wednesday 1st August 2025, 12:00 noon
Evaluation Period
August 2025
Outcome Notification
August 2025
Grant Award Target Start Date
September 2025
Grant Award End Date
31 March 2026 (Should funding be approved for further years; the Grant Agreement will be extended on an annual basis in line with funding approvals. Initial commitment is to 31st March 2026).
How to apply
To access documentation use the following link: https://procontract.due-north.com/Advert?advertId=987b58a5-ce4d-f011-813a-005056b64545&p=cca2f054-f21a-e711-80dd-005056b64545