A CI gate on real data¶
climval check runs comparisons declared in a configuration file and exits
with 0 when every threshold holds, 1 when a threshold fails and 2 when the
configuration or the data cannot be used. In a CI job the exit code decides
whether the job passes.
This notebook builds such a gate on real open data: ERA5 near-surface air
temperature as the reference, two CMIP6 historical simulations from the
Pangeo catalogue as candidates, over the IPCC AR6 land region Northern Europe
(NEU), 2005 to 2014, by season. It then runs climval check exactly as a CI
job would, and shows the Markdown that would appear as the job summary.
About 450 MB are read anonymously from public cloud storage; the cell that opens the models prints the exact amount.
import shutil
import subprocess
import sys
import warnings
from pathlib import Path
import dask
import gcsfs
import numpy as np
import pandas as pd
import regionmask
import xarray as xr
from IPython.display import Markdown, display
import climval
warnings.filterwarnings("ignore", category=FutureWarning)
warnings.filterwarnings("ignore", message=".*Consolidated metadata.*")
OUT = Path("output")
OUT.mkdir(exist_ok=True)
fs = gcsfs.GCSFileSystem(token="anon")
print("climval", climval.__version__, "| regionmask", regionmask.__version__)
climval 0.2.0 | regionmask 0.13.0
The reference: ERA5 monthly means for the region¶
As in the Europe example, ERA5 comes from the 1.5 degree, 6-hourly ARCO-ERA5 store, whose chunks hold two days of the whole globe. To keep this example small, one two-day chunk in three is read (two days every six days, all four analyses of each day), and monthly means are estimated from the sampled days. The sampling adds noise to the reference, which is measured below before the limits are judged. The result is written to a small NetCDF file: in a real CI job this file would be computed once, from the full record, and committed or cached.
region = climval.MaskRegion(regionmask.defined_regions.ar6.land, "NEU")
box = region.box()
print(region.name, "| bounding box lat", box.lat, "lon", box.lon)
N.Europe | bounding box lat (48.0, 72.6) lon (-10.0, 40.0)
ERA5_STORE = (
"gcp-public-data-arco-era5/ar/1959-2022-6h-240x121_equiangular_with_poles_conservative.zarr"
)
era5 = xr.open_zarr("gs://" + ERA5_STORE, storage_options={"token": "anon"}, chunks={})
t2m = era5["2m_temperature"]
step = t2m.encoding["chunks"][0]
times = pd.DatetimeIndex(era5.time.values)
i0 = int(np.searchsorted(times, np.datetime64("2005-01-01")))
i1 = int(np.searchsorted(times, np.datetime64("2015-01-01")))
EVERY = 3
blocks = [k for k in range(-(-i0 // step), i1 // step) if k % EVERY == 0]
index = np.concatenate([np.arange(k * step, (k + 1) * step) for k in blocks])
sizes = {
f["name"].rsplit("/", 1)[-1]: f["size"]
for f in fs.ls(ERA5_STORE + "/2m_temperature", detail=True)
}
era5_bytes = sum(sizes[f"{k}.0.0"] for k in blocks)
print(f"{len(blocks)} chunks, {era5_bytes / 1e6:.0f} MB to read")
608 chunks, 373 MB to read
sub = t2m.isel(time=index)
sub = sub.assign_coords(longitude=((sub.longitude + 180) % 360) - 180).sortby("longitude")
sub = sub.sel(
latitude=slice(box.lat[0] - 2, box.lat[1] + 2),
longitude=slice(box.lon[0] - 2, box.lon[1] + 2),
).transpose("time", "latitude", "longitude")
with dask.config.set(scheduler="threads", num_workers=8):
sub = sub.load()
monthly = sub.resample(time="MS").mean().rename("t2m")
monthly.attrs = {"units": "K", "long_name": "2 m temperature, monthly mean of sampled days"}
reference_file = OUT / "era5_t2m_monthly_neu_2005_2014.nc"
monthly.to_dataset().assign_attrs(
title="ERA5 2 m temperature, monthly means estimated from sampled days",
source="ARCO-ERA5, " + ERA5_STORE,
sampling=f"one 2-day chunk in {EVERY}, all four analyses per day",
licence="Contains modified Copernicus Climate Change Service information 2026",
).to_netcdf(reference_file)
print(
reference_file,
f"{reference_file.stat().st_size / 1e3:.0f} kB,",
monthly.sizes["time"],
"months",
)
output\era5_t2m_monthly_neu_2005_2014.nc 348 kB, 120 months
The two models¶
The gate below opens the models itself, through the catalogue. They are also opened here, to count the bytes read and to measure the sampling noise of the reference.
import intake
cat = intake.open_esm_datastore("https://storage.googleapis.com/cmip6/pangeo-cmip6.json")
MODELS = ["NorESM2-LM", "MPI-ESM1-2-LR"]
query = cat.search(
source_id=MODELS,
experiment_id="historical",
table_id="Amon",
variable_id="tas",
member_id="r1i1p1f1",
grid_label="gn",
)
dsets = query.to_dataset_dict(
xarray_open_kwargs={"consolidated": True, "chunks": {}},
storage_options={"token": "anon"},
progressbar=False,
)
models = {ds.attrs["source_id"]: ds["tas"].squeeze(drop=True) for ds in dsets.values()}
def bytes_for_period(zstore, da, first, last):
"""Bytes of the stored chunks that hold the period (whole global chunks)."""
store = zstore.replace("gs://", "").rstrip("/") + "/tas"
chunk = xr.open_zarr(zstore, storage_options={"token": "anon"})["tas"].encoding["chunks"][0]
years = da.time.dt.year.values
idx = np.nonzero((years >= first) & (years <= last))[0]
files = {f["name"].rsplit("/", 1)[-1]: f["size"] for f in fs.ls(store, detail=True)}
return sum(files[f"{c}.0.0"] for c in np.unique(idx // chunk))
cmip_bytes = sum(
bytes_for_period(row.zstore, models[row.source_id], 2005, 2014) for row in query.df.itertuples()
)
print(
f"ERA5 {era5_bytes / 1e6:.0f} MB, CMIP6 {cmip_bytes / 1e6:.0f} MB, "
f"total {(era5_bytes + cmip_bytes) / 1e6:.0f} MB read"
)
ERA5 373 MB, CMIP6 74 MB, total 447 MB read
How much of a score is sampling noise?¶
The ERA5 blocks are split into four interleaved subsets (block 1, 5, 9, ...; block 2, 6, 10, ...; and so on). Each subset holds one two-day chunk in twelve and gives its own reference; each model is scored against each of the four with the gate's settings. The spread of the four values estimates the sampling noise of a reference built from one chunk in twelve; the reference used by the gate holds four times as many days, so its noise is about half that spread (the standard deviation of the four values divided by the square root of four). The centred RMS difference also grows with the noise of the reference, so a sparser sample makes it larger on average.
SETTINGS = dict(
region="ar6.land:NEU",
period=("2005", "2014"),
by="season",
metrics=["crmse", "mean_bias", "pearson_r"],
samples=("climatology",),
)
noise_rows = []
with warnings.catch_warnings(), dask.config.set(scheduler="threads", num_workers=4):
warnings.simplefilter("ignore", UserWarning)
for part in range(4):
keep = np.concatenate(
[np.arange(j * step, (j + 1) * step) for j in range(part, len(blocks), 4)]
)
part_ref = sub.isel(time=keep).resample(time="MS").mean().rename("t2m")
part_ref.attrs = {"units": "K"}
for source, tas in models.items():
res = climval.compare(part_ref, tas, **SETTINGS)
table_ = res.to_dataframe()
for row in table_.itertuples():
noise_rows.append(
{
"model": source,
"subset": part + 1,
"group": row.group,
"metric": row.metric,
"value": row.value,
}
)
noise = pd.DataFrame(noise_rows)
noise = noise[noise.metric != "n_points"]
spread = noise.groupby(["model", "metric", "group"])["value"].agg(["mean", "std"])
spread["standard error of the gate's reference"] = spread["std"] / np.sqrt(4)
crmse_noise = spread.xs("crmse", level="metric").unstack("group")
crmse_noise.round(3)
| mean | std | standard error of the gate's reference | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| group | DJF | JJA | MAM | SON | all | DJF | JJA | MAM | SON | all | DJF | JJA | MAM | SON | all |
| model | |||||||||||||||
| MPI-ESM1-2-LR | 1.623 | 1.062 | 1.035 | 1.444 | 1.190 | 0.214 | 0.126 | 0.080 | 0.049 | 0.091 | 0.107 | 0.063 | 0.04 | 0.024 | 0.045 |
| NorESM2-LM | 2.019 | 1.062 | 1.211 | 0.888 | 1.066 | 0.218 | 0.089 | 0.099 | 0.043 | 0.014 | 0.109 | 0.044 | 0.05 | 0.021 | 0.007 |
The configuration¶
Three checks share a defaults block: the reference file, the region, the
period, seasonal groups and the thresholds. Each check names one model
through a query of the Pangeo CMIP6 catalogue. Each model is compared on the
coarser of its grid and the ERA5 grid (the default, target: coarser), which
for both models is the model grid, so the model is not interpolated to a
finer grid than it has. The seasons are contiguous by default, so DJF holds
the nine complete winters from December 2005 to February 2014. The third
check replaces the shared thresholds with one stricter limit, on the summer
(JJA) mean bias of NorESM2-LM, to show a failing check; it reuses the data
the first check opened. The limits are set for this demonstration; they are
not a recommendation for what a model should achieve.
config_text = """\
version: 1
defaults:
reference: {path: era5_t2m_monthly_neu_2005_2014.nc, name: ERA5}
variable: {reference: t2m, candidate: tas}
region: "ar6.land:NEU"
period: ["2005", "2014"]
by: season
samples: [space_time, climatology]
thresholds:
mean_bias: {abs_max: 1.5, sample: climatology, group: "*"}
pearson_r: {min: 0.9, sample: climatology, group: "*"}
crmse: {max: 2.0, sample: climatology, group: "*"}
checks:
- name: noresm2-lm
description: NorESM2-LM historical r1i1p1f1 against ERA5, AR6 Northern Europe land
candidate:
catalog: https://storage.googleapis.com/cmip6/pangeo-cmip6.json
query: {source_id: NorESM2-LM, experiment_id: historical, table_id: Amon,
variable_id: tas, member_id: r1i1p1f1, grid_label: gn}
storage_options: {token: anon}
open_kwargs: {consolidated: true}
- name: mpi-esm1-2-lr
description: MPI-ESM1-2-LR historical r1i1p1f1 against ERA5, AR6 Northern Europe land
candidate:
catalog: https://storage.googleapis.com/cmip6/pangeo-cmip6.json
query: {source_id: MPI-ESM1-2-LR, experiment_id: historical, table_id: Amon,
variable_id: tas, member_id: r1i1p1f1, grid_label: gn}
storage_options: {token: anon}
open_kwargs: {consolidated: true}
- name: noresm2-lm-summer
description: NorESM2-LM summer mean bias within 0.5 K, AR6 Northern Europe land
candidate:
catalog: https://storage.googleapis.com/cmip6/pangeo-cmip6.json
query: {source_id: NorESM2-LM, experiment_id: historical, table_id: Amon,
variable_id: tas, member_id: r1i1p1f1, grid_label: gn}
storage_options: {token: anon}
open_kwargs: {consolidated: true}
thresholds:
mean_bias: {abs_max: 0.5, sample: climatology, group: JJA}
"""
config = OUT / "climval.yml"
config.write_text(config_text, encoding="utf-8")
print(config_text)
version: 1
defaults:
reference: {path: era5_t2m_monthly_neu_2005_2014.nc, name: ERA5}
variable: {reference: t2m, candidate: tas}
region: "ar6.land:NEU"
period: ["2005", "2014"]
by: season
samples: [space_time, climatology]
thresholds:
mean_bias: {abs_max: 1.5, sample: climatology, group: "*"}
pearson_r: {min: 0.9, sample: climatology, group: "*"}
crmse: {max: 2.0, sample: climatology, group: "*"}
checks:
- name: noresm2-lm
description: NorESM2-LM historical r1i1p1f1 against ERA5, AR6 Northern Europe land
candidate:
catalog: https://storage.googleapis.com/cmip6/pangeo-cmip6.json
query: {source_id: NorESM2-LM, experiment_id: historical, table_id: Amon,
variable_id: tas, member_id: r1i1p1f1, grid_label: gn}
storage_options: {token: anon}
open_kwargs: {consolidated: true}
- name: mpi-esm1-2-lr
description: MPI-ESM1-2-LR historical r1i1p1f1 against ERA5, AR6 Northern Europe land
candidate:
catalog: https://storage.googleapis.com/cmip6/pangeo-cmip6.json
query: {source_id: MPI-ESM1-2-LR, experiment_id: historical, table_id: Amon,
variable_id: tas, member_id: r1i1p1f1, grid_label: gn}
storage_options: {token: anon}
open_kwargs: {consolidated: true}
- name: noresm2-lm-summer
description: NorESM2-LM summer mean bias within 0.5 K, AR6 Northern Europe land
candidate:
catalog: https://storage.googleapis.com/cmip6/pangeo-cmip6.json
query: {source_id: NorESM2-LM, experiment_id: historical, table_id: Amon,
variable_id: tas, member_id: r1i1p1f1, grid_label: gn}
storage_options: {token: anon}
open_kwargs: {consolidated: true}
thresholds:
mean_bias: {abs_max: 0.5, sample: climatology, group: JJA}
--validate checks the file without opening any data. A typo, an unknown
metric or a threshold on a sample that is not computed would exit with 2
here.
cli = [sys.executable, "-m", "climval.cli"]
done = subprocess.run([*cli, "check", str(config), "--validate"], capture_output=True, text=True)
print(done.stdout, "exit code", done.returncode)
climval check: output\climval.yml is valid (3 checks: noresm2-lm, mpi-esm1-2-lr, noresm2-lm-summer) exit code 0
Run the gate¶
This is the command a CI job runs. --output-dir writes the JSON result,
the Markdown summary and the HTML report.
# A report left over from an earlier run must not be read if this run fails.
shutil.rmtree(OUT / "report", ignore_errors=True)
done = subprocess.run(
[*cli, "check", str(config), "--output-dir", str(OUT / "report")],
capture_output=True,
text=True,
timeout=1200,
)
print(done.stdout)
print("exit code", done.returncode)
if done.returncode not in (0, 1):
print(done.stderr)
climval check: running noresm2-lm
climval check: running mpi-esm1-2-lr
climval check: running noresm2-lm-summer
========================================================================
climval check: output\climval.yml
========================================================================
PASS noresm2-lm (t2m/tas)
reference: era5_t2m_monthly_neu_2005_2014.nc
candidate: pangeo-cmip6.json [source_id=NorESM2-LM, experiment_id=historical, table_id=Amon, variable_id=tas, member_id=r1i1p1f1, grid_label=gn]
ok mean_bias [climatology, all] = -0.3536 (limit abs <= 1.5)
ok mean_bias [climatology, DJF] = 0.552 (limit abs <= 1.5)
ok mean_bias [climatology, MAM] = -0.02619 (limit abs <= 1.5)
ok mean_bias [climatology, JJA] = -1.054 (limit abs <= 1.5)
ok mean_bias [climatology, SON] = -0.7221 (limit abs <= 1.5)
ok pearson_r [climatology, all] = 0.9674 (limit >= 0.9)
ok pearson_r [climatology, DJF] = 0.9418 (limit >= 0.9)
ok pearson_r [climatology, MAM] = 0.9694 (limit >= 0.9)
ok pearson_r [climatology, JJA] = 0.9595 (limit >= 0.9)
ok pearson_r [climatology, SON] = 0.9728 (limit >= 0.9)
ok crmse [climatology, all] = 1.065 (limit <= 2)
ok crmse [climatology, DJF] = 1.912 (limit <= 2)
ok crmse [climatology, MAM] = 1.201 (limit <= 2)
ok crmse [climatology, JJA] = 1.053 (limit <= 2)
ok crmse [climatology, SON] = 0.8812 (limit <= 2)
10.7 s
PASS mpi-esm1-2-lr (t2m/tas)
reference: era5_t2m_monthly_neu_2005_2014.nc
candidate: pangeo-cmip6.json [source_id=MPI-ESM1-2-LR, experiment_id=historical, table_id=Amon, variable_id=tas, member_id=r1i1p1f1, grid_label=gn]
ok mean_bias [climatology, all] = -0.4057 (limit abs <= 1.5)
ok mean_bias [climatology, DJF] = -0.6979 (limit abs <= 1.5)
ok mean_bias [climatology, MAM] = -0.07895 (limit abs <= 1.5)
ok mean_bias [climatology, JJA] = -0.1532 (limit abs <= 1.5)
ok mean_bias [climatology, SON] = -0.6268 (limit abs <= 1.5)
ok pearson_r [climatology, all] = 0.9506 (limit >= 0.9)
ok pearson_r [climatology, DJF] = 0.9865 (limit >= 0.9)
ok pearson_r [climatology, MAM] = 0.956 (limit >= 0.9)
ok pearson_r [climatology, JJA] = 0.9548 (limit >= 0.9)
ok pearson_r [climatology, SON] = 0.9437 (limit >= 0.9)
ok crmse [climatology, all] = 1.193 (limit <= 2)
ok crmse [climatology, DJF] = 1.638 (limit <= 2)
ok crmse [climatology, MAM] = 0.9887 (limit <= 2)
ok crmse [climatology, JJA] = 1.03 (limit <= 2)
ok crmse [climatology, SON] = 1.449 (limit <= 2)
3.6 s
FAIL noresm2-lm-summer (t2m/tas)
reference: era5_t2m_monthly_neu_2005_2014.nc
candidate: pangeo-cmip6.json [source_id=NorESM2-LM, experiment_id=historical, table_id=Amon, variable_id=tas, member_id=r1i1p1f1, grid_label=gn]
FAIL mean_bias [climatology, JJA] = -1.054 (limit abs <= 0.5)
3.1 s
FAIL: threshold failure (2 of 3 checks passed, 1 failed, 0 errors; exit code 1)
Written: output\report\climval-check.json, output\report\climval-check.md, output\report\climval-check.html
exit code 1
The job summary¶
With --github-summary (or the GitHub Action) the same Markdown is appended
to $GITHUB_STEP_SUMMARY and appears on the run's summary page.
display(Markdown((OUT / "report" / "climval-check.md").read_text(encoding="utf-8")))
climval check¶
FAIL: threshold failure (2 of 3 checks passed, 1 failed, 0 errors; exit code 1)
Configuration: output\climval.yml (sha256 42d9a7fc070e)
| Status | Check | Variable | Limits passed | Reference | Candidate |
|---|---|---|---|---|---|
| PASS | noresm2-lm | t2m/tas | 15 of 15 | era5_t2m_monthly_neu_2005_2014.nc | pangeo-cmip6.json [source_id=NorESM2-LM, experiment_id=historical, table_id=Amon, variable_id=tas, member_id=r1i1p1f1, grid_label=gn] |
| PASS | mpi-esm1-2-lr | t2m/tas | 15 of 15 | era5_t2m_monthly_neu_2005_2014.nc | pangeo-cmip6.json [source_id=MPI-ESM1-2-LR, experiment_id=historical, table_id=Amon, variable_id=tas, member_id=r1i1p1f1, grid_label=gn] |
| FAIL | noresm2-lm-summer | t2m/tas | 0 of 1 | era5_t2m_monthly_neu_2005_2014.nc | pangeo-cmip6.json [source_id=NorESM2-LM, experiment_id=historical, table_id=Amon, variable_id=tas, member_id=r1i1p1f1, grid_label=gn] |
PASS: noresm2-lm (t2m/tas)¶
NorESM2-LM historical r1i1p1f1 against ERA5, AR6 Northern Europe land
NorESM2-LM r1i1p1f1 against ERA5, variable t2m in K.
| Result | Metric | Sample, group | Value | Units | Limit |
|---|---|---|---|---|---|
| pass | mean_bias | climatology, all | -0.3536 | K | abs <= 1.5 |
| pass | mean_bias | climatology, DJF | 0.552 | K | abs <= 1.5 |
| pass | mean_bias | climatology, MAM | -0.02619 | K | abs <= 1.5 |
| pass | mean_bias | climatology, JJA | -1.054 | K | abs <= 1.5 |
| pass | mean_bias | climatology, SON | -0.7221 | K | abs <= 1.5 |
| pass | pearson_r | climatology, all | 0.9674 | 1 | >= 0.9 |
| pass | pearson_r | climatology, DJF | 0.9418 | 1 | >= 0.9 |
| pass | pearson_r | climatology, MAM | 0.9694 | 1 | >= 0.9 |
| pass | pearson_r | climatology, JJA | 0.9595 | 1 | >= 0.9 |
| pass | pearson_r | climatology, SON | 0.9728 | 1 | >= 0.9 |
| pass | crmse | climatology, all | 1.065 | K | <= 2 |
| pass | crmse | climatology, DJF | 1.912 | K | <= 2 |
| pass | crmse | climatology, MAM | 1.201 | K | <= 2 |
| pass | crmse | climatology, JJA | 1.053 | K | <= 2 |
| pass | crmse | climatology, SON | 0.8812 | K | <= 2 |
Alignment steps
- inputs: reference_variable=t2m, candidate_variable=tas
- coordinates: reference={...}, candidate={"grid": "rectilinear", "detected": {"lat": "lat", "lon": "lon", "time": "time"}, "changes": []}
- target_grid: target=candidate, requested=coarser, regridded=["reference"]
- longitude: frame=[-180.0, 180.0], reference=-180..180 unchanged, candidate=0..360 re-expressed in frame
- region: region={...}, reference_points={"lat": 19, "lon": 35}, candidate_points={"lat": 13, "lon": 21}
- time: calendars={"reference": "proleptic_gregorian", "candidate": "noleap"}, native_frequency={"reference": "monthly (from time steps)", "candidate": "monthly (from time bounds)"}, relabelled_to_bound_midpoints=["candidate"], period=["2005", "2014"], frequency=monthly, matched_periods=120, unmatched_dropped={"reference": 0, "candidate": 0}, method=periods matched by calendar labels (year-month), not by timestamp, range=["2005-01-01T00:00:00.000000000", "2014-12-01T00:00:00.000000000"]
- regrid: source=reference, method=bilinear, target_grid={...}, source_grid={...}, note=the target grid is coarser than the source along lon: bilinear samples the source at the target points and does not average over target cells; regrid='conservative' gives cell means, target_points_outside_source={"lat": 0, "lon": 0}, target_points_in_edge_half_cell={"lat": 0, "lon": 0}, target_points_in_source_gaps={"lat": 0, "lon": 0}, action=bilinear interpolation with xarray.interp; target points within half a source cell beyond the outermost source centre take the edge value; points farther out, or inside a gap of the source grid, are NaN
- region_mask: region=N.Europe, cells_inside=180
- units: reference_units=K, candidate_units=K, mode=convert, action=units match
- missing: paired mask: a point counts only where reference and candidate are both valid
- weights: weights=cell area from bounds (latitude bounds from dataset bounds, longitude bounds from dataset bounds), time=each time step weighted by the length of its period in days (monthly, reference calendar)
PASS: mpi-esm1-2-lr (t2m/tas)¶
MPI-ESM1-2-LR historical r1i1p1f1 against ERA5, AR6 Northern Europe land
MPI-ESM1-2-LR r1i1p1f1 against ERA5, variable t2m in K.
| Result | Metric | Sample, group | Value | Units | Limit |
|---|---|---|---|---|---|
| pass | mean_bias | climatology, all | -0.4057 | K | abs <= 1.5 |
| pass | mean_bias | climatology, DJF | -0.6979 | K | abs <= 1.5 |
| pass | mean_bias | climatology, MAM | -0.07895 | K | abs <= 1.5 |
| pass | mean_bias | climatology, JJA | -0.1532 | K | abs <= 1.5 |
| pass | mean_bias | climatology, SON | -0.6268 | K | abs <= 1.5 |
| pass | pearson_r | climatology, all | 0.9506 | 1 | >= 0.9 |
| pass | pearson_r | climatology, DJF | 0.9865 | 1 | >= 0.9 |
| pass | pearson_r | climatology, MAM | 0.956 | 1 | >= 0.9 |
| pass | pearson_r | climatology, JJA | 0.9548 | 1 | >= 0.9 |
| pass | pearson_r | climatology, SON | 0.9437 | 1 | >= 0.9 |
| pass | crmse | climatology, all | 1.193 | K | <= 2 |
| pass | crmse | climatology, DJF | 1.638 | K | <= 2 |
| pass | crmse | climatology, MAM | 0.9887 | K | <= 2 |
| pass | crmse | climatology, JJA | 1.03 | K | <= 2 |
| pass | crmse | climatology, SON | 1.449 | K | <= 2 |
Alignment steps
- inputs: reference_variable=t2m, candidate_variable=tas
- coordinates: reference={...}, candidate={"grid": "rectilinear", "detected": {"lat": "lat", "lon": "lon", "time": "time"}, "changes": []}
- target_grid: target=candidate, requested=coarser, regridded=["reference"]
- longitude: frame=[-180.0, 180.0], reference=-180..180 unchanged, candidate=0..360 re-expressed in frame
- region: region={...}, reference_points={"lat": 19, "lon": 35}, candidate_points={"lat": 13, "lon": 27}
- time: calendars={"reference": "proleptic_gregorian", "candidate": "proleptic_gregorian"}, native_frequency={"reference": "monthly (from time steps)", "candidate": "monthly (from time bounds)"}, relabelled_to_bound_midpoints=["candidate"], period=["2005", "2014"], frequency=monthly, matched_periods=120, unmatched_dropped={"reference": 0, "candidate": 0}, method=periods matched by calendar labels (year-month), not by timestamp, range=["2005-01-01T00:00:00.000000000", "2014-12-01T00:00:00.000000000"]
- regrid: source=reference, method=bilinear, target_grid={...}, source_grid={...}, target_points_outside_source={"lat": 0, "lon": 0}, target_points_in_edge_half_cell={"lat": 0, "lon": 0}, target_points_in_source_gaps={"lat": 0, "lon": 0}, action=bilinear interpolation with xarray.interp; target points within half a source cell beyond the outermost source centre take the edge value; points farther out, or inside a gap of the source grid, are NaN
- region_mask: region=N.Europe, cells_inside=262
- units: reference_units=K, candidate_units=K, mode=convert, action=units match
- missing: paired mask: a point counts only where reference and candidate are both valid
- weights: weights=cell area from bounds (latitude bounds from dataset bounds, longitude bounds from dataset bounds), time=each time step weighted by the length of its period in days (monthly, reference calendar)
FAIL: noresm2-lm-summer (t2m/tas)¶
NorESM2-LM summer mean bias within 0.5 K, AR6 Northern Europe land
NorESM2-LM r1i1p1f1 against ERA5, variable t2m in K.
| Result | Metric | Sample, group | Value | Units | Limit |
|---|---|---|---|---|---|
| fail | mean_bias | climatology, JJA | -1.054 | K | abs <= 0.5 |
Alignment steps
- inputs: reference_variable=t2m, candidate_variable=tas
- coordinates: reference={...}, candidate={"grid": "rectilinear", "detected": {"lat": "lat", "lon": "lon", "time": "time"}, "changes": []}
- target_grid: target=candidate, requested=coarser, regridded=["reference"]
- longitude: frame=[-180.0, 180.0], reference=-180..180 unchanged, candidate=0..360 re-expressed in frame
- region: region={...}, reference_points={"lat": 19, "lon": 35}, candidate_points={"lat": 13, "lon": 21}
- time: calendars={"reference": "proleptic_gregorian", "candidate": "noleap"}, native_frequency={"reference": "monthly (from time steps)", "candidate": "monthly (from time bounds)"}, relabelled_to_bound_midpoints=["candidate"], period=["2005", "2014"], frequency=monthly, matched_periods=120, unmatched_dropped={"reference": 0, "candidate": 0}, method=periods matched by calendar labels (year-month), not by timestamp, range=["2005-01-01T00:00:00.000000000", "2014-12-01T00:00:00.000000000"]
- regrid: source=reference, method=bilinear, target_grid={...}, source_grid={...}, note=the target grid is coarser than the source along lon: bilinear samples the source at the target points and does not average over target cells; regrid='conservative' gives cell means, target_points_outside_source={"lat": 0, "lon": 0}, target_points_in_edge_half_cell={"lat": 0, "lon": 0}, target_points_in_source_gaps={"lat": 0, "lon": 0}, action=bilinear interpolation with xarray.interp; target points within half a source cell beyond the outermost source centre take the edge value; points farther out, or inside a gap of the source grid, are NaN
- region_mask: region=N.Europe, cells_inside=180
- units: reference_units=K, candidate_units=K, mode=convert, action=units match
- missing: paired mask: a point counts only where reference and candidate are both valid
- weights: weights=cell area from bounds (latitude bounds from dataset bounds, longitude bounds from dataset bounds), time=each time step weighted by the length of its period in days (monthly, reference calendar)
Reading the result¶
The JSON result holds the status, the exit code and, per check, every limit and the full comparison with its provenance. A later job step, a dashboard or a comparison with the previous run can read it.
import json
result = json.loads((OUT / "report" / "climval-check.json").read_text(encoding="utf-8"))
print("status", result["status"], "| exit code", result["exit_code"], "| counts", result["counts"])
for check in result["checks"]:
failed = [
f"{t['metric']} [{t['group']}] = {t['value']:.3f}"
for t in check["thresholds"]
if not t["passed"]
]
print(
f"{check['check']}: {check['status']}"
+ (f"; failed: {', '.join(failed)}" if failed else "")
)
status fail | exit code 1 | counts {'pass': 2, 'fail': 1, 'error': 0}
noresm2-lm: pass
mpi-esm1-2-lr: pass
noresm2-lm-summer: fail; failed: mean_bias [JJA] = -1.054
Each limit against its value, with the standard error from the sampling of
the reference estimated above. margin in standard errors is the distance
from the limit (positive inside, negative outside) in units of that error.
The standard error comes from only four subsets, so it has three degrees of
freedom and is itself uncertain: the two-sided 95% critical value of
Student's t with three degrees of freedom is about 3.18, so a margin needs
to be about three standard errors either way before it is clear of the
sampling noise. The table lists the failed limits and those within three
standard errors of their limit.
source_of = {
"noresm2-lm": "NorESM2-LM",
"noresm2-lm-summer": "NorESM2-LM",
"mpi-esm1-2-lr": "MPI-ESM1-2-LR",
}
judged = []
for check in result["checks"]:
for t in check["thresholds"]:
se = spread.loc[
(source_of[check["check"]], t["metric"], t["group"]),
"standard error of the gate's reference",
]
value = t["value"]
if "abs_max" in t["limits"]:
margin = t["limits"]["abs_max"] - abs(value)
elif "max" in t["limits"]:
margin = t["limits"]["max"] - value
else:
margin = value - t["limits"]["min"]
judged.append(
{
"check": check["check"],
"metric": t["metric"],
"group": t["group"],
"value": value,
"limit": t["limits"],
"passed": t["passed"],
"standard error": se,
"margin in standard errors": margin / se,
}
)
judged = pd.DataFrame(judged)
judged[(judged["margin in standard errors"].abs() < 3) | ~judged["passed"]].round(3)
| check | metric | group | value | limit | passed | standard error | margin in standard errors | |
|---|---|---|---|---|---|---|---|---|
| 3 | noresm2-lm | mean_bias | JJA | -1.054 | {'abs_max': 1.5} | True | 0.172 | 2.588 |
| 11 | noresm2-lm | crmse | DJF | 1.912 | {'max': 2.0} | True | 0.109 | 0.811 |
| 30 | noresm2-lm-summer | mean_bias | JJA | -1.054 | {'abs_max': 0.5} | False | 0.172 | -3.215 |
The summer limit of the third check fails by 3.2 standard errors: the JJA mean bias of NorESM2-LM is -1.05 K against an absolute limit of 0.5 K, with a standard error of 0.17 K. That is just beyond the 95% critical value of about 3.18 for three degrees of freedom, so the failure is unlikely to be an artefact of the sampled reference. The winter centred RMS difference of NorESM2-LM passes, 1.91 K against a limit of 2.0 K, but only 0.8 standard errors inside the limit, so that pass is not clear of the noise. The subset scores above show what sparser sampling does: references built from one chunk in twelve give this value as 2.02 K on average, above the limit, and a gate run on one of them could fail on sampling noise alone.
rows = [
{**row, "check": check["check"]}
for check in result["checks"]
for row in check["comparison"]["values"]
if row["sample"] == "climatology" and row["metric"] in ("mean_bias", "pearson_r", "crmse")
]
pd.DataFrame(rows).pivot_table(index=["check", "group"], columns="metric", values="value").round(3)
| metric | crmse | mean_bias | pearson_r | |
|---|---|---|---|---|
| check | group | |||
| mpi-esm1-2-lr | DJF | 1.638 | -0.698 | 0.986 |
| JJA | 1.030 | -0.153 | 0.955 | |
| MAM | 0.989 | -0.079 | 0.956 | |
| SON | 1.449 | -0.627 | 0.944 | |
| all | 1.193 | -0.406 | 0.951 | |
| noresm2-lm | DJF | 1.912 | 0.552 | 0.942 |
| JJA | 1.053 | -1.054 | 0.959 | |
| MAM | 1.201 | -0.026 | 0.969 | |
| SON | 0.881 | -0.722 | 0.973 | |
| all | 1.065 | -0.354 | 0.967 | |
| noresm2-lm-summer | DJF | 1.912 | 0.552 | 0.942 |
| JJA | 1.053 | -1.054 | 0.959 | |
| MAM | 1.201 | -0.026 | 0.969 | |
| SON | 0.881 | -0.722 | 0.973 | |
| all | 1.065 | -0.354 | 0.967 |
for check in result["checks"]:
steps = {s["step"]: s for s in check["comparison"]["provenance"]["steps"]}
print(
f"{check['check']}: comparison grid {steps['target_grid']['target']}, "
f"{steps['region_mask']['cells_inside']} AR6 NEU land cells"
)
noresm2-lm: comparison grid candidate, 180 AR6 NEU land cells mpi-esm1-2-lr: comparison grid candidate, 262 AR6 NEU land cells noresm2-lm-summer: comparison grid candidate, 180 AR6 NEU land cells
The same run from Python, without the command line:
from climval.gate import run_file
report = run_file("output/climval.yml")
report.exit_code, report.counts()
report.outcomes[0].comparison.to_dataframe()
What this shows¶
The printed console output, the job summary and the table above come from the same run: each limit, its value and whether it held. The exit code is what a CI job acts on. With limits like these in a repository, a change to a model configuration, a post-processing script or the data pipeline that moves one of these numbers past its limit makes the job fail, and the summary says which number moved.
The values describe two single-member historical simulations against ERA5 monthly means estimated from sampled days over AR6 Northern Europe land, 2005 to 2014. The sampling adds noise to the ERA5 seasonal means, measured above; a limit is only a reliable gate where the value is clear of it by a few standard errors. For a production gate, compute the reference from the full record once and commit or cache it, and set limits with the variability between ensemble members or reanalyses in mind.
In a workflow¶
- uses: northflowlabs/climval@v0.2.0
with:
config: climval.yml
version: "0.2.0"
extras: cloud,catalog,regions
See the GitHub Action for the inputs and outputs.
Data and licences¶
ERA5: Hersbach et al. (2020), https://doi.org/10.1002/qj.3803, read from ARCO-ERA5 (https://github.com/google-research/arco-era5). The Copernicus Climate Data Store publishes ERA5 under the CC-BY licence, that is Creative Commons Attribution 4.0 (https://creativecommons.org/licenses/by/4.0/; ERA5 hourly data on single levels, https://doi.org/10.24381/cds.adbb2d47). Contains modified Copernicus Climate Change Service information 2026: ARCO-ERA5 regridded the data to 1.5 degrees, and this notebook sampled them and averaged them to monthly means. Neither the European Commission nor ECMWF is responsible for any use that may be made of the Copernicus information or data it contains.
CMIP6: NorESM2-LM (NCC) and MPI-ESM1-2-LR (MPI-M), historical, r1i1p1f1, from the Pangeo CMIP6 collection, used under CC BY 4.0. Both datasets were published under CC BY-SA 4.0, and their files still say so (printed below); their modelling groups relaxed the licence to CC BY 4.0 in June 2022 (NorESM2-LM on 2022-06-03, MPI-ESM1-2-LR on 2022-06-16), as all CMIP6 modelling groups did by October 2022: see the CMIP6 Terms of Use (https://pcmdi.llnl.gov/CMIP6/TermsOfUse/TermsOfUse6-2.html) and the per-model licence history (https://wcrp-cmip.github.io/CMIP6_CVs/docs/CMIP6_source_id_licenses.html). The cell below prints the attributes recorded from each dataset. Eyring et al. (2016), https://doi.org/10.5194/gmd-9-1937-2016. We acknowledge the World Climate Research Programme, which, through its Working Group on Coupled Modelling, coordinated and promoted CMIP6, the climate modelling groups for producing and making available their model output, the Earth System Grid Federation for archiving the data and providing access, and the funding agencies who support CMIP6 and ESGF.
Region: IPCC AR6 reference region NEU, Iturbide et al. (2020), https://doi.org/10.5194/essd-12-2959-2020, through regionmask.
for check in result["checks"]:
attrs = check["comparison"]["provenance"]["inputs"]["candidate"]["global_attributes"]
print(
check["check"],
"|",
{
k: attrs.get(k)
for k in ("source_id", "institution_id", "variant_label", "further_info_url")
},
)
for source, ds in sorted((d.attrs["source_id"], d) for d in dsets.values()):
print(source, "| licence in the file:", ds.attrs.get("license", "")[:160], "...")
noresm2-lm | {'source_id': 'NorESM2-LM', 'institution_id': 'NCC', 'variant_label': 'r1i1p1f1', 'further_info_url': 'https://furtherinfo.es-doc.org/CMIP6.NCC.NorESM2-LM.historical.none.r1i1p1f1'}
mpi-esm1-2-lr | {'source_id': 'MPI-ESM1-2-LR', 'institution_id': 'MPI-M', 'variant_label': 'r1i1p1f1', 'further_info_url': 'https://furtherinfo.es-doc.org/CMIP6.MPI-M.MPI-ESM1-2-LR.historical.none.r1i1p1f1'}
noresm2-lm-summer | {'source_id': 'NorESM2-LM', 'institution_id': 'NCC', 'variant_label': 'r1i1p1f1', 'further_info_url': 'https://furtherinfo.es-doc.org/CMIP6.NCC.NorESM2-LM.historical.none.r1i1p1f1'}
MPI-ESM1-2-LR | licence in the file: CMIP6 model data produced by MPI-M is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). ...
NorESM2-LM | licence in the file: CMIP6 model data produced by NCC is licensed under a Creative Commons Attribution ShareAlike 4.0 International License (https://creativecommons.org/licenses). C ...