Skip to content

Thresholds and the CI gate

A threshold turns a metric value into pass or fail. In Python it is Comparison.check (or climval.evaluate_thresholds); from the command line and in CI it is climval check with a configuration file.

Thresholds in Python

verdict = result.check({
    "rmse": {"max": 2.0},                                     # space_time, all
    "mean_bias": {"abs_max": 1.0, "group": "*"},              # every group
    "pearson_r": {"min": 0.95, "sample": "climatology"},
    "std_ratio": {"min": 0.8, "max": 1.25, "sample": "climatology", "group": "JJA"},
})
verdict.passed            # True or False
verdict.failures          # the checks that failed
print(verdict.summary())

Each limit takes min, max and abs_max (any combination), and optionally sample (default space_time; climatology is the other scalar sample) and group (default all; "*" checks every group, for example every season). With an extra dimension such as member or model, every member is checked and must pass. A NaN value fails, because it cannot show that the limit holds. A limit must be a finite number, min may not be above max and abs_max may not be negative: anything else raises ValueError before any value is checked, because a NaN limit would pass every value and an inverted range would fail every value. A threshold that matches no value (a metric that was not computed, a group that does not exist) raises an error rather than passing silently. Metrics may be named by their aliases (bias for mean_bias, pearson for pearson_r, tss, centred_rmse; see Metrics), and a percentile bias by its percentile, percentile_bias_p99_9 for the 99.9th.

A configuration file

climval check reads YAML (.yml, .yaml), TOML (.toml) or JSON. Every check names a reference, a candidate and a variable, any compare options, and thresholds:

version: 1

defaults:                      # merged into every check; a check overrides them
  period: ["1995", "2014"]
  by: season                   # contiguous seasons: December counts in the next DJF
  min_coverage: 0.8            # the default, written out

checks:
  - name: europe-tas
    description: Near-surface temperature over Europe against ERA5
    reference: data/era5_t2m_monthly.nc          # relative to this file
    candidate:
      catalog: https://storage.googleapis.com/cmip6/pangeo-cmip6.json
      query:
        source_id: NorESM2-LM
        experiment_id: historical
        table_id: Amon
        variable_id: tas
        member_id: r1i1p1f1
      storage_options: {token: anon}
    variable: {reference: t2m, candidate: tas}
    region: {lat: [35, 72], lon: [-10, 40]}
    thresholds:
      mean_bias: {abs_max: 2.0, group: "*"}
      pearson_r: {min: 0.95, sample: climatology}
      crmse:
        - {max: 2.5}
        - {max: 1.5, sample: climatology}

  - name: nordic-land
    reference: data/era5_t2m_monthly.nc
    candidate: {path: "gs://my-bucket/run-42/tas.zarr", storage_options: {token: anon}}
    variable: {reference: t2m, candidate: tas}
    region: "ar6.land:NEU"
    thresholds:
      - {metric: mean_bias, abs_max: 1.5}
      - {metric: pearson_r, min: 0.9, sample: climatology, group: "*"}

Keys of a check

Key Value
name unique name, used with --check (default check-1, check-2, ...)
description free text shown in the reports
reference, candidate a path or URL, or a mapping (below)
variable a name, or {reference: NAME, candidate: NAME}
variables a list of those: one comparison per variable, same thresholds
region {lat: [S, N], lon: [W, E], name: ...}, [S, N], [S, N, W, E] or a regionmask name such as "ar6.land:NEU"
period [start, end], for example ["1995", "2014"]
freq h or <N>h for a whole number of hours that divides 24 (3h, 6h), D, MS or YS; default the coarser native frequency
min_coverage fraction in (0, 1] of a period's native steps that must be valid, per grid cell, when aggregating (default 0.8)
by season, month or year
seasons contiguous (default: December counts in the following DJF, incomplete seasons are left out) or calendar (every December, January and February form DJF)
regrid bilinear, nearest or conservative
target coarser (default), reference or candidate
time intersect or exact
units convert, strict, ignore or {reference: UNITS, candidate: UNITS}
weights area (default: cell area from the grid's bounds), cos_lat or none
metrics list of metric names or aliases (default: the default metrics; a threshold on any other metric, such as spearman_r or percentile_bias_p99_9, needs it listed here)
samples list of samples (default [space_time, climatology])
names [reference name, candidate name] for the reports
open_kwargs passed to the xarray opener for both inputs
thresholds a mapping metric: limits (limits may be a list), or a list of {metric: ..., min/max/abs_max, sample, group}

Keys of an input

Key Value
path path (relative to the configuration file), glob or URL to NetCDF, Zarr or CSV
catalog intake-esm catalogue (path or URL); needs query
query mapping passed to catalog.search(); must match exactly one dataset
sel mapping passed to Dataset.sel, for example {plev: 85000} or {member_id: r2i1p1f1}
storage_options fsspec options, for example {token: anon} for public Google Cloud buckets
open_kwargs further xarray open options, for example {engine: zarr}
name display name

Everything is validated before any data is opened: unknown keys, invalid choices, unknown metrics, thresholds on metrics or samples that are not computed, limits that are not finite numbers, min above max, a negative abs_max, and groups that the check's by does not produce are configuration errors (exit code 2). The groups are DJF, MAM, JJA and SON for by: season, Jan to Dec for by: month, a year such as 2001 for by: year, and all or "*" with any by. A year that is well formed but outside the data is only found when the data are read, and is then an error of that check.

Every check needs at least one threshold. A check without thresholds would pass whatever the values are, so it is a configuration error rather than a silent pass; to look at the values without gating, use climval compare.

PyYAML reads YAML 1.1, in which 1e-3 (no decimal point) is a string, not a number. Limits therefore also accept numeric strings such as "1e-3"; writing 1.0e-3 gives a number in every YAML reader.

Running it

climval check climval.yml                               # console summary
climval check climval.yml --output-dir climval-report   # .json, .md and .html
climval check climval.yml --json result.json --markdown summary.md --html report.html
climval check climval.yml --check europe-tas            # one check (repeatable)
climval check climval.yml --github-summary              # append to $GITHUB_STEP_SUMMARY
climval check climval.yml --validate                    # only validate the file
climval check climval.yml --verbose                     # tracebacks for errors

Exit codes

Code Meaning
0 every check ran and every threshold held
1 at least one threshold failed, and nothing else went wrong
2 the configuration is invalid, or a check could not run (missing file, failed query, alignment or units error)

An error takes precedence over a threshold failure. Every check runs even when an earlier one fails, so one report shows everything.

Outputs

  • Console: one block per check with each limit, the value and pass or fail, and a final line with the counts and the exit code.
  • JSON: the status, the exit code, the configuration path and its SHA-256, the environment, and per check the limits, the full comparison (every value and the provenance) or the error.
  • Markdown: a table of checks and a table of limits per check, failures first, with the alignment steps folded. It is written for a GitHub job summary.
  • HTML: a self-contained report with the same tables, every value and the alignment steps.

In Python

from climval.gate import load_config, run, run_file

report = run_file("climval.yml")
report.exit_code, report.counts()
report.to_json("result.json")
print(report.to_markdown())

Choosing limits

Limits encode a judgement about what is good enough for a purpose, and climval does not supply defaults. A common approach is to compute the values for a known good version (a previous release of the model, or several models of an ensemble), then set limits a margin away from them so that a regression fails. Keep the reference data fixed, and record why each limit has its value next to it in the file.

KGE and the percentile bias are computed on the Kelvin scale for temperatures, so a temperature bias looks small in them (about 1/280 of the value, since near-surface temperatures are about 280 K: a 1 K error in the 95th percentile is a percentile bias of about 0.36 percent), and their limits must be set accordingly.

Data for CI

CI runners have little memory and time. Compare small, fixed subsets: monthly means of a region, a reference file committed to the repository or cached between runs, and the candidate written by the job itself. The GitHub Action runs the gate and uploads the report.