Skip to content

The GitHub Action

The repository root holds a composite action that installs climval, runs climval check on a configuration file, writes the Markdown summary to the job summary and uploads the report as an artifact.

Copy-paste workflow

Save as .github/workflows/climval.yml:

name: Validate model output

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read

jobs:
  climval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Produce or download the candidate here, for example a short model run
      # that writes output/tas_monthly.nc.

      - name: climval check
        uses: northflowlabs/climval@v0.2.0
        with:
          config: climval.yml
          extras: cloud,catalog

Because the action is used at the release tag v0.2.0, it installs climval 0.2.0 from PyPI; see version below.

with a climval.yml next to it, for example:

version: 1
checks:
  - name: europe-tas
    reference: data/era5_t2m_monthly_europe.nc
    candidate: output/tas_monthly.nc
    variable: {reference: t2m, candidate: tas}
    region: {lat: [35, 72], lon: [-10, 40]}
    by: season
    thresholds:
      mean_bias: {abs_max: 2.0, group: "*"}
      pearson_r: {min: 0.95, sample: climatology}

The job fails when a threshold fails or a check cannot run. The job summary shows a table of checks and of every limit; the artifact climval-report holds climval-check.json, climval-check.md and climval-check.html.

Inputs

Input Default Meaning
config (required) path to the configuration file
version empty climval version from PyPI, for example 0.2.0. Empty: the version of the action's ref when that ref is a release tag (@v0.2.0 installs 0.2.0), otherwise (a branch, a commit SHA) the newest release. latest: always the newest release. source: install from the action's own checkout
extras empty pip extras, for example cloud,catalog
python-version 3.12 passed to actions/setup-python; empty skips it and uses the runner's system Python, which on ubuntu-24.04 refuses pip install (PEP 668), so leave it set unless an earlier step sets up Python
checks empty space-separated check names; empty runs all
args empty further arguments for climval check, for example --verbose
report-dir climval-report where the report files go
job-summary true write the Markdown summary to the job summary
upload-report true upload report-dir as an artifact
artifact-name climval-report artifact name (make it unique per job in a matrix)
fail-on threshold threshold: fail on exit codes 1 and 2; error: only on 2; never: never fail

Outputs

Output Meaning
exit-code 0 passed, 1 threshold failure, 2 configuration or data error
status pass, fail or error
report-dir directory of the report files
json path to the JSON result

Patterns

Report without gating, then decide in a later step:

      - name: climval check
        id: climval
        uses: northflowlabs/climval@v0.2.0
        with:
          config: climval.yml
          fail-on: never
      - name: Comment or label on failure
        if: steps.climval.outputs.status == 'fail'
        run: echo "A threshold failed; see the job summary"

Several configurations in a matrix, each with its own artifact:

    strategy:
      matrix:
        region: [europe, nordic, mediterranean]
    steps:
      - uses: actions/checkout@v4
      - uses: northflowlabs/climval@v0.2.0
        with:
          config: checks/${{ matrix.region }}.yml
          artifact-name: climval-${{ matrix.region }}

Cloud data needs only the cloud extra for public buckets (storage_options: {token: anon} in the configuration) and the catalog extra for intake-esm catalogues. For private buckets, set the credentials through the usual environment variables or a prior authentication step.

Pin the action to a release tag (@v0.2.0) so the gate does not change under you: the action and the climval it installs then come from the same release. If you pin the action to a commit SHA instead, also set version (for example version: "0.2.0"), because a SHA does not name a release and the default would install the newest one.

How this repository tests the action

The workflow .github/workflows/action.yml runs the action from the repository itself (uses: ./ with version: source) on the small synthetic files in tests/data/gate/: a passing configuration on Linux, Windows and macOS, a threshold failure that must fail the step with exit code 1, and a missing file that must report exit code 2.