Metrics¶
Every metric is computed over the points of one sample with weights \(w_i \ge 0\), which are zero where either input is missing. A weight is the cell area, times the length of the time step in days for monthly and yearly data when the sample pools time steps. Differences are candidate minus reference. With the weighted mean
the weighted standard deviations and covariance are population values:
Every metric is an xarray reduction, so dask arrays stay lazy and one
dask.compute call computes all values. The test suite checks rmse, mae,
mean_bias, pearson_r and spearman_r against xskillscore to a relative
tolerance of 1e-10, with and without weights and missing values, and the
metrics xskillscore does not have against independent NumPy definitions to
1e-9.
| Name | Definition | Units | Best |
|---|---|---|---|
mean_bias |
\(\langle c - r \rangle\) | data | 0 |
mae |
\(\langle \lvert c - r \rvert \rangle\) | data | 0 |
rmse |
\(\sqrt{\langle (c - r)^2 \rangle}\) | data | 0 |
crmse |
\(\sqrt{\langle ((c - \langle c \rangle) - (r - \langle r \rangle))^2 \rangle}\) | data | 0 |
nrmse |
\(\mathrm{rmse} / \sigma_r\) | 1 | 0 |
pearson_r |
\(\operatorname{cov}(r, c) / (\sigma_r \sigma_c)\) | 1 | 1 |
spearman_r |
pearson_r of the ranks (by name only) |
1 | 1 |
std_ratio |
\(\sigma_c / \sigma_r\) | 1 | 1 |
taylor_skill_score |
\(\dfrac{4 (1 + R)^4}{(s + 1/s)^2 (1 + R_0)^4}\), \(R\) = pearson_r, \(s\) = std_ratio, \(R_0 = 1\) |
1 | 1 |
kge |
\(1 - \sqrt{(R - 1)^2 + (\alpha - 1)^2 + (\beta - 1)^2}\), \(\alpha = \sigma_c / \sigma_r\), \(\beta = \langle c \rangle / \langle r \rangle\) | 1 | 1 |
percentile_bias_pN |
\(100\,(Q_c - Q_r) / \lvert Q_r \rvert\), \(Q\) the weighted N-th percentile | % | 0 |
n_points is reported with every sample: the number of points pooled, that
is points where both values are valid and the weight is positive. Points with
zero weight (for example from a zero area or time weight) take no part in any
metric and are not counted.
Metrics can be named by aliases wherever a name is taken (metrics=,
thresholds, climval check files): bias and me for mean_bias,
pearson for pearson_r, spearman for spearman_r, tss for
taylor_skill_score, centred_rmse and centered_rmse for crmse. Results
always use the canonical name. A percentile bias is named after its
percentile, with _ for a decimal point: percentile_bias_p95,
percentile_bias_p99_9 (percentile_bias_p99.9 is read as the same metric).
Notes on use¶
- Centred RMSE and the Taylor diagram.
crmse,std_ratioandpearson_rsatisfy \(\mathrm{crmse}^2 = \sigma_r^2 + \sigma_c^2 - 2 \sigma_r \sigma_c R\) (Taylor 2001, eq. 4), so a Taylor diagram can be drawn from theclimatologysample alone. The total error splits as \(\mathrm{rmse}^2 = \mathrm{mean\_bias}^2 + \mathrm{crmse}^2\). - Kling-Gupta efficiency (Gupta et al. 2009) uses a ratio of means, which
depends on where zero is on the unit scale. It was designed for
ratio-scale variables such as precipitation or runoff.
comparecomputes it for temperatures in degC or degF on the Kelvin scale, so the result does not depend on the temperature unit. - Percentile bias is relative to the reference percentile and has the
same dependence on the zero; it is computed on the Kelvin scale in the same
way. On samples that fit in dask's
array.chunk-size(128 MiB, 16.8 million float64 values) the percentile is exact; larger samples are estimated in the same single pass from weighted histograms with bins 2-20 of the value wide (method="auto", the default). The estimate is the inverse of the weighted empirical distribution function to within one bin width (2-20 of the value), clipped to the sample's minimum and maximum. That is not the same estimator as the exact method: for equal weights at most one sorted sample value lies between the two; for unequal weights xarray's weighted quantile interpolates differently, and the two can be a few sorted values apart.PercentileBias(95, method="exact")forces the exact computation (and warns when the sample exceedsarray.chunk-size), and the provenance records the method each sample used (metric_methods). Points with zero weight are left out by both. Percentile bias is NaN where the reference percentile is below 1% of the reference's weighted mean absolute value (min_relative=0.01): the 5th percentile of daily precipitation, for example, is near zero, and a relative difference to it says nothing. The provenance then lists the groups undermetric_notes["percentile_bias_undefined"]. - Spearman correlation needs every point of the sample in memory at once. It is available by name and not computed by default. Points with zero weight are not ranked.
- Constant fields. Pearson and Spearman r, normalised RMSE, the standard
deviation ratio, the Taylor skill score and KGE divide by a standard
deviation (KGE also by a mean). They are NaN when an input it divides by is
constant over the sample: when its weighted standard deviation is at most
1e-10 of its weighted mean absolute value. The two-pass standard deviation
of a constant field is about 1e-16 of the value, not zero, so a test for
zero would divide by rounding noise. The NumPy interface
(
BaseMetric.compute) applies the same rule. - Errors in metrics. An exception raised inside a metric reaches the caller as the same exception type (on Python 3.11 and later with a note naming the metric).
Custom metrics¶
Subclass climval.metrics.BaseMetric. Implement reduce with the weighted
helpers to stay lazy on dask, or compute on NumPy arrays for a quick
prototype.
from climval.metrics import BaseMetric
from climval.metrics.stats import wmean
class MeanSquaredError(BaseMetric):
"""Weighted mean squared difference."""
name = "mse"
higher_is_better = False
units_kind = None
def reduce(self, reference, candidate, weights, dims):
return wmean((candidate - reference) ** 2, weights, dims)
result = climval.compare(ref, cand, metrics=["rmse", MeanSquaredError()])
References¶
- Taylor, K. E. (2001). Summarizing multiple aspects of model performance in a single diagram. Journal of Geophysical Research: Atmospheres, 106(D7), 7183-7192. https://doi.org/10.1029/2000JD900719
- Gleckler, P. J., Taylor, K. E., and Doutriaux, C. (2008). Performance metrics for climate models. Journal of Geophysical Research: Atmospheres, 113(D6). https://doi.org/10.1029/2007JD008972
- Gupta, H. V., Kling, H., Yilmaz, K. K., and Martinez, G. F. (2009). Decomposition of the mean squared error and NSE performance criteria: Implications for improving hydrological modelling. Journal of Hydrology, 377(1-2), 80-91. https://doi.org/10.1016/j.jhydrol.2009.08.003