Pangeo and cloud data¶
climval reads data lazily and never loads a whole dataset. On cloud data the
usual pattern is: open the store with xarray, select what you need, and pass
the Dataset or DataArray to climval.compare. Only the chunks that hold the
selected region and period are read, when the metrics are computed.
Open the store yourself¶
import xarray as xr
import climval
era5 = xr.open_zarr(
"gs://gcp-public-data-arco-era5/ar/1959-2022-6h-240x121_equiangular_with_poles_conservative.zarr",
storage_options={"token": "anon"}, # public bucket, no credentials
chunks={}, # keep the stored chunks
)
t2m = era5["2m_temperature"].sel(time=slice("2010-01-01", "2010-12-31"))
Opening the store yourself is the most reliable route for remote Zarr: you choose the engine, the storage options and the chunks, and you can check sizes before anything is read.
Paths and URLs¶
compare also accepts paths and URLs. NetCDF and Zarr open lazily, local or
remote through fsspec (gs://, s3://, https://). A Zarr store is
recognised by a .zarr suffix or, without one, by its metadata (.zmetadata,
.zgroup or zarr.json), so a Pangeo CMIP6 store URL, which ends in a
version directory, works as given; open_kwargs={"engine": "zarr"} forces
Zarr. Storage options go in open_kwargs:
result = climval.compare(
"gs://my-bucket/reference.zarr",
"s3://my-bucket/candidate.zarr",
variable="tas",
open_kwargs={"storage_options": {"anon": True}},
)
open_kwargs applies to both inputs. When they need different options, open
them yourself.
Check sizes before you read¶
Cloud stores are chunked, and a chunk is read as a whole. A store chunked as one global field per time step reads the whole globe even for a small region. Look at the chunking and estimate the bytes first:
da = era5["2m_temperature"]
print(da.encoding["chunks"], da.dtype)
print(da.sel(time=slice("2010", "2010")).nbytes / 1e6, "MB uncompressed for one year")
With gcsfs the compressed size of each chunk is in the bucket listing:
gcsfs.GCSFileSystem(token="anon").ls(path, detail=True). The
Europe example does this before
reading.
Keep memory modest¶
- Subset in space and time before computing;
comparedoes this for its region and period, and dask reads only the chunks involved. compare(..., compute=False)returns a lazy result;.compute()runs one dask pass for every metric and sample.- Limit the threads when memory is tight:
with dask.config.set(scheduler="threads", num_workers=4): result.compute(). - On a cluster, create a
dask.distributed.Clientfirst; climval uses the active scheduler.
Sub-daily data¶
Sub-daily data keep their step: two 6-hourly inputs are compared every 6
hours, and hourly against 6-hourly data in 6-hour bins. A coarser freq
("D", "MS") averages them first, with the min_coverage rule (see
Calendars and frequencies). For long periods it is often
cheaper to compute daily or monthly means once, store them, and compare
those.