Methodology

Models

Model	Provider	Notes
IFS ENS		Physics-based ensemble.
HRES		The high-resolution (control) member of the IFS ensemble.
AIFS ENS		ECMWF's operational machine-learning ensemble. AIFS v2, as provided via ECMWF's open data.
GFS	NOAA	The U.S. Global Forecast System; physics-based, deterministic.
GEFS	NOAA	The U.S. Global Ensemble Forecast System. Scored to 10 days — see the notes below.
GraphCast	Google DeepMind	Machine-learning deterministic model, initialized from the ECMWF IFS analysis. Run by Brightband.
WeatherNext 2 (FGN)	Google DeepMind	Machine-learning ensemble, initialized from the ECMWF IFS analysis.
Aurora 1.5		Machine-learning ensemble, run by Brightband as 16 members on the 00 and 12 UTC cycles — see the notes below.
Atlas		Machine-learning ensemble, run by Brightband as 16 members on the 12 UTC cycle — see the notes below.
Climatology	—	Reference “no-skill” baseline built from ERA5 — see Climatology below.

Verification target

Every operational center produces its own analysis — its best estimate of the current atmospheric state, used to initialize the next forecast — and models drift relative to reality. We follow the standard operational convention: by default each model is scored against its own analysis, the lead-0 field of the same system valid at each verification time. This removes the systematic offset between one center's analysis and another's, so it is the fairest single choice for a “how good is this model in its own terms” score. Lead-0 error is ≈ 0 by construction, and the curves show how quickly each model departs from its own initial state.

Where a model publishes no analysis of its own, a defined reference is substituted: WeatherNext 2 (FGN) is initialized from the ECMWF IFS analysis, so its target is defined to be the IFS analysis, matching the model's own paper. The climatology baseline is likewise verified against the IFS analysis — a climatology has no analysis of its own. Other references (ECMWF IFS, NOAA GFS, ERA5) may eventually be used as separate targets for cross-comparison.

Precipitation is the exception: it is always verified against ERA5, never against a lead-0 analysis, because an analysis carries no accumulated-rainfall field — the accumulation at forecast step 0 is identically zero — so an own-analysis truth does not exist for it.

Deterministic metrics

• RMSE — area-weighted root-mean-square error between the forecast and the verifying analysis, aggregated over the selected region. Lower is better; reported in each variable's physical units. Computed for all evaluated variables.
• ACC — anomaly correlation coefficient: the correlation between forecast and verification anomalies, taken relative to a fixed climatology (WBX-standard era5-1990-2019-61d-linear, a 61-day linearly-weighted ERA5 1990–2019 climatology). Ranges up to 1 (perfect); higher is better. Currently computed for 500 hPa geopotential only.
• Wind-vector RMSE — for winds at pressure levels we report the vector RMSE sqrt(mean[(Δu)² + (Δv)²]), following the WeatherBench-2020 scorecard convention, rather than scoring the u and v components separately. Currently reported at 850 hPa, and shown as the Wind Vector 850 hPa variable.
• SEEPS — used for 24-hour precipitation only, in place of RMSE. See Precipitation below.

10 m wind speed is scored as a scalar speed. For ensemble means the speed is the mean of each member's speed, mean(√(u² + v²)), not the speed of the mean components — the latter is systematically low whenever members disagree on wind direction.

Probabilistic metrics (ensembles)

Ensemble systems are scored twice. Their ensemble-mean field is scored with the deterministic metrics above (those are the Mean models under Deterministic), and their members are scored directly with the metrics below (under Probabilistic).

The member scores use WBX's fair ensemble estimators — the versions corrected for finite ensemble size, so a score computed from \(K\) members estimates the value an arbitrarily large ensemble would have. At evaluation time we draw \(K = 10\) members from the full N-member perturbed forecast rather than reading all of them. Because the estimators are fair this is unbiased rather than merely approximate: a 10-member sample lands within about 1% of the full-ensemble value, with scatter in both directions, for roughly \(K/N\) of the data read. Members are exchangeable, so the first \(K\) are taken.

The truth \(y\) is the same lead-0 analysis field the deterministic scores use, so a system's probabilistic and deterministic curves are directly comparable. For the IFS and AIFS ensembles that field comes from the derived ensemble-mean store, because the members' own lead-0 planes are the perturbed initial conditions — one per member — rather than an analysis state.

Writing the \(K\) members at a grid point as \(x_1,\dots,x_K\), with mean \(\bar{x}\) and sample variance \(s^2 = \tfrac{1}{K-1}\sum_i (x_i-\bar{x})^2\):

• CRPS — Continuous Ranked Probability Score, the standard probabilistic accuracy score. Lower is better, in the variable's physical units. It decomposes into a skill term and a spread term,

for two independent draws \(X, X'\) from the forecast distribution, and is estimated from the members as

The \(1/(K(K-1))\) normalization on the second term — rather than \(1/K^2\) — is what makes the spread half unbiased for the infinite-ensemble value, and so what makes the score comparable across ensembles of different sizes.

• Spread — the ensemble's root-mean variance, \(\sqrt{\overline{s^2}}\), in the variable's units. This describes how wide the forecast distribution is, not how good it is: neither direction is “better” on its own, which is why spread is read against the ensemble-mean error rather than alone.
• Ensemble-mean RMSE — RMSE of the ensemble mean with the mean's own sampling noise removed, from the per-point squared error

The \(-s^2/K\) term corrects for the fact that a\(K\)-member mean is a noisier estimate of the ensemble's true mean than an arbitrarily large one; without it, small ensembles would look systematically worse. This is the bias-corrected counterpart of the deterministic RMSE reported for the Mean models, so the two differ slightly by construction.

• Spread–skill ratio — spread divided by the debiased ensemble-mean RMSE. A well-calibrated ensemble is about as spread out as it is wrong, so the ideal is 1: below 1 is under-dispersed (the ensemble is overconfident, its members too alike), above 1 is over-dispersed. It is dimensionless, and because the ideal sits inside the data range rather than at an axis end, the charts draw it as a dashed guide line at 1 instead of pinning the axis. The 0 h lead is omitted: a self-verified forecast has ~0 ensemble-mean error there, so the ratio divides by nearly nothing and carries no information.

Each statistic is aggregated with the same area weighting and per-region binning as the deterministic metrics — the per-point statistic is averaged first, and the square root (for spread and ensemble-mean RMSE) or the ratio (for spread–skill) is taken afterwards. The same average-first, root-or-ratio-last principle carries over to aggregation across forecast cycles — see Spatial & temporal aggregation below.

The 850 hPa wind is a different variable here. Deterministically it is scored as a vector RMSE over the u and v components, which has no probabilistic counterpart — the scores above are per-variable scalars. Instead each member carries its own u and v, so their hypotenuse is that member's wind speed, and the pair is scored as Wind Speed 850 hPa in m/s, exactly like 10 m wind speed.

Precipitation (SEEPS)

Precipitation is scored with SEEPS (Stable Equitable Error in Probability Space; Rodwell et al. 2010, Haiden et al. 2012) on 24-hour accumulated precipitation, rather than RMSE. SEEPS classifies each point as dry, light, or heavy relative to a local climatological wet/dry threshold and scores the forecast in probability space, so that different climates contribute comparably.

• The target is always ERA5. An analysis carries no accumulated rainfall — step-0 accumulation is identically zero — so precipitation cannot be verified against a lead-0 analysis. The truth is defined as the 24-hour sum of hourly ERA5 precipitation ending at each valid time, following the WeatherBench-2 right-labelled window convention.
• Scale. SEEPS is equitable: any uninformed forecast — always-dry, climatological, or random — has an expected score of exactly 1, and a perfect forecast scores 0. The no-skill baseline is therefore fixed at 1 by construction, with nothing to compute.
• Climatological parameters — the wet/dry threshold and the climatological dry fraction that define the categories — come from the WeatherBench-2 ERA5 1990–2019 climatology.
• Windowing. Only lead times of 24 h and greater are scored, since a full 24-hour accumulation window has to be available.
• ERA5 lag. ERA5 runs roughly 6 days behind real time, so freshly initialized forecasts produce no SEEPS scores until the verifying analysis catches up. Expect precipitation scores to populate on a ~7-day delay.

Climatology

Climatology plays two distinct roles:

• ACC baseline. ACC is measured relative to the WBX standard climatology — a 61-day linearly-weighted mean of ERA5 over 1990–2019 at native 0.25° resolution. Using this baseline is what makes our ACC directly comparable to published WeatherBench numbers.
• Skill floors. A plain climatology forecast is scored as a reference “no-skill” curve: a deterministic climatological-mean forecast for RMSE, and a probabilistic year-ensemble climatology for the ensemble metrics, in which each of the 30 years 1990–2019 at the same day-of-year and hour supplies one member. These mark the level a useful forecast has to beat; the deterministic climatology is the curve currently in the model list. Like every model without an analysis of its own, the climatology forecast is verified against the ECMWF IFS analysis. For SEEPS the floor is the constant 1 described above.

Spatial & temporal aggregation

Fields are evaluated on a fixed 0.25° global grid (1440×721; no regridding). Spatial means are area-weighted by grid-cell area, proportional to the cosine of latitude, so the densely packed cells near the poles do not dominate a global mean. Region boundaries follow the WBX public-benchmark convention. Scores are computed for a wider set of regions than the dashboard exposes; the selector here offers the global mean plus the tropics, the northern and southern hemispheres, North America, and Europe.

Note that 0.25° is a dissemination grid, not necessarily a model's native one: ECMWF's systems, for example, are produced on reduced Gaussian grids (N320 for the AIFS) and interpolated to the regular 0.25° grid for open data. Scoring these disseminated fields is standard practice, but the interpolation smooths fine structure, so some metrics here can differ slightly from results published on a model's native grid or under a different regridding.

In time, each stored score corresponds to a single forecast cycle. The Skill vs Lead and Scorecard views pool these values over all initialization times in the selected date window, per lead time — a skill estimate for the period — using a metric-appropriate aggregation rather than a uniform mean:

• RMSE, ensemble-mean RMSE, spread — the per-cycle values are squared, averaged, and re-rooted (root-mean-square). Because every cycle covers the same grid, this is exactly the score that would result from pooling all the cycles' points together; a plain mean of the per-cycle values would understate it.
• CRPS and SEEPS — a plain mean, which is already exact: both are linear means of per-point scores.
• ACC — the mean of the per-cycle Fisher z-transforms, mapped back with tanh. This is the standard way to average correlations; the exact pooled correlation is not recoverable from the per-cycle values alone.
• Spread–skill ratio — recomposed from its components rather than averaged: window-RMS spread divided by window-RMS ensemble-mean RMSE, over the cycles that report both. Per-cycle ratios do not average to the window ratio.

In short: ACC uses a standard simplification (the Fisher-z mean); every other window aggregation is exact — the pooled value is identical to what scoring the whole window's data in one pass would produce.

The home page's ranking score follows the same rule: per cycle, RMSE is pooled (root-mean-square) over the ranked lead times, so the window standings show an exact pooled RMSE over leads × cycles. Only each day's 12Z cycle is ranked — it is the cycle every model initializes with full Day-7 coverage (the 06/18 UTC ECMWF runs stop at 144 h), so a 30-day window is 30 cycles.

The Time Series view instead plots each cycle individually against its verification date at one fixed lead time, so you can see recent variability; no cross-cycle pooling is involved (the legend's avg uses the same per-metric pooling, except for the spread–skill ratio, where it is a plain mean of the plotted per-cycle ratios).

Lead times

Forecasts are scored every 6 hours out to 360 hours (15 days). The scorecard summarizes a subset — days 1, 3, 5, 7, 10, and 15.

Model-specific notes

• GEFS is scored to 10 days, and its upper-air fields are natively 0.5°. NOAA publishes only a small set of surface parameters for the GEFS ensemble mean at 0.25° (to 10 days) and keeps the pressure-level fields in a 0.5° product (to 16 days). We score all GEFS variables to 10 days so every variable shares one horizon, and map the 0.5° upper-air fields onto the common 0.25° grid by value replication rather than interpolation. Because GEFS's 500 hPa geopotential, 850 hPa temperature, and 850 hPa winds are then verified against a 0.5° analysis, those three scores are measured against a slightly smoother truth field than models whose upper air is natively 0.25°. Small differences there should not be over-interpreted.
• GFS precipitation ends at day 10. We aren't currently ingesting the accumulation files and instead calculate accumulated rainfall from PRATE surface fields.
• IFS off-cycle forecasts. The 06 and 18 UTC ECMWF ensemble cycles run to 144 h only; longer leads for those cycles are absent and are skipped rather than scored.
• Aurora runs 00 and 12 UTC only. We run Aurora 1.5 as a 16-member ensemble on the 00 and 12 UTC cycles; the 06 and 18 UTC cycles don't exist for it and are skipped rather than scored. Initialized via IFS Analysis.
• No Aurora precipitation scores. We run it on 6-hour timesteps, so the 1-hour accumulated rainfall fields the precipitation metrics are built from aren't available. Aurora is therefore absent from 24h Precipitation rather than scored on a coarser accumulation that wouldn't be comparable to the other models.
• Atlas runs 12 UTC only. We run Atlas as a 16-member ensemble on the 12 UTC cycle; the 00, 06, and 18 UTC cycles don't exist for it and are skipped rather than scored. Initialized via IFS Analysis.

Licensing & attribution

Every number on this dashboard is a statistic derived from third-party forecast and analysis data; the underlying fields are not redistributed here. Each source is credited per its license terms:

• ECMWF (IFS ENS, HRES, AIFS ENS). This service is based on data and products of the European Centre for Medium-Range Weather Forecasts (ECMWF), © 2026 ECMWF, www.ecmwf.int, obtained from ECMWF's real-time open data under CC BY 4.0 and modified here (verification statistics computed from the original fields). ECMWF does not accept any liability whatsoever for any error or omission in the data, their availability, or for any loss or damage arising from their use.
• Copernicus / ERA5. Contains modified Copernicus Climate Change Service information (ERA5) 2026. Neither the European Commission nor ECMWF is responsible for any use that may be made of the Copernicus information or data it contains. ERA5 (Hersbach et al. 2020) is the precipitation verification truth, an optional verification target, and the basis of the climatology baseline and the ACC reference climatology.
• NOAA (GFS, GEFS). NOAA data are in the public domain; forecast fields courtesy of NOAA/NWS/NCEP via the NOAA Open Data Dissemination program. The scores shown are computed from modified NOAA data, and nothing here states or implies endorsement by or affiliation with NOAA.
• Google DeepMind (GraphCast). GraphCast (Lam et al. 2023) model weights are © DeepMind Technologies Limited, made available under CC BY-NC-SA 4.0. Its forecasts are experimental machine-learning output: they are not based on data published by, produced in collaboration with, or endorsed by any government meteorological agency.
• Google DeepMind (WeatherNext 2 / FGN). Forecast data accessed via Google Cloud Storage. © 2024-5 Google LLC, whose machine learning models were used to create the experimental data made available under the following licence terms: storage.googleapis.com/weathernext-public/terms-of-use.pdf. This data is intended for experimental modelling only and is not intended, validated, or approved for real world use.
• Microsoft (Aurora). Aurora model weights are © Microsoft, released on Hugging Face under the MIT license. We run the model ourselves; the forecasts scored here are our own output, not a Microsoft product, and are experimental machine-learning output rather than an official forecast. Aurora is described in Bodnar et al. (2025), A Foundation Model for the Earth System, Nature.
• NVIDIA (Atlas). Atlas — the medium-range model of NVIDIA's Earth-2 family, trained on ERA5 — is released as nvidia/atlas-era5 under the NVIDIA Open Model License, which permits commercial use and claims no ownership of model outputs. We run the model ourselves; the forecasts scored here are our own output, not an NVIDIA product, and are experimental machine-learning output rather than an official forecast.

This dashboard is a verification tool for research and development purposes. The forecasts scored here in no way replace official forecasts, alerts, warnings, or notices issued by national meteorological agencies.

The home page's rank-per-cycle chart was inspired by ECMWF's AI Weather Quest leaderboards.

Operational WeatherBench by Brightband is distributed under the CC BY 4.0 license. That covers this dashboard and the derived scores it publishes; the third-party forecast and analysis data listed above remains under its own terms.

References

• WeatherBench-X, metric implementations — github.com/google-research/weatherbenchX
• WeatherBench-2, benchmark and conventions — Rasp et al. (2024)
• SEEPS — Rodwell et al. (2010); Haiden et al. (2012)
• ERA5 — Hersbach et al. (2020); ARCO-ERA5
• GraphCast — Lam et al. (2023)
• Aurora — Bodnar et al. (2025)