You can now see how the competing AI-based forecasts compare to leading dynamical models on the leaderboards page. Both the leaderboards and model score evolution graphs have been updated to include dynamical model results based on forecast data from the WMO Subseasonal to Seasonal (S2S) Database.
All entries appear under the “Dynamical_S2SDatabase” team name, featuring models from the following WMO Lead Centres:
-
CMA
-
ECCC
-
ECMWF
-
HMCR
-
JMA
-
KMA
-
NOAA
These serve as official reference benchmarks, they are not linked to registered teams, therefore appear with rank “N/A” and are not shown in the “Evolution of teams’ rankings” graph.
They are displayed in orange across both the leaderboards and the model score evolution graphs, allowing you to instantly compare AI-driven and dynamical approaches.
Learn more about the forecast and reforecast configurations on the dedicated Confluence page.
1 Like
Hello Olga,
this is a great addition, it is very useful. Thanks a lot for this.
I was interested to learn about eventual post-processing these forecast streams go through to be aligned with the competition. For example, if and how exactly they are calibrated to produce the probabilistic forecast for the specific quantiles values of the competition. I didn’t find this type of info in the confluence page.
All the best
Hi Harilaos,
At the moment there is no post-processing of dynamical forecasts. I only download operational forecasts and reforecasts, and then compute the forecasted probabilities within the lead-time dependent model-based quintile boundaries.
The following confluence page contains the selection of operational forecasts and reforecasts.
Kind regards,
Josh
1 Like
Hi Josh,
I see, I’ve checked the confluence page, thanks. So if I understand correctly, it is uncalibrated forecast (against reanalysis) with model climatologies that may differ in several ways (period, sample size etc.) between systems and compared the ERA5 based climatology/quantiles adopted by the competition for validation. This is probably not optimal for a strict benchmark but at least feasible. Do I get this right ?
Best
Hi Harilaos,
Your summary is correct. Whilst the comparison may not be perfect, it is restricted by the design of the operational forecast and reforecast frequency. Further thought is needed to think about calibration.
Kind regards,
Josh
Hi Josh,
Thanks for the confirmation. I was curious to know because we submit forecasts that is calibrated at daily scale (we want it to be useable for impact models) and we thus evaluate the quintile probabilities directly with the ERA5 based values you provide.
Best