Skill scores from traditional dynamical forecast models

Workshops, competitions and events AI Weather Quest
1

As promised, we are sharing the Ranked Probability Skill Scores (RPSSs) for dynamical forecast models. These scores will be updated weekly, throughout the training period, and the latest data is available in the attached file.

JJA25_DYNforecast_evaluation.xls.zip (24.3 KB)

Currently skill scores are provided for forecasts initialised up to week 6 (19th June 2025) of the testing JJA period.

Dynamical forecast data has been downloaded from the WMO lead centre for the sub-seasonal multi-model ensemble database . We are providing RPSSs for six dynamical models. For all models, except WASHINGTON, the reforecast period spans 2006 to 2016. Details regarding the selection of reforecasts and the configuration of each dynamical model can be found here. No bias correction has been performed as we are evaluating forecasted probabilistic values against each model climate.

The attached file is organised by forecast window, forecast variable and skill score type:

· fcwin1 = Week 3 forecast

· fcwin2 = Week 4 forecast

· tas , mslp, pr = Forecast variables

· period = period-aggregated RPSS.

· weekly = single weekly RPSS.

:desktop_computer: You can now evaluate your own forecast and compare it directly to dynamical forecast skill scores. Detailed guidance on forecast self-evaluation can be found in ReadTheDocs Python documentation. :magnifying_glass_tilted_left:

For precipitation, scores are masked where all historical quintiles are equal to zero. See the following forum post for more information.

:exclamation: If you encounter any issues or have questions regarding the self-evaluation process, feel free to reach out - we’re here to help!

2 Likes
5

Updated on 25th July!

6

Dear Joshua_Talib
When calculating probabilistic forecasts using ensemble prediction results the climatological threshold is derived from model hindcast.
Sub-seasonal forecast models - AI Weather Quest - ECMWF Confluence Wiki
in this page, Number of hindcast members per initialisation (full number used to compute climatology). for example EC, 11 (x11x2=242), the first “11” is from Rfc_size, the second “11” is from Hindcast years used, the “2” is from Hindcast date selection.
Models - S2S - ECMWF Confluence Wiki
in this page, HMCR Rfc_size is 1+10, but when calculating climatology only use 10. It is different from other model.

7

Hi @Peng_Lu,

You are correct to raise this error. I had made a mistake.

For HMCR I was only downloading nine perturbed reforecast members. This missed out a reforecast member.

I have rewritten the code so that 11 reforecasts (10 + 1) are downloaded every week. I will change the dynamical forecast scores (above) once all forecast initialisations have been downloaded.

Thanks for noticing the mistake. I’ve also updated Sub-seasonal forecast models - AI Weather Quest - ECMWF Confluence Wiki.

Kind regards,
Josh

8

Thank you for your detailed and clear explanation