Dear Joshua，
I have a question regarding forecast evaluation. I implemented the RPSS evaluation code locally and conducted a test on the ECMWF S2S tas forecast for lead days 19–25, with an initialization date of 6 November 2025. I found a discrepancy between my results and the published RPSS values. Specifically, I downloaded the observation data and the five quintile boundariesfor 24 November 2025 using retrieve_evaluation_data() and retrieve_evaluation_data.retrieve_20yr_quintile_clim(). In addition, I downloaded the ECMWF reforecast data initialized on 5 November and 7 November, covering a 20-year period (2005–2024). I then computed weekly means using step = slice(18, 25) and step = slice(16, 23) for the two initialization dates, respectively, and derived the five quintile bounds from the reforecast data. Using the provided function work_out_RPSS(), I obtained an RPSS value of 0.125, whereas the published value is 0.131. Could you please advise whether this discrepancy could be related to the definition of the valid time window or the time steps I selected ?
LiuCan
Hello LiuCan,
Thanks for your question! Joshua, the scientific lead of the Quest, is currently on leave. I will make sure to pass your question on to him upon his return on Monday 29th December.
Best regards,
Olga
Dear Olga,
Thank you for the information. I appreciate you forwarding my question to Joshua upon his return on the 29th.
Kind regards,
LiuCan
Hi @Can_Liu ,
Thanks for your hard work attempting to calculate the RPSS for ECMWF predictions. I think you have an error in your step selection.
For reforecast data initialised on 5th and 7th November I would select steps 19 to 26 and 17 to 24 respectively (assuming units of days). At the moment, the 5th November reforecast is using the same forecast period as the operational forecast, even though it is initialised a day earlier.
Happy to help further if you send me your code etc.
Thanks,
Josh
Hi Joshua,
Thank you for your detailed answer.
For operational forecast initialized on 6th November, I believe the lead days 19–25 (November 24th to 30th) should correspond to steps 17 to 24, given that step 0 is November 7th. I’m wondering why the same forecast period (steps 19 to 26) as November 5th is being used in this case.
If you have a moment, could you please clarify this? I’d appreciate it.
Best regards,
Liu Can
Dear Liu Can,
Sorry, I’ve got confused between the timesteps we select for operational and evaluation purposes.
For the operational forecast and the reforecasts, I always select steps 18 to 25 for the week 3 lead time. The time steps selected for the historical climatology should be based on lead time rather than valid time.
To summarise, for both forecast and reforecast, the same step, slice(18, 25), is chosen.
Kind regards,
Josh
Dear Joshua,
For the operational forecast initialized on November 6th, I believe the selection of steps 18 to 25 for the week 3 lead time may not align accurately with the intended period. According to the schedule, week 3 should cover November 24th to 30th. However, steps 18 to 25 correspond to November 25th to December 1st, resulting in a one-day shift.
Could you help verify the step selection for this initialization? Thank you.
Best regards,
Liu Can
Step 0 corresponds to the start of Thursday not Friday.
Josh
Dear Joshua,
Thank you very much for your detailed explanation. I realize now that the discrepancy was due to my oversight in aligning the time steps with the different S2S datasets. After adjusting the step selection accordingly and re-running the evaluation code, I have obtained results that are consistent with the released version.
Thank you again for your guidance and support.
Best regards,
LiuCan
