Hi all,
As we continue with the testing phase of the AI Weather Quest, we wanted to highlight ECMWF’s open data catalogue. Here you have access the most recent IFS and AIFS operational forecasts. Data is available for the previous three days and, most of the time, becomes accessible at 0700 UTC. Timestep 0 can be treated as ECMWF’s analysis.
You may also choose to initialise your models using dynamical sub-seasonal forecast data. Such data is available via ECMWF-hosted sub-seasonal forecast portal.
As written in today’s email to participants, we are currently in Week 9 of the JJA testing phase, which is the last week in which a submission will complete a full cycle of evaluation within the test phase 
We encourage all teams intending to compete to test their forecast submission workflows now.
Testing your submission ensures compatibility with the evaluation system and helps avoid errors during the competition. We particularly recommend practising automated submissions, as missed weeks during the competition will disqualify teams from period-aggregated RPSS (Ranked Probability Skill Score) evaluation.
Good luck,
Josh
Hi Josh,
Based on the links above, we only see the option to download data with up to 13 vertical levels. Does ECMWF provide real-time data with more than 13 pressure levels?
Thanks,
Kirsten
Hi Kirsten,
Currently only thirteen pressure levels (1000, 925, 850, 700, 600, 500, 400, 300, 250, 200, 150, 100, 50 hPa) are provided. All available data is explained on Open data | ECMWF .
If you think there are certain pressure levels that will improve your model development, I can put in a request to ECMWF Service Desk.
Kind regards,
Josh
Hi Josh,
Thanks for your quick response! We trained our model on the 32 pressure levels shown in the list below. Are you able to request these or pressure levels similar to the 32 listed below as well as 1hpa and 1000hpa?
[ 3.643466, 7.59482 , 14.356632, 24.61222 , 35.92325 , 43.19375 , 51.677499, 61.520498, 73.750958, 87.82123 , 103.317127, 121.547241, 142.994039, 168.22508 , 197.908087, 232.828619, 273.910817, 322.241902, 379.100904, 445.992574, 524.687175, 609.778695, 691.38943 , 763.404481, 820.858369, 859.534767, 887.020249, 912.644547, 936.198398, 957.48548 , 976.325407, 992.556095]
We were also wondering if you could request the variable ‘swvl1’. When a member of our team uses the python program provided to pull the data (ecmwf-opendata), they are not able to request this variable.
Lastly, is sst available in real time? If not, we can probably get away with skt.
Thanks!
Kirsten
Good morning Kirsten,
I’ve got the following reply from ECMWF service desk.
… Unfortunately you cannot request pressure levels to be added to the open data resource. At this time there are no plans to change the open data and release more pressure levels but I have noted the request for review in the future.
As part of the GRIB2 migration some parameters have now changed. In GRIB1, soil temperature, volumetric soil moisture and sea-ice temperature have different paramIds for each of the 4 soil layers. In GRIB2, only a single paramId is used with the layer indicated by the level key.
For example, in GRIB1, there is a paramId for Volumetric soil water layer 1 and a different paramId for Volumetric soil water layer 2, etc. In GRIB2, only a single paramId is used for Volumetric soil water, with the layer 1 having level=1 and layer 2 having level=2.
All of the above parameters can be found under levtype=sol rather than levetype=sfc with shortNames changing from swvl1-4 to ‘vsw’ with the relevant levlist =1/2/3/4. Please see the below example to retrieve volumetric soil water for level1. The Open Data webpage will reflect these changes as soon as possible.
import metview as mv
from ecmwf.opendata import Client
client = Client(“ecmwf”)
parameters = [‘vsw’]
filename = ‘swvl1_test.grib’
client.retrieve(
date=-1,
time=00,
step=0,
stream=“oper”,
type=“fc”,
levtype= “sol”,
levelist=“1”,
param=parameters,
target=filename
And for your final question, I have messaged one of our experts if Skin temperature can be used as a proxy to SST, as I am aware they are similar but not the same. I will let you know the answer when I receive a reply.
I hope this information is helpful and sorry regarding limited pressure levels.
Thanks,
Josh
Hello Josh,
Based on the info here, my understanding is that we will not be given any specific initialisation data similar to the training data we were given via the AI-WQ package. Am I correct in the way I understand things?
I am a bit confused about this because we were given preprocessed training data and now are expected to forecast based on raw data that we need to preprocess ourselves, which most likely will be mismatched to the training data.
Am I missing something?
Thank you in advance!
Asena
Hi Asena,
The training data provided to Quest participants is entirely optional and is based on the use of ERA5 to evaluate forecasts. As ERA5T is only accessible after five days, we cannot provide this dataset as suitable initial conditions. Therefore, we have not provided initial conditions to participants, so that participants can choose their own initial conditions and increase model diversity.
The python package readthedocs (Welcome to AI-WQ-package documentation! — AI_Weather_Quest 1.0 documentation) provides all the details regarding calculation of training dataset and evaluation data.
Josh