ERA-5 products of 20240907-20240912 still incomplete

[Richard_van_Hees]

Hi,

I have moved to the new CDS-beta without any serious problems. Thank you.
However, I noticed that with the old interface I could download complete products with 5 days delay, now this has become 10 days.

This is the request dictionary for the products:

{
            "product_type": "reanalysis",
            "format": "grib",
            "variable": [
                "100m_u_component_of_wind",
                "100m_v_component_of_wind",
                "10m_u_component_of_wind",
                "10m_v_component_of_wind",
            ],
            "year": f"{self.idate.year:04d}",
            "month": f"{self.idate.month:02d}",
            "day": f"{self.idate.day:02d}",
            "time": [f"{i:02d}:00" for i in range(24)],
        }

Then today I can download products of 2024-09-06 and older, but not products of 2024-09-07 until 2024-09-12. Is there a problem with the generation of these products?

Greetings, Richard

[Kevin_Marsh]

Hi Richard,
I can download ERA5 data for 10/9/2024 from CDS-Beta web and cds api:

% more request_latest.py
import cdsapi

dataset = “reanalysis-era5-single-levels”
request = {
‘product_type’: [‘reanalysis’],
‘variable’: [‘10m_u_component_of_wind’, ‘10m_v_component_of_wind’, ‘100m_u_component_of_wind’, ‘100m_v_component_of_wind’],
‘year’: [‘2024’],
‘month’: [‘09’],
‘day’: [‘10’],
‘time’: [‘00:00’, ‘01:00’, ‘02:00’, ‘03:00’, ‘04:00’, ‘05:00’, ‘06:00’, ‘07:00’, ‘08:00’, ‘09:00’, ‘10:00’, ‘11:00’, ‘12:00’, ‘13:00’, ‘14:00’, ‘15:00’, ‘16:00’, ‘17:00’, ‘18:00’, ‘19:00’, ‘20:00’, ‘21:00’, ‘22:00’, ‘23:00’],
‘data_format’: ‘grib’,
‘download_format’: ‘unarchived’
}

client = cdsapi.Client()
client.retrieve(dataset, request).download()

gives
% grib_ls c92182e55d477834232b1e843e1c8345.grib
c92182e55d477834232b1e843e1c8345.grib
edition centre typeOfLevel level dataDate stepRange dataType shortName packingType gridType
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 10v grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100u grid_simple regular_ll
1 ecmf surface 0 20240910 0 an 100v grid_simple regular_ll
96 of 96 messages in c92182e55d477834232b1e843e1c8345.grib

96 of 96 total messages in 1 files

I suspect you may be encountering the CDS cache issue:

How to avoid the CDS cache issue

If a user runs exactly the same CDS API request several times, and the data from the first time the request is submitted is still in the CDS cache. then the same data are retrieved from the cache, rather than being re-extracted from the dataset. For datasets which are updated on a daily basis (such as ERA5T), this can cause issues, as the later request may not be returned the data as expected. To avoid this, we suggest users to add the keyword 'nocache' in their CDS API request, with a rando…

[Richard_van_Hees]

Hi Kevin,

Many thanks for your help.
I was not aware of the potential CDS cache issue. I run my script once a day, only when I try to debug my script I encounter this problem.
I have added “nocache”: f"{random.randrange(1000):03d}" to the request, and my scripts downloads the products correctly.

Greetings, Richard

[Matteo_De_Felice12]

Another good reason to start creating a proper documentation for the CDS API!

[Kevin_Marsh]

Glad to help, Richard! Glad it is working for you now!