Retrieve daily ERA5/ERA5-Land data using the CDS API

While the CDS only has hourly and monthly ERA5/ERA5-Land data, It is possible to use the CDS API to request daily data from the CDS ERA5 daily application:

https://cds.climate.copernicus.eu/cdsapp#!/software/app-c3s-daily-era5-statistics?tab=app

Please note that to use this method, you need to use the correct syntax.

For valid keywords, see Table 2 of: https://datastore.copernicus-climate.eu/documents/app-c3s-daily-era5-statistics/C3S_Application-Documentation_ERA5-daily-statistics-v2.pdf

This script shows how the daily mean of 2m temperature for all months in 1979 can be retrieved:

import cdsapi
import requests
 
# CDS API script to use CDS service to retrieve daily ERA5* variables and iterate over
# all months in the specified years.
 
# Requires:
# 1) the CDS API to be installed and working on your system
# 2) You have agreed to the ERA5 Licence (via the CDS web page)
# 3) Selection of required variable, daily statistic, etc
 
# Output:
# 1) separate netCDF file for chosen daily statistic/variable for each month
 
c = cdsapi.Client(timeout=300)
 
# Uncomment years as required
 
years =  [
            '1979'
#           ,'1980', '1981',
#            '1982', '1983', '1984',
#            '1985', '1986', '1987',
#            '1988', '1989', '1990',
#            '1991', '1992', '1993',
#            '1994', '1995', '1996',
#            '1997', '1998', '1999',
#            '2000', '2001', '2002',
#            '2003', '2004', '2005',
#            '2006', '2007', '2008',
#            '2009', '2010', '2011',
#            '2012', '2013', '2014',
#            '2015', '2016', '2017',
#            '2018', '2019', '2020',
#            '2021'
]
 
 
# Retrieve all months for a given year.
 
months = ['01', '02', '03',
            '04', '05', '06',
            '07', '08', '09',
            '10', '11', '12']
 
# For valid keywords, see Table 2 of:
# https://datastore.copernicus-climate.eu/documents/app-c3s-daily-era5-statistics/C3S_Application-Documentation_ERA5-daily-statistics-v2.pdf
 
# select your variable; name must be a valid ERA5 CDS API name.
var = "2m_temperature"
 
# Select the required statistic, valid names given in link above
stat = "daily_mean"
 
# Loop over years and months
 
for yr in years:
    for mn in months:
        result = c.service(
        "tool.toolbox.orchestrator.workflow",
        params={
             "realm": "user-apps",
             "project": "app-c3s-daily-era5-statistics",
             "version": "master",
             "kwargs": {
                 "dataset": "reanalysis-era5-single-levels",
                 "product_type": "reanalysis",
                 "variable": var,
                 "statistic": stat,
                 "year": yr,
                 "month": mn,
                 "time_zone": "UTC+00:0",
                 "frequency": "1-hourly",
#
# Users can change the output grid resolution and selected area
#
#                "grid": "1.0/1.0",
#                "area":{"lat": [10, 60], "lon": [65, 140]}
 
                 },
        "workflow_name": "application"
        })
         
# set name of output file for each month (statistic, variable, year, month     
 
        file_name = "download_" + stat + "_" + var + "_" + yr + "_" + mn + ".nc"
         
        location=result[0]['location']
        res = requests.get(location, stream = True)
        print("Writing data to " + file_name)
        with open(file_name,'wb') as fh:
            for r in res.iter_content(chunk_size = 1024):
                fh.write(r)
        fh.close()

Hope that helps,

Kevin


Hi Kevin,

I am trying to retrieve ERA5-Land 2m_temperature daily mean of 1-hourly frequency with grid 0.1/0.1 over area N 50, W -126, S 24, E -66, but I keep getting "Failed (download widget)" from the application and "ValueError: Can't save empty DataArray." when running using the above script (after tweaking parameters). Retrieval works fine when I increase the area to N 90, W -180, S -90, E 180, so I am not sure what exactly is causing the request to fail, but I would like to avoid retrieving data globally if possible. Is there a way to make this work?

Hope to hear from you,

Jessica

I have the same problem. 
It seems the problem is with the size of the grid 0.1x0.1. For size 0.25 x 0.25 it works fine.


Hi Jessica, Karol

I think there may be an issue with the Toolbox; I will let you know when it is resolved,

Thanks,

Kevin

Hi Kevin


Any update on this? I am also getting the same error when trying to download at 0.1 deg resolution. It would be great to have the option to download daily data! 


Thanks!

Rutger

Hi Rutger,

The bug has been fixed but the code of the public application still need to be updated.

In the meanwhile the fixed app is accessible here:  https://cds.climate.copernicus.eu/apps/10207/era5-daily-statistics

Regards.

Vivien

Hi Kevin,

I have a problem when trying to save the file. I usually use r.download('/path/filename.nc'), but now I get this error:

AttributeError: 'list' object has no attribute 'download'

If I use instead c.download(r) I get a huge error message. I'd like to download the files in a specific folder, and I didn't understand well how you save them in your code. Is there any way in which I can do that?

Thanks in advance,

María Jesús




Hi Maria, 

Did you find a way to redirect downloads to an specific folder? I am trying to do the same.

Cheers,

Carlos. 


Hi Kevin

Get the same error as described above when trying to download 0.1 x 0.1 deg resolution data.

Thanks for looking into it. I agree. Having the option to tailor daily statistics from ERA5 will be absolutely great once we get around these issues...

Dear Martin,

I think the right statistic name is "daily_maximum".

Regarding the 0.1 x 0.1 deg resolution data it is under investigation.

Thank you for your patience.

Vivien

Thanks Vivien. My bad. Obviously haven't read the documentation correctly. Have removed my comment/error message on 'daily_max' from the above. 

Hi Kevin, 

when using the code above, I always get this error:

```

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/jsonrequest/requests.py", line 71, in jsonrequestcall
resp = coding.encode(req.callable(*req.args, **req.kwargs), register=encoders, **context)
File "/usr/local/lib/python3.6/dist-packages/cdsworkflows/submit_workflow.py", line 55, in submit_workflow
results = workflow_bare_func(**kwargs)
TypeError: application() missing 1 required positional argument: 'licence'

```


Could you please help with this?

Many thanks!


Ting

For those who are experiencing the save issue above, the workaround is actually simple but missed in the code snippet above:

in the request dict `"kwargs": {...}`, just add a keyword pair `"licence":'GPL'`.

the issue will be gone.

hope this helps.

Hi Ting,

Yes this is a valid work-around, however, it is no longer required as I have made the correction such that the licence field defaults to True for cdsapi users.

FYI, the purpose of the new key word argument was to ensure that non-registered CDS users are required to agree to the Copernicus licence when using the gui version of the application.

Thanks,

Eddy

Hi Kevin, 

When running the above python code my task is stuck in a Queued status for around two hours or so before being processed. The actual "In Progress" time is only a few mintues. This still occures when this is the only task being processed. 

When running the task throught the online app the task goes imediatly to "In Progress" and finishes reletivly quickly.

Just wanted to check and ask why this happend.


I also wasnted to ask is there is any parrelel processing occuring if multiple tasks are sent in at the same time. 


Thanks for your help,

-Thomas Ott

Hi Thomas,

Internally, the CDS manages the request load from the web interface, the CDS API and the Toolbox, as there are a limited number of processing slots available. The CDS is also undergoing constant evolution and system sessions to implement the changes can affect processing times.

You can see the CDS status via the 'live' page: 

https://cds.climate.copernicus.eu/live/

and you can see the number of simultaneous requests a user can submit at:

https://cds.climate.copernicus.eu/live/limits

Hope that helps,

Kevin

Hi Kevin,

Thanks for this, it will be great to have access to daily data.

I tried using your original script and it worked well to retrieve data globally at a 1 x 1 resolution. However, once I specify a different grid resolution (0.25 x 0.25) and particular sets of lat/long (as suggested in your script) I get different types of errors.

I also tried using the "License: GPL" fix suggested above by Ting Su but that does not seem to help.  

The queue times are quite long so it is difficult to try different script variations.

Any feedback will be much appreciated. 

Regards,

Carlos. 

Hi Carlos,

There were some CDS system session over the past week which affected performance and may have been the cause of the errors you saw/long queue times. 

Is it working ok for you now?

Thanks,

Kevin

Hi Kevin, 

Thank you for your reply. I am afraid I am experiencing the same issues in terms of the queue (at least yesterday and today). As such, I am not sure if the script is working properly. The queue is taking too long!

Any feedback will be much appreciated, 

Cheers,

Carlos. 

Thanks for providing the script for daily statistics, that's indeed really helpful!

I just tested the script above and it works like a charm. However, I noticed a reduction in download speed when changing the resolution. This is in particular strange since I want to download "low" resolution data (1.0 x 1.0 degree) compared to the default (0.25 x 0.25 degree). While the latter downloads about 1 month / min, I didn't obtain a single month after 10 min for the "low" resolution data.

Is this normal behavior?