Error in downloading 'reanalysis-era5-complete' dataset using the identical python code (CDS API)

Software and Tools API
, ,
1

Hello guys!

I am now coming across an elusive problem when downloading the 'reanalysis-era5-complete' dataset through the CDS API python code. The parameter I want to download is mean temperature tendency due to long-wave radiation (paramID=235002), which is not accessible on the CDS disk. Due to the large volume of daily data I need, I have used the loop statement to download only one month of daily data for one single request according to the efficiency tips, as shown in the python code below. Notwithstanding the slow speed, the data got successfully retrieved in the beginning. But when downloading the data with the date of '1982-03-01/to/1982-03-31', the request is failed after running for a while for the first time.

Then, I changed the value of the variable 'first_year' into 1983, trying to download data after the year of 1983. This time, the first six months of daily data can be downloaded successfully, then it failed with the date of '1983-07-01/to/1983-07-31'. And I repeated the above-mentioned processes several times, error message always emerged after some successful accesses. The content of error messages is either "the request you have submitted is not valid" or "an internal error occurred processing your request" with a 'permission denied: /cache/tmp' reason.

So, I am very confused why errors would appear even using the very identical python code, which has been successfully used to download data previously. Is it related to the cache issue in this post? Or is there any problem with my python code (I am also expecting any advice to download in a faster way)? Or, the parameter I download is missing on some specific dates in the ERA5 archive, causing these errors? 

Does anyone know what is wrong with it? Looking forward to any reply!

Kind regards,

JC Lee

download.py Expand source 
#!/usr/bin/env python
import cdsapi

first_year=1979

last_year=2020

nday=[31,28,31,30,31,30,31,31,30,31,30,31]


for year in range(first_year, last_year+1):

for month in range(1,13):

# print(‘{year:04d}-{month:02d}-01/to/{year:04d}-{month:02d}-{nday:02d}’.format(year=year, month=month, nday=nday[month-1]))

print(“=============================================”)

print(“Downloading {year:04d}.{month:02d}”.format(year=year, month=month))


    c = cdsapi.Client()
    c.retrieve('reanalysis-era5-complete', { # Requests follow MARS syntax
    'class': 'ea',
    'date': '{year:04d}-{month:02d}-01/to/{year:04d}-{month:02d}-{nday:02d}'.format(year=year, month=month, nday=nday[month-1]),
    'expver': '1',
    'levelist': '14/18/20/24/26/29/36/41/48/54/60/65/68/71/74/77/79/83/87/90/93/96/98/101/103/106/108/109/111/113/115/116/119/121/125/129/136',
    'levtype': 'ml',
    'param': '235002', # mean tmp tendency due to long-wave radiation
    'step': '0',
    'stream': 'oper',
    'time': '06:00:00/18:00:00', # or '06/18'
    'type': 'fc',
    'grid': '1.0/1.0',
    'area': '90/-180/0/180',
    'format': 'netcdf',
    }, "mttlwr.{year:04d}.{month:02d}.hourly.nc".format(year=year, month=month))</pre>
2

I also post the whole error messages as in the following pictures here.

1411×631 41.4 KB

999×152 7.74 KB

3

I'm getting a similar error from a piece of code that I use frequently to download ERA5 data for use as WRF boundary conditions. It stopped working yesterday at some point.

4

I have been getting a similar error today with a script I regularly use to retrieve daily ERA5T data. The error seemingly occurs randomly. Problems with the CDS server/API?

5

I even haven't been able to open the CDS website since just now. 

6

I made two requests at 07:20Z this morning: after almost 4 hours of waiting one ended correctly while the other failed with the [Errno 13] Permission denied: '/cache/tmp' error. 


Is anyone working on this problem?

Is there an open issue (I can't find CDS related issues on Jira)? 

7

Hi,

there was an issue on the system. It has been fixed now.

Please try to run the requests again.

Thanks

8

Great to hear that! It does work now! Thank you!

Will the system be intermittently broken lately?

9

Hi Michela,
thanks for your update.

I re-submitted the request which had failed but has been in the queue for more than an hour now (it's a very small request that usually doesn't take more than a few seconds).
For the moment I don't know if the problem is solved for me...

Thanks
Andrea

10

Hi Andrea,

I suspect that the CDS disruption at the weekend means that there is a large backlog of requests which will take a while to work through - this could mean that requests take longer than expected until the backlog clears,

Thanks,

Kevin