Random error while dowloading ERA5 with CDS-API

This type of error occurs a few times per day randomly. Looks like some problem with the status of the job.

2024-08-19 23:26:14,940 INFO Request ID is 9fdd9029-78cb-47e3-a929-05dcd17d7f62                           
2024-08-19 23:26:14,996 INFO status has been updated to accepted
2024-08-19 23:26:16,555 INFO status has been updated to running
2024-08-19 23:28:07,931 INFO Creating download object as as_source with files:
['data_0.nc']
2024-08-19 23:28:07,932 INFO status has been updated to successful
Traceback (most recent call last):
  File "get_meteo.py", line 86, in <module>
    client.retrieve(dataset, request, target)
  File "python3.10/site-packages/cads_api_client/legacy_api_client.py", line 149, in retrieve
    result = self.logging_decorator(self.client.submit_and_wait_on_result)(
  File "python3.10/site-packages/cads_api_client/legacy_api_client.py", line 134, in wrapper
    return func(*args, **kwargs)
  File "python3.10/site-packages/cads_api_client/api_client.py", line 77, in submit_and_wait_on_result
    return self.retrieve_api.submit_and_wait_on_result(
  File "python3.10/site-packages/cads_api_client/processing.py", line 455, in submit_and_wait_on_result
    return remote.make_results()
  File "python3.10/site-packages/cads_api_client/processing.py", line 260, in make_results
    raise ValueError(f"Result not ready, job is {status}")
ValueError: Result not ready, job is running
dataset = "reanalysis-era5-land"
client = cdsapi.Client()

# retrieve data
for nm in np.arange(len(vars_are)):
    var_is = str(vars_are[nm])
    for yr in np.arange(len(yrs_are)):
        year_is = str(yrs_are[yr])
        for mo in np.arange(len(months_are)):
            month_is = str(months_are[mo])

            request = {
                'variable': var_is,
                'year': year_is,
                'month': month_is,
                'day': days_are,
                'time': times_are,
                'data_format': 'netcdf',
                'download_format': 'unarchived',
                'area': [
                    lat_is+0.3, lon_is, lat_is, lon_is+0.5
                ]
            }
            target = 'meteo/ERA5_{var}_{year}_{month}.nc'.format(var=var_is, year=year_is, month=month_is)
            client.retrieve(dataset, request, target)

I met the same error, but before the error happened, 3 files have been download succesfully.

1 Like

I am having the same problem. I randomly receive a few files before the error occurs and crashes the code.

HEre is the error:

Traceback (most recent call last):
File “/net/rdu-nfssrv01/mnt/wpdata3/data_retrieval/grid/ecmwf_era5/Code/NA_rh.py”, line 69, in
c.retrieve(dataNames[level],
File “/wpdata3/sharedenv/anaconda-2023.09/lib/python3.11/site-packages/cads_api_client/legacy_api_client.py”, line 149, in retrieve
result = self.logging_decorator(self.client.submit_and_wait_on_result)(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/wpdata3/sharedenv/anaconda-2023.09/lib/python3.11/site-packages/cads_api_client/legacy_api_client.py”, line 134, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/wpdata3/sharedenv/anaconda-2023.09/lib/python3.11/site-packages/cads_api_client/api_client.py”, line 77, in submit_and_wait_on_result
return self.retrieve_api.submit_and_wait_on_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/wpdata3/sharedenv/anaconda-2023.09/lib/python3.11/site-packages/cads_api_client/processing.py”, line 455, in submit_and_wait_on_result
return remote.make_results()
^^^^^^^^^^^^^^^^^^^^^
File “/wpdata3/sharedenv/anaconda-2023.09/lib/python3.11/site-packages/cads_api_client/processing.py”, line 260, in make_results
raise ValueError(f"Result not ready, job is {status}")
ValueError: Result not ready, job is accepted

I also met the same error. The download breakdown about every 10 files. Another random error also occured.
Traceback (most recent call last):
File “/home/kcleung/ERA5Energy.py”, line 9, in
c.retrieve(
File “/home/kcleung/.local/lib/python3.11/site-packages/cads_api_client/legacy_api_client.py”, line 160, in retrieve
return result if target is None else result.download(target)
^^^^^^^^^^^^^^^^^^^^^^^
File “/home/kcleung/.local/lib/python3.11/site-packages/cads_api_client/legacy_api_client.py”, line 134, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/home/kcleung/.local/lib/python3.11/site-packages/cads_api_client/processing.py”, line 360, in download
multiurl.download(
File “/home/kcleung/.local/lib/python3.11/site-packages/multiurl/downloader.py”, line 111, in download
return Downloader(url, **kwargs).download(target)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/kcleung/.local/lib/python3.11/site-packages/multiurl/base.py”, line 135, in download
os.path.getsize(download) == size
AssertionError: File size mismatch 38939904 bytes instead of 52635311

This seems to be related to Potential race condition obtaining data from CDS-beta · Issue #110 · ecmwf/cdsapi · GitHub

3 Likes