Random error while dowloading ERA5 with CDS-API

This type of error occurs a few times per day randomly. Looks like some problem with the status of the job.

2024-08-19 23:26:14,940 INFO Request ID is 9fdd9029-78cb-47e3-a929-05dcd17d7f62                           
2024-08-19 23:26:14,996 INFO status has been updated to accepted
2024-08-19 23:26:16,555 INFO status has been updated to running
2024-08-19 23:28:07,931 INFO Creating download object as as_source with files:
['data_0.nc']
2024-08-19 23:28:07,932 INFO status has been updated to successful
Traceback (most recent call last):
  File "get_meteo.py", line 86, in <module>
    client.retrieve(dataset, request, target)
  File "python3.10/site-packages/cads_api_client/legacy_api_client.py", line 149, in retrieve
    result = self.logging_decorator(self.client.submit_and_wait_on_result)(
  File "python3.10/site-packages/cads_api_client/legacy_api_client.py", line 134, in wrapper
    return func(*args, **kwargs)
  File "python3.10/site-packages/cads_api_client/api_client.py", line 77, in submit_and_wait_on_result
    return self.retrieve_api.submit_and_wait_on_result(
  File "python3.10/site-packages/cads_api_client/processing.py", line 455, in submit_and_wait_on_result
    return remote.make_results()
  File "python3.10/site-packages/cads_api_client/processing.py", line 260, in make_results
    raise ValueError(f"Result not ready, job is {status}")
ValueError: Result not ready, job is running
dataset = "reanalysis-era5-land"
client = cdsapi.Client()

# retrieve data
for nm in np.arange(len(vars_are)):
    var_is = str(vars_are[nm])
    for yr in np.arange(len(yrs_are)):
        year_is = str(yrs_are[yr])
        for mo in np.arange(len(months_are)):
            month_is = str(months_are[mo])

            request = {
                'variable': var_is,
                'year': year_is,
                'month': month_is,
                'day': days_are,
                'time': times_are,
                'data_format': 'netcdf',
                'download_format': 'unarchived',
                'area': [
                    lat_is+0.3, lon_is, lat_is, lon_is+0.5
                ]
            }
            target = 'meteo/ERA5_{var}_{year}_{month}.nc'.format(var=var_is, year=year_is, month=month_is)
            client.retrieve(dataset, request, target)

I met the same error, but before the error happened, 3 files have been download succesfully.

1 Like

I am having the same problem. I randomly receive a few files before the error occurs and crashes the code.

HEre is the error:

Traceback (most recent call last):
File “/net/rdu-nfssrv01/mnt/wpdata3/data_retrieval/grid/ecmwf_era5/Code/NA_rh.py”, line 69, in
c.retrieve(dataNames[level],
File “/wpdata3/sharedenv/anaconda-2023.09/lib/python3.11/site-packages/cads_api_client/legacy_api_client.py”, line 149, in retrieve
result = self.logging_decorator(self.client.submit_and_wait_on_result)(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/wpdata3/sharedenv/anaconda-2023.09/lib/python3.11/site-packages/cads_api_client/legacy_api_client.py”, line 134, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/wpdata3/sharedenv/anaconda-2023.09/lib/python3.11/site-packages/cads_api_client/api_client.py”, line 77, in submit_and_wait_on_result
return self.retrieve_api.submit_and_wait_on_result(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/wpdata3/sharedenv/anaconda-2023.09/lib/python3.11/site-packages/cads_api_client/processing.py”, line 455, in submit_and_wait_on_result
return remote.make_results()
^^^^^^^^^^^^^^^^^^^^^
File “/wpdata3/sharedenv/anaconda-2023.09/lib/python3.11/site-packages/cads_api_client/processing.py”, line 260, in make_results
raise ValueError(f"Result not ready, job is {status}")
ValueError: Result not ready, job is accepted

I also met the same error. The download breakdown about every 10 files. Another random error also occured.
Traceback (most recent call last):
File “/home/kcleung/ERA5Energy.py”, line 9, in
c.retrieve(
File “/home/kcleung/.local/lib/python3.11/site-packages/cads_api_client/legacy_api_client.py”, line 160, in retrieve
return result if target is None else result.download(target)
^^^^^^^^^^^^^^^^^^^^^^^
File “/home/kcleung/.local/lib/python3.11/site-packages/cads_api_client/legacy_api_client.py”, line 134, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File “/home/kcleung/.local/lib/python3.11/site-packages/cads_api_client/processing.py”, line 360, in download
multiurl.download(
File “/home/kcleung/.local/lib/python3.11/site-packages/multiurl/downloader.py”, line 111, in download
return Downloader(url, **kwargs).download(target)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/home/kcleung/.local/lib/python3.11/site-packages/multiurl/base.py”, line 135, in download
os.path.getsize(download) == size
AssertionError: File size mismatch 38939904 bytes instead of 52635311

This seems to be related to Potential race condition obtaining data from CDS-beta · Issue #110 · ecmwf/cdsapi · GitHub

3 Likes

I have partly solved File size mismatch issue by modifying client.retrieve(dataset, request, target).download() into client.retrieve(dataset, request, target).

I am also getting this error. I see someone on the GitHub issue posted above regarding the race condition made a workaround to ignore the error and continue with the download. I’ll see if that works for me.

Thank you for your suggestion, it seems worked!

Did anyone else encounter this problem with the new version? I get the File size mismatch error since I changed to the new version. Sadly none of the fixes proposed in this thread seem to work. Any ideas?