Encountering 'ChunkedEncodingError' when downloading ERA5 data

Hi everyone,

I’ve been trying to download data using the cdsapi Python library from the Climate Data Store (CDS) API. Below is the code I used to attempt the download:

import cdsapi

dataset = "reanalysis-era5-single-levels"
request = {
    "product_type": ["reanalysis"],
    "variable": ["surface_pressure"],
    "year": ["2000"],
    "month": ["11"],
    "day": [
        "01", "02", "03",
        "04", "05", "06",
        "07", "08", "09",
        "10", "11", "12",
        "13", "14", "15",
        "16", "17", "18",
        "19", "20", "21",
        "22", "23", "24",
        "25", "26", "27",
        "28", "29", "30"
    ],
    "time": [
        "00:00", "01:00", "02:00",
        "03:00", "04:00", "05:00",
        "06:00", "07:00", "08:00",
        "09:00", "10:00", "11:00",
        "12:00", "13:00", "14:00",
        "15:00", "16:00", "17:00",
        "18:00", "19:00", "20:00",
        "21:00", "22:00", "23:00"
    ],
    "data_format": "netcdf",
    "download_format": "unarchived"
}

client = cdsapi.Client()
download_folder = "D:/ERA5_surface_pressure/"
filename = download_folder + "2000_11_suface_pressure.nc"
target = filename
client.retrieve(dataset, request, target)

And here is the error message I encountered:

d5f.nc:   6%|▌         | 56.0M/923M [05:52<5:48:50, 43.4kB/s]
5401cc67e7861e52d21a68d8b6addd5f.nc:   6%|▌         | 57.0M/923M [06:12<6:44:01, 37.4kB/s]
5401cc67e7861e52d21a68d8b6addd5f.nc:   6%|▌         | 57.0M/923M [06:33<6:44:01, 37.4kB/s]
                                                                                          
Traceback (most recent call last):
  File "D:\python\install\Lib\site-packages\urllib3\response.py", line 748, in _error_catcher
    yield
  File "D:\python\install\Lib\site-packages\urllib3\response.py", line 894, in _raw_read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(60498240 bytes read, 906996746 more expected)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "D:\python\install\Lib\site-packages\requests\models.py", line 820, in generate
    yield from self.raw.stream(chunk_size, decode_content=True)
  File "D:\python\install\Lib\site-packages\urllib3\response.py", line 1060, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "D:\python\install\Lib\site-packages\urllib3\response.py", line 977, in read
    data = self._raw_read(amt)
  File "D:\python\install\Lib\site-packages\urllib3\response.py", line 872, in _raw_read
    with self._error_catcher():
  File "D:\python\install\Lib\contextlib.py", line 155, in __exit__
    self.gen.throw(typ, value, traceback)
  File "D:\python\install\Lib\site-packages\urllib3\response.py", line 772, in _error_catcher
    raise ProtocolError(arg, e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(60498240 bytes read, 906996746 more expected)', IncompleteRead(60498240 bytes read, 906996746 more expected))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\python\install\MyScripts\test.py", line 39, in <module>
    client.retrieve(dataset, request, target)
  File "D:\python\install\Lib\site-packages\datapi\legacy_api_client.py", line 179, in retrieve
    return submitted if target is None else submitted.download(target)
  File "D:\python\install\Lib\site-packages\datapi\processing.py", line 659, in download
    robust_download(url, target)
  File "D:\python\install\Lib\site-packages\multiurl\http.py", line 479, in wrapped
    r = call(main_url, *args, **kwargs)
  File "D:\python\install\Lib\site-packages\datapi\processing.py", line 625, in _download
    multiurl.download(
  File "D:\python\install\Lib\site-packages\multiurl\downloader.py", line 111, in download
    return Downloader(url, **kwargs).download(target)
  File "D:\python\install\Lib\site-packages\multiurl\base.py", line 129, in download
    total = self.transfer(f, pbar)
  File "D:\python\install\Lib\site-packages\multiurl\http.py", line 133, in transfer
    for chunk in stream(chunk_size=self.chunk_size):
  File "D:\python\install\Lib\site-packages\requests\models.py", line 822, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(60498240 bytes read, 906996746 more expected)', IncompleteRead(60498240 bytes read, 906996746 more expected))

I’ve tried downloading different datasets and variables, but the issue persists. I’ve also checked my internet connection, but the issue seems unrelated to that.

Has anyone experienced a similar problem or could suggest potential solutions? Any help would be greatly appreciated.

Thanks a lot,
Fei