Hi everyone,
I’ve been trying to download data using the cdsapi Python library from the Climate Data Store (CDS) API. Below is the code I used to attempt the download:
import cdsapi
dataset = "reanalysis-era5-single-levels"
request = {
"product_type": ["reanalysis"],
"variable": ["surface_pressure"],
"year": ["2000"],
"month": ["11"],
"day": [
"01", "02", "03",
"04", "05", "06",
"07", "08", "09",
"10", "11", "12",
"13", "14", "15",
"16", "17", "18",
"19", "20", "21",
"22", "23", "24",
"25", "26", "27",
"28", "29", "30"
],
"time": [
"00:00", "01:00", "02:00",
"03:00", "04:00", "05:00",
"06:00", "07:00", "08:00",
"09:00", "10:00", "11:00",
"12:00", "13:00", "14:00",
"15:00", "16:00", "17:00",
"18:00", "19:00", "20:00",
"21:00", "22:00", "23:00"
],
"data_format": "netcdf",
"download_format": "unarchived"
}
client = cdsapi.Client()
download_folder = "D:/ERA5_surface_pressure/"
filename = download_folder + "2000_11_suface_pressure.nc"
target = filename
client.retrieve(dataset, request, target)
And here is the error message I encountered:
d5f.nc: 6%|▌ | 56.0M/923M [05:52<5:48:50, 43.4kB/s]
5401cc67e7861e52d21a68d8b6addd5f.nc: 6%|▌ | 57.0M/923M [06:12<6:44:01, 37.4kB/s]
5401cc67e7861e52d21a68d8b6addd5f.nc: 6%|▌ | 57.0M/923M [06:33<6:44:01, 37.4kB/s]
Traceback (most recent call last):
File "D:\python\install\Lib\site-packages\urllib3\response.py", line 748, in _error_catcher
yield
File "D:\python\install\Lib\site-packages\urllib3\response.py", line 894, in _raw_read
raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
urllib3.exceptions.IncompleteRead: IncompleteRead(60498240 bytes read, 906996746 more expected)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "D:\python\install\Lib\site-packages\requests\models.py", line 820, in generate
yield from self.raw.stream(chunk_size, decode_content=True)
File "D:\python\install\Lib\site-packages\urllib3\response.py", line 1060, in stream
data = self.read(amt=amt, decode_content=decode_content)
File "D:\python\install\Lib\site-packages\urllib3\response.py", line 977, in read
data = self._raw_read(amt)
File "D:\python\install\Lib\site-packages\urllib3\response.py", line 872, in _raw_read
with self._error_catcher():
File "D:\python\install\Lib\contextlib.py", line 155, in __exit__
self.gen.throw(typ, value, traceback)
File "D:\python\install\Lib\site-packages\urllib3\response.py", line 772, in _error_catcher
raise ProtocolError(arg, e) from e
urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(60498240 bytes read, 906996746 more expected)', IncompleteRead(60498240 bytes read, 906996746 more expected))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\python\install\MyScripts\test.py", line 39, in <module>
client.retrieve(dataset, request, target)
File "D:\python\install\Lib\site-packages\datapi\legacy_api_client.py", line 179, in retrieve
return submitted if target is None else submitted.download(target)
File "D:\python\install\Lib\site-packages\datapi\processing.py", line 659, in download
robust_download(url, target)
File "D:\python\install\Lib\site-packages\multiurl\http.py", line 479, in wrapped
r = call(main_url, *args, **kwargs)
File "D:\python\install\Lib\site-packages\datapi\processing.py", line 625, in _download
multiurl.download(
File "D:\python\install\Lib\site-packages\multiurl\downloader.py", line 111, in download
return Downloader(url, **kwargs).download(target)
File "D:\python\install\Lib\site-packages\multiurl\base.py", line 129, in download
total = self.transfer(f, pbar)
File "D:\python\install\Lib\site-packages\multiurl\http.py", line 133, in transfer
for chunk in stream(chunk_size=self.chunk_size):
File "D:\python\install\Lib\site-packages\requests\models.py", line 822, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(60498240 bytes read, 906996746 more expected)', IncompleteRead(60498240 bytes read, 906996746 more expected))
I’ve tried downloading different datasets and variables, but the issue persists. I’ve also checked my internet connection, but the issue seems unrelated to that.
Has anyone experienced a similar problem or could suggest potential solutions? Any help would be greatly appreciated.
Thanks a lot,
Fei