Hi!
I want to retrieve data from the a CDS dataset (more particularly this one: https://cds.climate.copernicus.eu/cdsapp#!/dataset/sis-agrometeorological-indicators?tab=overview) using the API, throught a jupyter notebook in python 3. I want to retrieve quite a lot of parameters, so this exceeds the 100 elements limit for each request. Thus I need to do for loops for each half month of my wanted time period, as suggested by the CDS efficiency tips.
Data gets downloaded but after some requests (honestly it seems quite random, I couldn't say how many), I get the following error :
2021-08-13 14:20:26,161 INFO Welcome to the CDS
2021-08-13 14:20:26,161 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/sis-agrometeorological-indicators
2021-08-13 14:20:26,162 DEBUG POST https://cds.climate.copernicus.eu/api/v2/resources/sis-agrometeorological-indicators {"variable": "2m_relative_humidity", "year": ["2020"], "month": ["09"], "day": ["16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30"], "time": ["18_00"], "format": "zip", "area": [-6.4, 34.3, -6.5, 34.4], "nocache": "123"}
2021-08-13 14:20:26,188 DEBUG https://cds.climate.copernicus.eu:443 "POST /api/v2/resources/sis-agrometeorological-indicators HTTP/1.1" 202 124
2021-08-13 14:20:26,190 DEBUG REPLY {'state': 'queued', 'request_id': '175cf049-55ed-4775-94df-aae46b8ed2e7', 'specific_metadata_json': {'top_request_origin': 'api'}}
2021-08-13 14:20:26,190 INFO Request is queued
2021-08-13 14:20:26,191 DEBUG Request ID is 175cf049-55ed-4775-94df-aae46b8ed2e7, sleep 1
2021-08-13 14:20:27,193 DEBUG GET https://cds.climate.copernicus.eu/api/v2/tasks/175cf049-55ed-4775-94df-aae46b8ed2e7
2021-08-13 14:20:27,216 DEBUG https://cds.climate.copernicus.eu:443 "GET /api/v2/tasks/175cf049-55ed-4775-94df-aae46b8ed2e7 HTTP/1.1" 200 None
2021-08-13 14:20:27,217 DEBUG REPLY {'state': 'failed', 'request_id': '175cf049-55ed-4775-94df-aae46b8ed2e7', 'error': {'message': 'an internal error occurred processing your request', 'url': 'http://copernicus-climate.eu/exc/exception', 'reason': "[Errno 13] Permission denied: '/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h'", 'context': {'traceback': 'Traceback (most recent call last):\n File "/opt/cds/adaptor/cdshandlers/adaptorlib/adaptorrequesthandler.py", line 66, in handle_request\n return super().handle_request(cdsinf, data_request, self.config)\n File "/opt/cds/cdsinf/python/lib/cdsinf/runner/requesthandler.py", line 113, in handle_request\n return handler(cdsinf, request, config)\n File "/opt/cds/adaptor/cdshandlers/archivecombiner/handler.py", line 63, in handle_retrieve\n config)\n File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 103, in combine\n di)\n File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 189, in process_one_input_archive\n di, extractor_class)\n File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 170, in dataitem_to_extractor\n allocation = di.as_file(use_server_filename=self.use_server_filenames)\n File "/opt/cds/cdsinf/python/lib/cdsinf/brokerclient.py", line 172, in as_file\n size=self.size) as allocation:\n File "/opt/cds/cdsinf/python/lib/cdsinf/cdsinf.py", line 564, in get_cache_file\n return self.cachemanager.get_file(self, **kwargs)\n File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 80, in get_file\n return self._allocate(cdsinf, kwargs, ReusableCacheAllocationFile)\n File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 104, in _allocate\n cdsinf, hints, alloc_class, cache, entryname)\n File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 136, in _allocate_in_cache\n directory=alloc_class._dir)\n File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 756, in create_entry\n os.makedirs(dirname, exist_ok=not directory or not new)\n File "/usr/lib64/python3.6/os.py", line 220, in makedirs\n mkdir(name, mode)\nPermissionError: [Errno 13] Permission denied: \'/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h\'\n'}, 'permanent': True, 'who': 'server'}, 'sent_to_rmq_at': '2021-08-13T14:20:26.533Z', 'specific_metadata_json': {'top_request_origin': 'api'}}
2021-08-13 14:20:27,218 INFO Request is failed
2021-08-13 14:20:27,218 ERROR Message: an internal error occurred processing your request
2021-08-13 14:20:27,219 ERROR Reason: [Errno 13] Permission denied: '/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h'
2021-08-13 14:20:27,220 ERROR Traceback (most recent call last):
2021-08-13 14:20:27,220 ERROR File "/opt/cds/adaptor/cdshandlers/adaptorlib/adaptorrequesthandler.py", line 66, in handle_request
2021-08-13 14:20:27,221 ERROR return super().handle_request(cdsinf, data_request, self.config)
2021-08-13 14:20:27,222 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/runner/requesthandler.py", line 113, in handle_request
2021-08-13 14:20:27,222 ERROR return handler(cdsinf, request, config)
2021-08-13 14:20:27,223 ERROR File "/opt/cds/adaptor/cdshandlers/archivecombiner/handler.py", line 63, in handle_retrieve
2021-08-13 14:20:27,223 ERROR config)
2021-08-13 14:20:27,224 ERROR File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 103, in combine
2021-08-13 14:20:27,225 ERROR di)
2021-08-13 14:20:27,225 ERROR File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 189, in process_one_input_archive
2021-08-13 14:20:27,226 ERROR di, extractor_class)
2021-08-13 14:20:27,229 ERROR File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 170, in dataitem_to_extractor
2021-08-13 14:20:27,229 ERROR allocation = di.as_file(use_server_filename=self.use_server_filenames)
2021-08-13 14:20:27,230 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/brokerclient.py", line 172, in as_file
2021-08-13 14:20:27,230 ERROR size=self.size) as allocation:
2021-08-13 14:20:27,231 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/cdsinf.py", line 564, in get_cache_file
2021-08-13 14:20:27,232 ERROR return self.cachemanager.get_file(self, **kwargs)
2021-08-13 14:20:27,232 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 80, in get_file
2021-08-13 14:20:27,233 ERROR return self._allocate(cdsinf, kwargs, ReusableCacheAllocationFile)
2021-08-13 14:20:27,233 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 104, in _allocate
2021-08-13 14:20:27,234 ERROR cdsinf, hints, alloc_class, cache, entryname)
2021-08-13 14:20:27,235 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 136, in _allocate_in_cache
2021-08-13 14:20:27,235 ERROR directory=alloc_class._dir)
2021-08-13 14:20:27,236 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 756, in create_entry
2021-08-13 14:20:27,236 ERROR os.makedirs(dirname, exist_ok=not directory or not new)
2021-08-13 14:20:27,237 ERROR File "/usr/lib64/python3.6/os.py", line 220, in makedirs
2021-08-13 14:20:27,240 ERROR mkdir(name, mode)
2021-08-13 14:20:27,240 ERROR PermissionError: [Errno 13] Permission denied: '/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h'
I'll add a code snippet below, but know that I've already read this topic and added a
|
|
but I still have the error. In order to see clearer if it is a cache problem or not, I also added
|
|
as suggested in this topic. Since the request is failed, it unfortunately does not display the 'location' alongside the 'request_id'. Thus I can't tell if it's from the cache but it would seem so when looking at the error message (PermissionError). For previous CDS requests, the request ID doesn't match the one in the location field, which means the result is provided by a cached one. Thus, the nocache argument fails. How could this be? See below to get a code snippet.
Other than that, why would I have a PermissionDenied error for cached data when the CDS frequently give cached data for specific requests?
Thank you for reading!
years_input = list(df_time['year'].unique()) #this gets a list of years to get data from cds = cdsapi.Client(quiet=False, debug=True)for y in range(int(min(years_input)), int(max(years_input)) + 1):
months_input = list(df_time[df_time[‘year’] == y][‘month’].unique())
years = [str(y)]
for m in range(min(months_input), max(months_input) + 1):
days_input = list(df_time[df_time[‘month’] == m][‘day’].unique())
months = [(lambda x: x.zfill(2))(str(m))]
idx_days = int(len(days_input)/2)
for days in [days_input[:idx_days], days_input[idx_days:]]:
days = [(lambda x: x.zfill(2))(str(x)) for x in days]# each half of month we do a request (otherwise there is the "Request Failed: Request has too many items" error # here I only left the max temp request but following this one, we have : mean temp, min temp, solar radiation flux 24h mean, relative humidity for # 6AM, 9AM, 12AM, 3PM, 6PM, precipitation 24h mean, wind speed 24h mean # request example cds.retrieve('sis-agrometeorological-indicators', { "variable": [ "2m_temperature" ], 'statistic': [ '24_hour_maximum', ], "product_type": "reanalysis", "year": years, "month": months, "day" : days, "format": "zip", "area": area, 'nocache': '123', }, 'era5_tmp.zip') with ZipFile('era5_tmp.zip', 'r') as zipObj: zipObj.extractall(folder_era)</pre> </div> </div>