We recently implemented a simple retrieval application to retrieve the Albedo product from the CDS
Here is the link to the CDS app
https://cds.climate.copernicus.eu/toolbox-editor/3086/retrieve_albedo_product_vgt
Where running the application, we encountered a memory issue (see the console log).
Traceback (most recent call last):
File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 49, in handle_request
result = cached(context.method, proc, context, context.args, context.kwargs)
File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached
result = proc(context, *context.args, **context.kwargs)
File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 118, in __call__
return p(*args, **kwargs)
File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 59, in __call__
return self.proc(context, *args, **kwargs)
File "/home/cds/cdsservices/services/retrieve.py", line 191, in execute
time_dimensions=time_dimensions, read_attributes=read_attributes
File "/opt/cdstoolbox/cdscompute/cdscompute/context.py", line 270, in call
return c.call(service, *args, **kwargs).value
File "/opt/cdstoolbox/cdsworkflows/cdsworkflows/future.py", line 76, in value
raise self._result
cdsworkflows.error.ClientError: {'traceback': 'Traceback (most recent call last):
File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 49, in handle_request
result = cached(context.method, proc, context, context.args, context.kwargs)
File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached
result = proc(context, *context.args, **context.kwargs)
File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 118, in __call__
return p(*args, **kwargs)
File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 59, in __call__
return self.proc(context, *args, **kwargs)
File "/home/cds/cdsservices/services/cdm_translate.py", line 61, in execute
retrieve_kwargs={'time_dimensions': time_dimensions, 'read_attributes': read_attributes}
File "/opt/cdstoolbox/cdscdm/cdscdm/gridded.py", line 710, in open_dataset
datasets += [xr.concat(variables[var], dim='time').sortby('time')]
File "/usr/local/lib/python3.6/site-packages/xarray/core/dataset.py", line 5322, in sortby
return aligned_self.isel(**indices)
File "/usr/local/lib/python3.6/site-packages/xarray/core/dataset.py", line 1969, in isel
var_value = var_value.isel(var_indexers)
File "/usr/local/lib/python3.6/site-packages/xarray/core/variable.py", line 1118, in isel
return self[key]
File "/usr/local/lib/python3.6/site-packages/xarray/core/variable.py", line 767, in __getitem__
data = as_indexable(self._data)[indexer]
File "/usr/local/lib/python3.6/site-packages/xarray/core/indexing.py", line 1294, in __getitem__
return array[key]
MemoryError: Unable to allocate array with shape (3, 15680, 40320) and data type float32
'}
We tested the code also by adding the option grid and size in the retrieval dictionnary but that does not seem to solve the issue.
Could you please provide us a way/hints or tips to overcome this issue
Best Regards,
Iskander