Data memory issue


We recently  implemented a simple retrieval application to retrieve the Albedo product from the CDS

Here is the link to the CDS app

https://cds.climate.copernicus.eu/toolbox-editor/3086/retrieve_albedo_product_vgt

Where running the application, we encountered a memory issue (see the console log).


Traceback (most recent call last):
  File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 49, in handle_request
    result = cached(context.method, proc, context, context.args, context.kwargs)
  File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached
    result = proc(context, *context.args, **context.kwargs)
  File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 118, in __call__
    return p(*args, **kwargs)
  File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 59, in __call__
    return self.proc(context, *args, **kwargs)
  File "/home/cds/cdsservices/services/retrieve.py", line 191, in execute
    time_dimensions=time_dimensions, read_attributes=read_attributes
  File "/opt/cdstoolbox/cdscompute/cdscompute/context.py", line 270, in call
    return c.call(service, *args, **kwargs).value
  File "/opt/cdstoolbox/cdsworkflows/cdsworkflows/future.py", line 76, in value
    raise self._result
cdsworkflows.error.ClientError: {'traceback': 'Traceback (most recent call last):
  File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 49, in handle_request
    result = cached(context.method, proc, context, context.args, context.kwargs)
  File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached
    result = proc(context, *context.args, **context.kwargs)
  File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 118, in __call__
    return p(*args, **kwargs)
  File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 59, in __call__
    return self.proc(context, *args, **kwargs)
  File "/home/cds/cdsservices/services/cdm_translate.py", line 61, in execute
    retrieve_kwargs={'time_dimensions': time_dimensions, 'read_attributes': read_attributes}
  File "/opt/cdstoolbox/cdscdm/cdscdm/gridded.py", line 710, in open_dataset
    datasets += [xr.concat(variables[var], dim='time').sortby('time')]
  File "/usr/local/lib/python3.6/site-packages/xarray/core/dataset.py", line 5322, in sortby
    return aligned_self.isel(**indices)
  File "/usr/local/lib/python3.6/site-packages/xarray/core/dataset.py", line 1969, in isel
    var_value = var_value.isel(var_indexers)
  File "/usr/local/lib/python3.6/site-packages/xarray/core/variable.py", line 1118, in isel
    return self[key]
  File "/usr/local/lib/python3.6/site-packages/xarray/core/variable.py", line 767, in __getitem__
    data = as_indexable(self._data)[indexer]
  File "/usr/local/lib/python3.6/site-packages/xarray/core/indexing.py", line 1294, in __getitem__
    return array[key]
MemoryError: Unable to allocate array with shape (3, 15680, 40320) and data type float32
'}


We tested the code also by adding the option grid and size in the retrieval dictionnary but that does not seem to solve the issue.

Could you please provide us a way/hints or tips to overcome this issue


Best Regards,

Iskander