How to avoid the CDS cache issue

Software and Tools API
,
1

If a user runs exactly the same CDS API request several times, and the data from the first time the request is submitted is still in the CDS cache. then the same data are retrieved from the cache, rather than being re-extracted from the dataset.

For datasets which are updated on a daily basis (such as ERA5T), this can cause issues, as the later request may not be returned the data as expected.

To avoid this, we suggest users to add the keyword 'nocache' in their CDS API request, with a random numeric string which is changed each time the request is submitted e.g.:

'nocache': '123'

Hope that helps!

Kevin

2

Thanks for this advice. When I try adding this to the request, it fails about half the time with the error shown below. I'm not sure what could be causing this - are there any requirements for the random string?


2021-05-15 07:42:33,451 INFO Welcome to the CDS
2021-05-15 07:42:33,452 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-single-levels
2021-05-15 07:42:33,591 INFO Request is queued
2021-05-15 07:42:34,989 INFO Request is failed
2021-05-15 07:42:34,990 ERROR Message: the request you have submitted is not valid
2021-05-15 07:42:34,995 ERROR Reason:  undefined value : c for parameter NUMBER - Values are : - No request
2021-05-15 07:42:34,997 ERROR   Traceback (most recent call last):
2021-05-15 07:42:35,000 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 55, in handle_request
2021-05-15 07:42:35,003 ERROR       result = cached(context.method, proc, context, context.args, context.kwargs)
2021-05-15 07:42:35,006 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached
2021-05-15 07:42:35,011 ERROR       result = proc(context, *context.args, **context.kwargs)
2021-05-15 07:42:35,013 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 118, in __call__
2021-05-15 07:42:35,016 ERROR       return p(*args, **kwargs)
2021-05-15 07:42:35,019 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 59, in __call__
2021-05-15 07:42:35,022 ERROR       return self.proc(context, *args, **kwargs)
2021-05-15 07:42:35,025 ERROR     File "/home/cds/cdsservices/services/mars.py", line 347, in internal
2021-05-15 07:42:35,027 ERROR       return mars(context, request, **kwargs)
2021-05-15 07:42:35,030 ERROR     File "/home/cds/cdsservices/services/mars.py", line 48, in mars
2021-05-15 07:42:35,032 ERROR       execute_mars(context, requests)
2021-05-15 07:42:35,035 ERROR     File "/home/cds/cdsservices/services/mars.py", line 192, in execute_mars
2021-05-15 07:42:35,037 ERROR       context.run_command("/usr/local/bin/mars", tmp, exception=MarsException)
2021-05-15 07:42:35,039 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/context.py", line 209, in run_command
2021-05-15 07:42:35,042 ERROR       raise exception(call, proc.returncode, output)
2021-05-15 07:42:35,044 ERROR   home.cds.cdsservices.services.mars.py.MarsException: undefined value : c for parameter NUMBER - Values are : - No request
3

Joseph,

I was experiencing the same issue you were and I have been having success with only passing numbers (as a string) as the value for the 'nocache' parameter. I have been generating a random string of three numbers and passing that. Here is the Python code I have been using:


import random
import string
...
digits = string.digits
rand_str = ''.join(random.choice(digits) for i in range(3))
...
'nocache':rand_str,


I don't have any knowledge of the inner workings of the cdsapi to know if this is a permanent solution, but it has been working for me throughout my testing today.

4

Hi Joseph, Anthony,

Thanks for your comments; there is a subtle 'feature' of the API which means it is better to use a random numeric string. I've updated the original forum posting accordingly.

Thanks,

Kevin