Best strategy to retrieve two years of model-level data (403 Client Error)

Datasets and Usage C3S - Datasets and Usage
1

Hi all, I am currently trying to retrieve two years of model level data from a series of locations.

Up until a month ago everything worked seamlessly for another set of locations, while now I get an error if I try to run the exact same request as I did before:

HTTPError: 403 Client Error: Forbidden for url: ``https://cds.climate.copernicus.eu/api/retrieve/v1/processes/reanalysis-era5-complete/execution
cost limits exceeded
Your request is too large, please reduce your selection.

I have obviously tried to reduce the request, but the only way I managed to make it work is by retrieving one month of data at the time.

This is beyond unpractical to do for all the locations and for two years of data: that would result in 144 individual request (and roughly a month ago, as I mentioned, I could easily do the exact same thing with just one request per location. Don’t know what changed).

What’s the best way to proceed? Why do I get this error which I wasn’t getting before?

My request from the API currently looks like this:

client = cdsapi.Client()

client.retrieve(
"reanalysis-era5-complete",
        {
           "date": "2020-01-01/2021-12-31"    # two years of data
           "levelist": "110/to/137",          # bottom 20 levels
           "levtype": "ml",
           "param": "130/131/132/133/152",    # five parameters
           "stream": "oper",
           "time": "00/to/23/by/1",           # hourly data
           "type": "an",
           "area": "72.5/-38.5/72.5/-38.5",   # just one gridpoint
           "grid": "0.25/0.25",              
           "format": "grib",
        },
    "my/path/here",
    )

Thanks!

2

Hi,
please see this: 403 Client Error while downloading 1 day of Europe constrained ERA5 data - #9 by Michela

Thanks

3

Hi Michela,

Thank you for your answer! I am not sure I understand how that solves my problem though. So there’s no way to request all the data at once (even if I could do so just few weeks ago)? I have to make 144 individual requests…? (I don’t care too much if the retrieval itself is slow: I just need the data). And I don’t need surface data, but model level data.

Thanks again!

4

Just an update: now it is impossible to even download one month of data. Could you please clairfy what’s going on with CDS? I have seen other similar posts. My workflow is really taking a strong hit from this recent change.

Thanks, Lorenzo

5

Hi Lorenzo,

I am not involved in CDS or Copernicus, but here to give you a small advice about MARS, as the title of your post is “Best strategy to retrieve the data…”.

While it does work, retrieving the data point by point from tapes probably wouldn’t be one of the best ways to get the data from MARS archive.

How does MARS work?
Data in MARS library is stored on tapes. Whenever you request the data the little robot goes to fetch the tape to give you the data.

Whenever you repeatedly ask the data from the same tape (by asking for the same data on different location) the system will give your request lower priority and you might end up queuing for more. (This is independent of the CDS’s additonal restrictions).

It would improve your waiting times (and reduce thenumber of the requests), if you put all your points in one area (or even get the global data), and then extract the points locally, after you download the data.

I hope this helps.

Milana

6

Hello, Lorenzo! :smiley:

I have the same error. Have you resolved it successfully?
Please let me know if you have found a solution. Even though my data request is just for one day, the error persists.

7

Hi,
the relevant team suggests for users downloading data in NetCDF to request 1 month 1 variable and all levels for the whole globe and after cut the area/grid point needed on your local machine. Please send requests sequentially, wait for one to finish and send the next.
Thanks

8

Hello,

Thank you all for your answers!

@Milana_Vuckovic this makes a big difference, thanks. I am now requesting data from a big bounding box encompassing all my locations and this reduced the number of requests without making the retrieval much slower (each file I am getting is roughly 5 Gb and takes around 90 min to retrieve)

@Michela I am downloading data in grib, does this matter for grib as well?
Right now I am requesting the bottom 20 levels, 5 variables, from a bounding box roughly spreading from Texas to Europe up to South Greenland. I am requesting 15 days at the time, everything above 15 days wouldn’t work.

@Brenda_Lopez see answer above. Wouldn’t know why one day of data doesn’t work… perhaps you could provide more details (or open a post)?
Anyway, for now I have created a script that submits a fifteen-day requests, then checks every X hours if it has finished, and once it’s finished it fetches it and submit the request for the following 15 day. Will continue like this until I download my 2 years…

Cheers

Lorenzo

9

Hi again. I tried incorporating all this suggestions in a Python package to handle the downloads more easily: Python package to (hopefully) retrieve data without cdsapi 403 Client Error

Hopefully useful for you

cheers

Lorenzo