403 Client Error while downloading 1 day of Europe constrained ERA5 data

Datasets and Usage C3S - Datasets and Usage
,
1

Hi all,

I am trying to download ERA5 data using api as a public user. I had downloaded the same dataset before a few times now and I had not encountered this error until now.

The downloading of the data is done through another package that downloads and process it and then erases the original files so that is why I have done it multiple times.

The resolution that I am requesting is 799, level list from 60 to 137, hourly analysis times and an output grid 0.25 for a domain constrained by North: 67.0°, West: -12.0°, South: 34.0°, East: 30.0° for two days on 2019. The request fails when asking for the “ERA5 hourly data on single levels from 1940 to present data”.

The error that I am getting is:

MARS Request failed!
403 Client Error: Forbidden for url: ``https://cds.climate.copernicus.eu/api/retrieve/v1/processes/reanalysis-era5-complete/execution
cost limits exceeded
Your request is too large, please reduce your selection.

Any hints would be greatly appreciated!

2

Facing the same issue. I was downloading the era5 data for one year. I had completed the downloads for 4 months and when i tried to do it for the 5th month it is showing this error.

I’m retrieving ERA5 model-level data via flex_extract (v7.1.2, using the new CDS API / ecmwf-datastores client). For the past month, model-level retrievals in 3-day chunks were completing reliably on two separate machines. As of the last 2 weeks, the same request size — and even 1-day chunks — now fail with:

403 Client Error: Forbidden for url: https://cds.climate.copernicus.eu/api/retrieve/v1/processes/reanalysis-era5-complete/execution
cost limits exceeded
Your request is too large, please reduce your selection.

Example failing request (model-level, type AN, from flex_extract’s MARS request log):slight_smile:

marsclass: EA
dataset: None
type: AN
levtype: ML
levelist: 1/to/137
date: 20170610/to/20170612
resol: 799
stream: OPER
area: 40.0/65.0/5.0/100.0
time: 00/01/02/…/23 (24 hourly steps)
step: 00
grid: 0.25/0.25
param: 130.128/133.128/131.128/132.128/077.128/246.128/247.128

3

Hey, same :slight_smile:

This happened and keeps happening when runing flex_extract with v7.1.2, and the following MARS request log:

marsclass: EA
dataset: None
type: AN
levtype: ML
levelist: 60/to/137
repres:
date: 20190423/to/20190425
resol: 799
stream: OPER
area: 67.0/-10.0/40.0/27.0
time: 00/01/02/03/04/05/06/07/08/09/10/11/12/13/14/15/16/17/18/19/20/21/22/23
step: 00
expver: 1
number: OFF
accuracy: 24
grid: 0.25/0.25
param: 130.128/133.128/131.128/132.128/077.128/246.128/247.128

4

Tried running it in 4 hour-intervals instead of the 24-hour retrievals. It is working now.

5

Thank you, it is running now as well with only 4 hours, but I had to reduce the area significantly, from -12° to 0°E and from 34° to 45°N on the bottom left corner.

6

Does anyone know if this is actually an intended change? I can’t find any communication/info/.. on this.

We are seeing downloads fail that are only ~1x1 degree, 10 variables, 24 time steps. Cutting this into even smaller chunks would be an enormous pain in the you-know-what :face_with_symbols_on_mouth:…

7

Now even 4-h intervals are failing for multi-day requests

8

Hey, I am having the same problem. Currently trying to download data for two years from 6 different locations (one gridpoint each), without success. If I decrease my requests to one month it seems to work, but I cannot submit 144 requests just to get two years of data.

@Bart_van_Stratum I also haven’t seen any official announcement on this. It’s pretty tedious, hopefully it’s something temporary that they will fix…

Looking forward to hearing about this

9

Hi,
for ERA5 hourly data on single level the suggestion is to retrieve one month at time. If you need data for grid points the suggestions is to download the data from the ERA5 single-levels dataset available in analysis-ready, cloud-optimised (ARCO) Zarr stores: ERA5 hourly time-series data on single levels from 1940 to present
If the variable is not available in the list, it is anyway better to ask for the whole globe and after cut the area/grid point needed on your local machine.

For ERA5 data on model levels as the data are store on tapes, to retrieve data efficiently (and get your data quicker!) you should retrieve all the data you need from one tape, then from the next tape, and so on.

As a rule of thumb everything shown on one page at parameter level in the MARS ERA5 catalogue is grouped together on one tape

  • For analysis fields this is one month of data with respect to one particular level type (e.g. surface).
  • For forecast fields on model levels this is limited to one single day.

A general suggestion is to send requests sequentially, wait for the first to finish and then send the next one.

Thanks

10

403 Forbidden (cost limits exceeded) when retrieving ERA5 2D wave spectra (param 251.140) via CDS API – request used to work a few days ago.

Hello,

I have been downloading ERA5 complete 2D wave spectra (reanalysis-era5-complete, param 251.140) via the CDS API for several months using a Python script. Until a few days ago, the same type of request was working without problems. Now, all my requests fail with:
403 Client Error: Forbidden for url: https://cds.climate.copernicus.eu/api/retrieve/v1/processes/reanalysis-era5-complete/execution
cost limits exceeded
Your request is too large, please reduce your selection.

My script used to work previously for month-long requests, but it now fails consistently across all months. Here is the CDS API request dictionary I am using:

dataset = “reanalysis-era5-complete”

request = {
“date”: “1992-01-01/to/1992-01-31”,
“grid”: “0.5/0.5”,
“class”: “ea”,
“dataset”: “era5”,
“direction”: “1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24”,
“domain”: “g”,
“expver”: “1”,
“frequency”: “1/2/3/4/5/6/7/8/9/10/11/12/13/14/15/16/17/18/19/20/21/22/23/24/25/26/27/28/29/30”,
“param”: “251.140”,
“stream”: “wave”,
“time”: “00/to/23/by/3”,
“type”: “an”,
“area”: [15, -60, -10, -25],
“data_format”: “netcdf”,
}

Each monthly request covers up to 31 days × 8 times × 24 directions × 30 frequencies over a regional area. The script sends these requests sequentially (one at a time).

I saw other threads mentioning cost limits exceeded and recent changes in limits for ERA5 requests, especially for NetCDF format. Could you please clarify:

  1. Have there been recent changes in the cost limits or selection rules for reanalysis-era5-complete (especially for 2D wave spectra)?

  2. Is there a recommended maximum temporal window (e.g. number of days or months) for this type of request?

  3. Are there specific recommendations for 2D wave spectra (param 251.140) regarding:

    • maximum number of days per request,

    • preferred format (GRIB vs NetCDF),

    • or any other constraints?

I can share my full MARS request if needed. Any guidance on how to adapt my requests to the current limits would be greatly appreciated.

Thank you in advance.

11

Hello, Sofía!

I have the same error. Have you resolved it successfully?
Please let me know if you have found a solution. Even though my data request is just for one day through flex_extract! Many months ago, I did the request without trouble. Downloading data hour by hour is not very efficient.

12

Hi,
the relevant team advises users that use NetCDF format to download wave spectra data either in 1 month but in chunks for frequencies and directions or for all frequencies and directions, 1 variable and chunks of days, say 10 or 20.
Thanks

13

Hello,

So, if I understand correctly, you recommend downloading the data in shorter intervals, such as 10 or 20 days at a time, rather than downloading an entire month at once?

I tested downloading the data day by day, and it worked. However, this approach is practically unfeasible, considering that I need approximately 40 years of data for a simulation.

I would therefore like to know whether there has been any recent change to the system or any new limitation regarding NetCDF file downloads, as I was previously able to download the data on a monthly basis without any issues.

Thank you in advance for your clarification. I look forward to hearing from you.

Best regards