As many users will have already noticed, a change has recently been implemented in some ERA5 datasets with respect to the retrieval of data in NetCDF format versus GRIB format.
The native ERA5 data is in GRIB format. When you download the data in NetCDF, additional processing is required which impacts the performance of the system. For this reason, a smaller amount of data in NetCDF can be retrieved rather than in GRIB format.
You can check whether your request is within scope using the data download form on the Climate Data Store (CDS) and update your API script accordingly.
The new constraints have been applied to the following ERA5 family datasets:
We will update this topic should other CDS datasets be added to the list above or changes to these constraints be made.
ECMWF Support
1 Like
Thanks for the update — though once again, it comes only after the changes have been implemented. This reactive communication style seems to be the norm lately, which is increasingly difficult for those of us relying on CDS in automated workflows.
While there’s no mailing list (yet), it’s still surprising that with all the user data already available, a more direct or proactive communication method hasn’t been considered. Relying solely on forum posts after-the-fact doesn’t exactly support smooth operations on our side.
A heads-up in advance — even just a basic notification system for planned changes — would help a lot in reducing unnecessary troubleshooting and downtime.
Hopefully this is something that can finally be prioritized — it’s clearly not a new concern.
10 Likes
THIS. Please.
An announcement in the forum.
Ideally a heads-up.
We’ve been using netcdf as a download format for more than 5 years now. It suddenly stopped working for our requests. Having to scrap our plans for the week and go into quick-fix and testing mode really takes the fun out of life and what we are doing.
Hello, we use NetCDF forecast data in our CNES (French Space Agency) project.
We also found new constraints for downloading ERA5 data : we needed to download for 370 dates.
Please find the encountered error :
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://cds.climate.copernicus.eu/api/retrieve/v1/processes/reanalysis-era5-pressure-levels/execution
cost limits exceeded
Your request is too large, please reduce your selection.
We tested that request with 100 dates several times to fulfill the 370 dates, this was OK.
It would have been preferable to be warned at least one month in advance for such change of policy, to be able to anticipate our software modifications.
On the other hand, can ESA ECMWF service provide super user capacity for downloading without constraints as we were used to in the past ?
This would avoid us an important software modification.
Thank you in advance for your feedback
CNES FORMATER/FLATSIM Exploitation team
You used to be able to calculate whether or not a request was small enough to process by multiplying the relevant factors (parameter-count, measurement-time-count and gridpoint-count).
“Additional processing” seems rather arbitrary however and not quantifiable. I could theoretically replace my requests based on months with smaller ones, but for some requests I’d rather work with bigger requests than a lot that have to be stitched back together. Is there some way to determine if a request would work before running it?
