A new CDS soon to be launched - expect some disruptions

,

UPDATE 04-Apr-2024:Despite ongoing disruptions, the CDS is now once again delivering data volumes near to its full capacity (as shown on CDS Live).

However, CDS performance is linked to users’ behaviour. To help alleviate the longer than normal waiting times, a decision has been made to implement an additional temporary measure to ERA5 and ERA5-Land hourly (single level/pressure level) data requests from now on until the use of the new data infrastructure is fully optimised.

All data requests asking for ERA5 and ERA5-Land hourly data will now be restricted to a value of half of the previous limit. The new limits will be 60000 ‘items’ and 6000 ‘items’ respectively, and these limits will apply to both CDS web form and CDS API requests.

Please note that we advise users to request one month of ERA5 or ERA5-Land data at a time, as this is the most efficient way to request these data from the CDS. The CDS web forms for these datasets will also be modified to implement this restriction.

We therefore ask all users to follow our guidelines. As a result, the CDS performance will improve, and our technical team will continue to optimise the new infrastructure in preparation for the release of the new system.

We sincerely apologise for these temporary restrictions; however they will lead to an improved service for all users.


UPDATE 20-Mar-2024: Despite ongoing disruptions, the CDS continues to deliver terabytes of data daily (as shown on [CDS Live](https://cds.climate.copernicus.eu/live/)). Temporary measures remain in place for the time being. We appreciate your patience and cooperation.

UPDATE on 27-Feb-2024: We are in the process of migrating our CDS to a new infrastructure, in order to provide a more resilient and scalable service in the future. Whilst this work is progressing well, we are aware that some users are facing longer waiting times than usual when requesting data from the CDS. The reasons behind this are mainly:

Ongoing work on our data infrastructure: We are transitioning our service to a new cloud infrastructure, which is leading to some temporary reductions in system performance. During this period we are making a concerted effort to minimise the impact on our users;

More users are making larger requests, in particular, requests with a long time range. Our system is optimised for handling requests of up to one month of data in many cases, e.g. for the ERA5 data.

To help alleviate the longer than normal waiting times, a decision has been made to implement a temporary measure from now on until the use of the new data infrastructure is fully optimised. All data requests asking for hourly data with the period of more than one month will be set with lower priorities in the queue, which means other requests will get executed earlier. If the CDS is affected heavily by such long period data requests, we will remove such requests from the queue. If this happens, please resubmit your requests with reduced period (one month) afterwards.

This is a difficult decision, but we believe it is in the best interests of our users. With this temporary measure in place, users should experience improved overall performance while our technical team optimises the new infrastructure. We sincerely apologise for this temporary degradation in service and are working hard to improve the situation.


UPDATE on 13-Feb-2024: Please read: CDS and ADS migrating to new infrastructure: Common Data Store (CDS) Engine

We are pleased to announce that the Climate Data Store (CDS) is being modernised to better manage the growing workload, surging computing demand and better serve its current and future users. This modernisation affects all layers of the infrastructure: the front-end web interface, the back-end software engine and the underlying cloud infrastructure hosting the service and core data repositories. The new modernised CDS is expected to be launched in Spring 2024.

While the modernisation process has up to now been transparent to users, it has reached the stage when data is being migrated onto the new infrastructure and this, unfortunately, is impacting services. This migration period will take weeks to complete, during which users should be expecting some disruption and degradation of service on the CDS such as:

  • longer queues
  • requests being cancelled
  • temporary closure of queues
  • requests failing with unusual error messages
  • catalogue temporarily offline

Relevant information (e.g. activation of new single sign-on ECMWF account) will be shared with CDS users nearer the time of the launch.

Users of the CDS Toolbox should however take note that the CDS Toolbox will be discontinued and will not be migrated to the new CDS infrastructure. New tool packages will be made available (post new CDS launch) to provide software tools for weather and climate workflows that simplify data access, analysis, visualisation and much more. 

We understand the inconvenience the disruptions are causing during this time. The CDS team is working hard at keeping disruption to a minimum. 

We will keep our users informed with the latest details on this announcement page. We therefore recommend users to "Watch" this page to receive email notifications of updates on this announcement.

We thank our users for their patience during this significant modernisation of the CDS which will enable exciting scientific developments and an overall better service to all ECMWF users.

CDS Team/ECMWF Support

2 Likes

any changes foreseen for cdsapi?

Is there any further update? I used to use this service on an AWS S3 bucket, but it has been fully retired:

https://registry.opendata.aws/ecmwf-era5/

It would be great to see a similar approach taken at ECMWF, via an S3 bucket. Then I don't need to have jobs running just waiting in the queue.

Thanks in advance!

With Open-Meteo, I now started to redistribute ERA5 and ERA5-Land on AWS Open-data. Updated daily. For any time-series analysis, this works well.

Ok that is great, I'll check it out. I think Open-Meteo is amazing btw (really, really amazing!!)

How many days are still required to finish the migration? 

1 Like

The API is completely unusable. Such a disruption with no mitigation strategy is hard to believe.

2 Likes

I totally support @Marco_Bacci 's comment. This disruption is getting problematic.
Can we expect some improvement before the migration?

2 Likes

I am encountering the same problem as the other users.

Can we access the new API in the beta version?

It would be helpful for us to get a head start on the migration process and avoid any disruptions on our end when the migration is complete.

Could you please provide us with more information to assist us with the transition?

1 Like

It would be nice to have more information about the migration.

How many weeks are we talking when you say "This migration period will take weeks to complete"?

There is any plan to mitigate the impact on the API?

Are there any changes foreseen to the cdsapi? If yes, when the new tools are going to be released? How long will the old tools remain operational?

1 Like

Are there any updates on this matter? It’s encouraging to hear that the CDS API is undergoing enhancements, yet the current interruption of the service poses challenges, particularly without clear indications of when operations will resume as usual. Can we anticipate how long this disruption will persist? Unfortunately, a vague timeframe like ‘Spring 2024’ lacks specificity; it would be helpful to know the expected duration in months.

3 Likes

I would also like to have some more information on the migration timeline and the service disruptions. Data requests/queue times for small datasets (<50 MB) from the KPI dashboard have ballooned from ~2 minutes to well over an hour, and from experience these requests are sometimes taking 6 hours to be fulfilled.

1 Like

I wanna know if there have been any changes to the wrapped methods in the existing Python cdsapi library besides verification aspects like keys. This implies that the code I’ve written may need to be modified again.

The lack of timeline is disruptive and baffling. Can datasets such as ERA5 be accessed anywhere else in their entirety? Or do we have to wait “weeks” in order to retrieve updated runs?

1 Like

I try to dowload data abaout “ERA5 hourly data on single levels from 1940 to present” but I cant pass the “please check mandatory field”.
ıs this problem is about the distruption? or am I wrong about selections?

please help me

Hi Patrick,
Do you know if the Open-Meteo API enables to download ERA5 data of areas (no single locations) and in GRIB format?
Thanks!

Hi Johan! Open-Meteo is designed to get time-series data for individual locations. The underlaying database is “rotated” to optimise fast access to the temporal dimension. Many applications for energy, agriculture, climate research or insurances “just” need fast data access to few locations.
However, you can request data for a list of coordinates. An extension to get data for a bounding box will follow in the next months. GRIB as an output format is not supported and it is unlikely that support will be added. If you self host the Open-Meteo API locally and use the provided Python SDK, you can quickly process larger areas of ERA5 or other weather models. Feel free to reach out GitHub Discussion sections!