UPDATE 09-May-2024: We are aware that many of you have been eagerly waiting for an update with respect to the launch of the new Climate Data Store (CDS) and the new Atmosphere Data Store (ADS) services.
Due to the high complexity of migrating all layers of the Data Stores (Cloud, Data, Software) onto new, state-of-the-art hardware and software infrastructures while keeping all systems running, delivering over 150TB of data a day and supporting 300k users, there have been some unavoidable delays and we thank you for your patience.
We are now in a position to inform you that, in preparation for the opening of the new Climate Data Store (CDS), built on the new Common Data Store Engine (CDS-E) software infrastructure, in a beta version, the final testing phase has started. At this stage, there may be some time periods when queued requests are temporarily put on hold before request processing can resume again.
Pending outcome of the tests, the intention is to open to users the CDS beta service by the end of this month (May 2024). The opening of the Atmosphere Data Store (ADS) beta will follow soon after.
Meanwhile, the current Climate and Atmosphere Data Stores will continue to operate until complete shutdown on 1st September 2024.
Stay tuned for further announcements!
UPDATE 04-Apr-2024:Despite ongoing disruptions, the CDS is now once again delivering data volumes near to its full capacity (as shown on CDS Live).
However, CDS performance is linked to usersā behaviour. To help alleviate the longer than normal waiting times, a decision has been made to implement an additional temporary measure to ERA5 and ERA5-Land hourly (single level/pressure level) data requests from now on until the use of the new data infrastructure is fully optimised.
All data requests asking for ERA5 and ERA5-Land hourly data will now be restricted to a value of half of the previous limit. The new limits will be 60000 āitemsā and 6000 āitemsā respectively, and these limits will apply to both CDS web form and CDS API requests.
Please note that we advise users to request one month of ERA5 or ERA5-Land data at a time, as this is the most efficient way to request these data from the CDS. The CDS web forms for these datasets will also be modified to implement this restriction.
We therefore ask all users to follow our guidelines. As a result, the CDS performance will improve, and our technical team will continue to optimise the new infrastructure in preparation for the release of the new system.
We sincerely apologise for these temporary restrictions; however they will lead to an improved service for all users.
UPDATE 20-Mar-2024: Despite ongoing disruptions, the CDS continues to deliver terabytes of data daily (as shown on [CDS Live](https://cds.climate.copernicus.eu/live/)). Temporary measures remain in place for the time being. We appreciate your patience and cooperation.
UPDATE on 27-Feb-2024: We are in the process of migrating our CDS to a new infrastructure, in order to provide a more resilient and scalable service in the future. Whilst this work is progressing well, we are aware that some users are facing longer waiting times than usual when requesting data from the CDS. The reasons behind this are mainly:
Ongoing work on our data infrastructure: We are transitioning our service to a new cloud infrastructure, which is leading to some temporary reductions in system performance. During this period we are making a concerted effort to minimise the impact on our users;
More users are making larger requests, in particular, requests with a long time range. Our system is optimised for handling requests of up to one month of data in many cases, e.g. for the ERA5 data.
To help alleviate the longer than normal waiting times, a decision has been made to implement a temporary measure from now on until the use of the new data infrastructure is fully optimised. All data requests asking for hourly data with the period of more than one month will be set with lower priorities in the queue, which means other requests will get executed earlier. If the CDS is affected heavily by such long period data requests, we will remove such requests from the queue. If this happens, please resubmit your requests with reduced period (one month) afterwards.
This is a difficult decision, but we believe it is in the best interests of our users. With this temporary measure in place, users should experience improved overall performance while our technical team optimises the new infrastructure. We sincerely apologise for this temporary degradation in service and are working hard to improve the situation.
UPDATE on 13-Feb-2024: Please read: CDS and ADS migrating to new infrastructure: Common Data Store (CDS) Engine
We are pleased to announce that the Climate Data Store (CDS) is being modernised to better manage the growing workload, surging computing demand and better serve its current and future users. This modernisation affects all layers of the infrastructure: the front-end web interface, the back-end software engine and the underlying cloud infrastructure hosting the service and core data repositories. The new modernised CDS is expected to be launched in Spring 2024.
While the modernisation process has up to now been transparent to users, it has reached the stage when data is being migrated onto the new infrastructure and this, unfortunately, is impacting services. This migration period will take weeks to complete, during which users should be expecting some disruption and degradation of service on the CDS such as:
- longer queues
- requests being cancelled
- temporary closure of queues
- requests failing with unusual error messages
- catalogue temporarily offline
Relevant information (e.g. activation of new single sign-on ECMWF account) will be shared with CDS users nearer the time of the launch.
Users of the CDS Toolbox should however take note that the CDS Toolbox will be discontinued and will not be migrated to the new CDS infrastructure. New tool packages will be made available (post new CDS launch) to provide software tools for weather and climate workflows that simplify data access, analysis, visualisation and much more.
We understand the inconvenience the disruptions are causing during this time. The CDS team is working hard at keeping disruption to a minimum.
We will keep our users informed with the latest details on this announcement page. We therefore recommend users to "Watch" this page to receive email notifications of updates on this announcement.
We thank our users for their patience during this significant modernisation of the CDS which will enable exciting scientific developments and an overall better service to all ECMWF users.
CDS Team/ECMWF Support