The new Climate Data Store Beta (CDS-Beta) is now live!

Anabelle · 17 July 2024 15:25

The Climate Data Store Beta (CDS-Beta) is now live!

https://cds-beta.climate.copernicus.eu/

Remember that you need to have an ECMWF account to use CDS-Beta. Your existing CDS credentials will not work in CDS-Beta! If you do not yet have an ECMWF account, please create one here, then with your ECMWF login account, login to CDS-Beta and follow instructions online.
API users need to upgrade their cdsapi package and update their credentials.

Important to note:

Please migrate to using CDS-Beta by September 26th, 2024 when current CDS will be shutdown.
CDS Toolbox will be discontinued and is not migrated to CDS-Beta.

We strongly recommend you read our Informative page in readiness for CDS-Beta.

With a launch of a new system on a state-of-the-art infrastructure, we understand that a natural perception is to assume that the new CDS Beta will be a visually revamped and fast performing system. This may not be your immediate experience. In fact, you may not initially see much difference between the current CDS and CDS Beta. The positive aspect of this is that it will help you to quickly get familiar with the new service and ease your move to using CDS Beta. While your interaction with the data won’t change too much, the key point here is what is new and at the heart of CDS Beta is the underlying technology, which you cannot see.

The modernisation of the Climate Data Store is a journey on which we embarked for the users and with the users. Without your support and your collaboration, this transition would simply not be achievable. We thank you all again for your patience during this significant modernisation of our Data Stores and eagerly await for your constructive feedback via our Support Portal!

ECMWF Support
On behalf of the Data Stores team

Anabelle · 18 July 2024 15:45

Your existing CDS credentials will not work in CDS-Beta!

If you do not yet have an ECMWF account, please create one here

Then with your ECMWF login account, login to CDS-Beta and follow instructions online.

user104 · 24 July 2024 09:09

Dear Anabelle,
I recently tried using the new beta version of the API and encountered an issue that I thought might interest you. Even when following the official example provided, I’m faced with an error. Here’s the code I used:

import cdsapi
client = cdsapi.Client()
dataset = 'reanalysis-era5-pressure-levels'
request = {
  'product_type': ['reanalysis'],
  'variable': ['geopotential'],
  'year': ['2024'],
  'month': ['03'],
  'day': ['01'],
  'time': ['13:00'],
  'pressure_level': ['1000'],
  'data_format': 'grib',
}
target = 'download.grib'
client.retrieve(dataset, request, target)

However, I’m getting the following error:

HTTPError: 404 Client Error: Not Found for url: https://cds-beta.climate.copernicus.eu/api/resources/reanalysis-era5-pressure-levels
During handling of the above exception, another exception occurred:

I’m puzzled as to why this is happening. Is it possible that the new version of the API is not ready for data downloads yet? I would appreciate any insights you might have on this issue.
Best regards,
W

Sascha_Hofmann · 24 July 2024 13:04

I am also getting an error trying to use the new CDS-beta. My error message is related to licensing

RuntimeError: 403 Client Error: required licences not accepted
required licences not accepted; please accept the following licences to proceed: [{'id': 'licence-to-use-copernicus-products', 'revision': 12}]

On my new CDS-beta account I have accepted all three terms & conditions

So I checked inside my ECMWF account (as a side note I find it confusing that the new CDS account has settings and while it says they’re taken from my ECMWF ACC it still looks like I could change them) and couldn’t find any T&Cs I could confirm. Maybe its on this link that is broken Keycloak Account Management

Michela · 24 July 2024 13:12

Hi,
you need to go to the ERA5 pressure levels webform (ERA5 hourly data on pressure levels from 1940 to present) and accepted the Licence to use Copernicus Products

Thanks

Sascha_Hofmann · 24 July 2024 13:30

Awesome worked! I feel like this could be part of the Accounts optional T&Cs? Definitely felt impossible to find

Jason_McCalla · 30 July 2024 05:40

I’ve been using the API mentioned here REST API for CDS and ADS - Software and Tools / API - Forum (ecmwf.int)

Swagger UI (copernicus.eu)

Is there a BETA alternative on the new platform

Alessio_Corsini · 30 July 2024 08:43

Hi
i’m use the api for take the info about the dataset, like this call:
example: https://cds.climate.copernicus.eu/api/v2.ui/resources/cems-glofas-historical

I would need to know if the structure of these APIs will be changed, because it seems that they now use a different structure, can anyone confirm if this will be the new call?
https://cds-beta.climate.copernicus.eu/api/retrieve/v1/processes/cems-glofas-historical

Marco_Bacci1 · 30 July 2024 09:08

It took them 7 months to make this upgrade, and now we have to get our things in order in one month - August - which is notoriously the month of summer vacations for everyone. This feels utterly disrespectful to me and any other worker.

Please migrate to using CDS-Beta by September 3rd, 2024 when current CDS will be shutdown.

Michela · 30 July 2024 11:27

Thank you for contacting us. At this point, to move to CDS-beta, all you need to do is to register a new account, update your CDS API and your key.

We were under a lot of pressure opening the new CDS and we shared as much information as we could on our forum at https://forum.ecmwf.int (check ‘Announcements’). We will make sure that all our users are supported when moving to the new CDS in coming weeks. If you have anything specific you need help, please raise this at our support portal

Thanks

Marco_Bacci1 · 30 July 2024 12:10

It’s still very poorly timed and so managed

Miguel_Dominguez · 31 July 2024 08:42

Good morning,

I am trying to fetch some data from the database but I get a syntax error when trying to run the cdsapi.Client() (my cdsapi version is 0.7.0, and I am running it in Python 3.7):

File “test.py”, line 3, in
client = cdsapi.Client()
File “/.venv/lib/python3.7/site-packages/cdsapi/api.py”, line 284, in new
import cads_api_client.legacy_api_client
File “/.venv/lib/python3.7/site-packages/cads_api_client/init.py”, line 25, in
from .api_client import ApiClient
File “/.venv/lib/python3.7/site-packages/cads_api_client/api_client.py”, line 7, in
from . import catalogue, config, processing, profile
File “/.venv/lib/python3.7/site-packages/cads_api_client/catalogue.py”, line 14, in
from . import processing
File “/.venv/lib/python3.7/site-packages/cads_api_client/processing.py”, line 95
if not (content := message.get(“content”)):
^
SyntaxError: invalid syntax

The issue is that “Python version 3.7 does not support assignment expressions”. I tried upgrading the cdsapi, but it is already up to date. Is there a way to solve this? I did not have any issues with the previous CDS version.

Thank you.

EDIT: It is related to the personal access token. In the previous CDS version, it contained a “:”, which redirected the software to the new version of the API client, not the legacy one. However, my new access key does not contain a “:”, so it redirects me to the legacy version of the software and, as a result, crashes. How can this be fixed?

Peter_Akers · 31 July 2024 13:08

Hi, I would also be very interested in a Swagger page for the new API. I get my data using python requests and it’s hard to tell what exactly I will need to update

Peter_Akers · 1 August 2024 12:37

For anyone wondering about the same thing, you can post a request for data using

https://cds-beta.climate.copernicus.eu/api/retrieve/v1/processes/reanalysis-era5-single-levels/execution

and a body such as

{
“inputs”: {
“product_type”: [
“reanalysis”
],
“variable”: [
“10m_u_component_of_wind”
],
“year”: [
“2022”
],
“month”: [
“03”
],
“day”: [
“20”
],
“time”: [
“07:00”
],
“data_format”: “netcdf”
}
}

It looks like the authorization has changed from basic to bearer token, I’m still working out how to generate this, I would be grateful if anyone could help with this

Wagner_Flauber_Lima · 1 August 2024 12:48

I am also getting an error trying to use the new CDS-beta.

Traceback (most recent call last):
File “./modelos_climaticos_tendencia.py”, line 193, in
raw,prob = mod.ecmwf(model,version[ind],tempo)
File “./modelos_climaticos_tendencia.py”, line 46, in ecmwf
client = cdsapi.Client()
File “/home/ubuntu/software/anaconda3/envs/pyvli/lib/python3.6/site-packages/cdsapi/api.py”, line 284, in new
import cads_api_client.legacy_api_client
File “/home/ubuntu/software/anaconda3/envs/pyvli/lib/python3.6/site-packages/cads_api_client/init.py”, line 25, in
from .api_client import ApiClient
File “/home/ubuntu/software/anaconda3/envs/pyvli/lib/python3.6/site-packages/cads_api_client/api_client.py”, line 7, in
from . import catalogue, config, processing, profile
File “/home/ubuntu/software/anaconda3/envs/pyvli/lib/python3.6/site-packages/cads_api_client/catalogue.py”, line 1
from future import annotations
^
SyntaxError: future feature annotations is not defined

Anabelle · 1 August 2024 12:53

To all API users,

Please make sure that you are using (or upgrade to) a supported version of Python i.e. Python 3.8 and above: Download Python | Python.org

ECMWF Support
On behalf of the Data Stores team

Ikrame_Boubaker · 2 August 2024 09:15

What are the differences between the old CDS version and the new one (CDS-Beta)?

Abhisek_Maiti · 3 August 2024 12:27

How to migrate things which relied upon the service API? For example I get “NotImplementedError: This is a beta version. This functionality has not been implemented yet” when I try the following:

result = c.service(
    "tool.toolbox.orchestrator.workflow",
    params={
        "realm": "user-apps",
        "project": ["app-c3s-daily-era5-statistics"],
        "version": "master",
        "kwargs": {
            "dataset": "reanalysis-era5-land",
            "product_type": "ensemble_mean",
            "variable": var,
            "statistic": stat,
            "year": years,
            "month": months,
            "time_zone": "UTC+00:0",
            "frequency": "1-hourly",
            "grid": "0.1/0.1",
            'area': [52.5, 4.5, 52, 5.25]
        },
    "workflow_name": "application"
    }
)

Qi_Wen · 5 August 2024 10:19

Downloading the code has been working fine, but one day running it suddenly reported an error "ProcessingFailedError: The job has failed.
We are unable to convert this GRIB data to netCDF, please download as GRIB and convert to netCDF locally.

The job failed with: RuntimeError"

import cdsapi

dataset = “reanalysis-era5-single-levels-monthly-means”
request = {
** ‘product_type’: [‘monthly_averaged_reanalysis’],**
** ‘variable’: [‘total_precipitation’],**
** ‘year’: [‘1979’,‘1980’, ‘1981’, ‘1982’, ‘1983’, ‘1984’, ‘1985’, ‘1986’, ‘1987’, ‘1988’, ‘1989’, ‘1990’, ‘1991’, ‘1992’, ‘1993’, ‘1994’, ‘1995’, ‘1996’, ‘1997’, ‘1998’, ‘1999’, ‘2000’, ‘2001’, ‘2002’, ‘2003’, ‘2004’, ‘2005’, ‘2006’, ‘2007’, ‘2008’, ‘2009’, ‘2010’, ‘2011’, ‘2012’, ‘2013’, ‘2014’, ‘2015’, ‘2016’, ‘2017’, ‘2018’, ‘2019’, ‘2020’, ‘2021’, ‘2022’, ‘2023’],**
** ‘month’: [‘01’, ‘02’, ‘03’, ‘04’, ‘05’, ‘06’, ‘07’, ‘08’, ‘09’, ‘10’, ‘11’, ‘12’],**
** ‘time’: [‘00:00’],**
** ‘data_format’: ‘netcdf’,**
** ‘area’: [50, -30, 0, 50]**
}

client = cdsapi.Client()
client.retrieve(dataset, request).download(r’I:\data\HadCRUT.5\ERA5_tp_1979-2023.nc’)