Informative page in readiness for new CDS-beta!

Announcements C3S - Announcements
1

The Climate Data Store (CDS) Beta will officially be opened to all following pre-launch preview for C3S partners and stakeholders.

What’s new on CDS Beta? All layers of the Data Stores are new!!!
The front-end web interface, the back-end software engine and the underlying cloud infrastructure hosting the service and core data repositories are all new.

And in addition to this, new features and functionalities await you:
• Integrated user community forum
• OGC compliant APIs
• Integrated Evaluation and Quality Control (EQC) function,
• Open-source expert Python packages (Earthkit) for climate and meteorological data handling and exploitation able to deploy & run anywhere
• Expanded training material supporting users on how to access and handle data effectively

CDS Beta will need a breaking-in period under live conditions – we cannot do this without your active involvement. We will therefore be counting on your feedback to consolidate our review and fine-tuning process of CDS Beta. This is key to the opening of a fully operational new CDS.

In readiness to start using CDS Beta , we strongly recommend that you check our informative page which also includes details about how to report bugs/issues and provide feedback.

ECMWF Support
on behalf of Data Stores Team

2 Likes
2

Dear Anabelle,

Can you please provide us with a link where we can create the SSO account? I tried creating the account on ECMWF, but it is not accepted either in the CDS or ADS.
This is the error I obtain using the ECMWF SSO account on the CDS:
Unrecognized username or password. Forgot your password?

BR,
M

1 Like
3

Hi M,
You will not be able to use your ECMWF account on the current CDS or ADS.
You will need your ECMWF account when logging on to the new CDS or new ADS.
Hang in there, the new CDS is very soon coming now :slight_smile:

2 Likes
4

The … is helpful but confusing. It describes old/new CDS and points to the newest cdsapi pypi page and the dataset download form. I have checked the way how to configure ~/.cdsapirc and how to request/retrieve data.

GitHub - ecmwf/cdsapi: Python API to access the Copernicus Climate Data Store (CDS) says that the ~/.cdsapirc files should look like:

url: https://cds.climate.copernicus.eu/api/v2
key: <UID>:<API key>

Where UD and APIkey should come from Log in | Copernicus Climate Data Store according to the readme file.

The download form suggests this way to request data (an example):

import cdsapi
c = cdsapi.Client()
c.retrieve(
    'reanalysis-era5-single-levels',
    {
        'product_type': 'reanalysis',
        'format': 'netcdf',
        'variable': '10m_u_component_of_wind',
        'year': '2018',
        'month': '01',
        'day': '01',
        'time': ['00:00', '06:00'],
    },
    'download.nc')

This all seems not to have changed with respect to say six months ago. Is this really the way to do it or will the respective pages still be updated? I am reading about an ECMWF account but this is not yet mentioned in the readme of cdsapi or anywhere else. I find this quite confusing. Furthermore, I would like to request deprecationwarnings if users retrieve data via a method that will become deprecated, if that was not already implemented.

1 Like
5

Dear Annabelle,
It is written in the announcement that "New tool packages will be made available (post new CDS launch) to provide software tools for weather and climate workflows that simplify data access, analysis, visualisation and much more. "

Do you know when these tools will be made available? Are they already available?

It would be great to know that in advance, because we are already in mid-July and you say there that " * The current CDS and ADS will be completely shutdown in September 2024. Should users still have API scripts pointing to the retired CDS and ADS systems, these will systematically break and no support will be provided."

How can we migrate the workflows from the cdstoolbox before having the tools to do so?

3 Likes
7

CDS-Beta is now live!
Please see our announcement: The new Climate Data Store Beta (CDS-Beta) is now live!

9

Dear Annabelle, I’ll repeat my question because I still cannot understand from your announcement:

It is written in the announcement that "New tool packages will be made available (post new CDS launch) to provide software tools for weather and climate workflows that simplify data access, analysis, visualisation and much more. "

Do you know when these tools will be made available? Are they already available?

How can we migrate the workflows from the cdstoolbox before having the tools to do so?

We are in late July, and many people take holidays in August. How can we anticipate the migration to the new infrastructure without knowing the tools that will be replaced and the workflows that we will have to rework entirely?

Thanks for your support :pray:

10

Hi Anabelle,

The new CDS is currently in BETA, i.e. cds-beta.climate.copernicus.eu

Come September, when it’s due to be live proper, will you be dropping the “Beta” part of the URL?

Permanently using something named ‘Beta’ doesn’t sit well with some.

If you do drop the ‘Beta’ and change to V3 or somesuch presumably the .cdsapirc contents will change and perhaps user keys.

Do you know the plans for this and any impact for us if/when the URL changes?

Thanks

Mark

11

The Beta will indeed be dropped .
You will not have to update the URL in your .cdsapirc files in the immediate future.
There will be no change to your API keys.

In other words, there should be no impact to users when we switch from CDS-Beta to CDS in September as both CDS-Beta and CDS urls will continue to co-exist for longer.

The Data Stores Team has made every effort to facilitate transition by respecting as much as possible backward compatibility.

We will of course post an announcement when CDS-beta becomes the new CDS. So keep watching the Forum :slight_smile:

12

Thanks for that, Anabelle

13

If there any update on when the ADS Beta will be live - in particular the
CAMS global atmospheric composition forecasts?

Thanks,

  • Colin
14

Dear @Anabelle ,

I currently use a CDS workflow that converts hourly data into daily averages. In the announcement of the new CDs, it is written that:

The CDS Toolbox will be discontinued and is not migrated to the new CDS.
…
New tool packages will be made available to provide software tools for weather and climate workflows that simplify data access, analysis, visualisation and much more.

Can you confirm whether there will be a replacement for this type of workflow, or will they be permanently discontinued?

There is only one month until the current CDS infrastructure is discontinued, and this month is in the middle of Summer vacations in the Northern Hemisphere. It would be greatly appreciated to know if I should wait for the release of new tools or commit to rewriting my application.

Thanks for your understanding,

15

The CDS Toolbox will not exist in CDS-Beta hence you will not be able to run your Toolbox scripts.
Your Toolbox scripts will not run outside of the current CDS platforms.

There will however be a new feature in CDS-Beta to generate ERA5 Daily means. We will post an announcement when we will have further details to share.

Hope this helps!

1 Like
16

@Anabelle - Is there any update on when the ADS Beta will be live - in particular the
CAMS global atmospheric composition forecasts?

17

Very dissapointing that there will be no replacement for the CDS toolbox, I’ve found it massively useful!

18

How do you get the status of a request using Python? Old documentation github doesnt seem to work anymore…?

Also, how do you get get the request ID, so the request can be downloaded at a later time?

Thanks.

19

Hi Fergus,

See cads-api-client/notebooks/cads_api_client_test.ipynb at main · ecmwf-projects/cads-api-client · GitHub, in particular Section 2 on getting the status.

Took some searching to find the new api to utilize i.e. migrating from the old cdsapi → cds_api_client.

Hope this helps :slight_smile:

Richard

20

Thanks Richard, took a bit of figuring out to work with the new api, but got worked out how to submit and not download.

Do you know how to download the data in a separate script using the requests request_uid? i can get all the required information using get_request(request_uid), but cant figure out how to download the request from there… any ideas?

edit: i figured it out, for anyone else here its:

client.download_result(request_uid, target, retry_options={})