The new Climate Data Store Beta (CDS-Beta) is now live!

[Peter_Zellner]

Hi all,
I have a similar issue. I am trying to download ERA5 Land data. When I submit the request via the GUI it starts processing directly. If I send the exact same request via python cdsapi it is queued for an hour or so. I checked the status and it goes into the MARS Archive queue of 190 parallel requests.

“The maximum number of requests that access the CDS-MARS archive is 190”

I think that if the requested dataset is not in the MARS Archive via GUI request, it cannot be in the MARS Archive for python API requests.
Might it be that API requests are treated differently than using the GUI?

[Mark_Elliott1]

Thanks for reply Peter,

I’d agree - both ERA5 and ERA5 Land via API are bad at the moment, I haven’t tried Web GUI.

It’s making things really difficult for us to test - we all have changes that need to be done and complete by September, and having to wait an hour or two per request isn’t helpful.

M

[Mark_Elliott1]

Yep - API requests are taking an hour or more in queue, yet Web UI requests are pretty much instant.

So, why is this - as suggested, different queues?

[xin_chen]

I have the same question.Do you solve it?

[Anil_Teela]

If I try convert nc file downloaded from beta api, it throws below error. Also, if file is downloaded from old api then its gets converted into gz without issue.
command:
NetCDFtoGZ.exe “<filepath_for_filenam.nc>”

Unhandled exception. System.Collections.Generic.KeyNotFoundException: The given key ‘add_offset’ was not present in the dictionary.
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
at Microsoft.Research.Science.Data.MetadataDictionary.get_Item(String key, SchemaVersion version)
at Microsoft.Research.Science.Data.MetadataDictionary.get_Item(String key)
at Microsoft.Research.Science.Data.Imperative.DataSetExtensions.GetAttr(DataSet dataset, Int32 variableId, String attributeName)
at Microsoft.Research.Science.Data.Imperative.DataSetExtensions.GetAttr(DataSet dataset, String variableName, String attributeName)
at NetCDFtoGZ.ERA5Handler.Handle(String args)
at NetCDFtoGZ.Program.Main(String args)

[Nicholas_Robinson]

Thanks Peter, so I’m guessing that is where my files will be when they are ready for download. What path would I need to use to request a file please?

[Peter_Akers]

https://cds-beta.climate.copernicus.eu/api/retrieve/v1/processes/reanalysis-era5-single-levels/execution

Above would be where you make your POST request. Body would be something like the below. You also need to include your private token as PRIVATE-TOKEN in the headers (many many thanks to @Koen_Hufkens for that one)

 {
"inputs": {
    "product_type": [
        "reanalysis"
    ],
    "variable": [
        "2m_temperature"],
    "year": [
        "2010"
    ],
    "month": [
        "04"
    ],
    "day": [
        "20"
    ],
    "time": [
        "00:00"
       
    ],
    "grid": [
        "1.0",
        "1.0"
    ],
    "data_format": "netcdf"
}

}

This is where you can get your job status:

https://cds-beta.climate.copernicus.eu/api/retrieve/v1/jobs/your-job-id/

Once response.status = ‘successful’, you can get your link here:

https://cds-beta.climate.copernicus.eu/api/retrieve/v1/jobs/your-job-id/results

[Peter_Richardson]

@Anabelle

• Please migrate to using CDS-Beta by September 16th, 2024 when current CDS will be shutdown.

Was this recently changed/edited from September 3 to 16? It would be nice to have a separate post announcing this change instead of an edit that might not be noticed.

[Jennifer_Newman]

Hello,

We’ve started to fetch data from the new API and we have noticed that the data is slightly different than what we obtained from the old API (particularly the solar irradiance variables). For example, here is a screenshot showing values of some of the solar irradiance variables for Lat = 62.75, Lon = 22.25, Time = 2024-08-13 04:00:00 UTC. Downward solar radiation (SSRD) is 122488.625 with the old API and 122496 with the new API. Any idea what would cause that difference?

[Anabelle]

Migration date to CDS-Beta has been extended to 26 September 2024 but this is final!

We posted a separate announcement earlier this morning: Decommissioning of CDS and ADS on September 26, 2024 - Final

[COG_Data]

I have raised the same issue

[Fuhua_Huang]

Hello, I am puzzled as to why there is no option to accept permission on the Complete ERA5 global atmospheric reanalysis download page？

Thanks very much!

[Programmer_Silvanus]

Hey, it seems that I know the root cause. Try this one:

client = cdsapi.Client(
“https://cds-beta.climate.copernicus.eu/api”,
f"{self.api_key}")

change self.api_key with your api key.

[Anna_Wendleder]

Hi CDS-Team,

the download via the python API is quite slow. Peter and Mark already pointed out that problem. Do you already have a solution?

Thanks a lot for your help

Anna

[Anil_Teella]

Hi @Anabelle , We have started sourcing data from Beta platform, However we see the Gridpoints coverage for Condition Parameter 'Sea Surface Temperature ’ aka ( sst ) is lower then current platform. Could you please make it consistent ? We are missing some of the main Grid points used .

Current Platform Grid points covered : 1,038,240
Beta Platform Grid points covered : 686,364

[Peter_Zellner]

@Anna_Wendleder, @Mark_Elliott1 , here’s also a seperate issue that seems to touch the same topic (I’ve referenced our conversation there as well):
ERA5-Land Hourly Data Download - Forever Waiting - Software and Tools / API - Forum (ecmwf.int)

[Igor_de_Amorim_Teixeira]

same issue here, and since im not exaclty good with python, trying to solve this issue in Grads, through a script that has over 2k lines is next to impossible. Why did they change that?

[Igor_de_Amorim_Teixeira]

is this normal? this waiting time?

[Anil_Teella]

Hi Copernicus team,

Could you please share any insights/ updates on same ? Appreciate your timely response as we are approaching current platform shutdown date.

Thanks

[Igor_de_Amorim_Teixeira]

They are not answering or replying our doubts.They will only say:go use the new site. Thing is , my entire script is written using data from the old site and when i try to run with the new site downloads it doesnt work. I thought i was going to finish my TCC this year, but now I doubt im finishing this anytime soon.