Unable to download corresponding data using the new API

user104 · 25 2024 06:36

Dear all,
I recently tried using the new beta version of the API and encountered an issue that I thought might interest you. Even when following the official example provided, I’m faced with an error. Here’s the code I used:

import cdsapi
client = cdsapi.Client()
dataset = 'reanalysis-era5-pressure-levels'
request = {
  'product_type': ['reanalysis'],
  'variable': ['geopotential'],
  'year': ['2024'],
  'month': ['03'],
  'day': ['01'],
  'time': ['13:00'],
  'pressure_level': ['1000'],
  'data_format': 'grib',
}
target = 'download.grib'
client.retrieve(dataset, request, target)

However, I’m getting the following error:

HTTPError: 404 Client Error: Not Found for url: https://cds-beta.climate.copernicus.eu/api/resources/reanalysis-era5-pressure-levels
During handling of the above exception, another exception occurred:

I’m puzzled as to why this is happening. Is it possible that the new version of the API is not ready for data downloads yet? I would appreciate any insights you might have on this issue.
Best regards,
Wang

Laurens_Stoop · 25 2024 08:59

Hi Wang,

No idea what is exactly wrong, but given the same remade request, I got the data. The only difference is the " instead of ’ around the dataset name (& use of Pathlib).

WIth the following code-block I was able to retreive the data:

import cdsapi

# nicer paths
from pathlib import Path
LOCATION_ERA5 = Path("c:/Data/")

# input
dataset = "reanalysis-era5-pressure-levels"
request = {
    'product_type': ['reanalysis'],
    'variable': ['geopotential'],
    'year': ['2024'],
    'month': ['03'],
    'day': ['01'],
    'time': ['13:00'],
    'pressure_level': ['1000'],
    'data_format': 'grib'
}

# Request call
client = cdsapi.Client()
target = LOCATION_ERA5 / 'download.grib'
client.retrieve(dataset, request, target)

Make sure to set your .cdsapirc file as shown on: How to API

user104 · 25 2024 09:55

Hi Laurens_Stoop,
Thank you for your prompt response. I have reviewed and followed the API usage tutorial for the Windows system as per the instructions provided on the website. However, when I attempted to utilize the tutorial for Linux, I encountered an error stating that I was missing a “uid”. Additionally, it appears that the Windows tutorial may be outdated.
To address this issue, I combined the updated URL with the information from the Windows tutorial and configured the .cdsapirc file as follows:

url: https://cds-beta.climate.copernicus.eu/api
key: User ID0:Personal Access Token

Unfortunately, this configuration did not resolve the problem. I would greatly appreciate any further guidance or assistance you could provide. Please let me know if there are any additional steps I need to take or if there are alternative solutions available.
Thank you once again for your help. I look forward to your response.

Best,
Wang

Michela · 25 2024 10:12

Hi,
did you accept the license as mentioned in this post?

Thanks

Laurens_Stoop · 25 2024 10:20

Make sure to login to the CDS-beta system. Both during the making of the request (to accept the license of the dataset) and during the visit to the guide page!
If you are not logged in, then your personal key won’t show on the ‘How-to’ page.

user104 · 25 2024 10:36

Dear Michela,
Thank you for your reply. I have obtained the necessary permissions (four in total). The web download is working, but the API is not.
Best regards,
Wang

user104 · 25 2024 10:39

Hi Laurens_Stoop,
Thanks for your response. I’ve logged in and registered on the new site, and used the new key as previously mentioned, but unfortunately, it still failed. I can’t seem to pinpoint the issue.QWQ

Laurens_Stoop · 25 2024 12:12

No worries, have you explicitely also accepted the license for the pressure level data of ERA5?
While I accepted the license for ERA5 single level variables, I needed to also accept the presure level data license. It seems that each dataset has its own license (for reasons).

For the solution maybe see also your other topic post on this : https://forum.ecmwf.int/t/new-website-usage-mistakes/3447/4

user104 · 25 2024 12:25

Yes, I’ve already accepted it. I suspect there might be an issue with the content of the .cdsapirc file. Here’s how I’ve set it up:

  url: https://cds-beta.climate.copernicus.eu/api
  key: UID (formatted as ********-****-****-****-************):Personal Access Token (I've tried both uppercase and lowercase, but both resulted in failure)

Are you also using a Windows system? Could you please help me check if this format is correct?
Thank you very much~

user104 · 25 2024 12:26

This is my download website page, it looks the same as yours.

Laurens_Stoop · 25 2024 12:30

Hi QWQ,

As I mentioned in the other post;

Maybe set your file as follows:

url: https://cds-beta.climate.copernicus.eu/api
key: [key here]

N.B.: The inclusion of your UID within the .cdsapirc file is gone for the CDS-beta system. In the past you indeed needed to list both user id & key, now just the key.

The old one looked like (i.e. DO NOT USE THIS IN CDS-BETA):
afbeelding
But the new one looks like (USE THIS):

The other topic post on this.

user104 · 25 2024 12:44

Hi Laurens,
Thank you for your suggestion. In fact, I tried this writing method, but Python gave me an error saying that the UID is missing.

user104 · 25 2024 12:47

Like this.

Laurens_Stoop · 25 2024 12:50

Are you using the cdsapi version 0.7.0?

Make sure to use the latest version of the cdsapi!

The line in the api.py file is incorrect for the code of version 0.7.0, see line 337 & 354 on:

github.com

ecmwf/cdsapi/blob/master/cdsapi/api.py

# (C) Copyright 2018 ECMWF.
#
# This software is licensed under the terms of the Apache Licence Version 2.0
# which can be obtained at http://www.apache.org/licenses/LICENSE-2.0.
# In applying this licence, ECMWF does not waive the privileges and immunities
# granted to it by virtue of its status as an intergovernmental organisation nor
# does it submit to any jurisdiction.

from __future__ import absolute_import, division, print_function, unicode_literals

import json
import logging
import os
import time
import uuid

import pkg_resources
import requests

try:

This file has been truncated. show original

user104 · 25 2024 13:05

Thank you for your suggestion. I updated this library at the beginning, and it also shows 0.7.0. It seems there’s no issue? I’ve also tried uninstalling and reinstalling this library, and it’s still the same version.
Wang

Laurens_Stoop · 25 2024 13:13

No real ideas anymore, sorry .

Last wild-card idea: Are you sure the .cdsapirc file has no extension (like a hidden file extension in windows)? mine was initially named .cdsapirc.txt & got me weird issues on windows.

Fix: open Windows explorer → go to the view tab → check the field of ‘file name extensions’ & then check if the .cdsapirc file located at C:\Users\[your_username]\ is indeed reading .cdsapirc.

user104 · 25 2024 13:26

Thank you so much for your help. But it seems that the format is also correct (cry.jpg).

Michela · 25 2024 13:28

Hi,
did you try to specify the url and key directly into the request as:

client = cdsapi.Client(url='https://cds-beta.climate.copernicus.eu/api', key='<YOUR-KEY>')

Please take the key from this link after you log in: Climate Data Store
Please also make sure that the required python packages are updated to the latest version.
Thanks

user104 · 25 2024 13:56

Thank you for your suggestion. I tried it, but it still failed. The Python error message is as shown in the figure.

Shubham_B · 29 2024 08:58

Hi, have you been able to resolve this? I’m facing this exact issue as well.