Is the new API going to be documented?

Peter_Akers · 28 2024 08:33

Hi, is there going to be any documentation for the new API or at least a Swagger page before access to the old one is switched off? I cant use the cdsapi package, and I imagine other users might be in the same situation. Any help much appreciated.

Thanks,

Peter Akers

Koen_Hufkens · 28 2024 15:23

I’m the develop or the R based {ecmwfr} package doing native API interfacing in R. I’m facing the same issues. While documentation was sparse in the original CDS, things seem to be even more opaque now. Limiting uptake in other systems, ecosystems of users.

@Peter_Akers check code here for endpoints (see services-ds.R in the R folder for the R6 class used: https://github.com/bluegreen-labs/ecmwfr)

Most pressing I’m still missing endpoints for:

account information (to validate SOS PAT)
download queues (for download management in intermittent download queues)

They are mentioned in the cadsapi code base under the profile section, but are non functional AFAIK.

Koen

Matteo_De_Felice12 · 29 2024 08:55

Also the documentation for the cdsapi module would be useful (see this discussion on the official github repo: Where is the full API documentation? · Issue #9 · ecmwf/cdsapi · GitHub )

Peter_Akers · 29 2024 09:19

Thanks Koen, yes authentication is the biggest issue for me, in particular what appears to be a change from Basic to Bearer Token. At the moment I’m trying to back-engineer the process using the ECMWF download UI, but it is not straightforward without documentation or at least an endpoint.

Koen_Hufkens · 29 2024 09:43

It is NOT Bearer Token it is a custom token in your header! Banged my head on that one for a while.

#  get the response for the query provided
response <- httr::GET(
  url,
  httr::add_headers(
    "PRIVATE-TOKEN" = key
  ),
  encode = "json"
)

Koen_Hufkens · 29 2024 09:46

Endpoints are buried in the cadsapi python package, which is sideloaded by cdsapi. They are only formally listed in the test routines, AFAIK.

github.com

ecmwf-projects/cads-api-client/blob/main/tests/test_10_processing.py

import json
import logging

import pytest
import responses
from responses.matchers import json_params_matcher

import cads_api_client

COLLECTION_ID = "reanalysis-era5-pressure-levels"
JOB_RUNNING_ID = "9bfc1362-2832-48e1-a235-359267420bb1"
JOB_SUCCESSFUL_ID = "9bfc1362-2832-48e1-a235-359267420bb2"
JOB_FAILED_ID = "9bfc1362-2832-48e1-a235-359267420bb3"

CATALOGUE_URL = "http://localhost:8080/api/catalogue"
COLLECTIONS_URL = "http://localhost:8080/api/catalogue/v1/datasets"
COLLECTION_URL = (
    "http://localhost:8080/api/catalogue/v1/collections/reanalysis-era5-pressure-levels"
)
PROCESS_URL = f"http://localhost:8080/api/retrieve/v1/processes/{COLLECTION_ID}"

This file has been truncated. show original

Koen_Hufkens · 29 2024 09:51

Additional references to account specific endpoints are made here:

Note, these seem not to function, or I haven’t been able to get them to work (due to an different base url?)

github.com

ecmwf-projects/cads-api-client/blob/main/cads_api_client/profile.py

from __future__ import annotations

from typing import Any

from . import processing


class Profile:
    supported_api_version = "v1"

    def __init__(self, url: str, headers: dict[str, Any] = {}) -> None:
        self.url = f"{url}/{self.supported_api_version}"
        self.headers = headers

    def profile(self) -> dict[str, Any]:
        url = f"{self.url}/account"
        response = processing.ApiResponse.from_request("get", url, headers=self.headers)
        return response.json

    def accept_licence(self, licence_id: str, revision: int) -> dict[str, Any]:

This file has been truncated. show original

Thomas_CHAMPSEIX · 29 2024 12:06

Hello !
With the help of @Koen_Hufkens and by activating the debug when using the python package cdsapi I was able to find the exact URL to call but I got 500 internal server error response. Here is all I found if it can help someone :

Python script with debug activated

import cdsapi

dataset = "cams-europe-air-quality-forecasts-optimised-at-observation-sites"
request = {
    'variable': ['particulate_matter_10um', 'particulate_matter_2.5um'],
    'country': ['france'],
    'type': ['mos_optimised'],
    'leadtime_hour': ['0-23'],
    'year': ['2024'],
    'month': ['01'],
    'day': ['18']
}

client = cdsapi.Client(debug=True)
client.retrieve(dataset, request).download()

Result in logs to see the URL working

2024-08-29 14:01:20,750 DEBUG CDSAPI {'url': 'http://ads-beta.atmosphere.copernicus.eu/api', 'key': '<YOUR-TOKEN>, 'quiet': False, 'timeout': 60, 'sleep_max': 120, 'retry_max': 500}
2024-08-29 14:01:20,750 DEBUG POST http://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/processes/cams-europe-air-quality-forecasts-optimised-at-observation-sites/execute {'variable': ['particulate_matter_10um', 'particulate_matter_2.5um'], 'country': ['france'], 'type': ['mos_optimised'], 'leadtime_hour': ['0-23'], 'year': ['2024'], 'month': ['01'], 'day': ['18']}
2024-08-29 14:01:21,275 DEBUG REPLY {"processID":"cams-europe-air-quality-forecasts-optimised-at-observation-sites","type":"process","jobID":"e3633244-04b3-40c7-9fb0-5843912877ce","status":"accepted","created":"2024-08-29T12:01:21.258084","updated":"2024-08-29T12:01:21.258084","links":[{"href":"https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/processes/cams-europe-air-quality-forecasts-optimised-at-observation-sites/execute","rel":"self"},{"href":"https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce","rel":"monitor","type":"application/json","title":"job status info"}],"metadata":{"datasetMetadata":{"messages":[]}}}
2024-08-29 14:01:21,275 INFO Request ID is e3633244-04b3-40c7-9fb0-5843912877ce
2024-08-29 14:01:21,275 DEBUG GET https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce
2024-08-29 14:01:21,341 DEBUG REPLY {"processID":"cams-europe-air-quality-forecasts-optimised-at-observation-sites","type":"process","jobID":"e3633244-04b3-40c7-9fb0-5843912877ce","status":"accepted","created":"2024-08-29T12:01:21.258084","updated":"2024-08-29T12:01:21.258084","links":[{"href":"https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce","rel":"self","type":"application/json"}],"metadata":{"log":[]}}
2024-08-29 14:01:21,342 INFO status has been updated to accepted
2024-08-29 14:01:21,342 DEBUG result not ready, waiting for 1.5 seconds
2024-08-29 14:01:22,843 DEBUG GET https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce
2024-08-29 14:01:22,913 DEBUG REPLY {"processID":"cams-europe-air-quality-forecasts-optimised-at-observation-sites","type":"process","jobID":"e3633244-04b3-40c7-9fb0-5843912877ce","status":"successful","created":"2024-08-29T12:01:21.258084","started":"2024-08-29T12:01:21.873506","finished":"2024-08-29T12:01:22.658882","updated":"2024-08-29T12:01:22.658882","links":[{"href":"https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce","rel":"self","type":"application/json"},{"href":"https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce/results","rel":"results"}],"metadata":{"log":[]}}
2024-08-29 14:01:22,913 INFO status has been updated to successful
2024-08-29 14:01:22,913 DEBUG GET https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce
2024-08-29 14:01:22,999 DEBUG REPLY {"processID":"cams-europe-air-quality-forecasts-optimised-at-observation-sites","type":"process","jobID":"e3633244-04b3-40c7-9fb0-5843912877ce","status":"successful","created":"2024-08-29T12:01:21.258084","started":"2024-08-29T12:01:21.873506","finished":"2024-08-29T12:01:22.658882","updated":"2024-08-29T12:01:22.658882","links":[{"href":"https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce","rel":"self","type":"application/json"},{"href":"https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce/results","rel":"results"}],"metadata":{"log":[]}}
2024-08-29 14:01:23,000 DEBUG GET https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce
2024-08-29 14:01:23,087 DEBUG REPLY {"processID":"cams-europe-air-quality-forecasts-optimised-at-observation-sites","type":"process","jobID":"e3633244-04b3-40c7-9fb0-5843912877ce","status":"successful","created":"2024-08-29T12:01:21.258084","started":"2024-08-29T12:01:21.873506","finished":"2024-08-29T12:01:22.658882","updated":"2024-08-29T12:01:22.658882","links":[{"href":"https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce","rel":"self","type":"application/json"},{"href":"https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce/results","rel":"results"}]}
2024-08-29 14:01:23,088 DEBUG GET https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/jobs/e3633244-04b3-40c7-9fb0-5843912877ce/results {}
2024-08-29 14:01:23,173 DEBUG REPLY {"asset":{"value":{"type":"application/zip","href":"https://object-store.os-api.cci2.ecmwf.int:443/cci2-prod-cache/aef65f98a90d0e8a737fe9838fd90009.zip","file:checksum":"83fe39b361e52f656cf7b9fa0c572692","file:size":27939,"file:local_path":"s3://cci2-prod-cache/aef65f98a90d0e8a737fe9838fd90009.zip"}}}

Curl command tried to validated URL and header

curl --request POST \
  --url https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/processes/cams-europe-air-quality-forecasts-optimised-at-observation-sites/execute \
  --header 'PRIVATE-TOKEN: <YOUR-TOKEN>' \
  --header 'content-type: application/json' \
  --data '{
    "variable": ["particulate_matter_10um", "particulate_matter_2.5um"],
    "country": ["france"],
    "type": ["mos_optimised"],
    "leadtime_hour": ["0-23"],
    "year": ["2024"],
    "month": ["08"],
    "day": ["04"]
}'

But response is 500 code and this body :

{
  "type": "internal server error",
  "title": "internal server error",
  "trace_id": "e20b4d4f-d766-4997-abaa-126553168236"
}

I suppose some header is missing but if we could have a documentation it would solve all this easily.

Koen_Hufkens · 29 2024 12:37

check your url / data encoding, which should be json. Also the forwarded data/body should be wrapped in an additional {“inputs”: {JSON_REQUEST}}. The debugger output doesn’t show this.

github.com

ecmwf-projects/cads-api-client/blob/ca975c86ec586125c5cd674ba33351b327603584/cads_api_client/processing.py#L78


      
              *args: Any,
              raise_for_status: bool = True,
              session: requests.Session | None = None,
              retry_options: dict[str, Any] = {"maximum_tries": 2, "retry_after": 10},
              **kwargs: Any,
          ) -> T_ApiResponse:
              if session is None:
                  session = requests.Session()
              method = kwargs["method"] if "method" in kwargs else args[0]
              url = kwargs["url"] if "url" in kwargs else args[1]
              inputs = kwargs.get("json", {}).get("inputs", {})
              logger.debug(f"{method.upper()} {url} {inputs}")
              response = multiurl.robust(session.request, **retry_options)(*args, **kwargs)
              logger.debug(f"REPLY {response.text}")
          
              if raise_for_status:
                  cads_raise_for_status(response)
              self = cls(response, headers=kwargs.get("headers", {}), session=session)
              self.log_messages()
              return self

Thomas_CHAMPSEIX · 29 2024 14:31

Thanks for the help (and warning) that was it now it works !

curl --request POST \
  --url https://ads-beta.atmosphere.copernicus.eu/api/retrieve/v1/processes/cams-europe-air-quality-forecasts-optimised-at-observation-sites/execute \
  --header 'PRIVATE-TOKEN: <YOUR-TOKEN>' \
  --header 'content-type: application/json' \
  --data '{
  "inputs": {
      "variable": ["particulate_matter_10um", "particulate_matter_2.5um"],
      "country": ["france"],
      "type": ["mos_optimised"],
      "leadtime_hour": ["0-23"],
      "year": ["2024"],
      "month": ["08"],
      "day": ["04"]
  }
}'

Peter_Akers · 29 2024 14:52

Thanks all, this is brilliant. I didn’t realise I could just use my personal access token in the headers, unlikely I would have realised that on my own