Separation of retrieve and download

Software and Tools API
1

Is there a way in the new CDS-API to separate the retrieve and the download similar to what is described here by fridgerator on Sep 2, 2020?

2

If by this you mean scheduling the download to pick up data on a later point, check here under Basic API documentation:

1 Like
3

This certainly works; however, I am retrieving and downloading with independent scripts. In order to not tie up the job queue with a job simply waiting for a request to process and complete, I am sending the requests first. I then log the requestID and run a new script to download the completed requests. Here is a snippet showing what I am doing now (aside from error checking, etc

session = requests.Session()
url = f"https://cds.climate.copernicus.eu/api/retrieve/v1/jobs/{request_id}"
headers={'User-Agent': 'datapi/0.1.1', 'PRIVATE-TOKEN': api_key}
session.headers.update(headers)
s = session.get(url)
# some checking here to make sure request exists and  is complete
# Query results to get URL for download
r = session.get(url + "/results")
# more checking
result = session.get(asset['href'])

If there is a better way to do this, let me know.

Thanks,
Eric

4

That’s what I meant, or see as the only option. There does not seem to be an option to outright list all request statuses, as there was before since the request id is included. If you do find it, let me know.

5

Got it - I read the first part of the post but missed the last part about the custom API implementations. Appreciate your help.

6

If you have the request ID, you can do the download later using this

    new_client = cdsapi.Client()
    result = new_client.client.get_remote(request_id)
    out = result.download(save_file)

Found in a reply to this post:

7

You can query https://cds.climate.copernicus.eu/api/retrieve/v1/jobs to get information on all your jobs, including their ID and status. Below code works for me.

with requests.Session() as session:

    session.headers = {
        "PRIVATE-TOKEN": YOUR_PRIVATE_KEY
        }

    r = session.get('https://cds.climate.copernicus.eu/api/retrieve/v1/jobs')
    
result = r.json()
for job in result.get("jobs"):
    print(job["jobID"], job["status"])

See here for more information: ECMWF APIs (FAQ, API + data documentation)

1 Like