Is there elsewhere to put the .cdsapirc file? Running it on Databricks

Software and Tools API
,
1

Hi everyone, I am trying to extract ERA5 data on Databricks. It turns out that the home path (home/) is not available in Databricks because I don't have access right to it. So, is there any way that I can manage to use my cdsapi and extract data in Databricks?

2

For who finds this, a sollution I found for databricks:

import os

# Set the API credentials as environment variables
os.environ['CDSAPI_URL'] = 'https://cds.climate.copernicus.eu/api/v2'
os.environ['CDSAPI_KEY'] = 'your_api_key_here'

Make sure to use the correct API key you find on How to use the CDS API | Copernicus Climate Data Store

1 Like
3

FYI, this is the setup we use on our Azure Databricks:

url = 'url: https://cds.climate.copernicus.eu/api/v2'
key = 'key: your_api_key_here'

with open('/root/.cdsapirc', 'w') as f:
    f.write('\n'.join([url, key]))
1 Like
4

Or even better by using the ‘dbutils.secrets’ functionality of Databrics like (as instructed by a dev within my team):

 # Set the API credentials as environment variables (NEATER)
os.environ['CDSAPI_URL'] = 'https://cds.climate.copernicus.eu/api/v2'
os.environ['CDSAPI_KEY'] = dbutils.secrets.get(scope = "keyvault_secretscope_name", key = "Copernicus-api-key")

@HANPU_YAO & @Matteo_De_Felice12 the downside of earlier methods is the ‘publication’ of your key.

1 Like
5

+1 , and should be implemented in the python library by default IMO.

1 Like