Model levels not downloading with ads-beta for EAC4

Dear support desk,

When I am trying to download CAMS EAC4 data (From CAMS global reanalysis (EAC4)) using API request, only one model level =1 is downloaded.

When using the ‘submit form’ option for the same request all the 60 model levels are included in the downloaded file.

Why the 60 model levels are only downloaded when using submit form and not via beta API ?

Additionally, the beta API script adds ‘data_format’: ‘grib’ line in the API script which should instead be ‘format’: ‘grib’

Any help is appreciated.

Thank you!
Prerita

Hi Prerita,

Thanks for the feedback. Could you post the exact request you’re using? Then we can investigate.

On the data_format issue, we’ve renamed that key but we’re supporting the old name for now in order to maintain backwards compatibility.

Luke Jones.

Hi Luke,
Thanks for your reply.

I am trying to download the 3-hourly CAMS aerosol data at all 60 model-levels from 2009 to 2019.
Since there is a limit for requesting the data, I have modified the API script to loop over each month of each year.

The issue is that all model levels are downloaded if I request only a single hour for a single day (00:00, for example), without the loop.
but when I request all the timesteps, only model level 1 is downloaded. I do not understand why this is so.

Only a manual submit form request works for a single month for all times for all model levels. But this is not a feasible method for downloading over 10 years of data.

Please see my API request pasted below. It only downloads hybrid : levels=1. whereas I have requested all 60.

I’d appreciate any suggestions/help!

import cdsapi
import datetime


# Initialize the CDS API client
client = cdsapi.Client()

# Define the dataset
dataset = "cams-global-reanalysis-eac4"

# Define the date range
start_date = datetime.datetime(2009, 1, 1)
end_date = datetime.datetime(2010, 12, 31)

# Loop through each day in the date range
while start_date <= end_date:
    # Format the current date
    current_date_str = start_date.strftime('%Y-%m-%d')
    
    # Update the request dictionary with the current date
    request = {
        'variable': [
            'dust_aerosol_0.03-0.55um_mixing_ratio', 'dust_aerosol_0.55-0.9um_mixing_ratio', 
            'dust_aerosol_0.9-20um_mixing_ratio', 'hydrophilic_black_carbon_aerosol_mixing_ratio', 
            'hydrophilic_organic_matter_aerosol_mixing_ratio', 'hydrophobic_black_carbon_aerosol_mixing_ratio', 
            'hydrophobic_organic_matter_aerosol_mixing_ratio', 'sea_salt_aerosol_0.03-0.5um_mixing_ratio', 
            'sea_salt_aerosol_0.5-5um_mixing_ratio', 'sea_salt_aerosol_5-20um_mixing_ratio', 
            'specific_humidity', 'temperature'
        ],
        'model_level': ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19','20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31', '32', '33', '34', '35', '36', '37', '38', '39','40', '41', '42', '43', '44', '45', '46', '47', '48', '49', '50', '51', '52', '53', '54', '55', '56', '57', '58', '59','60'],
        'date': current_date_str,  # Only request data for the current day
        'time': ['00:00', '03:00', '06:00', '09:00', '12:00', '15:00', '18:00', '21:00'],
        'format': 'grib'
    }

    # Construct the file name based on the current date
    file_name = '{}.grib'.format(current_date_str)
    
    # Perform the data retrieval and download
    print(f"Downloading data for {current_date_str}...")
    client.retrieve(dataset, request).download(file_name)
    
    # Move to the next day
    start_date += datetime.timedelta(days=1)

Hi Prerita,

I’m afraid I can’t reproduce this behaviour, at least not for date=2009-01-01. The request below gives me all 5760 fields as expected. Does this perhaps only happen for certain dates?

{
‘variable’: [
‘dust_aerosol_0.03-0.55um_mixing_ratio’, ‘dust_aerosol_0.55-0.9um_mixing_ratio’,
‘dust_aerosol_0.9-20um_mixing_ratio’, ‘hydrophilic_black_carbon_aerosol_mixing_ratio’,
‘hydrophilic_organic_matter_aerosol_mixing_ratio’, ‘hydrophobic_black_carbon_aerosol_mixing_ratio’,
‘hydrophobic_organic_matter_aerosol_mixing_ratio’, ‘sea_salt_aerosol_0.03-0.5um_mixing_ratio’,
‘sea_salt_aerosol_0.5-5um_mixing_ratio’, ‘sea_salt_aerosol_5-20um_mixing_ratio’,
‘specific_humidity’, ‘temperature’
],
‘model_level’: [‘1’, ‘2’, ‘3’, ‘4’, ‘5’, ‘6’, ‘7’, ‘8’, ‘9’, ‘10’, ‘11’, ‘12’, ‘13’, ‘14’, ‘15’, ‘16’, ‘17’, ‘18’, ‘19’,‘20’, ‘21’, ‘22’, ‘23’, ‘24’, ‘25’, ‘26’, ‘27’, ‘28’, ‘29’, ‘30’, ‘31’, ‘32’, ‘33’, ‘34’, ‘35’, ‘36’, ‘37’, ‘38’, ‘39’,‘40’, ‘41’, ‘42’, ‘43’, ‘44’, ‘45’, ‘46’, ‘47’, ‘48’, ‘49’, ‘50’, ‘51’, ‘52’, ‘53’, ‘54’, ‘55’, ‘56’, ‘57’, ‘58’, ‘59’,‘60’],
‘date’: ‘2009-01-01’,
‘time’: [‘00:00’, ‘03:00’, ‘06:00’, ‘09:00’, ‘12:00’, ‘15:00’, ‘18:00’, ‘21:00’],
‘format’: ‘grib’
}

Hi Luke,

Thanks for the response.
I believe the issue might be with certain dates.

But even if I try to download the data for any two days (2009-01-01-2009-01-02), as in your example, the downloaded grib file seems to be broken and gives the following error -

Warning (gribapiScanTimestep2): Record 25 (name=aermr04 id=4.210.192 lev1=2 lev2=0) timestep 2: Parameter not defined at timestep 1!
cdo    sinfo: Open failed on >adaptor.mars.internal-1723462746.2381923-31002-14-01ab8e9e-f2f8-4d4f-ace9-dc49366df336.grib<
              Unsupported file structure

If you try more than one date for the same request, it doesn’t work.

so my question is how can I download a whole year EAC4 data with all 60 model levels using API request.?

Thanks, Prerita

1 Like

Hi Prerita,

So you’re using cdo for the conversion? I just tried your request with those two dates and the cdo command “cdo -f nc copy input.grib output.nc” and it worked for me. Two things for you to check:

  • How many fields are in your input file? For me it’s 11520. You can find this out with the ecCodes command “grib_count input.grib”.
  • What version of cdo are you using? I’m using 2.4.0. You can find this out with “cdo -V”.

Luke.

It would also be useful if you posted the output of “grib_ls input.grib”, so I can see exactly which fields you have.

Another useful thing would also be your request ID, which is printed out when you make the request.

Hi Luke,

I figured out what the issue was.
The URL and key needed to be updated to ads-beta.
The old ad credentials still work, so there was no obvious error, except for the downloading of the model-level issue I faced. I believe this is still the case with the old ads datastore.

Thanks for your help.
Prerita

No problem, glad you got it working!