Hi,
Did something recently change in the CAMS NRT file structure? I no longer see the Total_column_hydrogen_peroxide_surface band that used to be present.
Thanks,
Simon
Hi Simon. Do you mean in the CAMS global forecast (CAMS global atmospheric composition forecasts)?
Total column hydrogen peroxide still looks present to me. If you’re not getting it then can you share your exact request?
Thanks,
Luke.
Thanks, Luke! I’m trying to replicate this using Climate Data Store, but even though I ran "pip install “cdsapi>=0.7.2” , I get
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://cds.climate.copernicus.eu/api/retrieve/v1/processes/cams-global-atmospheric-composition-forecasts/execution
process not found
dataset cams-global-atmospheric-composition-forecasts not found
The script is:
import cdsapi
dataset = “cams-global-atmospheric-composition-forecasts”
request = {
“pressure_level”: [“1000”],
“model_level”: [“1”],
“date”: [“2024-12-03/2024-12-03”],
“time”: [“00:00”],
“leadtime_hour”: [“0”],
“type”: [“forecast”],
“data_format”: “grib”,
“variable”: [“total_column_hydrogen_peroxide”]
}
client = cdsapi.Client()
client.retrieve(dataset, request).download()
However, we do have a working download client setup. I think the actual problem is that the output grib format changed somewhat, and now our old conversion routines didn’t work. I can try to adjust them, but I 'd like to understand what happened.
Eg, if I run gdalinfo on the new grib files, I get a lot of errors like this, which wasn’t the case before:
Warning 1: GRIB: Don’t understand the parameter table, since center 98-0 used
parameter table version 218 instead of 3 (international exchange).
Using default for now (which might lead to erroneous interpretation), but please email arthur.taylor@noaa.gov
about adding this table to his ‘degrib1.c’ and ‘grib1tab.c’ files.
Hi Simon,
I think your 404-not found error is probably because you’re using the wrong credentials in your .cdsapirc file. Note that the recent migration to the new system required you to update this file, and the same file can’t be used for both ADS and CDS (yet) because the datasets are different. To see what should be in there, go to Atmosphere Data Store and ensure you’re logged in. You’ll see the content in the section “Set up the CDS API personal access token”.
I don’t think the GRIB format has changed in any way, but I’m afraid I don’t know anything about gdlalinfo so I can’t really help you there. At ECMWF we use ecCodes (and packages built on top of it) for reading GRIB files.
Luke.
Hi Luke,
I do see differences with ecCodes. For a file retrieved earlier, when I read it with the script below, I get lines like
Centre: ecmf (European Centre for Medium-Range Weather Forecasts)
Parameter ID: 172
Parameter Name: None
(and many other parameter ids)
The new file just says many times:
Centre: ecmf (European Centre for Medium-Range Weather Forecasts)
Parameter ID: 218003
Parameter Name: None
Possibly more importantly, for the old files I get:
paramId: 165
parameterName: 10 metre U wind component
parameterUnits: m s**-1
parameterCategory: Not available
parameterNumber: Not available
shortName: 10u
name: 10 metre U wind component
while for the new file it’s
paramId: 218003
parameterName: 3
parameterUnits: 3
parameterCategory: Not available
parameterNumber: Not available
shortName: tc_h2o2
name: Total column hydrogen peroxide
(note parameterName and paramerUnits are now numeric)
My desktop eccodes version is 2.35.0, which isn’t that old. (To be honest, the in production we use an older version 2.27.0, but if even 2.35.0 is unhappy, maybe the old version isn’t the problem.)
Script:
import sys
from eccodes import *
def inspect_grib_metadata(filename):
with open(filename, ‘rb’) as f:
while True:
gid = codes_grib_new_from_file(f)
if gid is None:
break
# Get center and parameter table details
centre = codes_get(gid, 'centre')
centre_description = codes_get(gid, 'centreDescription')
#tablesVersion = codes_get(gid, 'tablesVersion')
# Get more detailed parameter information
param_id = codes_get(gid, 'paramId')
param_name = codes_get(gid, 'parameterName', 'unknown')
print(f"Centre: {centre} ({centre_description})")
#print(f"Tables Version: {tablesVersion}")
print(f"Parameter ID: {param_id}")
print(f"Parameter Name: {param_name}")
print("---")
codes_release(gid)
def check_center_details(filename):
with open(filename, ‘rb’) as f:
gid = codes_grib_new_from_file(f)
if gid is None:
return
# Get all centre-related information
keys = [
'centre', 'centreDescription',
'subCentre',
#'tablesVersion',
'localTablesVersion', 'productionStatusOfProcessedData'
]
for key in keys:
try:
value = codes_get(gid, key)
print(f"{key}: {value}")
except CodesInternalError:
print(f"{key}: Not available")
codes_release(gid)
def read_grib(filename):
with open(filename, ‘rb’) as f:
while True:
# Get the next message
gid = codes_grib_new_from_file(f)
if gid is None:
break
# Get some keys
shortName = codes_get(gid, 'shortName')
level = codes_get(gid, 'level')
date = codes_get(gid, 'date')
# Get the data values
values = codes_get_values(gid)
print(f"Parameter: {shortName}")
print(f"Level: {level}")
print(f"Date: {date}")
print(f"Number of values: {len(values)}")
print("---")
# Remember to delete the GRIB message
codes_release(gid)
def check_parameter_details(filename):
with open(filename, ‘rb’) as f:
gid = codes_grib_new_from_file(f)
if gid is None:
return
try:
# Parameter details
keys = [
'paramId', 'parameterName', 'parameterUnits',
'parameterCategory', 'parameterNumber',
'shortName', 'name'
]
for key in keys:
try:
value = codes_get(gid, key)
print(f"{key}: {value}")
except CodesInternalError:
print(f"{key}: Not available")
finally:
codes_release(gid)
Usage
file = sys.argv[1]
#inspect_grib_metadata(file)
#check_center_details(file)
check_parameter_details(file)
Hi Simon,
What this tells me is that your older file contains a mixture of parameters but your newer file contains only total column hydrogen peroxide. Is that not what you expect?
Luke.