Hi !
While downloading ERA5 data on pressure levels ( T u v) I noticed that
the data downloaded from the new site has new time format.
The reference time is 1/1/1970 while previously it was 1/1/1900.
ncview can not decipher it.
This is the error message from ncview:
dimension valid_time
ncview: netcdf_dim_value: unknown data type (10) for
dimension valid_time
In order to merge new and old data files, how do you solve the format problem ?
Many thanks for your help !
you can use CDO to reset the reference time, creating a new output file:
% cdo -setreftime,1900-01-01,00:00:00,1hour infile.nc outfile.nc
weird
cdo created new file with the following features:
ncdump -t -v valid_time dec-1950-fix.nc
netcdf dec-1950-fix {
dimensions:
valid_time = UNLIMITED ; // (90 currently)
longitude = 81 ;
latitude = 81 ;
pressure_level = 1 ;
variables:
double valid_time(valid_time) ;
valid_time:standard_name = “time” ;
valid_time:long_name = “time” ;
valid_time:units = “hours since 1970-1-1 00:00:00” ;
valid_time:calendar = “proleptic_gregorian” ;
valid_time:axis = “T” ;
double longitude(longitude) ;
longitude:standard_name = “longitude” ;
longitude:long_name = “longitude” ;
longitude:units = “degrees_east” ;
longitude:axis = “X” ;
double latitude(latitude) ;
latitude:standard_name = “latitude” ;
latitude:long_name = “latitude” ;
latitude:units = “degrees_north” ;
latitude:axis = “Y” ;
double pressure_level(pressure_level) ;
pressure_level:standard_name = “air_pressure” ;
pressure_level:long_name = “pressure” ;
pressure_level:units = “hPa” ;
pressure_level:positive = “down” ;
pressure_level:axis = “Z” ;
pressure_level:stored_direction = “d” ;
float t(valid_time, pressure_level, latitude, longitude) ;
t:standard_name = “air_temperature” ;
t:long_name = “Temperature” ;
t:units = “K” ;
t:_FillValue = NaNf ;
t:missing_value = NaNf ;
t:GRIB_dataType = “a” ;
t:GRIB_typeOfLevel = “i” ;
t:GRIB_stepType = “i” ;
t:GRIB_gridType = “r” ;
t:GRIB_cfName = “a” ;
t:GRIB_cfVarName = “t” ;
t:GRIB_gridDefinitionDescription = “L” ;
t:GRIB_iDirectionIncrementInDegrees = 0.25 ;
t:GRIB_jDirectionIncrementInDegrees = 0.25 ;
t:GRIB_latitudeOfFirstGridPointInDegrees = 40. ;
t:GRIB_latitudeOfLastGridPointInDegrees = 20. ;
t:GRIB_longitudeOfFirstGridPointInDegrees = 20. ;
t:GRIB_longitudeOfLastGridPointInDegrees = 40. ;
t:GRIB_missingValue = 3.40282346638529e+38 ;
t:GRIB_name = “T” ;
t:GRIB_shortName = “t” ;
t:GRIB_units = “K” ;
float u(valid_time, pressure_level, latitude, longitude) ;
u:standard_name = “eastward_wind” ;
u:long_name = “U component of wind” ;
u:units = “m s**-1” ;
u:_FillValue = NaNf ;
u:missing_value = NaNf ;
u:GRIB_dataType = “a” ;
u:GRIB_typeOfLevel = “i” ;
u:GRIB_stepType = “i” ;
u:GRIB_gridType = “r” ;
u:GRIB_cfName = “e” ;
u:GRIB_cfVarName = “u” ;
u:GRIB_gridDefinitionDescription = “L” ;
u:GRIB_iDirectionIncrementInDegrees = 0.25 ;
u:GRIB_jDirectionIncrementInDegrees = 0.25 ;
u:GRIB_latitudeOfFirstGridPointInDegrees = 40. ;
u:GRIB_latitudeOfLastGridPointInDegrees = 20. ;
u:GRIB_longitudeOfFirstGridPointInDegrees = 20. ;
u:GRIB_longitudeOfLastGridPointInDegrees = 40. ;
u:GRIB_missingValue = 3.40282346638529e+38 ;
u:GRIB_name = “U” ;
u:GRIB_shortName = “u” ;
u:GRIB_units = “m” ;
float v(valid_time, pressure_level, latitude, longitude) ;
v:standard_name = “northward_wind” ;
v:long_name = “V component of wind” ;
v:units = “m s**-1” ;
v:_FillValue = NaNf ;
v:missing_value = NaNf ;
v:GRIB_dataType = “a” ;
v:GRIB_typeOfLevel = “i” ;
v:GRIB_stepType = “i” ;
v:GRIB_gridType = “r” ;
v:GRIB_cfName = “n” ;
v:GRIB_cfVarName = “v” ;
v:GRIB_gridDefinitionDescription = “L” ;
v:GRIB_iDirectionIncrementInDegrees = 0.25 ;
v:GRIB_jDirectionIncrementInDegrees = 0.25 ;
v:GRIB_latitudeOfFirstGridPointInDegrees = 40. ;
v:GRIB_latitudeOfLastGridPointInDegrees = 20. ;
v:GRIB_longitudeOfFirstGridPointInDegrees = 20. ;
v:GRIB_longitudeOfLastGridPointInDegrees = 40. ;
v:GRIB_missingValue = 3.40282346638529e+38 ;
v:GRIB_name = “V” ;
v:GRIB_shortName = “v” ;
v:GRIB_units = “m” ;
// global attributes:
:CDI = “Climate Data Interface version ?? (http://mpimet.mpg.de/cdi)” ;
:history = “Wed Aug 07 00:22:50 2024: cdo -setreftime,1900-01-01,00:00:00,1hour dec-1950.nc dec-1950-fix.nc\nTue Aug 06 19:27:52 2024: cdo selyear,1950 dec1950-1959.nc dec-1950.nc\nP” ;
:institution = “European Centre for Medium-Range Weather Forecasts” ;
:Conventions = “CF-1.7” ;
:GRIB_centre = “ecmf” ;
:GRIB_centreDescription = “European Centre for Medium-Range Weather Forecasts” ;
:CDO = “Climate Data Operators version 1.9.3 (Overview - CDO - Project Management Service)” ;
data:
valid_time = “2020-01-01 12”, “2020-01-02 12”, “2020-01-03 12”,
“2020-01-04 12”, “2020-01-05 12”, “2020-01-06 12”, “2020-01-07 12”,
“2020-01-08 12”, “2020-01-09 12”, “2020-01-10 12”, “2020-01-11 12”,
“2020-01-12 12”, “2020-01-13 12”, “2020-01-14 12”, “2020-01-15 12”,
“2020-01-16 12”, “2020-01-17 12”, “2020-01-18 12”, “2020-01-19 12”,
“2020-01-20 12”, “2020-01-21 12”, “2020-01-22 12”, “2020-01-23 12”,
“2020-01-24 12”, “2020-01-25 12”, “2020-01-26 12”, “2020-01-27 12”,
“2020-01-28 12”, “2020-01-29 12”, “2020-01-30 12”, “2020-01-31 12”,
“2020-02-01 12”, “2020-02-02 12”, “2020-02-03 12”, “2020-02-04 12”,
“2020-02-05 12”, “2020-02-06 12”, “2020-02-07 12”, “2020-02-08 12”,
“2020-02-09 12”, “2020-02-10 12”, “2020-02-11 12”, “2020-02-12 12”,
“2020-02-13 12”, “2020-02-14 12”, “2020-02-15 12”, “2020-02-16 12”,
“2020-02-17 12”, “2020-02-18 12”, “2020-02-19 12”, “2020-02-20 12”,
“2020-02-21 12”, “2020-02-22 12”, “2020-02-23 12”, “2020-02-24 12”,
“2020-02-25 12”, “2020-02-26 12”, “2020-02-27 12”, “2020-02-28 12”,
“2020-11-30 12”, “2020-12-01 12”, “2020-12-02 12”, “2020-12-03 12”,
“2020-12-04 12”, “2020-12-05 12”, “2020-12-06 12”, “2020-12-07 12”,
“2020-12-08 12”, “2020-12-09 12”, “2020-12-10 12”, “2020-12-11 12”,
“2020-12-12 12”, “2020-12-13 12”, “2020-12-14 12”, “2020-12-15 12”,
“2020-12-16 12”, “2020-12-17 12”, “2020-12-18 12”, “2020-12-19 12”,
“2020-12-20 12”, “2020-12-21 12”, “2020-12-22 12”, “2020-12-23 12”,
“2020-12-24 12”, “2020-12-25 12”, “2020-12-26 12”, “2020-12-27 12”,
“2020-12-28 12”, “2020-12-29 12”, “2020-12-30 12” ;
The new downloaded data was dec 1950 ,
according to the suggested transformation, cdo should create ref time 1/1/1900
and dates 1/12/1950 - 31/12/1950
@Kevin_Marsh Another thing that changed in the output files is the variable name valid_time, instead of time. I know this can be quite easily be changed after downloading, but would it not be possible to rename it back to time for user convenience?
For me the most convenient rename method is using the netCDF4 api (cdo cannot be assumed to be available on all systems where my package runs):
nc = Dataset(file_out,'a')
nc.renameVariable("valid_time","time")
nc.renameDimension("valid_time","time")
However, the renameDimension for some reason replaces all time values with zeros, so this is not convenient. This is reported here: Error in renameDimension · Issue #1357 · Unidata/netcdf4-python · GitHub
I agree, it will be nice to keep the old format, if and when possible.
Backwards compatibility is a blessing.
xarray does a good job in renaming dimensions and variables. For me:
ds2 = (ds1.rename_dims({"valid_time":"time"})
.rename_vars({"valid_time":"time"})
.drop_vars(["expver", "number"], errors="ignore")
)
does the job in renaming “valid_time” to “time” and dropping some variables that I don’t need.
I was looking for solution of this issue and I found it. CDS API provide backwards compatibility. The problem is that a result of data is now in netcdf4 format (previous was in netcdf3). Here is description of changes made (not only time dimension name but also order of dimensions was changed):
https://confluence.ecmwf.int/display/CKB/GRIB+to+netCDF+conversion+on+new+CDS+and+ADS+systems
Summarising - if you would like to use previous netcdf format you will able to put
"data_format": "netcdf_legacy"
as a format parameter in your request.
I tried, tested and it works. I know it is deprecated format and it could be outdated in future but for now it solves my problems
Can you please help, I have downloaded single level ERA5 data of t2m hourly and I want to calculate daily mean using CDO, but the new ERA5 files are not working with CDO, how to resolve this, as file size is around 3GB and I can’t do it in any other way, 21 year data.
Unfortunately I do not help you because I never used CDO. For processing data I use Python and GIS-related tools and libraries. I hope you can find solution of your problem on this forum.