Hi !
While downloading ERA5 data on pressure levels ( T u v) I noticed that
the data downloaded from the new site has new time format.
The reference time is 1/1/1970 while previously it was 1/1/1900.
ncview can not decipher it.
This is the error message from ncview:
dimension valid_time
ncview: netcdf_dim_value: unknown data type (10) for
dimension valid_time
In order to merge new and old data files, how do you solve the format problem ?
Many thanks for your help !
you can use CDO to reset the reference time, creating a new output file:
% cdo -setreftime,1900-01-01,00:00:00,1hour infile.nc outfile.nc
weird
cdo created new file with the following features:
ncdump -t -v valid_time dec-1950-fix.nc
netcdf dec-1950-fix {
dimensions:
valid_time = UNLIMITED ; // (90 currently)
longitude = 81 ;
latitude = 81 ;
pressure_level = 1 ;
variables:
double valid_time(valid_time) ;
valid_time:standard_name = “time” ;
valid_time:long_name = “time” ;
valid_time:units = “hours since 1970-1-1 00:00:00” ;
valid_time:calendar = “proleptic_gregorian” ;
valid_time:axis = “T” ;
double longitude(longitude) ;
longitude:standard_name = “longitude” ;
longitude:long_name = “longitude” ;
longitude:units = “degrees_east” ;
longitude:axis = “X” ;
double latitude(latitude) ;
latitude:standard_name = “latitude” ;
latitude:long_name = “latitude” ;
latitude:units = “degrees_north” ;
latitude:axis = “Y” ;
double pressure_level(pressure_level) ;
pressure_level:standard_name = “air_pressure” ;
pressure_level:long_name = “pressure” ;
pressure_level:units = “hPa” ;
pressure_level:positive = “down” ;
pressure_level:axis = “Z” ;
pressure_level:stored_direction = “d” ;
float t(valid_time, pressure_level, latitude, longitude) ;
t:standard_name = “air_temperature” ;
t:long_name = “Temperature” ;
t:units = “K” ;
t:_FillValue = NaNf ;
t:missing_value = NaNf ;
t:GRIB_dataType = “a” ;
t:GRIB_typeOfLevel = “i” ;
t:GRIB_stepType = “i” ;
t:GRIB_gridType = “r” ;
t:GRIB_cfName = “a” ;
t:GRIB_cfVarName = “t” ;
t:GRIB_gridDefinitionDescription = “L” ;
t:GRIB_iDirectionIncrementInDegrees = 0.25 ;
t:GRIB_jDirectionIncrementInDegrees = 0.25 ;
t:GRIB_latitudeOfFirstGridPointInDegrees = 40. ;
t:GRIB_latitudeOfLastGridPointInDegrees = 20. ;
t:GRIB_longitudeOfFirstGridPointInDegrees = 20. ;
t:GRIB_longitudeOfLastGridPointInDegrees = 40. ;
t:GRIB_missingValue = 3.40282346638529e+38 ;
t:GRIB_name = “T” ;
t:GRIB_shortName = “t” ;
t:GRIB_units = “K” ;
float u(valid_time, pressure_level, latitude, longitude) ;
u:standard_name = “eastward_wind” ;
u:long_name = “U component of wind” ;
u:units = “m s**-1” ;
u:_FillValue = NaNf ;
u:missing_value = NaNf ;
u:GRIB_dataType = “a” ;
u:GRIB_typeOfLevel = “i” ;
u:GRIB_stepType = “i” ;
u:GRIB_gridType = “r” ;
u:GRIB_cfName = “e” ;
u:GRIB_cfVarName = “u” ;
u:GRIB_gridDefinitionDescription = “L” ;
u:GRIB_iDirectionIncrementInDegrees = 0.25 ;
u:GRIB_jDirectionIncrementInDegrees = 0.25 ;
u:GRIB_latitudeOfFirstGridPointInDegrees = 40. ;
u:GRIB_latitudeOfLastGridPointInDegrees = 20. ;
u:GRIB_longitudeOfFirstGridPointInDegrees = 20. ;
u:GRIB_longitudeOfLastGridPointInDegrees = 40. ;
u:GRIB_missingValue = 3.40282346638529e+38 ;
u:GRIB_name = “U” ;
u:GRIB_shortName = “u” ;
u:GRIB_units = “m” ;
float v(valid_time, pressure_level, latitude, longitude) ;
v:standard_name = “northward_wind” ;
v:long_name = “V component of wind” ;
v:units = “m s**-1” ;
v:_FillValue = NaNf ;
v:missing_value = NaNf ;
v:GRIB_dataType = “a” ;
v:GRIB_typeOfLevel = “i” ;
v:GRIB_stepType = “i” ;
v:GRIB_gridType = “r” ;
v:GRIB_cfName = “n” ;
v:GRIB_cfVarName = “v” ;
v:GRIB_gridDefinitionDescription = “L” ;
v:GRIB_iDirectionIncrementInDegrees = 0.25 ;
v:GRIB_jDirectionIncrementInDegrees = 0.25 ;
v:GRIB_latitudeOfFirstGridPointInDegrees = 40. ;
v:GRIB_latitudeOfLastGridPointInDegrees = 20. ;
v:GRIB_longitudeOfFirstGridPointInDegrees = 20. ;
v:GRIB_longitudeOfLastGridPointInDegrees = 40. ;
v:GRIB_missingValue = 3.40282346638529e+38 ;
v:GRIB_name = “V” ;
v:GRIB_shortName = “v” ;
v:GRIB_units = “m” ;
// global attributes:
:CDI = “Climate Data Interface version ?? (http://mpimet.mpg.de/cdi)” ;
:history = “Wed Aug 07 00:22:50 2024: cdo -setreftime,1900-01-01,00:00:00,1hour dec-1950.nc dec-1950-fix.nc\nTue Aug 06 19:27:52 2024: cdo selyear,1950 dec1950-1959.nc dec-1950.nc\nP” ;
:institution = “European Centre for Medium-Range Weather Forecasts” ;
:Conventions = “CF-1.7” ;
:GRIB_centre = “ecmf” ;
:GRIB_centreDescription = “European Centre for Medium-Range Weather Forecasts” ;
:CDO = “Climate Data Operators version 1.9.3 (Overview - CDO - Project Management Service)” ;
data:
valid_time = “2020-01-01 12”, “2020-01-02 12”, “2020-01-03 12”,
“2020-01-04 12”, “2020-01-05 12”, “2020-01-06 12”, “2020-01-07 12”,
“2020-01-08 12”, “2020-01-09 12”, “2020-01-10 12”, “2020-01-11 12”,
“2020-01-12 12”, “2020-01-13 12”, “2020-01-14 12”, “2020-01-15 12”,
“2020-01-16 12”, “2020-01-17 12”, “2020-01-18 12”, “2020-01-19 12”,
“2020-01-20 12”, “2020-01-21 12”, “2020-01-22 12”, “2020-01-23 12”,
“2020-01-24 12”, “2020-01-25 12”, “2020-01-26 12”, “2020-01-27 12”,
“2020-01-28 12”, “2020-01-29 12”, “2020-01-30 12”, “2020-01-31 12”,
“2020-02-01 12”, “2020-02-02 12”, “2020-02-03 12”, “2020-02-04 12”,
“2020-02-05 12”, “2020-02-06 12”, “2020-02-07 12”, “2020-02-08 12”,
“2020-02-09 12”, “2020-02-10 12”, “2020-02-11 12”, “2020-02-12 12”,
“2020-02-13 12”, “2020-02-14 12”, “2020-02-15 12”, “2020-02-16 12”,
“2020-02-17 12”, “2020-02-18 12”, “2020-02-19 12”, “2020-02-20 12”,
“2020-02-21 12”, “2020-02-22 12”, “2020-02-23 12”, “2020-02-24 12”,
“2020-02-25 12”, “2020-02-26 12”, “2020-02-27 12”, “2020-02-28 12”,
“2020-11-30 12”, “2020-12-01 12”, “2020-12-02 12”, “2020-12-03 12”,
“2020-12-04 12”, “2020-12-05 12”, “2020-12-06 12”, “2020-12-07 12”,
“2020-12-08 12”, “2020-12-09 12”, “2020-12-10 12”, “2020-12-11 12”,
“2020-12-12 12”, “2020-12-13 12”, “2020-12-14 12”, “2020-12-15 12”,
“2020-12-16 12”, “2020-12-17 12”, “2020-12-18 12”, “2020-12-19 12”,
“2020-12-20 12”, “2020-12-21 12”, “2020-12-22 12”, “2020-12-23 12”,
“2020-12-24 12”, “2020-12-25 12”, “2020-12-26 12”, “2020-12-27 12”,
“2020-12-28 12”, “2020-12-29 12”, “2020-12-30 12” ;
The new downloaded data was dec 1950 ,
according to the suggested transformation, cdo should create ref time 1/1/1900
and dates 1/12/1950 - 31/12/1950
@Kevin_Marsh Another thing that changed in the output files is the variable name valid_time, instead of time. I know this can be quite easily be changed after downloading, but would it not be possible to rename it back to time for user convenience?
For me the most convenient rename method is using the netCDF4 api (cdo cannot be assumed to be available on all systems where my package runs):
nc = Dataset(file_out,'a')
nc.renameVariable("valid_time","time")
nc.renameDimension("valid_time","time")
However, the renameDimension for some reason replaces all time values with zeros, so this is not convenient. This is reported here: Error in renameDimension · Issue #1357 · Unidata/netcdf4-python · GitHub
I agree, it will be nice to keep the old format, if and when possible.
Backwards compatibility is a blessing.
xarray does a good job in renaming dimensions and variables. For me:
ds2 = (ds1.rename_dims({"valid_time":"time"})
.rename_vars({"valid_time":"time"})
.drop_vars(["expver", "number"], errors="ignore")
)
does the job in renaming “valid_time” to “time” and dropping some variables that I don’t need.