New time format in ERA5 netcdf files

Hi !

While downloading ERA5 data on pressure levels ( T u v) I noticed that

the data downloaded from the new site has new time format.
The reference time is 1/1/1970 while previously it was 1/1/1900.

ncview can not decipher it.
This is the error message from ncview:

dimension valid_time
ncview: netcdf_dim_value: unknown data type (10) for
dimension valid_time

In order to merge new and old data files, how do you solve the format problem ?
Many thanks for your help !

1 Like

you can use CDO to reset the reference time, creating a new output file:

% cdo -setreftime,1900-01-01,00:00:00,1hour infile.nc outfile.nc

1 Like

weird
cdo created new file with the following features:
ncdump -t -v valid_time dec-1950-fix.nc
netcdf dec-1950-fix {
dimensions:
valid_time = UNLIMITED ; // (90 currently)
longitude = 81 ;
latitude = 81 ;
pressure_level = 1 ;
variables:
double valid_time(valid_time) ;
valid_time:standard_name = β€œtime” ;
valid_time:long_name = β€œtime” ;
valid_time:units = β€œhours since 1970-1-1 00:00:00” ;
valid_time:calendar = β€œproleptic_gregorian” ;
valid_time:axis = β€œT” ;
double longitude(longitude) ;
longitude:standard_name = β€œlongitude” ;
longitude:long_name = β€œlongitude” ;
longitude:units = β€œdegrees_east” ;
longitude:axis = β€œX” ;
double latitude(latitude) ;
latitude:standard_name = β€œlatitude” ;
latitude:long_name = β€œlatitude” ;
latitude:units = β€œdegrees_north” ;
latitude:axis = β€œY” ;
double pressure_level(pressure_level) ;
pressure_level:standard_name = β€œair_pressure” ;
pressure_level:long_name = β€œpressure” ;
pressure_level:units = β€œhPa” ;
pressure_level:positive = β€œdown” ;
pressure_level:axis = β€œZ” ;
pressure_level:stored_direction = β€œd” ;
float t(valid_time, pressure_level, latitude, longitude) ;
t:standard_name = β€œair_temperature” ;
t:long_name = β€œTemperature” ;
t:units = β€œK” ;
t:_FillValue = NaNf ;
t:missing_value = NaNf ;
t:GRIB_dataType = β€œa” ;
t:GRIB_typeOfLevel = β€œi” ;
t:GRIB_stepType = β€œi” ;
t:GRIB_gridType = β€œr” ;
t:GRIB_cfName = β€œa” ;
t:GRIB_cfVarName = β€œt” ;
t:GRIB_gridDefinitionDescription = β€œL” ;
t:GRIB_iDirectionIncrementInDegrees = 0.25 ;
t:GRIB_jDirectionIncrementInDegrees = 0.25 ;
t:GRIB_latitudeOfFirstGridPointInDegrees = 40. ;
t:GRIB_latitudeOfLastGridPointInDegrees = 20. ;
t:GRIB_longitudeOfFirstGridPointInDegrees = 20. ;
t:GRIB_longitudeOfLastGridPointInDegrees = 40. ;
t:GRIB_missingValue = 3.40282346638529e+38 ;
t:GRIB_name = β€œT” ;
t:GRIB_shortName = β€œt” ;
t:GRIB_units = β€œK” ;
float u(valid_time, pressure_level, latitude, longitude) ;
u:standard_name = β€œeastward_wind” ;
u:long_name = β€œU component of wind” ;
u:units = β€œm s**-1” ;
u:_FillValue = NaNf ;
u:missing_value = NaNf ;
u:GRIB_dataType = β€œa” ;
u:GRIB_typeOfLevel = β€œi” ;
u:GRIB_stepType = β€œi” ;
u:GRIB_gridType = β€œr” ;
u:GRIB_cfName = β€œe” ;
u:GRIB_cfVarName = β€œu” ;
u:GRIB_gridDefinitionDescription = β€œL” ;
u:GRIB_iDirectionIncrementInDegrees = 0.25 ;
u:GRIB_jDirectionIncrementInDegrees = 0.25 ;
u:GRIB_latitudeOfFirstGridPointInDegrees = 40. ;
u:GRIB_latitudeOfLastGridPointInDegrees = 20. ;
u:GRIB_longitudeOfFirstGridPointInDegrees = 20. ;
u:GRIB_longitudeOfLastGridPointInDegrees = 40. ;
u:GRIB_missingValue = 3.40282346638529e+38 ;
u:GRIB_name = β€œU” ;
u:GRIB_shortName = β€œu” ;
u:GRIB_units = β€œm” ;
float v(valid_time, pressure_level, latitude, longitude) ;
v:standard_name = β€œnorthward_wind” ;
v:long_name = β€œV component of wind” ;
v:units = β€œm s**-1” ;
v:_FillValue = NaNf ;
v:missing_value = NaNf ;
v:GRIB_dataType = β€œa” ;
v:GRIB_typeOfLevel = β€œi” ;
v:GRIB_stepType = β€œi” ;
v:GRIB_gridType = β€œr” ;
v:GRIB_cfName = β€œn” ;
v:GRIB_cfVarName = β€œv” ;
v:GRIB_gridDefinitionDescription = β€œL” ;
v:GRIB_iDirectionIncrementInDegrees = 0.25 ;
v:GRIB_jDirectionIncrementInDegrees = 0.25 ;
v:GRIB_latitudeOfFirstGridPointInDegrees = 40. ;
v:GRIB_latitudeOfLastGridPointInDegrees = 20. ;
v:GRIB_longitudeOfFirstGridPointInDegrees = 20. ;
v:GRIB_longitudeOfLastGridPointInDegrees = 40. ;
v:GRIB_missingValue = 3.40282346638529e+38 ;
v:GRIB_name = β€œV” ;
v:GRIB_shortName = β€œv” ;
v:GRIB_units = β€œm” ;

// global attributes:
:CDI = β€œClimate Data Interface version ?? (http://mpimet.mpg.de/cdi)” ;
:history = β€œWed Aug 07 00:22:50 2024: cdo -setreftime,1900-01-01,00:00:00,1hour dec-1950.nc dec-1950-fix.nc\nTue Aug 06 19:27:52 2024: cdo selyear,1950 dec1950-1959.nc dec-1950.nc\nP” ;
:institution = β€œEuropean Centre for Medium-Range Weather Forecasts” ;
:Conventions = β€œCF-1.7” ;
:GRIB_centre = β€œecmf” ;
:GRIB_centreDescription = β€œEuropean Centre for Medium-Range Weather Forecasts” ;
:CDO = β€œClimate Data Operators version 1.9.3 (Overview - CDO - Project Management Service)” ;
data:

valid_time = β€œ2020-01-01 12”, β€œ2020-01-02 12”, β€œ2020-01-03 12”,
β€œ2020-01-04 12”, β€œ2020-01-05 12”, β€œ2020-01-06 12”, β€œ2020-01-07 12”,
β€œ2020-01-08 12”, β€œ2020-01-09 12”, β€œ2020-01-10 12”, β€œ2020-01-11 12”,
β€œ2020-01-12 12”, β€œ2020-01-13 12”, β€œ2020-01-14 12”, β€œ2020-01-15 12”,
β€œ2020-01-16 12”, β€œ2020-01-17 12”, β€œ2020-01-18 12”, β€œ2020-01-19 12”,
β€œ2020-01-20 12”, β€œ2020-01-21 12”, β€œ2020-01-22 12”, β€œ2020-01-23 12”,
β€œ2020-01-24 12”, β€œ2020-01-25 12”, β€œ2020-01-26 12”, β€œ2020-01-27 12”,
β€œ2020-01-28 12”, β€œ2020-01-29 12”, β€œ2020-01-30 12”, β€œ2020-01-31 12”,
β€œ2020-02-01 12”, β€œ2020-02-02 12”, β€œ2020-02-03 12”, β€œ2020-02-04 12”,
β€œ2020-02-05 12”, β€œ2020-02-06 12”, β€œ2020-02-07 12”, β€œ2020-02-08 12”,
β€œ2020-02-09 12”, β€œ2020-02-10 12”, β€œ2020-02-11 12”, β€œ2020-02-12 12”,
β€œ2020-02-13 12”, β€œ2020-02-14 12”, β€œ2020-02-15 12”, β€œ2020-02-16 12”,
β€œ2020-02-17 12”, β€œ2020-02-18 12”, β€œ2020-02-19 12”, β€œ2020-02-20 12”,
β€œ2020-02-21 12”, β€œ2020-02-22 12”, β€œ2020-02-23 12”, β€œ2020-02-24 12”,
β€œ2020-02-25 12”, β€œ2020-02-26 12”, β€œ2020-02-27 12”, β€œ2020-02-28 12”,
β€œ2020-11-30 12”, β€œ2020-12-01 12”, β€œ2020-12-02 12”, β€œ2020-12-03 12”,
β€œ2020-12-04 12”, β€œ2020-12-05 12”, β€œ2020-12-06 12”, β€œ2020-12-07 12”,
β€œ2020-12-08 12”, β€œ2020-12-09 12”, β€œ2020-12-10 12”, β€œ2020-12-11 12”,
β€œ2020-12-12 12”, β€œ2020-12-13 12”, β€œ2020-12-14 12”, β€œ2020-12-15 12”,
β€œ2020-12-16 12”, β€œ2020-12-17 12”, β€œ2020-12-18 12”, β€œ2020-12-19 12”,
β€œ2020-12-20 12”, β€œ2020-12-21 12”, β€œ2020-12-22 12”, β€œ2020-12-23 12”,
β€œ2020-12-24 12”, β€œ2020-12-25 12”, β€œ2020-12-26 12”, β€œ2020-12-27 12”,
β€œ2020-12-28 12”, β€œ2020-12-29 12”, β€œ2020-12-30 12” ;

The new downloaded data was dec 1950 ,

according to the suggested transformation, cdo should create ref time 1/1/1900

and dates 1/12/1950 - 31/12/1950

@Kevin_Marsh Another thing that changed in the output files is the variable name valid_time, instead of time. I know this can be quite easily be changed after downloading, but would it not be possible to rename it back to time for user convenience?

For me the most convenient rename method is using the netCDF4 api (cdo cannot be assumed to be available on all systems where my package runs):

nc = Dataset(file_out,'a')
nc.renameVariable("valid_time","time")
nc.renameDimension("valid_time","time")

However, the renameDimension for some reason replaces all time values with zeros, so this is not convenient. This is reported here: Error in renameDimension Β· Issue #1357 Β· Unidata/netcdf4-python Β· GitHub

I agree, it will be nice to keep the old format, if and when possible.
Backwards compatibility is a blessing.

2 Likes

xarray does a good job in renaming dimensions and variables. For me:

ds2 = (ds1.rename_dims({"valid_time":"time"})
          .rename_vars({"valid_time":"time"})
          .drop_vars(["expver", "number"], errors="ignore")
      )

does the job in renaming β€œvalid_time” to β€œtime” and dropping some variables that I don’t need.

2 Likes

I was looking for solution of this issue and I found it. CDS API provide backwards compatibility. The problem is that a result of data is now in netcdf4 format (previous was in netcdf3). Here is description of changes made (not only time dimension name but also order of dimensions was changed):

https://confluence.ecmwf.int/display/CKB/GRIB+to+netCDF+conversion+on+new+CDS+and+ADS+systems

Summarising - if you would like to use previous netcdf format you will able to put
"data_format": "netcdf_legacy"
as a format parameter in your request.

I tried, tested and it works. I know it is deprecated format and it could be outdated in future but for now it solves my problems

3 Likes

Can you please help, I have downloaded single level ERA5 data of t2m hourly and I want to calculate daily mean using CDO, but the new ERA5 files are not working with CDO, how to resolve this, as file size is around 3GB and I can’t do it in any other way, 21 year data.

Unfortunately I do not help you because I never used CDO. For processing data I use Python and GIS-related tools and libraries. I hope you can find solution of your problem on this forum.