ERA5 CDS requests which return a mixture of ERA5 and ERA5T data

Thank you for sharing, Jian Tang. It worked smoothly and seamlessly.  You just save me from days of sleepless nights.

Just downloaded ERA5 data (ssrd) for March 2022 and the "expver" seems to be gone. Is this going to be gone for good? The problem is that when the number of dimensions vary over time, it is difficult to find a coherent way to process the data.

Hi,

March 2022 has no "expver" because there are only ERA5T data.


Thanks

Thanks Michela

How come? I thought there was only a 5 -day lag or am I misunderstanding something here?


Axel

I'm looking at a 10m u component of wind .nc I just downloaded for 2022. It has the expver dimension all right. But the values aren't all NaN. The expver 5 values start as NaN. Then at 2022-02-13 02:00 the values all become -1.7. They stay that way until they transition to sensible values at 2022-05-01 00:00. Then the expver 1 values become -1.7.

I guess my question is how can I tell when to use the value from expver 1 and when to use expver 5? Detecting NaN seems to not be sufficient.

hi John, Can you share the request you used to retrieve the data, please?

Thanks,

Kevin

Hi, try this. The transition from expver 1 to 5 in the resulting netcdf file was at time index 3625. I used Panoply to view it.

{
'product_type': 'reanalysis',
'format': 'netcdf',
'year': 2022,
'variable': ['10m_v_component_of_wind'],
'month': ['1', '2', '3', '4', '5', '6', '7'],
'day': ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15', '16', '17', '18', '19', '20', '21', '22', '23', '24', '25', '26', '27', '28', '29', '30', '31'],
'time': 'all'
}


Hi John,

Thanks for reporting this; I think this may be an issue with the grib to netCDF converter due to the size of the request. In this case, if requesting netCDF it may be better if you just request 1 month at a time (although months containing a mix of era5/era5T will need careful handling as only these will have the 'expver' dimension)

Hope that helps,

Kevin

Hi John,

expver is used to tell the difference between the initial release (expver=5, called ERA5T) and validated ERA5 data (expver=1). See the link below for details.

ERA5: data documentation#Dataupdatefrequency

In most cases, ERA5 is identical to ERA5T. Therefore, if you spot any unusual behaviour, please let us know.

Thank you,

Xiaobo

Hi.. I downloaded hourly data in (nc) file extension and open it in panoply, when export data in (csv) format the time was in Gregorian calendar.. how can convert hourly data from Gregorian calendar date to date calendar data?

 Hi, can someone help me. I can't remove the expver using cdo vertmean, cdo -sellevel,1. The error said:

Warning (cdfCheckVars): 5 dimensional variables are not supported, skipped variable z!
Warning (cdfInqContents): No data arrays found!
cdo    vertmean: Open failed on >geopotential.nlev.nc<
                 Unsupported file structure

This is the info from my nc.file.

ncdump -h GH.nlev.nc
netcdf GH.nlev {
dimensions:
        longitude = 1440 ;
        latitude = 721 ;
        level = 14 ;
        expver = 2 ;
        time = 538 ;
variables:
        float longitude(longitude) ;
                longitude:units = "degrees_east" ;
                longitude:long_name = "longitude" ;
        float latitude(latitude) ;
                latitude:units = "degrees_north" ;
                latitude:long_name = "latitude" ;
        int level(level) ;
                level:units = "millibars" ;
                level:long_name = "pressure_level" ;
        int expver(expver) ;
                expver:long_name = "expver" ;
        int time(time) ;
                time:units = "hours since 1900-01-01 00:00:00.0" ;
                time:long_name = "time" ;
                time:calendar = "gregorian" ;
        short z(time, expver, level, latitude, longitude) ;
                z:scale_factor = 4.86892538277471 ;
                z:add_offset = 156350.109482621 ;
                z:_FillValue = -32767s ;
                z:missing_value = -32767s ;
                z:units = "m**2 s**-2" ;
                z:long_name = "Geopotential" ;
                z:standard_name = "geopotential" ;


Thank you.

hi, I think the method described in an earlier comment may help. Just use the small python script from Jin Tang:

(replace era5.tp.20200801.nc era5.tp.20200801.copy.nc with the name of your input/output files)

import xarray as xr
ERA5 = xr.open_mfdataset('era5.tp.20200801.nc',combine='by_coords')
ERA5_combine =ERA5.sel(expver=1).combine_first(ERA5.sel(expver=5))
ERA5_combine.load()
ERA5_combine.to_netcdf("era5.tp.20200801.copy.nc")

to remove the expver dimension in your downloaded data file.


Running it on this file:

ncdump -h test_in.nc        

netcdf test_in {

dimensions:

longitude = 1440 ;

latitude = 721 ;

expver = 2 ;

time = 12 ;

variables:

float longitude(longitude) ;

longitude:units = "degrees_east" ;

longitude:long_name = "longitude" ;

float latitude(latitude) ;

latitude:units = "degrees_north" ;

latitude:long_name = "latitude" ;

int expver(expver) ;

expver:long_name = "expver" ;

int time(time) ;

time:units = "hours since 1900-01-01 00:00:00.0" ;

time:long_name = "time" ;

time:calendar = "gregorian" ;

short t2m(time, expver, latitude, longitude) ;

t2m:scale_factor = 0.00182472256828442 ;

t2m:add_offset = 257.866053886274 ;

t2m:_FillValue = -32767s ;

t2m:missing_value = -32767s ;

t2m:units = "K" ;

t2m:long_name = "2 metre temperature" ;


cdo info  test_in.nc

    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter ID

     1 : 2023-01-01 00:00:00       1  1038240       0 :      218.55      277.46      313.44 : -1            

     2 : 2023-01-01 00:00:00       5  1038240 1038240 :                     nan             : -1            

     3 : 2023-02-01 00:00:00       1  1038240       0 :      225.35      276.67      307.91 : -1            

     4 : 2023-02-01 00:00:00       5  1038240 1038240 :                     nan             : -1            

     5 : 2023-03-01 00:00:00       1  1038240       0 :      218.47      276.55      310.27 : -1            

     6 : 2023-03-01 00:00:00       5  1038240 1038240 :                     nan             : -1            

     7 : 2023-04-01 00:00:00       1  1038240       0 :      209.07      277.40      311.98 : -1            

     8 : 2023-04-01 00:00:00       5  1038240 1038240 :                     nan             : -1            

     9 : 2023-05-01 00:00:00       1  1038240       0 :      210.15      278.76      313.10 : -1            

    10 : 2023-05-01 00:00:00       5  1038240 1038240 :                     nan             : -1            

    11 : 2023-06-01 00:00:00       1  1038240       0 :      198.08      280.23      311.80 : -1            

    12 : 2023-06-01 00:00:00       5  1038240 1038240 :                     nan             : -1            

    13 : 2023-07-01 00:00:00       1  1038240       0 :      198.80      280.80      317.66 : -1            

    14 : 2023-07-01 00:00:00       5  1038240 1038240 :                     nan             : -1            

    15 : 2023-08-01 00:00:00       1  1038240       0 :      199.11      281.23      314.89 : -1            

    16 : 2023-08-01 00:00:00       5  1038240 1038240 :                     nan             : -1            

    17 : 2023-09-01 00:00:00       1  1038240       0 :      198.98      280.38      316.13 : -1            

    18 : 2023-09-01 00:00:00       5  1038240 1038240 :                     nan             : -1            

    19 : 2023-10-01 00:00:00       1  1038240 1038240 :                     nan             : -1            

    20 : 2023-10-01 00:00:00       5  1038240       0 :      207.78      279.84      310.16 : -1            

    21 : 2023-11-01 00:00:00       1  1038240 1038240 :                     nan             : -1            

    22 : 2023-11-01 00:00:00       5  1038240       0 :      216.54      278.59      310.12 : -1            

    23 : 2023-12-01 00:00:00       1  1038240 1038240 :                     nan             : -1            

    24 : 2023-12-01 00:00:00       5  1038240       0 :      230.68      278.14      309.12 : -1  


gives:

 ncdump -h test_in_flatten.nc 

netcdf test_in_flatten {

dimensions:

time = 12 ;

latitude = 721 ;

longitude = 1440 ;

variables:

float t2m(time, latitude, longitude) ;

t2m:_FillValue = NaNf ;

t2m:units = "K" ;

t2m:long_name = "2 metre temperature" ;

float longitude(longitude) ;

longitude:_FillValue = NaNf ;

longitude:units = "degrees_east" ;

longitude:long_name = "longitude" ;

float latitude(latitude) ;

latitude:_FillValue = NaNf ;

latitude:units = "degrees_north" ;

latitude:long_name = "latitude" ;

int time(time) ;

time:long_name = "time" ;

time:units = "hours since 1900-01-01" ;

time:calendar = "gregorian" ;



% cdo info  test_in_flatten.nc

    -1 :       Date     Time   Level Gridsize    Miss :     Minimum        Mean     Maximum : Parameter ID

     1 : 2023-01-01 00:00:00       0  1038240       0 :      218.55      277.46      313.44 : -1            

     2 : 2023-02-01 00:00:00       0  1038240       0 :      225.35      276.67      307.91 : -1            

     3 : 2023-03-01 00:00:00       0  1038240       0 :      218.47      276.55      310.27 : -1            

     4 : 2023-04-01 00:00:00       0  1038240       0 :      209.07      277.40      311.98 : -1            

     5 : 2023-05-01 00:00:00       0  1038240       0 :      210.15      278.76      313.10 : -1            

     6 : 2023-06-01 00:00:00       0  1038240       0 :      198.08      280.23      311.80 : -1            

     7 : 2023-07-01 00:00:00       0  1038240       0 :      198.80      280.80      317.66 : -1            

     8 : 2023-08-01 00:00:00       0  1038240       0 :      199.11      281.23      314.89 : -1            

     9 : 2023-09-01 00:00:00       0  1038240       0 :      198.98      280.38      316.13 : -1            

    10 : 2023-10-01 00:00:00       0  1038240       0 :      207.78      279.84      310.16 : -1            

    11 : 2023-11-01 00:00:00       0  1038240       0 :      216.54      278.59      310.12 : -1            

    12 : 2023-12-01 00:00:00       0  1038240       0 :      230.68      278.14      309.12 : -1    

So the expver dimension is removed.