# ERA5 CDS requests which return a mixture of ERA5 and ERA5T data **URL:** \ **Category:** C3S - Datasets and Usage\ **Tags:** c3s, cds, era5\ **Created:** [18 February 2020 09:29 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781 "2020-02-18T09:29:43Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Michela](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Michela](https://forum.ecmwf.int/u/Michela)\ **Post date:** [18 February 2020 09:29 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/1 "2020-02-18T09:29:43Z")
The ERA5 hourly and monthly data are made available with a 3 month delay. This means that after a month has passed, another month's worth of ERA5 data is written to the dataset. ERA5T (near real time) preliminary data are used to fill the gap between the end of the ERA5 data and 5 days before the present date. The oldest month of these is overwritten each month as new ERA5 data become available. So as an example, say we have a current date  of 15th February 2020: - ERA5 data are currently from 1/1/1979 - 30/11/2019 (instantaneous variables)  and 1/1/1979 - 1/12/2019 (00-06 UTC, accumulated variables) - ERA5T data (with a 5 day delay) are from 1/12/2019- 10/2/2020 (instantaneous variables)  and 1/12/2019 (07-23 UTC, accumulated variables)- 10/2/2020 For requests which return a mixture of ERA5 and ERA5T data  (such as for data from the 1st of the month), instantaneous variables (e.g temperature) come from ERA5T (which has 'experiment version'  of 5) while accumulated variables (fluxes, precipitation) come from both datasets with the following structure: - 00-06 UTC on 1 day of the month from ERA5 (expver 1) - 07-23 UTC on 1 day of the month (and the following dates up to 5 day from present) from ERA5T (expver 5) When these data are converted to netCDF a new dimension is created called **expver** containing 1 and 5 **.** Moreover, a single time coordinate is used which covers the entire requested period. ``` dimensions: longitude = 1440 ; latitude = 721 ; expver = 2 ; time = 24 ; variables: float longitude(longitude) ; longitude:units = "degrees_east" ; longitude:long_name = "longitude" ; float latitude(latitude) ; latitude:units = "degrees_north" ; latitude:long_name = "latitude" ; int expver(expver) ; expver:long_name = "expver" ; int time(time) ; time:units = "hours since 1900-01-01 00:00:00.0" ; time:long_name = "time" ; time:calendar = "gregorian" ; short tp(time, expver, latitude, longitude) ; tp:scale_factor = 9.06276558810304e-07 ; tp:add_offset = 0.0296950577259784 ; tp:_FillValue = -32767s ; tp:missing_value = -32767s ; tp:units = "m" ; tp:long_name = "Total precipitation" ; data: expver = 5, 1 ; … } ``` Both expver dimensions use the full time extent of time coordinate but the expver 1 data only covers the first 7 timesteps, the remaining timesteps are 'padded' with empty fields. For the expver 5 data, the first 7 timesteps are padded with empty fields, with the remaining timesteps coming from the ERA5T data. When the last ERA5 data are released, they will overwrite the ERA5T data for the entire month and for accumulated variables for 00-06 in next month. This process will be repeated each month. ---
**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\ **Post date:** [18 February 2020 10:09 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/2 "2020-02-18T10:09:49Z")
Notice for the time being, if you download only ERA5, or ERA5T, the above mentioned dimension 'expver' will not appear. This makes it difficult to tell the difference between ERA5 and ERA5T. ---
**Author:** ![UnknownUser](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/u/35a633/32.png) [@UnknownUser](https://forum.ecmwf.int/u/UnknownUser)\ **Post date:** [1 March 2020 14:50 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/3 "2020-03-01T14:50:42Z")
It seems that if one requests hourly total precipitation ERA5 data for  1 January 2020, the file contains both expver versions (1 and 5) and the file size is doubled (around 99 MB). For other days, the expver dimension does not appear.  ---
**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\ **Post date:** [2 March 2020 09:14 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/4 "2020-03-02T09:14:53Z")
Thank you for reporting this, Julia. We are looking into a long term solution now. Unfortunately it will take some time. ---
**Author:** ![Michela](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Michela](https://forum.ecmwf.int/u/Michela)\ **Post date:** [3 March 2020 10:48 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/5 "2020-03-03T10:48:42Z")
As pointed out **above** , only mixed ERA5/ERA5T data has 'expver'. When users consider accumulated variables the file has "the following structure: - 00-06 UTC on 1 day of the month from ERA5 (expver 1) - 07-23 UTC on 1 day of the month (and the following dates up to 5 day from present) from ERA5T (expver 5)" So for your case, data for 00-06 UTC  of 1 January 2020 is ERA5 while the rest of data are ERA5T. Data for 2 January 2020 are only ERA5T so 'expver' does not appear. Moreover, please pay attention "Both expver dimensions use the full time extent of time coordinate but the expver 1 data only covers the first 7 timesteps, the remaining timesteps are 'padded' with empty fields." This means that the empty fields contain NaN values. ---
**Author:** ![Alberto\_Troccoli](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/a/f475e1/32.png) [@Alberto\_Troccoli](https://forum.ecmwf.int/u/Alberto_Troccoli)\ **Post date:** [18 March 2020 00:59 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/6 "2020-03-18T00:59:07Z")
Hello Michela and Xiaobo I can see why you want to keep two expver but I think it's making things more complicated than needed for users. Moreover, the introduction of the two experiments is breaking codes, with consequent time loss trying to first identify the issue, and then find a (not-so-striaghtforward) solution. My suggestion would be to get rid of the two expver and just communicate when changes to ERA5T, when they become ERA5, are made, as anyway already indicated in [Release of ERA5T](https://prod.ecmwf-forum-prod.compute.cci2.ecmwf.int/t/release-of-era5t/1136) Could this solution – i.e. merging expver 1 with 5, so no expver dimension/parameter appears in the retrieval – be implemented please? I think it'd be much cleaner if this was done at your end. Thank you very much Alberto ---
**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\ **Post date:** [18 March 2020 11:02 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/7 "2020-03-18T11:02:17Z")
FYI Alberto, we are thinking about to have 'expver' as a dimension for all ERA5 and ERA5T data. ---
**Author:** ![Carl\_Svboda](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/c/e9a140/32.png) [@Carl\_Svboda](https://forum.ecmwf.int/u/Carl_Svboda)\ **Post date:** [9 March 2020 12:25 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/8 "2020-03-09T12:25:25Z")
I have the same issue as Julia Wagemann when downloading SurfaceSolarRadiation for all available 2020 timesteps, the first six hrs of 01/01/2020 are expver = 1, but the rest of the 2020 timesteps are expver = 5 ---
**Author:** ![Michela](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/m/898d66/32.png) [@Michela](https://forum.ecmwf.int/u/Michela)\ **Post date:** [9 March 2020 12:38 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/9 "2020-03-09T12:38:33Z")
Yes, this is because Surface Solar Radiation is an accumulated parameter and January is a month with ERA5 and ERA5T mixed data. For these reasons, the file has the following structure: - 00-06 UTC on 1 day of the month from ERA5 (expver 1) - 07-23 UTC on 1 day of the month (and the following dates up to 5 day from present) from ERA5T (expver 5)" So also in your case, data for 00-06 UTC of 1 January 2020 is ERA5 while the rest of data are ERA5T. Data for 2 January 2020 are only ERA5T so 'expver' does not appear. Moreover, please pay attention to the empty fields which contain NaN values. This happens because "Both expver dimensions use the full time extent of time coordinate but the expver 1 data only covers the first 7 timesteps, the remaining timesteps are 'padded' with empty fields. For the expver 5 data, the first 7 timesteps are padded with empty fields, with the remaining timesteps coming from the ERA5T data." ---
**Author:** ![Richard\_Berg](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/r/9dc877/32.png) [@Richard\_Berg](https://forum.ecmwf.int/u/Richard_Berg)\ **Post date:** [9 March 2020 20:52 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/10 "2020-03-09T20:52:42Z")
Semi-related to this topic... Is there any documentation for why the most recent data for ERA5T instantaneous variables are available only from 0 - 21Z for the most recent available day, and the accumulated variables are available through 06Z the following day? ---
**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\ **Post date:** [10 March 2020 10:32 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/11 "2020-03-10T10:32:49Z")
I understand this is the best we can do for the time being. ---
**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\ **Post date:** [10 March 2020 11:37 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/12 "2020-03-10T11:37:35Z")
Our technical team commented: "the accumulated fields are forecast fields from forecast starting at 18h , while the instantaneous fields are analysis fields from the 9-21h assimilation window." ---
**Author:** ![Richard\_Berg](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/r/9dc877/32.png) [@Richard\_Berg](https://forum.ecmwf.int/u/Richard_Berg)\ **Post date:** [11 March 2020 01:29 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/13 "2020-03-11T01:29:13Z")
I see, thank you for the quick response. And are the accumulated and instantaneous data released through CDS at 18Z and 21Z, respectively, or is there some specific lag (computational) time for each? ---
**Author:** ![Luiz\_Angelo\_Steffene](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/l/a3d4f5/32.png) [@Luiz\_Angelo\_Steffene](https://forum.ecmwf.int/u/Luiz_Angelo_Steffene)\ **Post date:** [28 December 2020 20:52 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/14 "2020-12-28T20:52:23Z")
Did someone found a simple way to get rid of expver dimension, or at least to filter out Era5 and Era5T data on mars scripts? As Alberto told, this is breaking a lot of codes. In my case, I download Ozone Total Column in a "monthly" base to automatically generate maps with NCL. However, the presence of expver adds a dimension that NCL can't understand, and I can't get rid of it  ![(sad)](https://confluence.ecmwf.int/s/-xehvhq/9012/8yg2g7/_/images/icons/emoticons/sad.svg) (could not find an easy way: I can remove the expver variables but the dimension remains).   ---
**Author:** ![Gifty\_Attiah](https://forum.ecmwf.int/user_avatar/forum.ecmwf.int/gifty_attiah/32/81_2.png) [@Gifty\_Attiah](https://forum.ecmwf.int/u/Gifty_Attiah)\ **Post date:** [13 January 2021 16:01 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/15 "2021-01-13T16:01:57Z")
Hi, is there a solution to removing the expver dimension? I have the same situation with the November, 2020 data and it is messing up all my codes which works perfectly on data from 2003 till date. I will really appreciate the help if anyone knows a way to achieve this. Thanks  ---
**Author:** ![Kevin\_Marsh](https://forum.ecmwf.int/user_avatar/forum.ecmwf.int/kevin_marsh/32/368_2.png) [@Kevin\_Marsh](https://forum.ecmwf.int/u/Kevin_Marsh)\ **Post date:** [14 January 2021 09:57 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/16 "2021-01-14T09:57:41Z")
Hi, At the moment i think the easiest way is to retrieve the ERA5 and ERA5T data in separate requests, by careful selection of the dates. In this way you would get 2 netCDF files without the 'expver' dimension which you can then merge if required, Thanks Kevin ---
**Author:** ![marco\_venturini](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/m/7993a0/32.png) [@marco\_venturini](https://forum.ecmwf.int/u/marco_venturini)\ **Post date:** [20 January 2021 17:59 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/17 "2021-01-20T17:59:13Z")
Hi, If you are looking for a Python workaround, you can use Xarray function _reduce__(np.nansum, 'expver')_. In this way you can collapse the dimension summing each other the two expver arrays, that perfectly match (the one is NaN when the other got a value). I know that it isn't politically correct, but with 1 row you avoid tons of code stop working. ---
**Author:** ![Gifty\_Attiah](https://forum.ecmwf.int/user_avatar/forum.ecmwf.int/gifty_attiah/32/81_2.png) [@Gifty\_Attiah](https://forum.ecmwf.int/u/Gifty_Attiah)\ **Post date:** [20 January 2021 22:29 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/18 "2021-01-20T22:29:05Z")
Hi. Thank you all for the helpful responses. I appreciate the suggestion [marco venturini](https://confluence.ecmwf.int/display/~marco.venturini@amigoclimate.com). That is a solution I can work around. Thanks ---
**Author:** ![Shahid\_Mehmood](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/s/54ee81/32.png) [@Shahid\_Mehmood](https://forum.ecmwf.int/u/Shahid_Mehmood)\ **Post date:** [17 June 2021 00:10 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/19 "2021-06-17T00:10:21Z")
cdo --reduce\_dim -copy [in.nc](http://pc_dly_mms_2011.nc) out[.nc](http://test.nc) worked well for me and it removed expver dimension  ---
**Author:** ![Jian\_Tang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/j/b9bd4f/32.png) [@Jian\_Tang](https://forum.ecmwf.int/u/Jian_Tang)\ **Post date:** [22 July 2021 07:19 UTC](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781/20 "2021-07-22T07:19:37Z")
this will do the trick import xarray as xr ERA5 = xr.open\_mfdataset('[era5.tp.20200801.nc](http://era5.tp.20200801.nc)',combine='by\_coords') ERA5\_combine =ERA5.sel(expver=1).combine\_first(ERA5.sel(expver=5)) ERA5\_combine.load() ERA5\_combine.to\_netcdf("[era5.tp.20200801.copy.nc](http://era5.tp.20200801.copy.nc)") from [https://unseen-open.readthedocs.io/\_/downloads/en/latest/pdf/](https://unseen-open.readthedocs.io/_/downloads/en/latest/pdf/) [Next page](https://forum.ecmwf.int/t/era5-cds-requests-which-return-a-mixture-of-era5-and-era5t-data/781.md?page=2)