I'm reading some NaN values for ERA5-Land Solar Surface Radiation Downwards on 2018/08/27
at location (32.3, -86.9) from 0 - 6 hours. I think I downloaded this data previously and
and it did not contain NaN values, so I was wondering if these are intended to be blank?
from GRIB file
--------------
In [8]: xarray.open_dataset("/tmp/solar_201808_32.3_-86.9.grib", engine="cfgrib").ssrd.sel(
latitude=32.3, longitude=-86.9, time="2018-08-27")
Out[8]:
<xarray.DataArray 'ssrd' (step: 24)>
array([9.998750e+03, nan, nan, nan, nan,
1.000000e+04, 1.000000e+04, 1.000000e+04, 1.000000e+04, 1.000000e+04,
1.000000e+04, 8.096800e+04, 6.839000e+05, 2.008996e+06, 4.061476e+06,
6.352708e+06, 9.096770e+06, 1.135856e+07, 1.347785e+07, 1.585578e+07,
1.767849e+07, 1.907706e+07, 2.003201e+07, 2.035096e+07], dtype=float32)
from netCDF file
----------------
In [20]: xarray.open_dataset("/tmp/solar_201808_32.3_-86.9.nc").ssrd.sel(
latitude=32.3, longitude=-86.9, time=pandas.date_range(datetime.date(2018, 8, 27),
datetime.date(2018, 8, 28), freq="H"))
Out[20]:
<xarray.DataArray 'ssrd' (time: 25)>
array([2.199908e+07, 1.014600e+04, nan, nan, nan,
nan, 1.014600e+04, 1.014600e+04, 1.014600e+04, 1.014600e+04,
1.014600e+04, 1.014600e+04, 8.107800e+04, 6.838120e+05, 2.009003e+06,
4.061529e+06, 6.352746e+06, 9.096953e+06, 1.135852e+07, 1.347785e+07,
1.585576e+07, 1.767860e+07, 1.907698e+07, 2.003212e+07, 2.035113e+07],
dtype=float32)
I am also seeing the NaN values in the first 6 hours of ERA5-Land data. Every single forecast, the first 6 hours have a handful of extra NaN values for SSRD. Has any reason for this been given?
hi Cathryn,
can you share the request you used to retrieve the data, please?
Thanks,
Michela
c.retrieve(
'reanalysis-era5-land',
{
'variable': [
'10m_u_component_of_wind', '10m_v_component_of_wind', '2m_dewpoint_temperature',
'2m_temperature', 'snow_depth', 'snowfall',
'snowmelt', 'surface_net_solar_radiation', 'surface_pressure',
'surface_solar_radiation_downwards', 'total_precipitation',
],
'year': 2010,
'month': 01,
'day': 02,
'time': [
'00:00', '01:00', '02:00',
'03:00', '04:00', '05:00',
'06:00', '07:00', '08:00',
'09:00', '10:00', '11:00',
'12:00', '13:00', '14:00',
'15:00', '16:00', '17:00',
'18:00', '19:00', '20:00',
'21:00', '22:00', '23:00'
],
'format': 'grib',
},
outputFileFinal)
Hi Cathryn, I ran your request for 3 variables (ssrd,tp,snsr) and the grib file contains
% cdo info test3vars.grib
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID
1 : 2010-01-02 00:00:00 0 6483600 4270737 : 0.0000 6.7921e+06 3.3273e+07 : 176.128
2 : 2010-01-02 00:00:00 0 6483600 4270737 : 0.0000 1.7855e+07 4.1079e+07 : 169.128
3 : 2010-01-02 00:00:00 0 6483600 4270737 : 0.0000 0.0016304 0.31887 : 228.128
4 : 2010-01-02 01:00:00 0 6483600 4270739 : 0.0000 1.5303e+05 3.7536e+06 : 176.128
5 : 2010-01-02 01:00:00 0 6483600 4270737 : 0.0000 5.6068e+05 3.9311e+06 : 169.128
6 : 2010-01-02 01:00:00 0 6483600 4270737 : 0.0000 5.8033e-05 0.020378 : 228.128
7 : 2010-01-02 02:00:00 0 6483600 4270740 : 0.0000 3.4324e+05 7.4105e+06 : 176.128
8 : 2010-01-02 02:00:00 0 6483600 4270737 : 0.0000 1.1801e+06 7.7778e+06 : 169.128
9 : 2010-01-02 02:00:00 0 6483600 4270737 : 0.0000 0.00011777 0.036021 : 228.128
10 : 2010-01-02 03:00:00 0 6483600 4270742 : 0.0000 5.7390e+05 1.0709e+07 : 176.128
11 : 2010-01-02 03:00:00 0 6483600 4270737 : 0.0000 1.8642e+06 1.1387e+07 : 169.128
12 : 2010-01-02 03:00:00 0 6483600 4270737 : 0.0000 0.00017909 0.046740 : 228.128
13 : 2010-01-02 04:00:00 0 6483600 4270743 : 0.0000 8.4012e+05 1.4075e+07 : 176.128
14 : 2010-01-02 04:00:00 0 6483600 4270737 : 0.0000 2.6066e+06 1.5278e+07 : 169.128
15 : 2010-01-02 04:00:00 0 6483600 4270737 : 0.0000 0.00024377 0.054088 : 228.128
16 : 2010-01-02 05:00:00 0 6483600 4270737 : 0.0000 1.1429e+06 1.7551e+07 : 176.128
17 : 2010-01-02 05:00:00 0 6483600 4270737 : 0.0000 3.4072e+06 1.8949e+07 : 169.128
18 : 2010-01-02 05:00:00 0 6483600 4270737 : 0.0000 0.00030913 0.069972 : 228.128
19 : 2010-01-02 06:00:00 0 6483600 4270737 : 0.0000 1.4840e+06 2.0565e+07 : 176.128
20 : 2010-01-02 06:00:00 0 6483600 4270737 : 0.0000 4.2655e+06 2.2111e+07 : 169.128
21 : 2010-01-02 06:00:00 0 6483600 4270737 : 0.0000 0.00037486 0.087644 : 228.128
22 : 2010-01-02 07:00:00 0 6483600 4270737 : 0.0000 1.8578e+06 2.3249e+07 : 176.128
23 : 2010-01-02 07:00:00 0 6483600 4270737 : 0.0000 5.1730e+06 2.5103e+07 : 169.128
24 : 2010-01-02 07:00:00 0 6483600 4270737 : 0.0000 0.00044155 0.098348 : 228.128
Could the software you are using to view the data be adding extra (empty) steps at the start of the file? can you please share an example of the results you are observing?
Thanks,
Kevin
Sorry I responded to this last week but I guess it didn't go through. Your output is showing the exact issue. I've bolded and underlined the issue with the differeing number of missing values at different time steps. Is your land mask not consistent for parameter 176.128? We need to subtract off the previous hour from the current hour in this data in order to get an hourly value, it causes NaN values when the number of NaNs differs between timesteps.
-1 : Date Time Level Gridsize Miss : Minimum Mean Maximum : Parameter ID 1 : 2010-01-02 00:00:00 0 6483600 4270737 : 0.0000 6.7921e+06 3.3273e+07 : 176.128 2 : 2010-01-02 00:00:00 0 6483600 4270737 : 0.0000 1.7855e+07 4.1079e+07 : 169.128 3 : 2010-01-02 00:00:00 0 6483600 4270737 : 0.0000 0.0016304 0.31887 : 228.128 4 : 2010-01-02 01:00:00 0 6483600 4270739 : 0.0000 1.5303e+05 3.7536e+06 : 176.128 5 : 2010-01-02 01:00:00 0 6483600 4270737 : 0.0000 5.6068e+05 3.9311e+06 : 169.128 6 : 2010-01-02 01:00:00 0 6483600 4270737 : 0.0000 5.8033e-05 0.020378 : 228.128 7 : 2010-01-02 02:00:00 0 6483600 4270740 : 0.0000 3.4324e+05 7.4105e+06 : 176.128 8 : 2010-01-02 02:00:00 0 6483600 4270737 : 0.0000 1.1801e+06 7.7778e+06 : 169.128 9 : 2010-01-02 02:00:00 0 6483600 4270737 : 0.0000 0.00011777 0.036021 : 228.128 10 : 2010-01-02 03:00:00 0 6483600 4270742 : 0.0000 5.7390e+05 1.0709e+07 : 176.128 11 : 2010-01-02 03:00:00 0 6483600 4270737 : 0.0000 1.8642e+06 1.1387e+07 : 169.128 12 : 2010-01-02 03:00:00 0 6483600 4270737 : 0.0000 0.00017909 0.046740 : 228.128 13 : 2010-01-02 04:00:00 0 6483600 4270743 : 0.0000 8.4012e+05 1.4075e+07 : 176.128 14 : 2010-01-02 04:00:00 0 6483600 4270737 : 0.0000 2.6066e+06 1.5278e+07 : 169.128 15 : 2010-01-02 04:00:00 0 6483600 4270737 : 0.0000 0.00024377 0.054088 : 228.128 16 : 2010-01-02 05:00:00 0 6483600 4270737 : 0.0000 1.1429e+06 1.7551e+07 : 176.128 17 : 2010-01-02 05:00:00 0 6483600 4270737 : 0.0000 3.4072e+06 1.8949e+07 : 169.128 18 : 2010-01-02 05:00:00 0 6483600 4270737 : 0.0000 0.00030913 0.069972 : 228.128 19 : 2010-01-02 06:00:00 0 6483600 4270737 : 0.0000 1.4840e+06 2.0565e+07 : 176.128 20 : 2010-01-02 06:00:00 0 6483600 4270737 : 0.0000 4.2655e+06 2.2111e+07 : 169.128 21 : 2010-01-02 06:00:00 0 6483600 4270737 : 0.0000 0.00037486 0.087644 : 228.128 22 : 2010-01-02 07:00:00 0 6483600 4270737 : 0.0000 1.8578e+06 2.3249e+07 : 176.128 23 : 2010-01-02 07:00:00 0 6483600 4270737 : 0.0000 5.1730e+06 2.5103e+07 : 169.128 24 : 2010-01-02 07:00:00 0 6483600 4270737 : 0.0000 0.00044155 0.098348 : 228.128