UserError: Restricted access to ERA5T. Please, check that that your date selection is valid

Access and Login C3S - Access and Login
, ,
21

Just want to highlight that the current behaviour is also problematic when querying multiple years of data ending in the current time frame. Queries for multiple years would select months which are by definition not available, resulting in the above error. By default one can now not query multiple years which includes months in the current time frame which have not yet come to pass.

In a practical sense the new behaviour limits use to the previous year, upon which the last year's data has to be parsed separately. This is real troubling for any workflow which requires multiple years upon until fairly recent time frames.


In short, this query will work as it will query a month (January) which is available in both years.

{
  "area": [
    1,
    -1,
    -1,
    1
  ],
  "dataset_short_name": "reanalysis-era5-single-levels",
  "day": "01",
  "format": "netcdf",
  "month": "01",
  "product_type": "reanalysis",
  "target": "download.nc",
  "time": "00:00",
  "variable": "2m_temperature",
  "year": [
    "2021",
    "2022"
  ]
}


In this query I want data for the whole year of 2021 but in doing so, any data queried for 2022 will also include "unavailable" months and therefore this query fails.

{
  "area": [
    1,
    -1,
    -1,
    1
  ],
  "dataset_short_name": "reanalysis-era5-single-levels",
  "day": "01",
  "format": "netcdf",
  "month": [
    "01",
    "02",
    "03",
    "04",
    "05",
    "06",
    "07",
    "08",
    "09",
    "10",
    "11",
    "12"
  ],
  "product_type": "reanalysis",
  "target": "download.nc",
  "time": "00:00",
  "variable": "2m_temperature",
  "year": [
    "2021",
    "2022"
  ]
}



22

We could all get around this new problem if there was something like a "date_begin" and "date_end" set of keywords, instead of the less specific "year", "month", "day", "time" keywords.  For example, something like "date_begin": "2001-01-01 00:00".  "date_end": "2022-05-31 23:00".  Is there any such functionality already?

23

Hi everyone

Just to mention I am having the same issue as the other users above (getting the 'Restricted' error, error code -2) when trying to download data for 2022. Thanks ECMWF for your support.

24

Milana, many thanks, these are good ideas.  In the first example: Is there any limit to the number of separate dates that can be specified in one request (other than the total data size of the request)?  If not, this looks like a good approach.  The middle example would work, but sends a separate request for each date, so maybe not the best for a long date range.  In the third example: I was not aware of the "/" syntax in the date value.  I will experiment with that.  Again, thanks!

25

Dear users,

an issue was recently discovered with the way the 5 day embargo period was being applied to requests for ERA5 data from the CDS. This has now been resolved, however a consequence of this is that now any request which includes embargoed data will fail with an error message:

Mars server task finished in error; UserError: Restricted access to ERA5T. Please, check that your date selection is valid. For more information, visit https://climate.copernicus.eu/climate-reanalysis [mars]; Error code is -2; Request failed; Some errors reported (last error -2)

In the past, such request would have completed without the error message. Consequently, users will have to amend their requests to exclude any embargoed dates (i.e. any dates less than 5 days before the current date).
This change has been implemented in order to ensure access is as it should be, and we apologise for any inconvenience caused.

Regards

ECMWF Support

26

Dear Michela,

According to my tests the issue still persists, without ECMWF Support offering a solution. In short, as outlined above, if people use the CDS dataset browser to format their queries for a date range which includes multiple years (including a recent month) downloads will still fail by default. No provisions are made in the user interface, or warnings provided.

More so, the solutions which were proposed by users and DO WORK using the date: "YYYY-MM-DD/YYYY-MM-DD" syntax were forcibly removed! I consider this a huge disservice to those looking for a solution. Basically, for multi-year queries including the current year (a common instance I would think) things are broken STILL.

Please provide an actionable solutions,

Best,

Koen

27

Is there any update on this issue?

28

Hi All,

I discovered a CDS API feature that allows the user to make requests over multiple months/years without triggering this error. It's possible to specify a specific start and end date for the request, and so long as the end date is more than 5 days in the past, the request should work. Here is some example syntax using the Python cdsapi:


import cdsapi
cds = cdsapi.Client()
cds.retrieve(
    'reanalysis-era5-single-levels', {
        "area": [
            40.75,
            -74.25,
            40.5,
            -74
        ],
        "variable": "surface_runoff",
        "product_type": "reanalysis",
        "date": "2021-06-24/2022-06-24",
        'time': [
            '00:00', '01:00', '02:00',
            '03:00', '04:00', '05:00',
            '06:00', '07:00', '08:00',
            '09:00', '10:00', '11:00',
            '12:00', '13:00', '14:00',
            '15:00', '16:00', '17:00',
            '18:00', '19:00', '20:00',
            '21:00', '22:00', '23:00',
        ],
        "format": "grib"
    }, 'download.grib')

This request will retrieve the ERA5 surface runoff data from the previous year for a small area as a GRIB. Hope this helps!

29

Hi Evan,

just to say that the recommended way to specify dates in your CDS API request (CDS ERA5 datasets) is to use the  

'year', 'month', 'day'
keywords in your request, as using 'date' with a date range may give unexpected results.

Also, for this dataset, it is most efficient to request 1 month of data at a time (e.g. via a loop in the script)

Thanks,

Kevin

30

Hello Kevin,


The solutions with specifying the month and exact days need you to know in advance what are the latest dates of ERA5T that are really available. The previous behaviour in which we could request all days of the latest month and cdsapi would automatically return everything that is available was easy to get the latest dates that are really available.


Let's say I want to download all days and hours in july 2022 without knowing in advance what are the dates that are really available, is there any ways that it can be done?


Thank you,

31

Hi Francois,

No, now the only way to identify what ERA5T data are available is to look at the CDS Download data form to see the latest dates. 

Thanks

Kevin

32

Kevin, under this new behavior, it would be very helpful to have a simple API call that returns the latest date/time of available data for a particular data set.  Many of us have developed code within automated processes that downloads ERA5 data, so if the only way to determine available data is to click check boxes on the menu-driven download page, none of our codes can work.  Out of desperation, people might write code that requests every hour of the last 7 days or so and see which ones fail, and thereby determine the latest data available, but I don't think that would be good for the system.  I think a simple API call to determine date of latest data is what's needed.

33

I'd like to reiterate what Mark (and others) have said. The new system requires users to revamp their ETL codebases to either use unreliable/unpredictable workarounds (specifying by date instead of year/month/day) or overload the server with requests (Mark's out of desperation solution) for a very simple piece of information. This is ideal for nobody.


Given all this, I would like to humbly petition that you all implement a simple API call returning the latest final and provisional data availability. I appreciate this implies real work on ECMWF's side.

34

Hi Mark, Robert, 

Thanks for your feedback. This feature has been recorded as a C3S 'user requirement' and so will help determine the way the CDS developed in the future,

Hope that helps,

Kevin

35

Hi all,

Is there a way to download all the latest available data automatically? I think now it is difficult because the last available date/hour is unknown (for operational purposes with the API). I'm using R.

Thank you in advance!
Diego

36

Hi all,

The latest ERA5T available data can be calculated as below:

  • Take your local time, say CEST 23:09 on 18/8/22
  • Work out UTC hours, 21:09 on 18/8/22
  • Use the UTC time and date and minus 5 days, in this case, 21:09 on 13/8/22
  • ERA5T data is available up to 21 UTC 13/8/22

To tell the difference between ERA5 and ERA5T, you can always download data in GRIB and check the value of 'expver'.

Hope this helps.

Xiaobo


37

Hello all,

Are there any plans to add a check if data is ERA5 or ERA5T via the API? I realize you can check by downloading a file and reading the GRIB headers, however, knowing before downloading would be very helpful and save a lot of time. Especially when downloading large files. 


Thanks,

Travis

38

Hi Travis,

So far, you can download a small data in GRIB and check 'expver' in it. '1' means ERA5 and '5' means ERA5T.

ERA5 is normally around 3-month behind the real-time.

Hope this helps.

Xiaobo

39

Hello Yang,

I want to download the " [Mean temperature tendency due to short-wave radiation" and the “Mean temperature tendency due to long-wave radiation” in Table 13, and my codes are as follows:
import cdsapi

c = cdsapi.Client()

yearStart = 1979 # adjust to your requirements
yearEnd = 2022 # adjust to your requirements

years = range(yearStart, yearEnd+1)

for y in years:
target = ‘/mnt/nas/ERA5/monthly/1x1/LWHR/lwhr.’+str(y)+‘.1x1.grib’
#target = ‘/mnt/nas/ERA5/monthly/packed/T.’+str(y)+‘.0.25x0.25.v5.1.nc’ # specifies the output file name
print(target)
year = str(y)
date = year + ‘-01-01/’ + year + ‘-12-01’
print(date)
c.retrieve(
‘reanalysis-era5-complete’, #‘reanalysis-era5.1-pressure-levels-monthly-means’
{
“year”: year,
“levtype”: “ml”,
‘format’: ‘grib’,
“param”: “235002”,
“stream”: “moda”,
“type”: “fc”,
“date”: date,
# ‘grid’: ‘1.0/1.0’
},
target)

but there are some wrong information :
2024-07-03 21:28:29,625 INFO Welcome to the CDS
2024-07-03 21:28:29,625 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-complete
2024-07-03 21:28:29,847 INFO Request is queued
2024-07-03 21:28:38,753 INFO Request is failed
2024-07-03 21:28:38,753 ERROR Message: an internal error occurred processing your request
2024-07-03 21:28:38,753 ERROR Reason: mars - ERROR - 20240703.132835 - Mars server task finished in error
mars - ERROR - 20240703.132835 - AccessError: Restricted access to ERA5T. Please, check that your date selection is valid. For more information, visit Climate reanalysis | Copernicus [cdsfdb]
mars - ERROR - 20240703.132835 - Error code is -2
mars - ERROR - 20240703.132835 - Multibase error -2
mars - ERROR - 20240703.132835 - Request failed
mars - ERROR - 20240703.132835 - Some errors reported (last error -2)
2024-07-03 21:28:38,753 ERROR Traceback (most recent call last):
2024-07-03 21:28:38,753 ERROR File “/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py”, line 59, in handle_request
2024-07-03 21:28:38,753 ERROR result = cached(context.method, proc, context, context.args, context.kwargs)
2024-07-03 21:28:38,753 ERROR File “/opt/cdstoolbox/cdscompute/cdscompute/caching.py”, line 108, in cached
2024-07-03 21:28:38,753 ERROR result = proc(context, *context.args, **context.kwargs)
2024-07-03 21:28:38,753 ERROR File “/opt/cdstoolbox/cdscompute/cdscompute/services.py”, line 124, in call
2024-07-03 21:28:38,753 ERROR return p(*args, **kwargs)
2024-07-03 21:28:38,753 ERROR File “/opt/cdstoolbox/cdscompute/cdscompute/services.py”, line 60, in call
2024-07-03 21:28:38,753 ERROR return self.proc(context, *args, **kwargs)
2024-07-03 21:28:38,753 ERROR File “/home/cds/cdsservices/services/mars/mars.py”, line 53, in external
2024-07-03 21:28:38,753 ERROR return mars(context, request, **kwargs)
2024-07-03 21:28:38,753 ERROR File “/home/cds/cdsservices/services/mars/mars.py”, line 20, in mars
2024-07-03 21:28:38,753 ERROR execute_mars(context, requests, info)
2024-07-03 21:28:38,753 ERROR File “/home/cds/cdsservices/services/mars/execute_mars.py”, line 74, in execute_mars
2024-07-03 21:28:38,753 ERROR raise Exception(error_lines)
2024-07-03 21:28:38,754 ERROR Exception: mars - ERROR - 20240703.132835 - Mars server task finished in error
2024-07-03 21:28:38,754 ERROR mars - ERROR - 20240703.132835 - AccessError: Restricted access to ERA5T. Please, check that your date selection is valid. For more information, visit Climate reanalysis | Copernicus [cdsfdb]
2024-07-03 21:28:38,754 ERROR mars - ERROR - 20240703.132835 - Error code is -2
2024-07-03 21:28:38,754 ERROR mars - ERROR - 20240703.132835 - Multibase error -2
2024-07-03 21:28:38,754 ERROR mars - ERROR - 20240703.132835 - Request failed
2024-07-03 21:28:38,754 ERROR mars - ERROR - 20240703.132835 - Some errors reported (last error -2)
Traceback (most recent call last):
File “ERA5-monthly-longwave_heatingrate01.py”, line 33, in
target)
File “/home1/xuxr/.local/lib/python3.7/site-packages/cdsapi/api.py”, line 364, in retrieve
result = self._api(“%s/resources/%s” % (self.url, name), request, “POST”)
File “/home1/xuxr/.local/lib/python3.7/site-packages/cdsapi/api.py”, line 521, in _api
% (reply[“error”].get(“message”), reply[“error”].get(“reason”))
Exception: an internal error occurred processing your request. mars - ERROR - 20240703.132835 - Mars server task finished in error
mars - ERROR - 20240703.132835 - AccessError: Restricted access to ERA5T. Please, check that your date selection is valid. For more information, visit Climate reanalysis | Copernicus [cdsfdb]
mars - ERROR - 20240703.132835 - Error code is -2
mars - ERROR - 20240703.132835 - Multibase error -2
mars - ERROR - 20240703.132835 - Request failed
mars - ERROR - 20240703.132835 - Some errors reported (last error -2).

Could you give me some help please? Thank you very much!

40

Hi
Use the ERA5 catalogue to get the correct form for your request
https://apps.ecmwf.int/data-catalogues/era5/?axis_month=1979-01&axis_levelist=3&axis_param=235001&class=ea&stream=moda&expver=1&type=fc&decade=1970&levtype=ml
and click ‘view MARS request’ to see the CDS API script.
in your case, you do not need to include ‘year’ key/value pair as this generates an error, but you do need to include ‘levelist’ with the model levels you require