# Why it is much slower downloading from 'reanalysis-era5-complete'?

**URL:** <https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471>\
**Category:** C3S - Access and Login\
**Tags:** api, c3s, era5\
**Created:** [19 June 2019 14:00 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471 "2019-06-19T14:00:35Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\
**Post date:** [19 June 2019 14:00 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/1 "2019-06-19T14:00:35Z")

</div>

As title, when compared with downloading other ERA5 datasets from the CDS.

This is because data request for 'reanalysis-era5-complete'&nbsp; will retrieve data from ECMWF MARS, a tape based archiving system. While other ERA5 datasets on the CDS are hosted on the CDS itself, which is disk based. Notice the volume of ERA5 is so huge that it is possible to only copy over most popular data from ECMWF MARS to the CDS.

- [ERA5 hourly data on single levels from 1979 to present](https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview)
- [ERA5 hourly data on pressure levels from 1979 to present](https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels?tab=overview)
- [ERA5 monthly averaged data on single levels from 1979 to present](https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels-monthly-means?tab=overview)
- [ERA5 monthly averaged data on pressure levels from 1979 to present](https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-pressure-levels-monthly-means?tab=overview)

---

<div class="post-metadata">

**Author:** ![Mikhail\_Pichugin](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/m/97f17d/32.png) [@Mikhail\_Pichugin](https://forum.ecmwf.int/u/Mikhail_Pichugin)\
**Post date:** [20 June 2019 01:30 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/2 "2019-06-20T01:30:38Z")

</div>

My data request for 'reanalysis-era5-complete' (I need temperature and humidity for 137 levels on 1 Jan 2015) is in the Queued status for 46 hours. Is this acceptable?

---

<div class="post-metadata">

**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\
**Post date:** [20 June 2019 08:14 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/3 "2019-06-20T08:14:19Z")

</div>

Hi Mikhail,

Could you let me have your request ID or your CDS user ID so that I can have a detailed look at your request? You can contact us by sending an email to copernicus-support AT ecmwf DOT int.

Thank you,

Xiaobo

---

<div class="post-metadata">

**Author:** ![Michael\_Shaw](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@Michael\_Shaw](https://forum.ecmwf.int/u/Michael_Shaw)\
**Post date:** [8 July 2019 15:29 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/4 "2019-07-08T15:29:45Z")

</div>

Is it safe to assume the same is true for&nbsp;reanalysis-era5-single-levels?&nbsp; It takes on order days to download just a month of runoff, for example.

---

<div class="post-metadata">

**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\
**Post date:** [8 July 2019 15:36 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/5 "2019-07-08T15:36:41Z")

</div>

Hi Michael,

It should not be. In fact we have received reports from several users and our technical team is looking into the problem now.

I'll keep you updated.

Thank you,

Xiaobo

---

<div class="post-metadata">

**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\
**Post date:** [8 July 2019 15:51 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/6 "2019-07-08T15:51:48Z")

</div>

Just to confirm&nbsp;reanalysis-era5-single-levels data is stored on the CDS disks.

---

<div class="post-metadata">

**Author:** ![Michell\_Fontenelle\_G](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/m/e79b87/32.png) [@Michell\_Fontenelle\_G](https://forum.ecmwf.int/u/Michell_Fontenelle_G)\
**Post date:** [30 July 2019 14:46 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/7 "2019-07-30T14:46:16Z")

</div>

I'm having the same problems... For instance, to download 1 day is taking sometimes several hours. The same problems are happening with single levels. Is it some kind of internal problem? Is there any kind of estimation when it'll be fixed?

---

<div class="post-metadata">

**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\
**Post date:** [30 July 2019 15:02 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/8 "2019-07-30T15:02:38Z")

</div>

Michell, you should see better performance downloading single level ERA5 data. Let me know if that's not the case. Thank you.

---

<div class="post-metadata">

**Author:** ![Xiaolong\_Liu](https://forum.ecmwf.int/user_avatar/forum.ecmwf.int/xiaolong_liu/32/67_2.png) [@Xiaolong\_Liu](https://forum.ecmwf.int/u/Xiaolong_Liu)\
**Post date:** [12 August 2019 10:57 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/9 "2019-08-12T10:57:30Z")

</div>

Hi Xiaobo,

I request the dataset&nbsp;'reanalysis-era5-complete' recently. Normally it will cost about 90 minutes when I request one-month data. While my last request has timed out seriously.&nbsp;This is my request ID:&nbsp;Request ID: e5ff686a-d3cc-4004-8e07-4f3287ec9881 and my UID is 19396. It already lasts 25 hours showed in my requests list.&nbsp; Can you check for me?

Thank you.&nbsp;

---

<div class="post-metadata">

**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\
**Post date:** [12 August 2019 12:53 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/10 "2019-08-12T12:53:46Z")

</div>

Hi Xiaolong,

You can now check status of our system by going to [https://cds.climate.copernicus.eu/live/queue](https://cds.climate.copernicus.eu/live/queue). If you log into the CDS, you should be able to see your requests with the URL. Let me know if this helps.

Kind regards,

Xiaobo

---

<div class="post-metadata">

**Author:** ![Xiaolong\_Liu](https://forum.ecmwf.int/user_avatar/forum.ecmwf.int/xiaolong_liu/32/67_2.png) [@Xiaolong\_Liu](https://forum.ecmwf.int/u/Xiaolong_Liu)\
**Post date:** [12 August 2019 13:05 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/11 "2019-08-12T13:05:17Z")

</div>

Hi, it is still in process and already running more than 1day.

 ![](https://forum.ecmwf.int/uploads/default/original/1X/374d08f74ca4485ed6460d74881c39e0454c7684.png)

---

<div class="post-metadata">

**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\
**Post date:** [12 August 2019 13:46 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/12 "2019-08-12T13:46:23Z")

</div>

Hi Xiaolong,

This depends on how busy our system is. For example, at the moment I type, I can see 30 requests are sharing the resources the same as 'reanalysis-era5-complete', with 473 requests queuing.

Whenever possible, you are recommended to download ERA5 data which is hosted on the CDS physically. This includes all ERA5 datasets you can see from the CDS catalogue. In general, if you do not need model level data, you should avoid requesting data from 'reanalysis-era5-complete'.

I hope this helps.

Xiaobo

---

<div class="post-metadata">

**Author:** ![Xiaolong\_Liu](https://forum.ecmwf.int/user_avatar/forum.ecmwf.int/xiaolong_liu/32/67_2.png) [@Xiaolong\_Liu](https://forum.ecmwf.int/u/Xiaolong_Liu)\
**Post date:** [12 August 2019 14:00 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/13 "2019-08-12T14:00:23Z")

</div>

Hi Xiaobo,

Yeah, I know that it only costs a few minutes when downloading the single-level data. It seems what I can do is just waiting. Sorry, I posted a wrong Request ID.&nbsp;

Thanks for your response anyway.

Xiaolong

---

<div class="post-metadata">

**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\
**Post date:** [12 August 2019 14:32 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/14 "2019-08-12T14:32:27Z")

</div>

No problem and thank you for your patience.

---

<div class="post-metadata">

**Author:** ![X\_DW](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/41988e/32.png) [@X\_DW](https://forum.ecmwf.int/u/X_DW)\
**Post date:** [6 April 2022 13:31 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/15 "2022-04-06T13:31:50Z")

</div>

Hi, Xiaobo

My requests have stayed here for several hours, And In this page the reason for queuing is Unable to establish. How can i make my request run again. Thank you.

[Climate Data Store | Live (copernicus.eu)](https://cds.climate.copernicus.eu/live/queue)

And can I speed up my data retrieving by add more 'time' on my request per time. Since every time I add my 'time' to more than one, The error message will show:

Exception: the request you have submitted is not valid. Expected 1, got 137.; Request failed; Some errors reported (last error -1).

---

<div class="post-metadata">

**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\
**Post date:** [12 August 2019 13:50 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/16 "2019-08-12T13:50:26Z")

</div>

BTW, I checked requestid 'e5ff686a-d3cc-4004-8e07-4f3287ec9881' and noticed it was completed.

---

<div class="post-metadata">

**Author:** ![Michael\_Shaw](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@Michael\_Shaw](https://forum.ecmwf.int/u/Michael_Shaw)\
**Post date:** [11 December 2019 17:49 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/17 "2019-12-11T17:49:47Z")

</div>

Hello.

I am still seeing that pulling hourly runoff from cds/Copernicus is potentially prohibitively slow.&nbsp; Perhaps one year of data per day.&nbsp; Yet it varies, and apparently at least in part as a function of user load/demand.

And, it occasionally hangs up, seemingly somewhat at random.

Please advise?

Here is a sample script (which appears to work just fine through sometimes a few months, then I get a timeout, sometimes through years, then I get a timeout…).&nbsp; I follow it with a sample error I have gotten after, e.g., a timeout.

"import cdsapi

c = cdsapi.Client()

def is\_leap\_year(year):

&nbsp; &nbsp; return (year % 4 == 0) and (year % 100 != 0) or (year % 400 == 0)

def days\_in\_month(month):

&nbsp; &nbsp; if month == 1 or month == 3 or month == 5 or month == 7 or month == 8 or month == 10 or month == 12:

&nbsp; &nbsp; &nbsp; &nbsp; return 31

&nbsp; &nbsp; elif month == 2:

&nbsp; &nbsp; &nbsp; &nbsp; if is\_leap\_year(year):

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; return 29

&nbsp; &nbsp; &nbsp; &nbsp; else:

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; return 28

&nbsp; &nbsp; else:

&nbsp; &nbsp; &nbsp; &nbsp; return 30

for year in range(1979,2020):

&nbsp;&nbsp; for mon in range(1,13):

&nbsp; &nbsp; &nbsp; for day in range(1,days\_in\_month(mon)+1):

&nbsp; &nbsp; &nbsp; &nbsp; if (day \< 10):

&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; theday="0"+str(day)

&nbsp; &nbsp; &nbsp; &nbsp; else:

&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; theday=str(day)

&nbsp; &nbsp; &nbsp; &nbsp; if (mon \< 10):

&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; themon="0"+str(mon)

&nbsp; &nbsp; &nbsp; &nbsp; else:

&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; themon=str(mon)

&nbsp; &nbsp; &nbsp; &nbsp; for time in range(0,24):

&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; if (time \< 10):

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; thetime="0"+str(time)

&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; else:

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; thetime=str(time)

&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; print("Year, month, day, time: ",str(year),themon,theday,thetime)

&nbsp;&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; c.retrieve("reanalysis-era5-single-levels", {

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "product\_type": &nbsp; "reanalysis",

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "format": &nbsp; &nbsp; &nbsp; &nbsp; "netcdf",

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "variable": &nbsp; &nbsp; &nbsp; "runoff",

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "year": &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; str(year),

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "month":&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; themon,

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "day":&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; theday,

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; "time": &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; thetime

&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; }, "output."+str(year)+str(themon)+str(theday)+str(thetime)+".nc")"

Error:

“requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='cds.climate.copernicus.eu', port=443): Read timed out. (read timeout=60)”

Sorry if I'm doing something wrong!&nbsp; Thanks much in advance for any advice.

Best,

Michael

---

<div class="post-metadata">

**Author:** ![Michael\_Shaw](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@Michael\_Shaw](https://forum.ecmwf.int/u/Michael_Shaw)\
**Post date:** [12 December 2019 18:50 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/18 "2019-12-12T18:50:02Z")

</div>

This&nbsp; **appears** &nbsp;to now be working without any issues, now.&nbsp; Wondering if you made a change on your end?&nbsp; Whereas yesterday when I submitted multiple instances (to get multiple temporal chunks simultaneously) it seemed to slow each instance down (potentially to a timeout and crash of scripts),&nbsp; **seems** &nbsp; now that all instances are running along at ok rates, now.

Thanks for whatever you may have done on your end...and for your responsiveness either way!

---

<div class="post-metadata">

**Author:** ![Michael\_Shaw](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/m/8edcca/32.png) [@Michael\_Shaw](https://forum.ecmwf.int/u/Michael_Shaw)\
**Post date:** [13 December 2019 20:51 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/19 "2019-12-13T20:51:05Z")

</div>

As it turns out, I'm still seeing arbitrary snags, timeouts, and crashes of long term data pulls.

Anyone have a solution to pull the whole hourly archive of ERA5 runoff which doesn't timeout, drop files, etc?

Thanks very much in advance.

---

<div class="post-metadata">

**Author:** ![Xiaobo\_Yang](https://forum.ecmwf.int/letter_avatar_proxy/v4/letter/x/e495f1/32.png) [@Xiaobo\_Yang](https://forum.ecmwf.int/u/Xiaobo_Yang)\
**Post date:** [16 December 2019 09:07 UTC](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471/20 "2019-12-16T09:07:25Z")

</div>

Hi Michael,

I'll have a look and then get back to you - unfortunately this will take a while.

Kind regards,

Xiaobo

[Next page](https://forum.ecmwf.int/t/why-it-is-much-slower-downloading-from-reanalysis-era5-complete/1471.md?page=2)