ECMWF MARS retrieve speed(archived operational forecast data)

Datasets and Usage
1

I am downloading ECMWF MARS archived operational forecast data with python WebAPI. The downloading process is fine but the speed is very slow with pressure level data. When I download Surface data (270M per file) it takes only 2 minutes, while when I download PL data (~400M per file) it takes 2 hours. The downloading speed is high (about 5M/s) but after each request has been active, it takes hours to start downloading. This speed is not that low at the beginning, but it takes longer and longer. Therefore, Could anyone tell me if there is any way to make it faster? Can I download data with lower resolution?

248c6c3d22c228e5b6d4088433b456a
248c6c3d22c228e5b6d4088433b456a1872×790 149 KB

#!/usr/bin/env python
from ecmwfapi import ECMWFService
import os
#setup the date
server = ECMWFService("mars")
year = '2022'
month = '01'
#day=['01','02','03']
day = ['01']
step = [f"{i:03}" for i in range(0, 241, 6)]

for k in day:
    print(k+"th day")
    for s in step:
        server.execute(
            {
            "class": "od",
            "date": year+month+k,
            "expver": "1",
            "levtype": "sfc",
            "param": "129.128/165.128/166.128/167.128/168.128/172.128/134.128/151.128/235.128/31.128/34.128/141.128/139.128/170.128/183.128/236.128/39.128/40.128/41.128/42.128/33.128", 
            "step": s,
            "stream": "oper",
            "time":"00:00:00",
            "type": "fc",
            },
            "ECMWF_SFC_"+year+month+k+"00_"+s+".grb")

        server.execute(
            {
            "class": "od",
            "date": year+month+k,
            "expver": "1",
            "levtype": "pl",
            "levelist":"1/2/3/5/7/10/20/30/50/70/100/150/200/250/300/400/500/600/700/800/850/900/925/950/1000",
            "param": "129.128/130.128/157.128/131.128/132.128", 
            "step": s,
            "stream": "oper",
            "time":"00:00:00",
            "type": "fc",
            },
            "ECMWF_PL_"+year+month+k+"00_"+s+".grb")
2

Hello,

The speed of processing of your request will depend on many factors, not just the size of the data that is downloaded.

First thing that I can see, in both requests you should not loop over steps. Your data is on tapes so in this case one forecast date (start date) is on one tape, so long story short, you should put all your steps in one request.
When you hit the same tape many times during one day, your requests will be given lower priority and you might need to wait longer.

Second thing. One reason why your PL request takes longer is because of the wind fields. U and V (and 10U and 10V) are not stored in MARS. They are calculated from vorticity and divergence. So in the first request you request only 10u and 10v, so it needs to be calculated once. However in second request you ask for u and v for each PL so they need to be calculated many more times, and this is what takes time.

I hope this helps.

Best regards,
Milana