Optimizing MARS request for historical HRES

Software and Tools
,
1

Hello,

I’m retrieving several years of historical wave forecast data from the operational MARS archive (class=od, type=fc, stream=wave, expver=1).
My workflow sends one request per day × run (00/12) × region. Each request extracts an area and stores the output as NetCDF.

Example structure:

for date in dates:

    for time in TIMES:

        for region in REGIONS:

            server.execute(req, filename)

I cannot group multiple runs (00,12) in the same request because NetCDF output is causing conflicts when both runs contain overlapping forecast steps.

As you can imagine, the process becomes very slow. And if I request multiple days at once, MARS starts reading more tapes, which becomes even slower.

Any recommendation for a more efficient and faster request strategy?
At the current speed, downloading multiple years is not feasible.

Thanks!

2

Hi Manuel,

Yes, downloading multiple years can be slow.

A few things you can do to make it a bit faster:

  • Download global data and both 00/12 runs in one request as GRIB files.
  • Then you can split the file locally to 00 and 12 using ecCodes, earthkit or Metview (or some other tool)
  • You can also locally crop to areas you are interested in using earthkit or Metview
  • and convert to netCDF in the end (if you use ecCodes’ grib_to_netcdf command it will be exactly the same MARS is using to do it)

You should avoid getting the data from more tapes in one request, you have discovered this already.

You should also avoid hitting the same tape multiple times, as not only this is not efficient, MARS will also punish you for this, and if you do this frequently, your ‘efficiency’ will go down and you will be queuing for longer.

You can find some hints on efficient MARS request in the documentation as well: Guidelines to write efficient MARS requests - User Documentation - ECMWF Confluence Wiki

I hope this helps,
Milana

3

Hi Milana,

Thanks a lot for the tips on making MARS requests more efficient. This really helps. I’ll apply your suggestions and check again the documentation you shared.