Tigge downloads through ecds api taking too long

Datasets and Usage
1

Hi all,

I am a PhD researcher using the ECDS to retrieve historical TIGGE ECMWF ensemble forecasts for academic research. I have been experiencing several issues during large historical downloads and would like to ask whether these behaviors are expected or if there is a recommended strategy for this type of retrieval.

My dataset consists of:

  • Dataset: tigge-forecasts
  • Origin: ECMWF
  • Forecast type: perturbed forecast (all 50 members)
  • Variable: total precipitation
  • Initialization time: 12 UTC
  • Forecast lead times: 24 to 336 hours (D+1 to D+14)
  • Time period: 2007 to present
  • Spatial subset over Brazil (or smaller regions)

Initially, I tried multi-year requests, but they were rejected due to request size limitations. I then switched to monthly requests, which are generally accepted.

However, I have observed several issues:

  1. Some requests remain in “running” status for more than 24 hours without completing.
  2. Some requests disappear from the ECDS portal, after which the API returns 404 - job not found.
  3. I encountered MARS errors caused by damaged tapes (for example tape XD024500), which prevented retrieval of some historical dates. (when I last accessed the status page, this issue seems to be resolved, though I haven’t tested the particular dates for download again)
  4. During periods of heavy usage, I observed that only one TIGGE request appears to run simultaneously for my account, while additional requests remain indefinitely in accepted status.
  5. I also experienced temporary connection issues (RemoteDisconnected, DNS resolution failures, etc.), although I understand these may have been related to temporary service maintenance.

At the moment, I am using one worker (through cdsswarm) only and monthly requests, since this seems to be the most reliable approach.

I would appreciate your advice on the following questions:

  1. Is there a recommended request size for historical TIGGE retrievals (monthly, biweekly, daily, etc.)?
  2. Are requests that remain in “running” status for many hours likely to be stuck, or should they simply be left running?
  3. Is there a more efficient way to retrieve long historical TIGGE archives than the standard ECDS API?
  4. For damaged tapes such as XD024500, is there an estimated recovery schedule, or should those dates simply be retried periodically?

My goal is simply to download the complete historical archive for this single variable and ensemble configuration for scientific research.

Thank you very much for your help.