Hi all,
Over the past few days, I’ve been noticing frequent 503 errors when requesting AIFS data from the s3://ecmwf-forecasts S3 bucket.
aws s3 cp --no-sign-request s3://ecmwf-forecasts/20251101/00z/aifs-ens/0p25/enfo/README.txt ~/Downloads/README.txt
gives back
download failed: s3://ecmwf-forecasts/20251101/00z/aifs-ens/0p25/enfo/README.txt to Downloads/README.txt An error occurred (SlowDown) when calling the GetObject operation (reached max retries: 2): Please reduce your request rate.
More complicated data requests are even less likely to succeed. Is there an overwhelming user request volume on that bucket?
Thanks,
Evan
Hi Evan,
This data storage is in charge of AWS, so we at ECMWF have no visibility if there is a bigger than usual demand on the service.
If it is of any help, when the download fails from AWS, you can try downloading directly from ECMWF, using opendata api Python client: GitHub - ecmwf/ecmwf-opendata: A package to download ECMWF open data
I hope this helps!
Milana
@Evan_Schechter , I was having similar issues that I first noticed on 30 Oct. Some discussion is here ECMWF AWS issues? (Oct 30th) · blaylockbk/Herbie · Discussion #487 · GitHub .
As an alternative to AWS, Google also appears to keep a copy of IFS/AIFS Open Data. See [1]. It’s now available as a “source” in Herbie (GitHub - blaylockbk/Herbie: Download numerical weather prediction datasets (HRRR, RAP, GFS, IFS, etc.) from NOMADS, NODD partners (Amazon, Google, Microsoft), ECMWF open data, and the University of Utah Pando Archive System.).
[1] I’m a new user here, so I can only include 2 links in a post. If you follow the first link I posted above, my latest comment has a link to PR #489, which has details on Google’s copy of the data.