Dear ECMWF Copernicus
At Open-Meteo, I am tackling similar challenges with a larger range of ERA5 variables and additional datasets, including ERA5-Land, CERRA, ECMWF IFS HRES, and various national weather service datasets.
To maximize efficiency, I opted against traditional formats like NetCDF, HDF5, or Zarr, instead developing a novel chunked, compressed format optimized for time-series compression. Benefits include significantly smaller data sizes and faster compression/decompression speeds. The downside is that it’s not yet a widely adopted format, but we are actively developing Python, Rust, Swift, and TypeScript WASM libraries. Open-Meteo is fully open-data on AWS and open-source.
Your engineers are likely encountering similar challenges with handling massive datasets, IO bottlenecks, storage constraints, and memory limitations—issues that will only intensify with ERA5-Land and the upcoming ERA6. If you’re interested in collaborating on converting the entire Copernicus database into a chunked time-series architecture, feel free to reach out!