Tiago Quintino (ECWMF) gave this presentation at PASC19, Zürich, Switzerland (12-14 June 2019).
Abstract: Starting 2014, ECMWF has embarked on a 10 year research programme on HPC Scalability, aiming to achieve Exascale numerical weather prediction systems by 2025. ECMWF operational forecast generates massive amounts of I/O in short bursts, accumulating to tens of TiB in hourly windows. From this output, millions of user-defined daily products are generated and disseminated to member states and commercial clients all over the world. These products are processed from the raw output of the IFS model, within the time critical path and under strict delivery schedule. Upcoming rise in resolution and growing popularity will increase both the size and number of these products. Based on expected model resolution upgrades, by 2020 we estimate the operational model will output over 100 TiB/day and need to archive over 400 TiB/day. Given that the I/O workload is already one of the strongest bottlenecks in ECMWF's workflow, this is one of the main challenges to reach Exascale NWP. We present the latest ECMWF developments in model I/O, product generation and storage, and how we are reworking our operational workflows to adapt to forthcoming new architectures and memory-storage hierarchies. In particular, we present recent developments in the integration of a domain-specific object store to ECMWF's operational pipeline.
(Authors: Tiago Quintino, Simon Smart, Baudouin Raoult - all ECMWF)
