Tiago Quintino (ECMWF) gave this presentation at the Fifth Symposium on High Performance Computing for Weather, Water, and Climate at the 99th Annual Meeting of the AMS American Meteorological Society, Phoenix, Arizona, USA, 6-10 January 2019.
A recording of the presentation can be found here: https://ams.confex.com/ams/2019Annual/recordingredirect.cgi/id/49927?entry_password=null&uniqueid=Paper349781
Abstract:
ECMWF's operational forecast generates massive I/O in short bursts, currently approaching 100 TiB per day, in two hour-long windows. From this output, millions of user-defined daily products are generated and disseminated to member states and commercial clients all over the world.
Currently, the IFS model and the product generation system use the HPC parallel file-system as their prefered I/O systems. In addition, research experiments and climate model runs rely on parallel file-system for their temporary storage, before archival to the tape systems.
As ECMWF aims to achieve Exascale NWP by 2025, we expect to handle around 1 PiB of model data per day and generate 100's of millions daily products. This poses a strong challenge to a complex workflow that is already facing I/O bottlenecks.
To help tackle this challenge, ECMWF has developed a high-performance distributed object-store that manages the model output, for the needs of our NWP and Climate simulations, making data available via scientific meaningful requests.
We will present how ECMWF is leveraging this technology to address current performance issues in our operations, while at the same time preparing for technology changes in the hardware and system landscape and the convergence between HPC and Cloud provisioning.
(Authors: Tiago Quintino, Simon Smart, James Hawkes, and Baudouin Raoult - all of ECMWF, Reading, U.K)