NEXTGenIO was represented and the 2018 Cray Users' Group Meeting on 24th May 2018. Michèle Weiland, the project manager, took part in the panel entitled 'Future of HPC Storage and IO' with others from across the industry. The panel abstract is given below.
The future landscape of HPC storage and I/O is rapidly evolving as new hardware devices such as flash and nonvolatile memory and new interconnects such as NVMeoF and Omni-Path become economically viable. Using these technologies, the longtime albatross of checkpointing seems to have finally been solved with various burst buffer systems. With this challenge resolved, users are increasingly enabled to focus on analysis routines which tend to have an increased demand for IOPS as opposed to bandwidth. A wider perspective considering entire workflows of scientific computing has emerged and the long dominance of simulation is diminishing.
At the same time, the largest storage systems in the world are now being deployed by commercial cloud and other hyperscale vendors, driving new requirements for storage and I/O that are dissimilar with the needs of traditional HPC but perhaps synergistic with these HPC emerging analysis workloads. As the exascale era approaches, the HPC industry must now leverage the lessons learned from cloud/hyperscale storage systems and the opportunities presented by new solid-state storage media to continue to scale.
This panel discussion will explore how new paradigms in I/O might address the needs of future exascale workloads. Have burst buffers fully solved the checkpointing problem and is analysis emerging as an equally important challenge? Is POSIX truly a dead end for scalable I/O, or can various tricks and consistency relaxations give parallel file systems another decade? If HPC must move beyond POSIX, how can lessons learned from hyperscale object stores be applied to moving us forward? And what role does I/O middleware play in bridging the semantic gap between application I/O and the underlying storage system? These questions will guide this panel discussion to provide future prospects in I/O software, systems, and hardware.