NEXTGenIO Systemware White Paper

This White Paper summarises NEXTGenIO’s work on creating and adapting schedulers (Slurm and NORNS), filesystems (GekkoFS and dataClay) and programming models (PyCOMPSs) to the NEXTGenIO architecture. All the software has been released and is available on the prototype.

In this document, we present an overview of the different components that have been developed, together with evaluations, conclusions and lessons learned. The White Paper describes the new functionality that was developed for Slurm, in particular around the use of NVRAM and workflows. The scheduler has been enhanced to be able to take advantage of the persistent memory (for example by scheduling related jobs on the same nodes).

The White Paper also introduces the NORNS data scheduler, designed to manage data transfers asynchronously between different backends. NORNS is fully integrated with SLURM.

Finally, we present the ad-hoc filesystem GekkoFS, which allows the creation of a distributed filesystem from the local space of each computing node, and the dataClay object store and its integration with PyCOMPSs. Thanks to this integration, it is possible for both Python and Java workloads to take advantage of NVRAM.