Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O

In HPC I/O middleware like the Adaptable I/O System (ADIOS) often mediates data transfers between applications. The metadata I/O generated by such systems often presents significant scaling and performance limitations. This work seeks improvement opportunities for metadata I/O by leveraging the DAOS storage systems, a recent storage system solution deployed on high-end systems such as the Aurora supercomputer. We investigate the tradeoffs and the design space for integrating I/O engines for the ADIOS middleware based on the different storage mechanisms supported by DAOS. We present a new DAOS-Array-ChunkSize-Aligned engine which provides up to 2.3x improved performance than when using the existing DAOS-POSIX interface, without requiring any applications modifications.

---

Contributors:

• Greg Eisenhauer
• Norbert Podhorszki
• Dmitry Ganyushin
• Ada Gavrilovska