Accelerating HPC and AI with DAOS Storage

DAOS (see https://docs.daos.io/) is an open-source scale-out object store designed from the ground up to deliver extremely high bandwidth/IOPS and low latency I/Os to the most demanding data-intensive workloads. It aims at supporting nextgen HPC workflows combining simulation, big data and AI in a single storage tier. DAOS presents a rich and scalable storage interface that allows efficient storage of both structured and unstructured data. DAOS supports multiple application interfaces including a parallel filesystem, Hadoop/Spark connector, TensorFlow-IO, native Python dictionary bindings, HDF5, MPI-IO as well as domain-specific data models like SEGY. Many DAOS deployments are underway including a 230PB/25TB/s installation connected to the ALCF’s Aurora system and a 1PB DAOS system for LRZ’s SuperMUC-NG phase 2.

This BoF will be the opportunity for members of the DAOS community to share experience running and using DAOS and also brainstorm on possible enhancements (e.g. new OS support, storage management, GPU acceleration, AI framework integration) that should be considered for future DAOS versions. The current roadmap will be presented to the audience and short lightning talks from various community members will be used to spark the discussion on a broad set of topics.

---

Contributors:

• Johann Lombardi (Intel Corp.)
• Kevin Harms (ALCF)
• Michael Hennecke (Intel (Deutschland) GmbH)
• Mohamad Chaarawi (Intel Corporation)
• Adrian Jackson (Edinburgh Parallel Computing Centre (EPCC))