LUSTRE in HPC, BigData and AI: Status, New Features and Roadmap

Contributors:

• Frank Baetke (EOFS)
• Hugo Falter (ParTec AG)
• Olaf Gellert (Deutsches Klimarechenzentrum (DKRZ))
• Kevin Harms (Argonne National Laboratory)
• Peter Jones (Whamcloud/DDN)
• Jacques-Charles Lafoucriere (French Alternative Energies and Atomic Energy Commission)
• Sarah M. Neuwirth (Goethe-University Frankfurt)

---

Abstract:

LUSTRE is the leading open-source and open-development file system for HPC and now supports many HPC and AI/ML infrastructures even beyond its traditional stronghold in academia and scientific research. Around two-thirds of the top 100 supercomputers use LUSTRE. It is a community-developed technology with contributors from around the world and will become a key component of upcoming Exascale Systems. Beyond research and academic installations it has gained wide adoption in segments like financial services, the energy sector, manufacturing, life sciences and digital content creation. At the ISC 2021 DIGITAL BoF developers, administrators, users and solution providers will gather to discuss the current roadmap, new features, innovative developments and deployments as well as new challenges and the current status of the LUSTRE community that continues to drive LUSTRE forward. We will focus on the recent feature developments and discuss how they will shape the next years of LUSTRE. Examples of important and requested features are: FLR Erasure Coding, Pool Quota, Auto Restriping, Client Encryption etc. We will address benefits, experiences and issues up to LUSTRE 2.14.x but will also discuss features planned for 2.15 and future releases. The session will start with very short presentations about the community and its product LUSTRE. A key part will be a presentation of the LUSTRE roadmap from the current release up to the end of 2022 and related discussion topics with a special focus on features, requirements and challenges. We will also cover how to get involved as a LUSTRE developer/contributor.