Developing Robust and Scalable Next Generation Workflows Applications and Systems

Workflow applications are critical to scientific discovery. Technology trends and the convergence of traditional High Performance Computing (HPC) with new simulation, analysis, and machine learning (ML) approaches provides unprecedented opportunities. Traditional approaches to workflow applications and systems development have scalability and robustness limits. The ExaWorks project is building a robust workflows SDK with robust and high-performance technologies as well as well-defined and scalable component interfaces which can be leveraged by new and existing workflow applications and systems.

This tutorial will present the ExaWorks SDK, and its constituent components: Flux, Parsl, RADICAL-Cybertools (RCT), and Swift/T. These components are widely used, and available tools for developing workflow applications. This tutorial will outline modern workflow motifs on HPC platforms (e.g., ensemble campaigns, ML-in-the-loop), illustrate science examples of these motifs, and discuss solutions using the ExaWorks SDK. One third of the tutorial is dedicated to presentations from experts, and two thirds are dedicated to hands-on exercises. Attendees will gain practical knowledge to develop best workflow practices to manage large-scale campaigns on the largest supercomputers. At the end of the tutorial, they will be able to apply these tools and techniques to their advanced workflows with minimal programming effort.

---

Contributors:

• Daniel Laney (Lawrence Livermore National Laboratory)
• Kyle Chard (Argonne National Laboratory)
• Shantenu Jha (Brookhaven National Laboratory)
• Justin Wozniak (Argonne National Laboratory)
• Rafael Ferreira da Silva (Oak Ridge Nationational Laboratory)
• Ben Clifford (CQX Limited)
• Andre Merzky (Brookhaven National Laboratory)