Introduction to the eFlows4HPC Software Stack and HPC Workflows as a Service Methodology

The tutorial will give an overview of the eFlows4HPC software stack and of the HPC Workflows as a Service (HPCWaaS) methodology. The eFlows4HPC project (https://eflows4hpc.eu/) aims to widen access to HPC to newcomers and to simplify the development, deployment and execution of complex workflows in HPC systems. It proposes to simplify this process in two ways. From one side, the eFlows4HPC software stack aims to provide the required functionalities to manage the workflow's lifecycle. On the other side, it introduces the HPCWaaS concept, which leverages the software stack to widen access to HPC systems. This service offering tries to bring the Function as a Service (FaaS) concept to the HPC environments hiding the complexity of an HPC Workflow execution to end users. The tutorial will be composed of two parts. First, it will present an overview of the project software stack. Then, it will introduce how workflows are developed in the project through three different components: computational aspects (PyCOMPSs), data transfers (Data Pipelines) and overall topology (TOSCA). The second part will be more practical, showing how to generate specific containers that leverage the HPC systems features (hands-on component), move data from repositories to the HPC systems, and deploy the workflows with Alien4Cloud.