Distributed HPC Workflows with Covalent

Workflow orchestration in the field of high performance computing has increased in popularity in the last few years. This is due to a confluence of trends facilitating, and sometimes requiring distributed computing at scale. On one hand, individual supercomputing centers are beginning to reach scaling limits due to excessive power consumption, and it is increasingly common for users to have access to multiple clusters. On the other hand, the introduction of cloud-based quantum computing has made distributed computing and hybrid-cloud computing mainstream to new audiences previously unfamiliar with cloud-based computing. We discuss how users can leverage workflow orchestration tools like Covalent to overcome the operational challenges of distributed high performance computing at scale. Covalent allows users to deploy software to a variety of compute backends, interfacing with workload managers such as Kubernetes, Slurm, AWS/Azure/GCP Batch, as well as with the latest quantum computers using a hybrid-quantum scheduler. Covalent manages tasks across distributed systems, and also gives insight into how, where, and when tasks execute through a powerful user interface. This allows users to rapidly redeploy workloads, pause and restart dispatches, and even define dynamic workflows whose execution patterns are determined at runtime.