Parallel Works Activate for Enterprise Research and Collaboration: Weather Model Workflow Case Study

Enterprise research and development teams that use high performance computing (HPC) and machine learning (ML) often encounter difficulties collaborating due to the need to span many different storage and compute resources (i.e. on-premise batch-scheduler clusters, cloud service providers, and virtualized environments) as well as empower practitioners with a wide range of backgrounds and skills. Parallel Works ACTIVATE provides a single control plane for HPC and ML that allows teams to share both the resources and the workflows that automate these complex tasks. This unique confluence is a highly configurable foundation for a huge variety of HPC and ML applications while also enforcing cost and security guardrails and providing interactive (i.e. graphical) support for teams. Workflows include establishing remote desktop and notebook sessions for individual use but can be extended to the orchestration of large scale HPC such as the launching of large MPI simulations that span multiple compute nodes. Here, we bring the potential to accelerate collaboration via Parallel Works ACTIVATE to life with a demonstration of how this platform simplifies the running and analysis of a weather model (WRF v4.5) across a team. The weather model and all its files are publicly available; a central public repository acts as a workflow plug-in for ACTIVATE. The workflow leverages open source tooling (Spack) to bootstrap its software stack via an environment as code approach. The result is that the workflow is easy to run on different cloud service providers. Over the course of this presentation, ACTIVATE is configured to host a group to share resources and track the near-real-time cloud costs. Then, ACTIVATE is used to launch and monitor the simulation, transfer the simulation outputs to another collaborator’s resource, interactively visualize the results, and finally, track the cost of the simulation. While the majority of this presentation involves manual, interactive usage of ACTIVATE, we also discuss briefly how ACTIVATE workflows can be launched in a fully programmatic fashion via API calls.