EGI Federated HPC Service
Tuesday, June 10, 2025 3:00 PM to Thursday, June 12, 2025 4:00 PM · 2 days 1 hr. (Europe/Berlin)
Foyer D-G - 2nd floor
Project Poster
Community EngagementHeterogeneous System ArchitecturesHPC in the Cloud and HPC Containers
Information
Poster is on display.
EGI is federation of computing and storage resource providers united by a mission of delivering advanced computing and data analytics services for research and innovation. From advanced HTC and Cloud computing to Data spaces and data analytics tools, EGI changes the way researchers open and innovate their science, and operate in digital environments.
EGI has set up a Working Group dedicated to creating a new service offer to enable and facilitate access to federated HPC providers within the EGI infrastructure. This effort involves defining policies and developing technical solutions in several areas, including: federated user access; monitoring and accounting of the HPC providers; portable execution of container-based applications across cloud, HTC, and HPC environments; and establishing resource brokering and allocation practices for international use cases.
This poster presents with initial technical architecture of the EGI HPC service, designed to simplify the execution of containerised workloads and enable offloading them from cloud-native environments to HPC systems. Based on Kubernetes—the widely adopted standard for container management in cloud environments— and the interLink software, it provides an abstraction layer that enables running pods on remote resources. It offers an easy-to-use interface that can be customised by the HPC systems to align with the provider's internal policies and configurations. Two integration modes are foreseen, both offering the same functionality towards the user, but with varying degree of integration of the HPC provider so the service can adapt to the existing policies and restrictions of each site.
Relying on Kubernetes allows users to exploit existing data science frameworks that are already Kubernetes-ready, like JupyterHub or Kubeflow which can offload heavy computation to the remote HPC site without major changes. A JupyterHub setup running on the cloud spawning individual notebook servers on a EuroHPC site (Vega) has been setup and demonstrated the technical feasibility of the service.
EGI's HPC Federation Working Group welcomes new Infrastructure providers, User communities
and Technical providers to join and contribute to shaping the EGI service. Come and join us!
Contributors:
EGI is federation of computing and storage resource providers united by a mission of delivering advanced computing and data analytics services for research and innovation. From advanced HTC and Cloud computing to Data spaces and data analytics tools, EGI changes the way researchers open and innovate their science, and operate in digital environments.
EGI has set up a Working Group dedicated to creating a new service offer to enable and facilitate access to federated HPC providers within the EGI infrastructure. This effort involves defining policies and developing technical solutions in several areas, including: federated user access; monitoring and accounting of the HPC providers; portable execution of container-based applications across cloud, HTC, and HPC environments; and establishing resource brokering and allocation practices for international use cases.
This poster presents with initial technical architecture of the EGI HPC service, designed to simplify the execution of containerised workloads and enable offloading them from cloud-native environments to HPC systems. Based on Kubernetes—the widely adopted standard for container management in cloud environments— and the interLink software, it provides an abstraction layer that enables running pods on remote resources. It offers an easy-to-use interface that can be customised by the HPC systems to align with the provider's internal policies and configurations. Two integration modes are foreseen, both offering the same functionality towards the user, but with varying degree of integration of the HPC provider so the service can adapt to the existing policies and restrictions of each site.
Relying on Kubernetes allows users to exploit existing data science frameworks that are already Kubernetes-ready, like JupyterHub or Kubeflow which can offload heavy computation to the remote HPC site without major changes. A JupyterHub setup running on the cloud spawning individual notebook servers on a EuroHPC site (Vega) has been setup and demonstrated the technical feasibility of the service.
EGI's HPC Federation Working Group welcomes new Infrastructure providers, User communities
and Technical providers to join and contribute to shaping the EGI service. Come and join us!
Contributors:
Format
On DemandOn Site
