Accelerating Generative AI with PyTorch

Generative Artificial Intelligence (Generative AI) has emerged as a transformative force in creating realistic data, images, and content. By leveraging deep learning (DL) techniques, generative AI models learn patterns and structures from large datasets and autonomously produce novel outputs without explicit programming for each possible outcome. Even though these models are usually pre-trained on massive datasets, fine-tuning for specific tasks or performing inference still requires considerable number of computational resources, necessitating strategic optimizations to unlock their full potential. This tutorial, powered by EuroCC 2, aims to equip participants with a comprehensive understanding of the computational challenges inherent in Generative AI. Moreover, it seeks to present strategies for identifying and mitigating bottlenecks, exclusively leveraging PyTorch’s native features without resorting to additional languages like C++ or CUDA. The tutorial encompasses a diverse array of topics, including hands-on sessions on setting up and optimizing a generic DL pipeline on a supercomputer, an exploration of mainstream Generative AI models, and optimization strategies for inference specifically tailored to Generative AI. Additionally, it investigates advanced subjects such as sparsification and model parallelism for models with a large number of parameters. By the end of this tutorial, participants will be capable at navigating the challenges of accelerating Generative AI using the powerful tools and techniques inherent in PyTorch.