Modern Mixed-Precision Methods: Hardware Perspectives, Algorithms, Kernels, and Solvers

This tutorial will expose the audience to the rapidly expanding landscape of mixed- and multi- precision methods. The ongoing cross-pollination between high-performance computing (HPC) and machine learning (ML) is leading to intelligent computational steering of large-scale simulations. More importantly for this tutorial, sharing of the hardware platforms and exploiting their wide range of computational modes has led to proliferation of multiple representations of floating-point data— and increased interest in new methods that exploit them. Against the backdrop of high-performance libraries produced by internet-scale companies, hardware vendors, national laboratories, and academic institutions, we will show the recent algorithmic progress in exploiting multiple precisions for increased efficiency in performance, communication, and/or storage. The techniques presented in the tutorial employ floating-point representations such as limited precision, quantized integers, and modular precision ecosystems. Note that lossless or lossy compression techniques are not covered. A portion of this tutorial will cover the fundamentals of the HPC software development in order to introduce the audience to some of the low-level details of coding for multiple precisions on modern hardware, including accelerators. The hardware focus of the tutorial will feature floating-point representation and performance of the recent supercomputing and industrial computing CPUs and accelerators, as well as less mainstream devices meant for more specific tasks at limited power envelopes.