Modern Mixed-Precision Methods: Hardware Perspectives, Algorithms, Kernels, and Solvers

This tutorial will guide the audience through the rapidly expanding landscape of mixed- and multi- precision methods. The ongoing cross-pollination between high-performance computing (HPC) and machine learning (ML) is leading to intelligent computational steering of large-scale simulations. More importantly for this tutorial, sharing of the hardware platforms and exploiting their wide range of computational modes has led to proliferation of multiple representations of floating-point data. What followed was an increased interest in new methods that exploit them. Against the backdrop of high-performance libraries produced by internet-scale companies, hardware vendors, national laboratories, and academic institutions, we will show the recent algorithmic progress in exploiting multiple precisions for increased efficiency in achieved performance, required communication, and/or optimized storage needs. The techniques presented in the tutorial employ floating-point representations such as limited precision, quantized integers, and modular precision ecosystems, among others. Note that the lossless or lossy compression techniques are not covered as their algorithmic and accuracy considerations are substantially divergent to mixed-precision aspects. A portion of this tutorial will cover the fundamentals of the HPC software development in order to introduce the audience to some of the low-level details of coding for multiple precisions on modern hardware, including accelerators. The hardware focus of the tutorial will feature floating- point representation and performance of the recent supercomputing and industrial computing CPUs and accelerators, as well as less mainstream devices meant for more specific tasks at constraint power envelopes.