Compression for Scientific & Engineering Data

Large-scale numerical simulations, observations and experiments are generating very large datasets that are difficult to analyze, store and transfer. Data compression is an attractive and efficient technique to significantly reduce the size of scientific datasets. This tutorial reviews the state of the art in lossy compression of scientific datasets, presents a large diversity of use-cases, discusses in detail two lossy compressors (SZ and ZFP), and introduces compression error assessment metrics and tools (Z-Checker) to analyze the difference between original and decompressed datasets. The tutorial will offer hands-on exercises using SZ, ZFP and Z-checker. The tutorial addresses the following questions: Why lossy compression? How does lossy compression work? How measure and control compression error? The tutorial leverages use-cases of real-world compressors and scientific datasets to illustrate the different compression techniques and their performance. Participants will also have the opportunity to learn how to use SZ, ZFP and Z-Checker for their own datasets. The tutorial is given by two of the leading teams in this domain and targets primarily beginners interested in learning about lossy compression for scientific and engineering data. This half-day tutorial is improved from the evaluations of the highly rated tutorials given on this topic at ISC an SC.

---

Contributors:

• Franck Cappello (Argonne National Laboratory)
• Peter Lindstrom (Lawrence Livermore National Laboratory)
• Sheng Di (Argonne National Laboratory)