Memory Symphony: Orchestrating Heterogeneity for High-Performance Computing

Memory subsystem, as a pivotal element within HPC systems, influences both cost and performance. Despite persistent challenges such as the memory wall and memory bandwidth wall, ongoing advancements in new memory technologies, including non-volatile memories and high-bandwidth memories, present heterogeneous memory systems as a promising solution for realizing large-scale memory subsystems while meeting the requirements of cost, capacity, and performance. Heterogeneous memory often consists of multiple memory tiers exhibiting different characteristics. In recent years, heterogeneous memory systems have evolved from HBM-DRAM systems to later byte-addressable NVM-DRAM systems, and with the latest advancements in CXL-interconnected disaggregated memory tiers, significantly expanding the landscape. Yet, while hardware-managed caches have simplified programmability, their efficacy wanes when confronted with complex memory access patterns and large memory footprints, as seen in real-world applications. Thus, a growing focus shifts towards software-based data placement strategies, where a synthesis of workload insights and system-level understanding becomes crucial for performance optimization. This talk presents the memory utilization and patterns on existing leadership supercomputers, brings insights into the future design of heterogeneous memory systems, and then delves into approaches that harness the synergy between workload characteristics and system-level attributes for addressing the challenge in programming heterogeneous memory.