The Future of Benchmarks in Supercomputing

As supercomputing welcomes new workflows of simulations, data science and artificial intelligence in the Exascale era, the goal of this workshop is to pose, engage, debate, and address the question - "How should the supercomputing community evolve performance benchmarks?". The workshop will be organized as presentations and panel discussions with audience participation that will invite active members of the Top500, HPCG, MLPerf, TeraSort, etc. and key personnel from industry, academia, and government to discuss the value, need and desire for evolving the benchmark suite that is inclusive and accommodative of emerging applications to guide future supercomputing system design and architecture. This workshop is a follow-on to a well-attended (>100 participants) birds-of-a-feather session at SC23 under the same name. Stalwarts from the community presented their viewpoints, and a significant majority of participants expressed a desire to continue deeper conversations on the following topics - (i) Design of benchmarks (representative, cost-conscious, etc.), (ii) Choice of metrics (energy efficiency, I/O inclusive, etc.), (iii) Agility to evolve (problem size, newer kernels, mini-apps, containerized harness etc.), and (iv) Articulation of purpose (ranking list, co-design, marketing). This workshop will organize presentations and panels to further the community engagement around these topics.