What AI-driven Data Movement can bring to the HPC World!

This presentation addresses the following topic(s):

• Machine learning and HPC: Each worthy of its own investment, or better together? (Alternately: Marriage for the ages, or heading for divorce?)
• What under-appreciated aspects of data management are the real secrets to scalable computing?

Managing datasets is often considered as time consuming. Moving large number of files or petabyte volumes of data can quickly turn into a nightmare if done without method. Artificial Intelligence (AI) and machine learning (ML) can help to get out of the “rsync ice age” and enter an age where data management is data-aware or data-driven.

In the period of hybrid and multi-cloud infrastructures, it has never been more important to understand your datasets and usage patterns. Their analysis opens the way to promising solutions and value extraction. Here are a few examples to illustrate what we are working on and the benefits for HPC:

- Massive dataset migrations: thanks to analytics and AI we can analyze each dataset usage and apply the best data migration strategy and achieve optimal storage cutover. Different approaches can be implemented depending on storage sources, data types, dataset analytics, …

- Data-driven data management: pre-positioning of files at a specific location with the help of machine learning to deliver the highest possible performances: o When you have 100PB+ on tape, prepositioning, deduping, encapsulating small files make all the difference, o When you have 100PB+ on Cloud: migrating to the nearest node, managing multi-cloud, moving scratch compute results, managing dark data etc.

Machine Learning (ML) and Artificial Intelligence (AI) applied to complex data movement technical tasks quickly deliver concrete improvements in term of performances, efficiency and reduce overall complexity for end-users. These technologies are not merely concepts; Atempo provides solutions to unlock complex data management problems.