A framework for Better ORGanized Earth System model data - BORGES

Climate modeling increasingly strives for higher resolutions in its simulations in space and, such, in time. Accordingly, the amount of data handled and produced by competitive Earth system models is ever increasing, posing an increasing challenge to the hardware and software used in this context. We here introduce the architecture for the large-scale data system for Better ORGanized Earth System model Data (BORGES).

BORGES is being built around the modern, competitive Earth system model, ICON, at the Max Planck Insitute for Meteorology in Hamburg especially for large ensembles of experiments as well as storm-resolving model runs. In a classical HPC context the system retrieves large data streams and the associated metadata from the model. For High-Performance Data Analytics these streams are distributed to further post-processing proccesses. For storing the data stream BORGES builts on the domain-sepecific object storage Field DataBase, which has been developed at the European Centre for Medium-Range Weather Forcast. The data system thereby makes sure that metadata is stored consistently with the stream data in classical databases. As part of a research workflow, it thereby has to account for the variety of needs by a large number of users. Accordingly, in its data flow BORGES tries to find an efficient compromise between a centralized database architecture and computing jobs distributed on the HPC cluster.

This project is part of the Weather, Climate, Hydrological and Farming Pilot of the ACROSS (HPC Big DAta ArtifiCial intelligence across Stack PlatfoRM TOwardS ExaScale) EuroHPC project.

---

Contributors:

• Reinhard Budich (Max-Planck-Institut für Meteorologie)
• Luis Kornblueh (Max-Planck-Institut für Meteorologie)
• Jairo Alonso Segura Bermudez (Max-Planck-Institut für Meteorologie)
• Sven Willner (Max-Planck-Institut für Meteorologie)