Conveners
THCR MC16 Data Management and Analytics
- Gary Croke (Thomas Jefferson National Accelerator Facility)
- Chris Roderick (European Organization for Nuclear Research)
Effective data management is crucial for accessible and usable research data. This presentation will describe the data infrastructure at the European XFEL, built on a four-layer storage architecture designed for high-throughput handling. The first layer, Online storage, acts as a high-speed cache for data rates up to 15 GB/s. The second layer, High-Performance Storage, supports real-time...
The ESRF has initiated an ambitious program to automate data processing and management across all 45 of its beamlines. Since the successful installation and commissioning of the ESRF-EBS new storage ring in 2020, there has been a significant increase in X-ray flux. A comprehensive strategy has been developed to optimize the use of the increasing quantities of data generated by the new advanced...
At the Square Kilometre Array Observatory (SKAO), monitoring data is ingested from distributed subsystems via the Tango Controls archiver, with attribute data stored in the Engineering Data Archive (EDA). The EDA uses a PostgreSQL database with the TimescaleDB extension, offering a performant solution for time-series storage. However, as SKAO infrastructure scales, PostgreSQL becomes...
Data is one of the key deliverables of the ITER machine.Since 2019, the ITER data handling system has gradually been extended to cope with the commissioning of new plant systems and new needs.This contribution gives an overview of the different sub-systems which compose the data handling ecosystem from data archivers to visualization tools.We will summarize the short- and long-term storage for...
Large-scale research facilities such as the European XFEL rely on multiple distributed services to manage complex experimental workflows. Consistent, reliable and timely metadata exchange between these services is essential to ensure automation and data integrity.
This paper presents myMdC, the centralised metadata backbone that supports scientific data services in European XFEL. myMdC...
