Effective data management is crucial for accessible and usable research data. This presentation will describe the data infrastructure at the European XFEL, built on a four-layer storage architecture designed for high-throughput handling. The first layer, Online storage, acts as a high-speed cache for data rates up to 15 GB/s. The second layer, High-Performance Storage, supports real-time...
The ESRF has initiated an ambitious program to automate data processing and management across all 45 of its beamlines. Since the successful installation and commissioning of the ESRF-EBS new storage ring in 2020, there has been a significant increase in X-ray flux. A comprehensive strategy has been developed to optimize the use of the increasing quantities of data generated by the new advanced...
At the Square Kilometre Array Observatory (SKAO), monitoring data is ingested from distributed subsystems via the Tango Controls archiver, with attribute data stored in the Engineering Data Archive (EDA). The EDA uses a PostgreSQL database with the TimescaleDB extension, offering a performant solution for time-series storage. However, as SKAO infrastructure scales, PostgreSQL becomes...
Data is one of the key deliverables of the ITER machine.Since 2019, the ITER data handling system has gradually been extended to cope with the commissioning of new plant systems and new needs.This contribution gives an overview of the different sub-systems which compose the data handling ecosystem from data archivers to visualization tools.We will summarize the short- and long-term storage for...
As data volumes and complexity continue to rise at European XFEL, the need for integrated, sustainable metadata solutions continues to be critical. We introduce myMdC - a centralized metadata catalogue available at https://in.xfel.eu/metadata * - and highlight its key role in supporting the facility’s data management strategy.
In operation since the first day of...
