9–13 Sept 2024
Wanda Realm Beijing
Asia/Shanghai timezone
Pre-press Proceedings: https://proceedings.ihep.ac.cn/ibic2024/

Research on visualization and indexing of data based on the ELK stack

WEP50
11 Sept 2024, 14:20
1h 30m
China Hall 3

China Hall 3

Poster Presentation MC7: Data Acquisition and Processing Platforms WEP: Wednesday Poster Session

Speaker

Dr Yukun Li (Institute of High Energy Physics)

Description

This paper presents a comprehensive solution for the real-time collection and analysis of BPM telemetry data using Kafka and the ELK stack. It includes the transmission of PV variables from BPM electronic devices to the Kafka message queue, thus realizing a powerful and scalable data streaming process. By retrieving JSON formatted data from Kafka using the ELK stack, efficient data indexing and visualization in Kibana are achieved. The paper details the architectural design, implementation details, and the advantages of using Kafka as a BPM data dissemination center. This integration not only enhances the performance and reliability of the data processing pipeline but also provides physicists and engineers with powerful tools for the real-time visualization and monitoring of BPM data. Our approach has shown significant improvements in data accessibility, searchability, and real-time analytics, offering profound implications for future research and development in the instrumentation and diagnostics of particle accelerators.

I have read and accept the Privacy Policy Statement Yes

Primary author

Dr Yukun Li (Institute of High Energy Physics)

Co-authors

Jianshe Cao (Institute of High Energy Physics) Qiang Ye (Institute of High Energy Physics) Mr Yaoyao Du (Institute of High Energy Physics)

Presentation materials