Speaker
Description
Large language models are becoming increasingly relevant for accelerator operations, where they assist with common tasks like retrieving historical data, preparing analysis scripts, and coordinating multi-step procedures. At the Advanced Light Source (ALS), these operators use their personal jargon (e.g. “sector 4 beam current”) to search for the correct PV name from numerous channels, resulting in countless variations of naming conventions. Strong scores on general-purpose benchmarks do not indicate how well a model maps operator jargon to facility-specific EPICS process variable~(PV) identifiers. Building on the semantic channel-finding benchmark, we evaluate chat-based large language models on two tasks using 101 ALS expert query–PV pairs. The first probes query-level grounding via single-item testing. The assessment is executed with varying inference-time cues, scored by character-wise correspondence (Levenshstein ratio). The second probes structural understanding by requiring the model to infer character-sequence mapping from the global naming-token vocabulary under prescribed edge-count budgets. We report precision, recall, combined retrieval score (F1), and token overlap (Jaccard similarity). Applied to 27 models, these evaluations split PV retrieval from structural understanding of hierarchical naming patterns, and offer strong dependency of end-to-end PV identification on the ALS control system's naming conventions.
Funding Agency
This work was supported by the Director of the Office of Science of the U.S.Department of Energy under Contract No. DEAC02-05CH11231.
| In which format do you inted to submit your paper? | LaTeX |
|---|---|
| Preprint marking on your proceeding paper | I wish my paper to be marked as preprint. |
| I no longer wish to present this contribution, please withdraw it. | Keep my contribution |