Speakers
Description
Language models are becoming increasingly relevant for accelerator operations, where they assist with common tasks such as retrieving historical data, preparing analysis scripts, and coordinating multi-step procedures. Strong scores on general-purpose benchmarks do not indicate how well a model maps operator jargon to facility-specific EPICS process variable~(PV) identifiers. Building on the semantic channel-finding benchmark, we evaluate chat-based language models on two tasks using 101 Advanced Light Source (ALS) expert query–PV pairs. The first probes query-level grounding via a leave-one-out protocol with varying inference-time cues, scored by minimum number of single-character edits (Levenshtein similarity). The second probes structural understanding by requiring the model to infer token–token adjacencies from the global naming-token vocabulary under prescribed edge-count budgets; we report precision, recall, F1, and Jaccard overlap.
Applied to 27 models, these evaluations separate PV retrieval from structural understanding of hierarchical naming patterns, and reveal the strong dependency of end-to-end PV identification on the naming conventions of the ALS control system.
Funding Agency
This work was supported by the Director of the Office of Science of the U.S.Department of Energy under Contract No. DEAC02-05CH11231.
| In which format do you inted to submit your paper? | LaTeX |
|---|---|
| Preprint marking on your proceeding paper | I wish my paper to be marked as preprint. |
| I no longer wish to present this contribution, please withdraw it. | Keep my contribution |