Publications

J. Kaiser1, C. Xu2, A. Eichler1, A. Santamaria Garcia2, O. Stein1, E. Bründermann2, W. Kuropka1, H. Dinter1, F. Mayet1, T. Vinatier1, F. Burkart1, H. Schlarb1 1Deutsches Elektronen-Synchrotron DESY, 2 Karlsruhe Institute of Technology KIT arXiv Abstract Online tuning of real-world plants is a complex optimisation problem that continues to require manual intervention by experienced human operators. Autonomous tuning is a rapidly expanding field of research, where learning-based methods, such as Reinforcement Learning-trained Optimisation (RLO) and Bayesian optimisation (BO), hold great promise for achieving outstanding plant performance and reducing tuning times. Which algorithm to choose in different scenarios, however, remains an open question. Here we present a comparative study using a routine task in a real particle accelerator as an example, showing that RLO generally outperforms BO, but is not always the best choice. Based on the study’s results, we provide a clear set of criteria to guide the choice of algorithm for a given tuning task. These can ease the adoption of learning-based autonomous tuning solutions to the operation of complex real-world plants, ultimately improving the availability and pushing the limits of operability of these facilities, thereby enabling scientific and engineering advancements. ...

X. Chen, X. Qi, C. Su, Y. He, Z. Wang, K. Sun, C. Jin, W. Chen, S. Liu, X. Zhao, D. Jia, M. Yi Chinese Academy of Sciences arXiv Abstract The superconducting linear accelerator is a highly flexiable facility for modern scientific discoveries, necessitating weekly reconfiguration and tuning. Accordingly, minimizing setup time proves essential in affording users with ample experimental time. We propose a trend-based soft actor-critic(TBSAC) beam control method with strong robustness, allowing the agents to be trained in a simulated environment and applied to the real accelerator directly with zero-shot. To validate the effectiveness of our method, two different typical beam control tasks were performed on China Accelerator Facility for Superheavy Elements (CAFe II) and a light particle injector(LPI) respectively. The orbit correction tasks were performed in three cryomodules in CAFe II seperately, the time required for tuning has been reduced to one-tenth of that needed by human experts, and the RMS values of the corrected orbit were all less than 1mm. The other transmission efficiency optimization task was conducted in the LPI, our agent successfully optimized the transmission efficiency of radio-frequency quadrupole(RFQ) to over 85% within 2 minutes. The outcomes of these two experiments offer substantiation that our proposed TBSAC approach can efficiently and effectively accomplish beam commissioning tasks while upholding the same standard as skilled human experts. As such, our method exhibits potential for future applications in other accelerator commissioning fields. ...

** Simon Hirlaender, Lukas Lamminger, Giovanni Zevi Della Porta, Verena Kain** Abstract Reinforcement learning (RL) is a promising direction in machine learning for the control and optimisation of particle accelerators since it learns directly from experience without needing a model a-priori. However, RL generally suffers from low sample efficiency and thus training from scracth on the machine is often not an option. RL agents are usually trained or pre-tuned on simulators and then transferred to the real environment. In this work we propose a model-based RL approach based on Gaussian processes (GPs) to overcome the sample efficiency limitation. Our RL agent was able to learn to control the trajectory at the CERN AWAKE (Advanced Wakefield Experiment) facility, a problem of 10 degrees of freedom, within a few interactions only. To date, numerical optimises are used to restore or increase and stabilise the performance of accelerators. A major drawback is that they must explore the optimisation space each time they are applied. Our RL approach learns as quickly as numerical optimisers for one optimisation run, but can be used afterwards as single-shot or few-shot controllers. Furthermore, it can also handle safety and time-varying systems and can be used for the online stabilisation of accelerator operation.This approach opens a new avenue for the application of RL in accelerator control and brings it into the realm of everyday applications. ...

F. M. Velotti1, B. Goddard1, V. Kain1, R. Ramjiawan1, G. Z. Della Porta1 and S. Hirlaender2 1CERN, 2University of Salzburg Machine Learning: Science and Technology Abstract To improve the performance-critical stability and brightness of the electron bunch at injection into the proton-driven plasma wakefield at the AWAKE CERN experiment, automation approaches based on unsupervised machine learning (ML) were developed and deployed. Numerical optimisers were tested together with different model-free reinforcement learning (RL) agents. In order to avoid any bias, RL agents have been trained also using a completely unsupervised state encoding using auto-encoders. To aid hyper-parameter selection, a full synthetic model of the beamline was constructed using a variational auto-encoder trained to generate surrogate data from equipment settings. This paper describes the novel approaches based on deep learning and RL to aid the automatic setup of a low energy line, as the one used to deliver beam to the AWAKE facility. The results obtained with the different ML approaches, including automatic unsupervised feature extraction from images using computer vision are presented. The prospects for operational deployment and wider applicability are discussed. ...

X. Chen, Y. Jia, X. Qi, Z. Wang, Y. He Chinese Academy of Sciences Physical Review Accelerators and Beams Abstract Recently, reinforcement learning (RL) algorithms have been applied to a wide range of control problems in accelerator commissioning. In order to achieve efficient and fast control, these algorithms need to be highly efficient, so as to minimize the online training time. In this paper, we incorporated the beam position monitor trend into the observation space of the twin delayed deep deterministic policy gradient (TD3) algorithm and trained two different structure agents, one based on physical prior knowledge and the other using the original TD3 network architecture. Both of the agents exhibit strong robustness in the simulated environment. The effectiveness of the agent based on physical prior knowledge has been validated in a real accelerator. Results show that the agent can overcome the difference between simulated and real accelerator environments. Once the training is completed in the simulated environment, the agent can be directly applied to the real accelerator without any online training process. The RL agent is deployed to the medium energy beam transport section of China Accelerator Facility for Superheavy Elements. Fast and automatic orbit correction is being tested with up to ten degrees of freedom. The experimental results show that the agents can correct the orbit to within 1 mm. Moreover, due to the strong robustness of the agent, when a trained agent is applied to different lattices of different particles, the orbit correction can still be completed. Since there are no online data collection and training processes, all online corrections are done within 30 s. This paper shows that, as long as the robustness of the RL algorithm is sufficient, the offline learning agents can be directly applied to online correction, which will greatly improve the efficiency of orbit correction. Such an approach to RL may find promising applications in other areas of accelerator commissioning. ...

D. Meier1, L. V. Ramirez1, J. Völker1, J. Viefhaus1, B. Sick2, G. Hartmann1 1Helmholtz-Zentrum Berlin, 2University of Kassel Physical Review Accelerators and Beams Abstract Superconducting photoelectron injectors are promising for generating highly brilliant pulsed electron beams with high repetition rates and low emittances. Experiments such as ultrafast electron diffraction, experiments at the Terahertz scale, and energy recovery linac applications require such properties. However, optimizing the beam properties is challenging due to the high number of possible machine parameter combinations. This article shows the successful automated optimization of beam properties utilizing an already existing simulation model. To reduce the required computation time, we replace the costly simulation with a faster approximation with a neural network. For optimization, we propose a reinforcement learning approach leveraging the simple computation of the derivative of the approximation. We prove that our approach outperforms standard optimization methods for the required function evaluations given a defined minimum accuracy. ...

L. Grech1, G. Valentino1, D. Alves2 and Simon Hirlaender3 1University of Malta, 2CERN, 3University of Salzburg Frontiers in Physics Abstract The Beam-Based Feedback System (BBFS) was primarily responsible for correcting the beam energy, orbit and tune in the CERN Large Hadron Collider (LHC). A major code renovation of the BBFS was planned and carried out during the LHC Long Shutdown 2 (LS2). This work consists of an explorative study to solve a beam-based control problem, the tune feedback (QFB), utilising state-of-the-art Reinforcement Learning (RL). A simulation environment was created to mimic the operation of the QFB. A series of RL agents were trained, and the best-performing agents were then subjected to a set of well-designed tests. The original feedback controller used in the QFB was reimplemented to compare the performance of the classical approach to the performance of selected RL agents in the test scenarios. Results from the simulated environment show that the RL agent performance can exceed the controller-based paradigm. ...

J. Kaiser, O. Stein, A. Eichler Deutsches Elektronen-Synchrotron DESY 39th International Conference on Machine Learning Abstract In recent work, it has been shown that reinforcement learning (RL) is capable of solving a variety of problems at sometimes super-human performance levels. But despite continued advances in the field, applying RL to complex real-world control and optimisation problems has proven difficult. In this contribution, we demonstrate how to successfully apply RL to the optimisation of a highly complex real-world machine – specifically a linear particle accelerator – in an only partially observable setting and without requiring training on the real machine. Our method outperforms conventional optimisation algorithms in both the achieved result and time taken as well as already achieving close to human-level performance. We expect that such automation of machine optimisation will push the limits of operability, increase machine availability and lead to a paradigm shift in how such machines are operated, ultimately facilitating advances in a variety of fields, such as science and medicine among many others. ...

N. Madysa, V. Kain, R. Alemany Fernandez, N. Biancacci, B. Goddard, F. M. Velotti CERN 13th Particle Accelerator Conference Abstract High intensities in the Low Energy Ion Ring (LEIR) at CERN are achieved by stacking several multi-turn injec- tions from the pre-accelerator Linac3. Up to seven consec- utive 200 μs long, 200 ms spaced pulses are injected from Linac3 into LEIR. Two inclined septa, one magnetic and one electrostatic, combined with a collapsing horizontal or- bit bump allows a 6-D phase space painting via a linearly ramped mean momentum along the Linac3 pulse and in- jection at high dispersion. The already circulating beam is cooled and dragged longitudinally via electron cooling (e- cooling) into a stacking momentum to free space for the fol- lowing injections. For optimal intensity accumulation, the electron energy and trajectory need to match the ion energy and orbit at the e-cooler section. ...

T. Boltz Karlsruhe Insitute of Technology KIT PhD thesis Abstract At the time this thesis is written, the world finds itself amidst and partly in the process of recovering from the COVID-19 pandemic caused by the SARS-Cov-2 virus. One major contribution to the worldwide efforts of bringing this pandemic to an end are the vaccines developed by different research teams all around the globe. Produced in a remarkably short time frame, a crucial first step for the discovery of these vaccines was mapping out the atomic structure of the proteins making up the virus and their interactions. Due to the bright X-rays required in the process, synchrotron light sources play an active role in the ongoing efforts of accomplishing that goal. Synchrotron light sources are particle accelerators that are capable of providing intense electromagnetic radiation by accelerating packages of electrons, called bunches, and forcing them on curved trajectories. Besides the support of research on the SARS-Cov-2 virus, the remarkable properties of synchrotron radiation lead to a multitude of applications in a variety of scientific fields such as materials science, geology, biology and medicine. As a special form of synchrotron radiation, this thesis is concerned with the coherent synchrotron radiation (CSR) generated by short electron bunches in a storage ring. At wavelengths larger than the size of the emitting electron structure, the particles within a bunch radiate coherently. This coherent emission of synchrotron radiation scales with the number of involved particles and can thus enhance the intensity of the emitted radiation by several orders of magnitude. As a consequence, modern synchrotron light sources, such as the Karlsruhe Research Accelerator (KARA) at the Karlsruhe Institute of Technology (KIT), are deliberately operating with short bunch lengths to extend the radiated CSR spectrum to higher frequencies and to increase the intensity of the emitted radiation. Yet, the continuous reduction of the bunch length at high beam intensities eventually leads to complex longitudinal dynamics caused by the self-interaction of the electron bunches with their own emitted CSR. This phenomenon, generally referred to as micro-bunching or micro-wave instability, can lead to the formation of dynamically changing micro-structures within the charge distribution of the electron bunches and thus to a uctuating emission of CSR. Moreover, it can cause oscillations of the bunch length and the energy spread, which can be detrimental to the operation of a synchrotron light source. On the other hand, as electron structures smaller than the full electron bunch, the micro-structures created by the instability lead to an increased emission of CSR at frequencies up to the THz frequency range. The instability can thus also be beneficial for a variety of applications that rely on intense radiation in that particular frequency range. ...