University of Trieste

N. Bruchon University of Trieste PhD thesis Abstract The research carried out in particle accelerator facilities does not concern only particle and condensed matter physics, although these are the main topics covered in the field. Indeed, since a particle accelerator is composed of many different sub-systems, its proper functioning depends both on each of these parts and their interconnection. It follows that the study, implementation, and improvement of the various sub-systems are fundamental points of investigation too. In particular, an interesting aspect for the automation engineering community is the control of such systems that usually are complex, large, noise-affected, and non-linear. ...

F. H. O’Shea1, N. Bruchon2, G. Gaio1 1Elettra Sincrotrone Trieste, 2University of Trieste Physical Review Accelerators and Beams Abstract In this article we report on the application of a model-free reinforcement learning method to the optimization of accelerator systems. We simplify a policy gradient algorithm to accelerator control from sophisticated algorithms that have recently been demonstrated to solve complex dynamic problems. After outlining a theoretical basis for the functioning of the algorithm, we explore the small hyperparameter space to develop intuition about said parameters using a simple number-guess environment. Finally, we demonstrate the algorithm optimizing both a free-electron laser and an accelerator-based terahertz source in-situ. The algorithm is applied to different accelerator control systems and optimizes the desired signals in a few hundred steps without any domain knowledge using up to five control parameters. In addition, the algorithm shows modest tolerance to accelerator fault conditions without any special preparation for such conditions. ...

S. Hirlaender1, N. Bruchon2 1University of Salzburg, 2University of Trieste arXiv Abstract Reinforcement learning holds tremendous promise in accelerator controls. The primary goal of this paper is to show how this approach can be utilised on an operational level on accelerator physics problems. Despite the success of model-free reinforcement learning in several domains, sample-efficiency still is a bottle-neck, which might be encompassed by model-based methods. We compare well-suited purely model-based to model-free reinforcement learning applied to the intensity optimisation on the FERMI FEL system. We find that the model-based approach demonstrates higher representational power and sample-efficiency, while the asymptotic performance of the model-free method is slightly superior. The model-based algorithm is implemented in a DYNA-style using an uncertainty aware model, and the model-free algorithm is based on tailored deep Q-learning. In both cases, the algorithms were implemented in a way, which presents increased noise robustness as omnipresent in accelerator control problems. ...

V. Kain1, S. Hirlander1, B. Goddard1, F. M. Velotti1, G. Z. Della Porta1, N. Bruchon2, G. Valentino3 1CERN, 2University of Trieste, 3University of Malta Physical Review Accelerators and Beams Abstract Numerical optimization algorithms are already established tools to increase and stabilize the performance of particle accelerators. These algorithms have many advantages, are available out of the box, and can be adapted to a wide range of optimization problems in accelerator operation. The next boost in efficiency is expected to come from reinforcement learning algorithms that learn the optimal policy for a certain control problem and hence, once trained, can do without the time-consuming exploration phase needed for numerical optimizers. To investigate this approach, continuous model-free reinforcement learning with up to 16 degrees of freedom was developed and successfully tested at various facilities at CERN. The approach and algorithms used are discussed and the results obtained for trajectory steering at the AWAKE electron line and LINAC4 are presented. The necessary next steps, such as uncertainty aware model-based approaches, and the potential for future applications at particle accelerators are addressed. ...

N. Bruchon1, G. Fenu1, G. Gaio2, M. Lonza2, F. H. O’Shea2, F. A. Pellegrino1, E. Salvato1 1University of Trieste, 2Elettra Sincrotrone Trieste Electronics Abstract Optimal tuning of particle accelerators is a challenging task. Many different approaches have been proposed in the past to solve two main problems—attainment of an optimal working point and performance recovery after machine drifts. The most classical model-free techniques (e.g., Gradient Ascent or Extremum Seeking algorithms) have some intrinsic limitations. To overcome those limitations, Machine Learning tools, in particular Reinforcement Learning (RL), are attracting more and more attention in the particle accelerator community. We investigate the feasibility of RL model-free approaches to align the seed laser, as well as other service lasers, at FERMI, the free-electron laser facility at Elettra Sincrotrone Trieste. We apply two different techniques—the first, based on the episodic Q-learning with linear function approximation, for performance optimization; the second, based on the continuous Natural Policy Gradient REINFORCE algorithm, for performance recovery. Despite the simplicity of these approaches, we report satisfactory preliminary results, that represent the first step toward a new fully automatic procedure for the alignment of the seed laser to the electron beam. Such an alignment is, at present, performed manually. ...

N. Bruchon, G. Fenu, G. Gaio, M. Lonza, F. A. Pellegrino, E. Salvato University of Trieste 23rd International Conference on Mechatronics Technology Abstract The optimization of particle accelerators is a challenging task, and many different approaches have been proposed in years, to obtain an optimal tuning of the plant and to keep it optimally tuned despite drifts or disturbances. Indeed, the classical model-free approaches (such as Gradient Ascent or Extremum Seeking algorithms) have intrinsic limitations. To overcome those limitations, Machine Learning techniques, in particular, the Reinforcement Learning, are attracting more and more attention in the particle accelerator community. The purpose of this paper is to apply a Reinforcement Learning model-free approach to the alignment of a seed laser, based on a rather general target function depending on the laser trajectory. The study focuses on the alignment of the lasers at FERMI, the free-electron laser facility at Elettra Sincrotrone Trieste. In particular, we employ Q-learning with linear function approximation and report experimental results obtained in two setups, which are the actual setups where the final application has to be deployed. Despite the simplicity of the approach, we report satisfactory preliminary results, that represent the first step toward a fully automatic procedure for seed laser to the electron beam. Such a superimposition is, at present, performed manually. ...

N. Bruchon1, G. Gaio2, G. Fenu1, M. Lonza2, F. A. Pellegrino1, E. Salvato1 1University of Trieste, 2Elettra Sincrotrone Trieste 17th International Conference on Accelerator and Large Experimental Physics Control Systems Abstract Reinforcement Learning (RL) is one of the most promis-ing techniques in Machine Learning because of its modestcomputational requirements with respect to other algorithms.RL uses an agent that takes actions within its environmentto maximize a reward related to the goal it is designed toachieve. We have recently used RL as a model-free approachto improve the performance of the FERMI Free ElectronLaser. A number of machine parameters are adjusted tofind the optimum FEL output in terms of intensity and spec-tral quality. In particular we focus on the problem of thealignment of the seed laser with the electron beam, initiallyusing a simplified model and then applying the developedalgorithm on the real machine. This paper reports the resultsobtained and discusses pros and cons of this approach withplans for future applications. ...