Control of Continuous Quantum Systems with Many Degrees of Freedom based on Convergent Reinforcement Learning

Abstract

With advances in digital technology in recent years, parallel computation utilizing GPUs has achieved remarkable efficiency, which has made it possible to use large-scale machine learning algorithms with large datasets, and deep learning, which is machine learning utilizing deep neural networks trained with millions to billions of data and parameters, has become increasingly popular and found its use for various tasks including facial recognition, language translation and combinatorial problems, etc., achieving considerably many record-breaking results. As a powerful generic tool to solve problems, deep learning has been applied to physical problems as well, and has become an important alternative tool for scientists to solve a number of important problems. With the development of experimental quantum technology over the past several decades, quantum control has attracted increasing attention due to the realization of controllable artificial quantum systems, which have opened up a new regime of complex quantum systems that can be engineered. Quantum control has found its use in controlled quantum-chemical processes and artificial quantum systems including quantum dots, superconducting qubits, trapped ions, and cavity optomechanical systems, etc., which are of considerable importance for future technology as candidates for sensors and quantum-computational devices. However, because quantum-mechanical systems are often too difficult to analytically deal with, heuristic strategies and generic numerical algorithms which search for proper control protocols are adopted. Therefore, deep learning, especially deep reinforcement learning, is a promising generic candidate solution for the control problems. Although there have already been a few successful examples of applications of deep reinforcement learning to quantum control problems, most of the existing reinforcement learning algorithms intrinsically suffer from instabilities and unsatisfactory reproducibility, and as a consequence, they typically require a large amount of fine-tuning and a large computational budget, both of which limit their applicability to quantum control problems and require expertise in machine learning. To resolve the issue of instabilities of the reinforcement learning algorithms, in this thesis, we first investigate the non-convergence issue of Q-learning, which is one of the most efficient reinforcement learning strategies. Then, we investigate the weakness of existing convergent approaches that have been proposed, and we develop a new convergent Q-learning algorithm, which we call the convergent deep Q network (C-DQN) algorithm, as an alternative to the conventional deep Q network (DQN) algorithm. We prove the convergence of the C-DQN algorithm, and since the algorithm is scalable and computationally efficient, we apply it to a standard reinforcement learning benchmark, the Atari 2600 benchmark, to demonstrate its effectiveness. We show that when the DQN algorithm fail, the C-DQN algorithm still learns successfully. Then, we apply the algorithm to the measurement-feedback cooling problems of a quantum-mechanical quartic oscillator