Reinforcement Learning for Control and Management of Communications and Computer Infrastructures

Kin K. Leung received his BS degree from the Chinese University of Hong Kong in 1980, and his MS and PhD degrees from University of California, Los Angeles (UCLA), in 1982 and 1985, respectively. He joined AT&T Bell Labs in New Jersey in 1986 and worked at its successor companies until 2004. Since then, he has been the Tanaka Chair Professor in the Electrical and Electronic Engineering (EEE), and Computing Departments at Imperial College in London. He serves as the Head of Communications and Signal Processing Group in the EEE Department at Imperial.

His current research focuses on optimization and machine-learning techniques for system design and control of large-scale communication networks and computer infrastructures. He also works on multi-antenna and cross-layer designs for wireless networks. He received the Distinguished Member of Technical Staff Award from AT&T Bell Labs in 1994, and was a co-recipient of the 1997 Lanchester Prize Honorable Mention Award. He was elected as an IEEE Fellow in 2001. He received the Royal Society Wolfson Research Merits Award from 2004 to 2009 and became a member of Academia Europaea in 2012. Along with his co-authors, he also received a number of best paper awards at major conferences, including the IEEE PIMRC 2012, ICDCS 2013 and ICC 2019. He serves as a member (2009-11) and the chairman (2012-15) of the IEEE Fellow Evaluation Committee for Communications Society. He was a guest editor for the IEEE JSAC, IEEE Wireless Communications and the MONET journal, and as an editor for the JSAC: Wireless Series, IEEE Transactions on Wireless Communications and IEEE Transactions on Communications. Currently, he is an editor for the ACM Computing Survey and International Journal on Sensor Networks.

Reinforcement learning (RL) is a very versatile technique that has been widely applied to many domains. Deep RL techniques have been proposed to solve various routing, service-placement, and power allocation problems on the software defined networks (SDN) as well as in the software defined coalitions (SDC) developed in the DAIS ITA Program https://dais-ita.org/pub. This seminar begins with a brief introduction to RL for communications and computer systems. For illustration purposes, the speaker presents use of RL to arrange how the domain controllers should synchronize with each other to maximize the benefits of synchronization in distributed SDN. Specifically, the RL technique is combined with deep neural network to train a smart controller synchronization policy, which we call the Deep-Q (DQ) Scheduler. Evaluation results show that DQ Scheduler significantly outperforms other algorithms for inter-domain routing tasks. As reflected in the DQ work, a very challenging issue for deep RL is the huge state and action spaces, which increase the model complexity and training time beyond practical feasibility. To address this issue, we have started to propose new techniques with a common goal of reducing model complexity and training time for deep RL. In particular, we propose a method to decouple the actions from the state space for the value-function learning process and a relatively simple transition model is learned to determine the action that causes the associated state transition. It has been found that the state-action separable RL (sasRL) can greatly reduce training time without noticeable performance degradation. The seminar will conclude by highlighting the open issues for use of RL for control and management of large-scaled communications and computer systems.