Policy Gradient and Actor-Critic Approaches to Quickest Change Detection

This work builds on the recently proposed end-to-end deep learning approach to quickest change detection (QCD), called DeepQCD, that is based on supervised learning of the binary labels (corresponding to no change or change) associated with the data sample at each time instant. Inspired by the sequential nature of reinforcement learning, our new approach data-driven QCD is to learn a probabilistic policy that maximizes the total reward over a data sample trajectory. Under such a framework, we develop three new deep learning-based QCD algorithms – one policy gradient (PG) QCD algorithm, and two actor-critic (AC) QCD algorithms. These QCD algorithms employ a neural network consisting of recurrent layers that keep an internal state summarizing the time-series data seen so far and update the state as new data sample comes in, and dense layers that map the internal state into a decision probability and/or estimated cumulative reward. Extensive experiments are performed on both synthetic data and real-world data, and the results demonstrate the superior performance of the proposed PG-QCD and AC-QCD methods compared to state-of-the-art learning-based and non-learning-based QCD methods. Moreover, among the three proposed methods, the AC-QCD method that uses a baseline performs the best.