Demand response, or demand-side management, improves grid stability by increasing demand flexibility, and shifts peak demand towards periods of peak renewable energy generation by providing consumers with economic incentives. Reinforcement learning has been utilized to control diverse energy systems such as electric vehicles, HVAC systems, smart appliances, or batteries. The future of demand response greatly depends on its ability to prevent consumer discomfort and integrate human feedback into the control loop. Reinforcement learning is a potentially model-free algorithm that can adapt to its environment, as well as to human preferences by directly integrating user feedback into its control logic.
We reviewed all the literature about the use of reinforcement learning, in urban energy systems and for demand response applications in the smart grid [1]. Our review shows that although many papers consider human comfort and satisfaction, most of them focus on single-agent systems with demand-independent electricity prices and a stationary environment. However, when electricity prices are modeled as demand-dependent variables, there is a risk of shifting the peak demand rather than shaving it. Therefore, there is a need to further explore the applicability of reinforcement learning in multi-agent systems, which can participate in demand response. Reinforcement learning control algorithms have been tested in physical systems in only a small fraction of the articles we have reviewed. Therefore, in order to prove the reliability and adaptability of reinforcement learning algorithms, more real-world experiments need to be conducted with state of the art reinforcement learning methods. We observed that most of the studies are not easily reproducible, and so it is rather challenging to compare the performance of the controllers. Further standardization is needed in both the investigated control problems, and the used methods and simulation tools. We have proposed a basic framework to help in this standardization.
[1] Vázquez-Canteli, J.R., and Nagy, Z., “Reinforcement Learning for Demand Response: A Review of algorithms and modeling techniques”, Applied Energy 235, 1072-1089, 2019 (published in a special section: Progress in Applied Energy – reserved to the top 3% of the articles).
