CityLearn: A Tutorial on Reinforcement Learning Control for Grid-Interactive Efficient Buildings and Communities
The CityLearn tutorial at RLEM'23 will help participants get acquainted with the CityLearn OpenAI Gym environment, developed for easy implementation and benchmarking of control algorithms, e.g., rule-based control, model predictive control or deep reinforcement learning control in the demand response, building energy and grid-interactive community domain. By the end of the tutorial, participants will learn how to design their own simple or advanced control algorithms to provide energy flexibility, and acquire familiarity with the CityLearn environment for extended use in personal projects.
The primary learning outcome for participants is to gain familiarity with CityLearn environment, its application programming interface (API) and dataset offerings for extended use in academic research or personal projects. Other secondary outcomes are to:
- Understand how electrification, distributed energy resources e.g. batteries, photovoltaic (PV) systems and smart controls provide a promising pathway to decarbonization and energy flexibility.
- Learn how to design and optimize their own rule-based control (RBC) agent for battery management using readily available knowledge of a building's energy use.
- Identify the challenges surrounding the generalizability of an RBC agent and how reinforcement learning (RL) can mitigate these challenges.
- Train their own RL Tabular Q-Learning algorithm.
- Evaluate the performance of a standard model-free deep RL algorithm in optimizing key performance indicators (KPIs) that are targeted at quantifying energy flexibility, environmental and economic costs.
- Learn the effect of different control algorithms and their parameters in improving these KPIs.
- Make submission to The CityLearn Challenge 2023.
The target audience for this tutorial includes the following:
- Academic, private and commercial researchers or professionals that are interested in sustainable artificial intelligence (AI)-related pathways to electrification and building decarbonization.
- I enthusiasts with at least beginner level of expertise in programming or data science whom may have or pick interest in solving control problems or are interested in learning about a new reinforcement learning (RL) environment that deviates from popular simpler problems e.g. the Atari, MuJoCo and Box2D Gym environments, to real-world problems with urgency like climate change mitigation and decarbonization of the electric grid infrastructure and building end-uses.
The CityLearn tutorial has a fairly low entry level and participants do not need to have prior experience in reinforcement learning (RL) nor use of a Gym environment. However, participants need to have at least, beginner knowledge in Python or other similar high-level scripting language. Also, participants should have a computer that is able to launch a Google Colab notebook in the browser or a Jupyter notebook locally.
The workshop is divided into the following parts (note that durations are flexible):
|5m||Overview, Learning Outcomes, Climate Impact, Target Audience and Prerequisites|
|20m||Background on Grid-Interactive Efficient Buildings, Energy Flexibility and CityLearn|
|15m||Overview of Hands-On Experiments, Setting up Development Environment, Dataset Description and Key Performance Indicators for Evaluation|
|10m||Experiment 1: Build your Custom Rule-Based Controller|
|10m||Experiment 2: An Introduction to Tabular Q-Learning Algorithm as an Adaptive Controller|
|10m||Experiment 3.1: Optimize a Soft-Actor Critic Reinforcement Learning Controller|
|15m||Experiment 3.2: Tune your Soft-Actor Critic Agent|
|20m||Make Submission to The CityLearn Challenge 2023 Control and Forecast Tracks|
|5m||Next Steps and Concluding Remarks|