(12e) Safe and Adaptive Strategies for Optimizing Battery Fast-Charging Protocols Via Reinforcement Learning

Demand for lithium-ion batteries in the transportation sector and stationary energy storage systems has significantly increased due to the growing inclination towards sustainable energy sources [1]. However, the excessively long charging time stands as one of the major barriers that limit the wider adoption of lithium-ion batteries. While battery fast-charging can be achieved by applying large charging currents, this often causes rapid escalations of battery temperature, which can initiate unwanted side reactions and result in dramatic battery degradation [2]. Traditional battery fast-charging methods can be categorized into heuristic rule-based and model-based strategies [3]. Heuristic rule-based strategies, such as the constant-current constant-voltage (CCCV) charging protocol, are prevalent in practical applications but do not consider battery dynamics and physical constraints. This can lead to suboptimal charging protocols and adversely impact the battery life. Model-based charging strategies, such as those based on electrochemical models, have better optimality and robustness than heuristic rule-based methods. Nonetheless, the complexity of battery models (e.g., electrochemical model) make the resultant optimization challenging to solve [4]. Also, both categories of methods cannot handle the drifting nature of battery dynamics [5].

Deep reinforcement learning (DRL) methods have recently been used to develop adaptive battery charging protocols to overcome the above issues [6-7]. However, existing DRL-based methods often fail to strictly ensure battery safety due to the inherent difficulty of DRL to incorporate hard constraints (mainly via soft constraints in the reward function). In this work, we present an adaptive and safe DRL-based optimization of fast-charging protocols for batteries that can strictly respect system constraints. For this new adaptive and safe DRL method, we append a safety layer after the actor network before passing the action (charging current) to the environment (batteries). The safety layer will project any unsafe action to a safety region, which is constructed by solving a constrained optimization. Such a safety region is contingent on battery dynamics and ambient conditions and thus changes over time. To capture a reliable safety region in real-time, we propose a novel adaptive Gaussian process (GP) model as the surrogate of the battery. Such adaptive GP modeling consists of a static GP model to capture the baseline behavior of batteries and a dynamic GP model to capture any real-time changes in battery dynamics due to environmental variations and battery aging. The static GP is trained with data collected from the first few charging cycles and the dynamic GP is trained by using the new data from the current cycle.

Finally, we have tested the performance of the proposed adaptive and safe RL method for optimizing battery fast-charging protocols, under both static and varying ambient environments. For the static environment, we compared the charging performance between heuristic rule-based strategy CCCV, conventional DRL algorithm, twin-delayed deep deterministic policy gradient algorithm (TD3), safe TD3 (static GP-based), and our adaptive safe TD3 (static and dynamic GP-based). The charging time for the battery to reach from 10% to 80% state-of-charge for CCCV, TD3, safe TD3, and adaptive safe TD3 is 33.33, 16.83, 18.83, and 22 minutes, respectively. Although the TD3 has a shorter charging time than other methods, it demonstrated significant temperature and voltage violations (16^oC and 0.34 V higher than temperature and voltage upper bounds) during the training. On the other hand, with a slightly longer charging time, both safe and adaptive TD3 could meet the temperature and voltage constraints. We then tested our method on a dynamically changing environment, where we incorporated a diffusion-limited solid electrolyte interphase (SEI) growth model to introduce battery degradation and aging while gradually increasing the ambient temperature from 10^oC to 36^oC. In a dynamic environment, the safe TD3 cannot strictly maintain safety constraints (7^oC higher than upper bound). In contrast, the proposed adaptive and safe GP can successfully adapt to the environmental changes without any temperature or voltage constraint violations.

Reference

[1] Tomaszewska, Anna, Zhengyu Chu, Xuning Feng, Simon O'kane, Xinhua Liu, Jingyi Chen, Chenzhen Ji et al. "Lithium-ion battery fast charging: A review." ETransportation1 (2019): 100011.

[2] Zhang, Sheng S. "Challenges and strategies for fast charge of Li‐ion batteries." ChemElectroChem7, no. 17 (2020): 3569-3577.

[3] Wei, Zhongbao, Zhongyi Quan, Jingda Wu, Yang Li, Josep Pou, and Hao Zhong. "Deep deterministic policy gradient-DRL enabled multiphysics-constrained fast charging of lithium-ion battery." IEEE Transactions on Industrial Electronics69, no. 3 (2021): 2588-2598.

[4] Yadu, Ankit, Subramanian Swernath Brahmadathan, Samarth Agarwal, D. B. Sreevatsa, Sangheon Lee, and Youngju Kim. "On-device personalized charging strategy with an aging model for Lithium-ion batteries using deep reinforcement learning." IEEE Transactions on Automation Science and Engineering(2023).

[5] Dong, Guangzhong, Yuyao Feng, Yunjiang Lou, Mingming Zhang, and Jingwen Wei. "Data-driven fast charging optimization for Lithium-ion battery using Bayesian optimization with fast convergence." IEEE Transactions on Transportation Electrification124, no. 4 (2023):

[6] Park, Saehong, Andrea Pozzi, Michael Whitmeyer, Hector Perez, Won Tae Joe, Davide M. Raimondo, and Scott Moura. "Reinforcement learning-based fast charging control strategy for li-ion batteries." In 2020 IEEE Conference on Control Technology and Applications (CCTA), pp. 100-107, 2020.

[7] Park, Saehong, Andrea Pozzi, Michael Whitmeyer, Hector Perez, Aaron Kandel, Geumbee Kim, Yohwan Choi, Won Tae Joe, Davide M. Raimondo, and Scott Moura. "A deep reinforcement learning framework for fast charging of li-ion batteries." IEEE Transactions on Transportation Electrification8, no. 2 (2022): 2770-2784.