(196a) Enhancing Oil Recovery Efficiency with Constrained Reinforcement Learning for Waterflooding Optimization

As we strive for environmental sustainability, finding ways to optimize oil extraction has become crucial to meet our energy needs more sustainably. With the discovery of new oil reserves becoming increasingly challenging, maximizing the output from existing fields through enhanced recovery techniques is a necessity [1]. Waterflooding stands out as a pivotal secondary recovery method, wherein the injection of water into reservoirs displaces trapped oil, facilitating its extraction [2,3]. However, this process is remarkably water-intensive, with three barrels of water consumed for every barrel of oil produced, resulting in a daily consumption of 250 million barrels of water worldwide [4]. This excessive water usage not only poses significant environmental challenges but also leads to substantial annual costs exceeding $40 billion for water management [5]. By optimizing waterflooding operations, we can dramatically reduce water consumption, cut costs, extend the productive life of oil fields, and strike a sustainable balance between economic and environmental perspectives. Nonetheless, achieving this optimization is challenging due to the highly nonlinear reservoir models and thousands of process constraints involved in decision-making. This makes the direct deterministic optimization of the secondary oil recovery processes computationally intractable.

In this study, we introduce a constrained reinforcement learning-based (RL) algorithm to address the nonlinear dynamic optimization of waterflooding of oil reservoirs. This framework incorporates a highly accurate surrogate model of the reservoir with a classification model that maps the feasible region of the reservoir operation to ensure higher economic gains, better computational efficiency, and a feasible operation schedule for the process. We formulate the problem using a parametrized well-pressure control approach to maximize the net present value (NPV) of reservoir operations. During the offline phase, we gather data from reservoir simulations to train a deep feedforward neural network (FFNN) with high predictive accuracy (R² > 0.99 in blind tests) using 20,000 samples. To improve the model further, we augment it with a mechanistic model, creating a hybrid system that forms the backbone of our RL framework. For constraint handling, we incorporate various binary classification algorithms to model and filter out the infeasible samples [6,7]. The classifier with the highest accuracy is utilized to filter out any infeasible solutions from the search space. Building upon the framework described above, which incorporates the surrogate model and binary classifier, we employ a model-free reinforcement learning approach to learn the optimal policy for setting bottom-hole pressures. The RL agent iteratively selects pressure settings, obtains liquid flow rate estimates from the surrogate model, and calculates rewards based on predicted NPV. The binary classifier ensures only feasible pressure settings progress to the surrogate model for obtaining new data and rewards. The reward function is designed to impose penalties for actions that violate constraints, ensuring the exploration of viable solutions. We explore various model-free RL algorithms, including value-based, policy-based, and actor-critic methods, to iteratively refine the policy and maximize cumulative NPV while adhering to constraints. The performance of the developed constrained RL algorithm is tested on one low-dimensional and one high-dimensional benchmark reservoir study. Finally, the results are compared to those in the literature where the optimization process is conducted using explicit constraint-handling strategies [8].

References

1. Muggeridge, A., Cockin, A., Webb, K., Frampton, H., Collins, I., Moulds, T., and Salino, P., Recovery rates, enhanced oil recovery and technological limits. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2014, Volume 372, 20120320.

2. Ahmed, T., Principles of Waterflooding. Reservoir engineering handbook, 2018, 901-1093.

3. United States Department of Energy, Office of Fossil Energy: Enhanced Oil Recovery, (n.d.). http://energy.gov/fe/science­innovation/oil­gas­research/enhanc...

4. Fakhru’l-Razi, A., Pendashteh, A., Abdullah, L.C., Biak, D.R.A., Madaeni, S.S. and Abidin, Z.Z., 2009. Review of technologies for oil and gas produced water treatment. Journal of hazardous materials, 170(2-3), pp.530-551.

5. Bailey, B., Crabtree, M., Tyrie, J., Elphick, J., Kuchuk, F., Romano, C., and Roodhart, L., Water control. Oilfield review, 2000, Volume 12, 30-51.

6. Beykal, B., Aghayev, Z., Onel, O., Onel, M. and Pistikopoulos, E.N., 2022. Data-driven Stochastic Optimization of Numerically Infeasible Differential Algebraic Equations: An Application to the Steam Cracking Process. In Computer Aided Chemical Engineering(Vol. 49, pp. 1579-1584). Elsevier.

7. Dias, L.S. and Ierapetritou, M.G., 2019. Data-driven feasibility analysis for the integration of planning and scheduling problems. Optimization and Engineering, 20, pp.1029-1066.

8. Beykal, B., Boukouvala, F., Floudas, C.A., Sorek, N., Zalavadia, H. and Gildin, E., 2018. Global optimization of grey-box computational systems using surrogate functions and application to highly constrained oil-field operations. Computers & Chemical Engineering, 114, pp.99-110.