(413h) Transfer Learning Applied to an Industrial Wastewater Treatment Unit

Ece Serenat Koksal ^a, Erdal Aydin ^a,b,*

^a Department of Chemical and Biological Engineering, Koc University, Istanbul 34450, Turkey

^b Koc University Tupras Energy Center (KUTEM), Koc University, Istanbul 34450, Turkey

^* eaydin@ku.edu.tr

Abstract

Recurrent neural networks (RNN) are a kind of upgraded version of artificial neural networks which use a memory able to keep the information of prior inputs. When systems are dynamic, RNNs may increase the prediction capacity of the model since they use sequential or time series data. Simple RNNs often suffer from “vanishing gradient problem” where partial derivative of the objective function gets closer to zero at some point when there are multiple hidden layers. As a solution, a type of recurrent neural network architecture, long-short term memory (LSTM), may be suggested which has cells in the hidden layers and three gates that control the flow of information (Hochreiter, 1998).

Transfer learning consists of transferring knowledge of the related tasks which can handle the issue of scarcity of labeled data where small dataset size is insufficient to train a network with high performance (Zhuang et al., 2021). Parameter-based transfer learning consists of transferring knowledge, weights in particular, to a task where data is scarce. The transferred knowledge of pretrained model with good performance sustains an initialization for the main model which is the target neural network to obtain a better performance. The training procedure is followed by fine tuning step where both transferred and target network models are trained with low learning rate. Transfer learning is useful especially when the number and kind of features of the transferred knowledge are different from the target model and where mixing the datasets is not possible.

Although transfer learning has found several application steps especially for image classification, its application to regression problems is scarce, especially for actual chemical engineering problems where data quality, size and type are questionable.

In this work, transfer learning is applied to an industrial wastewater plant to predict dissolved oxygen at activated sludge. Dissolved oxygen is a critical parameter for this process since oxygen is consumed by aerobic bacteria at activated sludge. Thus, designing a valid and efficient soft sensor is a relevant task for the industry for both monitoring and control. In addition to artificial neural networks, recurrent neural networks are also used for training of the source and target tasks. Prediction performance of the plant is increased by the related tasks such as an open-source model where the main physics of the process is captured. In addition to the open-source simulation model (Alex et al., 1999), process data of other industrial wastewater plants is also used. Training performance based on MSE slightly decreases in case of transfer learning. However, test MSE is up to 30% smaller when knowledge of the related tasks is transferred to the refinery’s target model. When number of neurons is increased, the improvement in the test performance is reduced. On the other hand, test MSE sharply decreases when transfer learning algorithm is applied.

References

Alex, J., Beteau, J. F., Copp, J. B., Hellinga, C., Jeppsson, U., Marsili-Libelli, S., Pons, M. N., Spanjers, H., & Vanhooren, H. (1999). Benchmark for evaluating control strategies in wastewater treatment plants. http://www.ensic.u-nancy.fr/costwwtp

Hochreiter, S. (1998). The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions. International Journal of Uncertainty, Fuziness and Knowledge-Based Systems, 6(2), 107–116. https://doi.org/https://doi.org/10.1142/S0218488598000094

Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., & He, Q. (2021). A Comprehensive Survey on Transfer Learning. In Proceedings of the IEEE (Vol. 109, Issue 1, pp. 43–76). Institute of Electrical and Electronics Engineers Inc. https://doi.org/10.1109/jproc.2020.3004555