(632g) Multi-Source Transfer Learning for Accelerating Modeling of Chemical Processes

Transfer learning has attracted increasing attention in modeling complex nonlinear chemical processes in recent years [1]. In conventional transfer learning methods [2], a pre-trained model obtained from a single-source domain is utilized in the development of the model for the target process. However, in practice, there could exist multiple source processes that have similar configurations as the target process and can be used to generate a large training dataset for pre-trained models. Despite some successful applications of transfer learning methods with multiple sources, negative transfer that leads to degradation in the performance of final models may occur due to improper selection of multiple sources. Therefore, it is imperative to investigate how to strike a balance between the knowledge obtained in the selected datasets and the negative transfer performance that occurs when the sample size increases [3]. Generalization error for domain adaptation plays a vital role in the selection of the pre-trained model in the sense that it measures the performance of an ML model obtained with the source datasets on the target dataset [4]. Several optimization algorithms have been proposed for the data selection of ML models based on the generalization error bound [5]. However, at this stage, no theoretical results have been derived for the generalization error of pre-trained models developed using a multi-source dataset, and therefore, the design of a source data selection algorithm for multi-source transfer learning lacks theoretical guidance.

In this work, an optimization-based multi-source transfer learning scheme is developed for modeling of nonlinear chemical processes. Specifically, a transfer learning neural network model for a target process with limited data is developed using the pre-trained model obtained with multiple source processes. Since the performance of transfer learning models depends on the quality of the pre-trained models, we propose a novel Bayesian optimization problem to optimize the selection of multi-source data for the pre-trained models by first deriving a generalization error bound for multi-source domain adaptation using -discrepancy distance. Subsequently, the optimization problem is formulated using the theoretical error bound to select the optimal set of multiple sources, which can be used to develop the pre-trained model that provides a good initial guess of the weight parameters for transfer learning model. Finally, a simulation study of a chemical reactor process in Aspen Plus Dynamics is conducted to illustrate the effectiveness of the optimization-based multi-source transfer learning scheme.

References:

[1] Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., Xiong, H., & He, Q. (2020). A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1), 43-76.

[2] Jiang, J., Shu, Y., Wang, J., & Long, M. (2022). Transferability in deep learning: A survey. arXiv preprint arXiv:2201.05867.

[3] Sun, S., Shi, H., & Wu, Y. (2015). A survey of multi-source domain adaptation. Information Fusion, 24, 84-92.

[4] Hoffman, J., Mohri, M., & Zhang, N. (2018). Algorithms and theory for multiple-source adaptation. Advances in neural information processing systems, 31.

[5] Awasthi, P., Cortes, C., & Mohri, M. (2024). Best-effort adaptation. Annals of Mathematics and Artificial Intelligence, 1-46.