(361h) Discovering Novel Materials Using Combinatorial Chemistry-Based Reinforcement Learning

The discovery of new materials with desired properties is important because it can lead to significant advancements in technology and applications. Moreover, the discovery of these new materials tends to have properties that are hard to coexist with or have a very high or low value of specific properties. For example, in the case of transparent electrodes, two properties, transparent and electric, are challenging to exist simultaneously. Here, the problem is that in contrast to conventional materials, few (or any) known samples exhibit these characteristics. Because of this, chemists find it challenging to gain insights or knowledge from the existing materials that might be used to derive the molecular structures of the desired materials.

In recent years, machine learning has provided a new methodology for screening novel materials with high performance, developing quantitative structure-activity relationships (QSARs) and other models, predicting the properties of materials, and finding new materials [1]. However, since most machine learning-based models are probability distribution-based models, such as conditional variational autoencoders (cVAEs) [2] and conditional generative adversarial networks (cGANs) [3], there is a limit to finding materials with extreme characteristic values, since there are very few corresponding molecular samples. In contrast, reinforcement learning is a type of machine learning which learns policy (or order of action) that maximizes reward among selectable actions. Unlike probability distribution-based models, reinforcement learning relies solely on the reward, so it does not matter if a few existing molecules satisfy the target property. Combinatorial chemistry is a methodology that can generate a large number of molecules by combining fragments according to chemical rules. An example of combinatorial chemistry is the Breaking of retrosynthetically interesting chemical substructures (BRICS) [4]. Applying the BRICS rule, molecules can be broken into fragments, or new molecular structures can be generated by combining those fragments.

In this study, we present reinforcement learning to provide combinatorial chemistry a strategy for choosing molecular fragments that directs the molecule that is generated toward the target. With a randomly selected initial fragment, our model selects the next fragment to be combined, considering the chemical rule so that the model can only generate chemically valid molecules. The policy is learned by giving a higher reward if the target properties are satisfied. This learned policy efficiently searches large chemical spaces using combinatorial chemistry. We empirically showed that our model is suitable for discovering novel materials by conducting experiments on several cases, and compared the result with two probability distribution-learning models, conditional recurrent neural network (cRNN) [5] and generative chemical transformer (GCT) [6]. Moreover, after showing that our model performs well by implementing it to a simple problem, we discuss how to improve the performance of the reinforcement learning model and implement the model in real-world problems.

[1] Liu, Y., et al., Materials discovery and design using machine learning. Journal of Materiomics, 2017. 3(3): p. 159-177.

[2] Sohn, K., H. Lee, and X. Yan, Learning structured output representation using deep conditional generative models. Advances in neural information processing systems, 2015. 28.

[3] Mirza, M. and S. Osindero, Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.

[4] Degen, J., et al., On the Art of Compiling and Using'Drug‐Like'Chemical Fragment Spaces. ChemMedChem: Chemistry Enabling Drug Discovery, 2008. 3(10): p. 1503-1507.

[5] Kotsias, P.-C., et al., Direct steering of de novo molecular generation with descriptor conditional recurrent neural networks. Nature Machine Intelligence, 2020. 2(5): p. 254-265.

[6] Kim, H., J. Na, and W.B. Lee, Generative chemical transformer: neural machine learning of molecular geometric structures from chemical language via attention. Journal of chemical information and modeling, 2021. 61(12): p. 5804-5814.