(652b) Building Confidence in Machine Learning Models: A Case Study in Deep Learning-Based Image Analytics for Characterization of Pharmaceutical Samples

Machine Learning (ML) tools for different tasks in pharmaceutical Research and Development (R&D) are increasingly popular. A quick search elucidates a large number of submissions from both industry and academia on the topic in the AIChE 2022 meeting confex repository, especially to the Pharma 4.0 session. These studies and ML applications cover a range of focus areas from drug substance to drug product as well as from discovery to development (including analytical and process R&D) plus manufacturing and control. Indeed, artificial intelligence and specifically ML approaches to it are considered major contributors to the industry 4.0 revolution across different sectors. In the analytical space AI-based tools, many forms of which are already in use, can enable fast and robust data analysis, which is particularly beneficial for high-throughput and data-rich measurements.

In the highly regulated pharmaceuticals industry, applying such innovative solutions go hand in hand with regulatory influence. This reinforces the need that these novel AI/ML tools and their internal decision-making mechanisms should be accessible, as much as possible, to various stakeholders including scientists, process engineers, and regulatory agencies. Historically, chemical engineers have dealt with this for a long time by incorporating physics into their models. However, model transparency is not a given in some modern AI/ML cases where models can be highly complex and accurate but at the expense of interpretability, including obscured correlations to underlying physicochemical phenomena. A key question that we must ask about every model is: What is the confidence in the model? The answer to which might go beyond simply reporting accuracy on some test data.

In this talk, we use a common case, developing ML models for data analysis in image-based characterization of particulate samples (e.g., amorphous solid dispersions), to highlight some of the aforementioned model confidence issues and provide suggestions for tackling them. We use this example to discuss some of the opportunities and pitfalls in using state-of-the-art models such as deep neural networks, including the common misunderstanding of them being fully “black box” and “unexplainable” models. We illustrate how model interpretation and competency methods such as Integrated Gradients can help explaining model predictions and provide insight into the model’s internal decision-making processes. We argue that such insight is crucial for understanding model limitations, anticipating its potential future failure modes, and building confidence and trust for future wide-spread use. This understanding also allows incorporating the necessary changes in the model development process to appropriately deal with challenging future test cases.