Autonomous Image Segmentation for Single Nanoparticle Tracking in Liquid Phase Transmission Electron Microscopy

Solutions of nanoparticles are extensively used in catalysis, energy storage, drug delivery, and inkjet printing. To understand the structure-property relationship of such solutions, it is necessary to analyze the interactions and motions of nanoparticles in the liquid environment at the single particle level. Liquid-phase transmission electron microscopy (Liquid Phase TEM) is a promising technique for studying the dynamics of these solutions. Liquid Phase TEM is an emerging in-situ microscopy technique that allows us to visualize the dynamics of nanoparticles in their native liquid environment with an unprecedented spatial (nanometer) and temporal (millisecond) resolution. However, the analysis of liquid-phase TEM images is challenging due to the presence of noise caused by the thick liquid layer and the interaction of the liquid cell window with the incident imaging electrons. This lowers the contrast between the nanoparticles and the liquid background, making it challenging to identify and track the nanoparticles accurately. Moreover, with the rapid advancement of microscopy technologies, vast amounts of data are generated. To address this data deluge resulting from multimodal in-situ microscopy data, automated algorithms for efficient data processing are needed. Traditional image segmentation methods, such as ones based on thresholding, are time-consuming and labor-intensive. In recent years, machine learning-based approaches such as deep learning architectures have shown remarkable performance in image segmentation. These models can be trained on diverse datasets and hence can implement image segmentation on wide variety of samples, including those with low contrast or high noise. Furthermore, these models eliminate the need for manual parameter adjustments, streamlining the image segmentation pipeline and removing any human bias from the image analysis. Here, we developed an unsupervised autoencoder deep learning model for image denoising and segmentation in liquid phase TEM. The algorithm identifies a single nanoparticle by isolating the region of interest, removing background noise, and outputting the boundary of nanoparticles, position, and orientation. To train and test the model, tens of thousands of real, raw TEM images were used as input and the corresponding segmented images generated by the traditional image segmentation thresholding algorithm developed previously were used as the output training dataset for this model. This algorithm can be used to study the dynamics of nanoparticles of various shapes and sizes in the liquid phase. The results from this project will be an essential part of an artificial intelligence-based workflow for automating electron microscopes to analyze samples faster, more efficiently, and without any human intervention.