PiSCAT: A Python Package for Interferometric Scattering Microscopy

1 Max Planck Institute for the Science of Light, 91058 Erlangen, Germany. 2 Max-Planck-Zentrum für Physik und Medizin, 91058 Erlangen, Germany. 3 Department of Computer Science, Friedrich-Alexander University Erlangen-Nürnberg, 91058 Erlangen, Germany. 4 Department of Physics, Friedrich-Alexander University Erlangen-Nürnberg, 91058 Erlangen, Germany. 5 Erlangen National High Performance Computing Center (NHR@FAU). DOI: 10.21105/joss.04024


Summary
Interferometric scattering (iSCAT) microscopy allows one to image and track nano-objects with a nanometer spatial and microsecond temporal resolution over arbitrarily long measurement times (Lindfors et al., 2004;Taylor & Sandoghdar, 2019b, 2019a. A key advantage of this technique over the well-established fluorescence methods is the indefinite photostability of the scattering phenomenon in contrast to the photobleaching of fluorophores. This means that one can perform very long measurements. Moreover, scattering processes are linear and thus do not saturate. This leads to larger signals than is possible from a single fluorophore. As a result, one can image at a much faster rate than in fluorescence microscopy. Furthermore, the higher signal makes it possible to localize a nano-object with much better spatial precision. The remarkable sensitivity of iSCAT, however, also brings about the drawback that one obtains a rich speckle-like background from other nano-objects in the field of view.

Statement of need
The aim of this project is to explore the potential of powerful computer vision tools (such as background correction, segmentation, and object tracking) to address challenges in the analysis of iSCAT images. The use of existing libraries and packages turns out not to be satisfactory. Here, we present a software platform implemented in Python for processing the iSCAT images, named PiSCAT. In order to make this framework accessible to a wide range of applications, we have developed a GUI (Graphical User Interface), which enables the users to conduct the analysis of interest, regardless of their programming skills. Reading different image or video formats, background correction (e.g removing fixed pattern noise (FPN), differential rolling average (DRA)), point spread function (PSF) detection, linking and extraction of trajectories of target PSFs are amongst the key features of PiSCAT. PiSCAT is also optimized for parallel coding to decrease the analysis time.

Concept and Structure of PiSCAT
PiSCAT is a unified python-based package that presents classical computer vision and machine learning analysis for various applications such as unlabeled protein detection or three-dimensional (3D) tracking of nanoparticles. Protein detection based on the linear relationship of contrast and particle mass is one of the popular iSCAT applications (Dastjerdi et al., 2021;Piliarik & Sandoghdar, 2014). But due to camera hardware limitations, a protein cannot be visualized in a single frame (see Figure 1a-(I)). To remove the static background, this method subtracts the averages of two consecutive batches (see Figure 1a-(II)). Additionally, we need algorithms to localize and track particles in spatial and temporal domains to correctly read the contrasts and provide a histogram from them (see Figure 1a-(III)). Another application of iSCAT is to monitor the diffusion of single nanoparticles such as lipid vesicles across the surface of giant unilamellar vesicles (GUVs; (Spindler et al., 2018); see also Figure 1a-(IV)). As in most iSCAT signal processing steps, we first separate the nanoparticle signals from the background signal present in the raw videos. In comparison to diffusing nanoparticles, the slowly varying GUV signal can be estimated using temporal median and then removed during background correction. Although this method partially removes background rings and can reveal nanoparticles as small as 20 nm, one is still left with a residual background signal that creates some difficulties for quantitative tracking. To address this issue, we developed a background correction routine based on clustering of the similar frames in a video to get around small fluctuations in the background signal. In order to compensate slightly stronger fluctuations in the background signal, we additionally perform affine image registration (Spindler et al., 2018) (see Figure 1a-(V)). High frame rate imaging in iSCAT provides us enough signal redundancy in time so that we can afford to select the frames in a video sparsely. We therefore call this routine Sparse In Time Affine Registration (SITAR). (IV) iSCAT image of a part of GUV which shows a Newton ring-like pattern. The GUV signal is treated as the background. After its removal one obtains (V) in which a yellow arrow points to the iPSF of a gold nanoparticle. In (IV) and (V), the scale bars correspond to 4 µm. Such iPSFs can be localized across several frames and linked together to obtain a 3D trajectory as shown in (VI). Inset image illustrates 3D position of several particles that are color coded differently. (b) Modelling iPSF images of a nanoparticle travelling in the axial direction using iPSF module: (I) 2D image of the iPSF of a particle positioned at 3 µm with the focus positioned at 3 µm. The particle height is swept axially to form an axial stack made of 2D iPSF images. This stack is then sliced along its meridional plane and plotted in (II). The amplitude and differential phase of the same plane is shown in (III) and (IV) respectively.
PiSCAT provides several algorithms for localization and tracking of nanoparticles (see Figure 1a-(VI)). The current version offers several image processing techniques to mainly accomplish the following: 1) Background correction that improves the distinction of the signal of interest from the background, 2) Localizing particles in (X,Y,Z axis and time). To achieve this, we need the combination of different computer vision and machine learning algorithms with physical models such as the iSCAT point-spread function (iPSF) model (see Figure 1b-(I-IV)) (Mahmoodabadi et al., 2020). Figure 2 depicts the structure of the PiSCAT package. PiSCAT uses a set of dependencies (e.g. PyQtGraph, scikit-learn/image (Pedregosa et al., 2011), Trackpy (Allan et al., 2021) and Astropy (Astropy Collaboration et al., 2018)) that assist us in achieving various goals. The PiSCAT repository contains this list.

Ongoing Research and Dissemination
Our future work aims to add GPU kernels by using code generation technologies. This kind of design helps us to add real-time processing and different deep neural networks (DNN) and machine learning to this package in order to facilitate feature extraction and distinguish target signals from the noisy background even at low SNR.