PyMatting: A Python Library for Alpha Matting

An important step of many image editing tasks is to extract specific objects from an image in order to place them in a scene of a movie or compose them onto another background. Alpha matting describes the problem of separating the objects in the foreground from the background of an image given only a rough sketch. We introduce the PyMatting package for Python which implements various approaches to solve the alpha matting problem. Our toolbox is also able to extract the foreground of an image given the alpha matte. The implementation aims to be computationally efficient and easy to use. The source code of PyMatting is available under an open-source license at https://github.com/pymatting/pymatting.


Introduction
For an image I with foreground pixels F and background pixels B, alpha matting asks to determine opacities α, such that the equality holds for every pixel i. This problem is ill-posed since, for each pixel, we have three equations (one for each color channel) with seven unknown variables. The implemented methods rely on a trimap, which is a rough classification of the input image into foreground, background and unknown pixels, to further constrain the problem. Giving the alpha matte, foreground estimation aims to extract the foreground F from image I.

Implemented Methods for Alpha Estimation
Closed-form Matting: Levin et al. (2007) show that assuming local smoothness of pixel colors yields a closed-form solution to the alpha matting problem.
KNN Matting: Lee and Wu (2011) and Chen et al. (2013) use nearest neighbor information to derive closed-form solutions to the alpha matting problem which they note to perform particularly well on sparse trimaps.
Large Kernel Matting: He et al. (2010) propose an efficient algorithm based on a large kernel matting Laplacian. They show that the computational complexity of their method is independent of the kernel size.
Random Walk Matting: Grady et al. (2005) use random walks on the pixels to estimate alpha. The calculated alpha of a pixel is the probability that a random walk starting from that pixel will reach a foreground pixel before encountering a background pixel. Learning Based Digital Matting: Zheng and Kambhamettu (2009) estimate alpha using local semisupervised learning. They assume that the alpha value of a pixel can be learned by a linear combination of the neighboring pixels.

Implemented Methods for Foreground Estimation
Closed-form Foreground Estimation: For given α, the foreground pixels F can be determined by making additional smoothness assumptions on F and B. Our toolbox implements the foreground estimation by Levin et al. (2007).
Multi-level Foreground Estimation: Furthermore, the PyMatting toolbox implements a novel multilevel approach for foreground estimation. For this method our toolbox also provides GPU implementations for OpenCL and CUDA.

Installation and Code Example
The following code snippet demonstrates the usage of the library: The estimate alpha cf method implements closed-form alpha estimation (Levin et al., 2007), whereas the estimate foreground cf method implements the closed-form foreground estimation (Levin et al., 2007). The method stack images can be used to compose the foreground onto a new background.
More code examples at different levels of abstraction can be found in the documentation of the toolbox. PyMatting can be easily installed through pip.

Performance Comparison
Since all of the considered methods require to solve a large sparse systems of linear equations, an efficient solver is crucial for good performance. Therefore, the PyMatting package implements the conjugate gradients method (Hestenes et al., 1952) together with different preconditioners that improve convergence: Jacobi, V-cycle (Lee and Wu, 2014) and thresholded incomplete Cholesky decomposition (Jones and Plassmann, 1995). To evaluate the performance of our implementation, we calculate the mean squared error on the unknown pixels of the benchmark images of Rhemann et al. (2009). Figure 2 shows the mean squared error to the ground truth alpha matte. Our results are consistent with the results achieved by the authors implementation (if available). We compare the computational runtime of our solver with other solvers: PyAMG (Olson and Schroder, 2018), UMFPACK (Davis, 2004), AMGCL (Demidov, 2019), MUMPS (Amestoy et al., 2001(Amestoy et al., , 2006, Eigen (Guennebaud et al., 2010) and SuperLU (Li et al., 1999). Figure 4 shows that our implemented conjugate gradients method in combination with the incomplete Cholesky decomposition preconditioner outperforms the other methods by a large margin. For the iterative solver, we use an absolute tolerance of 10 −7 , which we scale with the number of known pixels, i.e. pixels that are either marked as foreground or background in the trimap. The benchmarked linear system arises from the matting Laplacian by Levin et al. (2007). Figure 3 shows that our solver also outperforms the other solvers in terms of memory usage.

Compatibility and Extendability
The PyMatting package has been tested on Windows 10, Ubuntu 16.04 and macOS 10.15.2. The package can be easily extended by adding new definitions of the graph Laplacian matrix. We plan on continuously extending our toolbox with new methods.