spam: Software for Practical Analysis of Materials

Olga Stamati1, Edward Andò1, Emmanuel Roubin1, Rémi Cailletaud1, 2, Max Wiebicke1, 3, Gustavo Pinzon1, Cyrille Couture1, Ryan C. Hurley4, Robert Caulk1, Denis Caillerie1, Takashi Matsushima5, Pierre Bésuelle1, Félix Bertoni6, Tom Arnaud6, Alejandro Ortega Laborin1, Riccardo Rorato7, Yue Sun8, Alessandro Tengattini1, 9, Olumide Okubadejo1, Jean-Baptiste Colliat8, Mohammad Saadatfar10, Fernando E. Garcia11, Christos Papazoglou1, Ilija Vego1, Sébastien Brisard12, Jelke Dijkstra13, and Georgios Birmpilis13


Statement of need
Advanced experimental materials science is entering a new era thanks to the performance and availability of tomography allowing full-3D studies, and often 3D-timeseries.However in the analysis of these advanced measurements, the attractiveness of closed-source and black-box solutions to 3D measurement problems seems to be precipitating a reproducibility crisis in this domain.A radically open approach to measurement science with the disclosure of analysis tools will allow validation and verification of tools and results, rendering the whole chain more robust.Furthermore, the hope is that there is a community-driven improvement of the tools with time.

Summary
Spam, the Software for the Practical Analysis of Materials is a Python library that has evolved to cover needs of data analysis from 3D x-ray tomography work and correlated random fields with mechanical applications.Spam is first and foremost a measurement package that extends the extremely convenient framework of NumPy (Van der Walt, Colbert, & Varoquaux, 2011) and SciPy (Virtanen et al., 2020) by providing or accelerating tools for the materialscience/mechanics oriented analysis of 2D images or 3D volumes representing field measurements.Typical uses are either the measurement of displacements fields between images of a deforming sample from which strains can be computed, or the characterisation of a particular microstructure (correlation length or particle orientation).The package is organised into a library of Python tools which are expected to be used in user-written scripts and a number of more sophisticated standalone scripts.The tools are organised as follows: • DIC: toolkit for Digital Image/Volume Correlation.This toolkit provides tools to measure the deformation between two images/volumes.A robust registration tool based on the same-modality version of Tudisco, Jailin, et al. (2017) is provided, as well as some cutting edge tools such as multi-modal registration that implements Roubin, Andò, & Roux (2019).A version of Global-DVC as per Mendoza, Neggers, Hild, & Roux (2019) is in the process of being developed.The DIC toolkit is presented in the introduction, discussed in detail in tutorials covering theory, practice and discrete DIC, and illustrated in the examples gallery • deformation: toolkit for manipulating deformation functions called Φ (expressed in homogeneous coordinates), as well as the more usual in continuum mechanics transformation gradient tensor F .Tools are also provided to compute fields of F from a displacement field measured on a regular grid (coming from a "local" correlation for example), either using square/cubic finite element shape functions, or the method proposed in Geers, De Borst, & Brekelmans (1996).For a displacement field on an irregular grid (for example defined at particle centres coming from a "discrete" correlation), a Delaunay triangulation based method (Bagi & others, 1996;Catalano, Chareyre, & Barthélémy, 2014;Zhang & Regueiro, 2015) is implemented.Both finite (large) strain and infinitesimal (small) strain frameworks are implemented for both regular and irregular grids.The deformation toolkit is presented (together with DIC) in the introduction, discussed in detail in the continuum mechanics refresher, the start of DIC theory and strain tutorial, and is used throughout the DIC examples (see above) • excursions: toolkit for the excursion set of correlated random fields theory (Adler, 2008).It includes functions that give the analytical predictions of the global descriptors (or Lipschitz-Killing curvatures) of excursions in spaces of arbitrary dimensions (Roubin & Colliat, 2016;Roubin et al., 2015a) along with the generation of correlated random fields using RandomFields in R through rpy2 (Gautier, 2020;R Core Team, 2018;Schlather, Malinowski, Menck, Oesting, & Strokorb, 2015).The excursions toolkit is illustrated in the examples gallery • filters: toolkit of 3D filters that provide some functionality missing in scipy.ndimage.filters such as the computation of a local hessian, or functions which are slow.For example, for the computation of a local variance, the spam-provided function is more than 100 times faster than using the generic filter with variance in scipy.ndimage.
The filters toolkit is used in the projection tutorial, and illustrated examples gallery • helpers: toolkit of internal helper functions such as the parsers for the scripts, as well as tools for reading and writing TSV and VTK files.The latter partially uses meshio (Schlömer et al., 2020) • kalisphera: wrapper for C++ version of kalisphera (Tengattini & Andò, 2015) which generates analytically-correct partial-volume spheres which are useful for testing discrete analysis code (see label below).The kalisphera toolkit is presented in the introduction, is used in the contacts tutorial, and is illustrated in the examples gallery • label: toolkit to measure and manipulate "labelled" images, where discrete particles are labelled with integer voxel patches.The computation of standard quantities such as the volume, centre of mass, and moment of inertia tensor of each particle can be done very conveniently and somewhat faster and with a smaller memory footprint than what is available in scipy.ndimage.An estimation of an ellipse fitting of each particle is implemented with the algorithm from Ikeda, Nakano, & Nakashima (2000).Tools for characterising inter-particle contacts based on the work of Wiebicke, Andò, Herle, & Viggiani (2017).A wrapper for ITK's morphological watershed (Beare & Lehmann, 2006;Schroeder, Ng, & Cates, 2003) is also provided.The label toolkit is presented in the introduction, discussed in detail in tutorials covering the base toolkit and contacts, and illustrated in the examples gallery • measurements: toolkit implementing the measurement of covariance, porosity and global descriptors (volume, perimeter, surface area, and Euler characteristic).The mea surements toolkit is used in the examples gallery • mesh: toolkit for the creation or manipulation of meshes -in spam tetrahedral meshes are principally used.Meshers based on Gmsh (Geuzaine & Remacle, 2009) are used through pygmsh and weighted Delaunay triangulation (Laguerre triangulation) is provided through an interface with CGAL (The CGAL Project, 2020) -alpha-shapes are also implemented to help clean up badly-shaped tetrahedra.In addition, a set of projection functions creates meshes able to represent heterogeneities (phases and interfaces of a given meso/micro structure) based on binary or trinary images or continuous fields (level set) with outputs easily convertible to any FE software (Roubin et al., 2015b;Stamati, Roubin, Andò, & Malecot, 2019).The mesh toolkit is presented in the introduction, at the heart of the projection tutorial, and illustrated in the examples gallery • plotting: toolkit of plotting tools based on matplotlib (Hunter, 2007) for creating complex plots such as a 3D orientation plot.The plotting toolkit is presented in the introduction and used throughout tutorials and examples • visual: toolkit of graphical helper functions for scripts A number of scripts are available to be called from the command line.Currently, the most-used scripts are related to image correlation: • spam-ldic and spam-regularStrain: A "local" image correlation script, working for series of greyscale 2D or 3D images where kinematics are measured on independent points spread on a regular grid, accompanied by a strain computation script.spam-ldic is presented in the scripts page and is the subject of the DIC practice.spam-regular Strain is presented in the scripts page and is used in the strain tutorial • spam-ddic and spam-discreteStrain: A "discrete" image correlation script (Andò, Hall, Viggiani, Desrues, & Bésuelle, 2012;Hall et al., 2010), working on greyscale 3D images plus a "labelled" image of the reference configuration.This script also has its own strain calculation based on a triangulation of grain centres.spam-ddic is presented in the scripts page and is the subject of the discrete DIC tutorial • spam-gdic (in beta test): A "global" image correlation script, where the displacement field between two 3D images is computed as a global problem expressed on a tetrahedral mesh • spam-mmr and spam-mmr-graphical: A pair of "multi-modal registration" scripts (command-line and graphical) allowing 3D images of different modalities (e.g., a neutron tomography and an x-ray tomography of the same sample) to be registered (i.e., aligned)

Technical details
Spam is based on simple Python data types, avoiding complex data structures, and all functions have a reasonable and safe set of default parameters, with required parameters kept to a minimum.Spam has a number of different use cases: • Interactive use, such as in iPython or Jupyter.Many outputs from 3D analysis in materials science are highly sensitive to the parameters used, encouraging a "live" exploration of optimal settings • Embedded use, such as importing in user-written Python scripts • Standalone use, particularly of the more complex spam-scripts.These chain together a number of functions and are intended to be called from a command line, and produce output as live plots, or files saved to disk Given the large data volumes often encountered in 3D analysis, critical parts of the code are written in C/C++ wrapped with appropriate Python calling functions which are responsible for checking input sanity.The current wrapping method is with pybind11 (Jakob, Rhinelander, & Moldovan, 2017).
Building on the large number of functions already available in NumPy ( Van der Walt et al., 2011), SciPy (Virtanen et al., 2020), scikit-image (Van der Walt et al., 2014) and making use of tifffile (Gohlke, 2020) and meshio (Schlömer et al., 2020), the spam project adds a large amount of functionality, which means that a number of advanced forms of data analysis can be chained together in ways that are otherwise complex, requiring the combination of many different tools.
Spam uses unittest to check each commit with a coverage of more than 90% of lines of code covered as of June 2020.Details of the coverage are available on the GitLab repository.

Online documentation
The documentation for this project is available online at this address: https://ttk.gricad-pages.univ-grenoble-alpes.fr/spam/index.html There are three main components to the documentation: • The module index is built automatically by sphinx, and function headers are written in such a way (following the NumPy norm) that a brief description appears in the module index • A "Gallery of Examples" using sphinx-gallery where downloadable Python scripts or Jupyter notebooks are executed during the compilation of the documentation and whose results are visible • A series of "Tutorials" with longer and more detailed explanations are available, which cover the mathematical/mechanics background of the functions provided, and some longer examples of using the provided tools

State of the field
Other software packages exist for material science analysis, such as the popular commercial software Avizo, which is closed-source, cannot be inspected, and therefore has limited trust.Other open-source packages include ITK (Schroeder et al., 2003), which is a quite complex ecosystem, and ImageJ/Fiji (Rueden et al., 2017), which is not fully 3D and not well-suited to scripting or running remotely.All of the above have some version of the label toolkit which allows discrete objects to be characterised as well as some parts of the measurements toolkit.
On the specific topic of Digital Image/Volume Correlation, many pieces of software are available for 2D, surface and 3D image correlation.The spam.DIC.registercorrelation engine uses a well-known mathematical framework dating back to (Lucas & Kanade, 1981), but distinguishes itself by being non-rigid, 2/3D compatible, clearly documented and relatively fast.
The discrete and multimodal image correlation are much less common.A number of other image correlation codes exist (this is really not an exhaustive list): • CMV and CMV_3D: Developed at Laboratoire Navier (Bornert et al., 2004)

Use in existing work
Spam has already enabled research progress on a number of fronts, resulting in the following publications:

•
Roubin & Colliat (2016): Use of excursions toolkit to predict percolation threshold in n-dimensional Euclidean spaces (Figures1 and 2)• Stamati, Roubin, Andò, & Malecot (2018): Use of filters toolkit to identify aggregates in concrete (Figure5)•Stamati et al. (2019): Use of spam-ldic and spam-ddic scripts to measure deformation in a concrete sample subjected to a tension test (Figure8) and mesh projection functions to conduct the FE analysis (Figure2) • Stavropoulou et al. (2019): Use of spam-ldic script to measure deformation of a claystone (Figures 10 and 11) • Wiebicke, Andò, Šmilauer, Herle, & Viggiani (2019): Use of kalisphera (Figure 5) and label toolkits to benchmark sand-grain contact measurements (Figure 8 and others), provides an example script • Andò, Dijkstra, Roubin, Dano, & Boller (2019): Use of spam-ddic script and the label toolkit to measure small displacements in a creep test on sand, see Figure 4 • Roubin et al. (2019): Application of Multi Modal Registration to concrete (Figure 4 onwards) • Hurley & Pagan (2019): Use of deformation toolkit to measure deformation in concrete (strain in Figure 2b) • Wiebicke, Andò, Viggiani, & Herle (2020): Use of label toolkit for the analysis of interparticle contacts (Figure 1) as well as the plotting toolkit to plot the distribution of orientations (Figures 7 and 8) • Stavropoulou et al. (2020): Use of spam-mmr and spam-gdic scripts (Figures 2, 3, 4 and Figure 7 respectively), and the mesh toolkit to measure water absorption in claystone (Figures 8 and 9) Developers or curious users are encouraged to clone the Git repository.Solid build instructions exist for Debian-based environments (in principle any GNU/Linux distribution should work) and for some versions of OSX.Compilation for Windows has not been attempted given the large number of dependencies.However, users have been able to get spam to work using Ubuntu in the Windows Subsystem for Linux (WSL).
spam is available for Linux users with Python 3 as a PyPI package installed via pip.