DIRECT: Deep Image REConstruction Toolkit

DIRECT is a Python, end-to-end pipeline for solving inverse problems emerging in image processing. It is built with PyTorch (Paszke et al., 2019) and stores state-of-the-art deep learning imaging inverse problem solvers for solving inverse problems such as denoising, dealiasing, and reconstruction. By defining a base forward linear or non-linear operator, DIRECT can be used for training models for recovering images such as MRIs from partially observed or noisy input data. Additionally, it provides the user with the functionality to load saved weights of pre-trained models to be used for inference. Furthermore, it offers functions for preparing and pre-processing data such as .h5 files into PyTorch datasets compatible with the software’s training pipeline, but also allows for flexibility to work with any kind of PyTorch dataset. Additionally, in order for the user to view the process of their experiments, it allows for continuous visualisation of training and validation metrics as well as image predictions utilising Tensorboard (examples are illustrated in Figures 1 and 2).


Summary
DIRECT is a Python, end-to-end pipeline for solving inverse problems emerging in image processing. It is built with PyTorch (Paszke et al., 2019) and stores state-of-the-art deep learning imaging inverse problem solvers for solving inverse problems such as denoising, dealiasing, and reconstruction. By defining a base forward linear or non-linear operator, DIRECT can be used for training models for recovering images such as MRIs from partially observed or noisy input data. Additionally, it provides the user with the functionality to load saved weights of pre-trained models to be used for inference. Furthermore, it offers functions for preparing and pre-processing data such as .h5 files into PyTorch datasets compatible with the software's training pipeline, but also allows for flexibility to work with any kind of PyTorch dataset. Additionally, in order for the user to view the process of their experiments, it allows for continuous visualisation of training and validation metrics as well as image predictions utilising Tensorboard (examples are illustrated in Figures 1 and 2).

Statement of need
A plethora of image processing problems arising in biology, chemistry, and medicine can be defined as inverse problems. Inverse problems aim in recovering a signal x ∈ X (e.g. an image) that cannot be directly observed from a set of measurements y ∈ Y and is subject to a given corruption process known as the forward model: where A denotes the forward operator and n is some measurement noise, often assumed to be additive and normally distributed. Equation 1 is usually ill-posed and therefore an explicit solution is hard to find. Instead, inverse problems in imaging are typically solved by minimizing an objective function J which is consisted of a data-fidelity term and a regularization term R (also known as Variational Problems): (2)

Accelerated Parallel MRI Reconstruction
Accelerated Parallel Magnetic Resonance Image (MRI) Reconstruction, that is, reconstructing an MR image from a set of partially observed (or sub-sampled) k-space measurements from multiple receiver coils (parallel imaging (Larkman & Nunes, 2007)), is par excellence an example of inverse problems. The base forward operator of accelerated MRI reconstruction is usually the two or three-dimensional Fast Fourier Transform (FFT) denoted as F.
More specifically, let be the sub-sampled k-space measurements acquired from n c receiver coils, where where S i denotes a (usually unknown or estimated) coil sensitivity map, property of each individual coil, and U a retrospective binary sub-sampling mask operator which simulates the sub-sampling process in clinical settings. Then, the corresponding inverse problem for Accelerated Parallel MRI Reconstruction replaces (2) with the following optimization problem Conventional approaches employed for solving (4) include Compressed Sensing algorithms (CS) (Candes et al., 2006;Donoho, 2006;Lustig et al., 2007), SENSE (Pruessmann et al., 1999), and GRAPPA (Griswold et al., 2002). Deep learning-based imaging inverse problem solvers have shown to outperform these conventional techniques by outputting reconstructed images with higher fidelity from highly sub-sampled k-space measurements Lønning et al., 2019;Pal & Rathi, 2021).

Functionality
DIRECT stores PyTorch MRI datasets and data-loaders, multiple retrospective sub-sampling schemes, MRI-related transforms and evaluation metrics, and several state-of-the-art DL baselines that can be applied to the task of solving the inverse problem of Accelerated Parallel MRI Reconstruction, which make it a perfect tool for research in this domain. In addition to the currently implemented methods and already-stored baselines, the user can easily incorporate into DIRECT their own code following the current implementations.
DIRECT also allows for easy and flexible experimentation. For an experiment, the user simply needs to define a configuration file that comprises the experiment parameters. See Configuration File below for a configuration file template. DIRECT can be employed for training and/or validating models on multiple machines and GPUs as it is integrated with PyTorch's torch.distributed module and NVIDIA's cuDNN (Chetlur et al., 2014).

Configuration File
All experiment parameters can be specified in a configuration file. These include model, dataset, sub-sampling scheme, physics, training, and validation. Each configuration file should be saved with the .yaml extension. The following is a template example of a configuration file:

Research projects using DIRECT
DIRECT is the main software used for research by the MRI Reconstruction team of the Innovation Centre for Artificial Intelligence (ICAI) -AI for Oncology group of the Netherlands Cancer Institute (NKI).

Challenges
DIRECT has been used for MRI Reconstruction result submissions in the fastMRI challenge (Muckley et al., 2021) and the Multi-Coil MRI Reconstruction challenge (Beauferris et al., 2021).

Publications
Papers using DIRECT include Yiasemis et al. (2022) (presented in SPIE Medical Imaging Conference 2022) and Yiasemis et al. (2021) (to be presented in CVPR Conference 2022).