This code provides access to multi-GPU computing from PETSc-based applications, using NVIDIA's AmgX library. The PETSc (Portable, Extensible Toolkit for Scientific Computation) library, developed by Argonne National Laboratory, is a very popular high-performance software through for scientific applications modeled by partial differential equations (Balay et al. 2016). It executes in distributed systems via message-passing communications with MPI. NVIDIA's AmgX library is a multi-GPU linear-algebra package that supports multigrid preconditioners and Krylov solvers. Because PETSc lacks support for modern heterogeneous platforms (GPU+CPU), AmgX can be a good option for adding GPU computing to PETSc applications. Incorporating AmgX into PETSc applications is not straightforward, however, due to the difference between the two libraries in design, underlying data formats, and usage. The purpose of AmgXWrapper is to bridge PETSc and AmgX, helping PETSc-application developers use AmgX in their software without a thorough understanding of the AmgX API. With AmgXWrapper, developers may need only a few lines of code modification to add AmgX solvers in legacy PETSc applications. The wrapper also features implicit data transfer when there are mismatched numbers of CPU cores and GPU devices in a computing node. This allows exploiting all possible resources on modern heterogeneous platforms. We have presented an example of using AmgX and AmgXWrapper to accelerate an existing PETSc-based CFD code (Chuang and Barba 2017).
Balay, Satish, Shrirang Abhyankar, Mark F. Adams, Jed Brown, Peter Brune, Kris Buschelman, Lisandro Dalcin, et al. 2016. “PETSc Users Manual.” ANL-95/11 - Revision 3.7. Argonne National Laboratory. http://www.mcs.anl.gov/petsc.
Chuang, Pi-Yueh, and Lorena A. Barba. 2017. “Using Amgx to Accelerate Petsc-Based Cfd Codes.” Figshare. doi:10.6084/m9.figshare.5018774.v1.