Launcher: A simple tool for executing high throughput computing workloads

Summary

Launcher (Wilson and Fonner 2014, Wilson (2016), Wilson (2017)) is a utility for performing simple, data parallel, high throughput computing (HTC) workflows on clusters, massively parallel processor (MPP) systems, workgroups of computers, and personal machines. It can be easily run from userspace or installed for system-wide use on shared resources. Launcher will perform automatic process binding on multi-/many-core architectures where hwloc (“Portable Hardware Locality (Hwloc)” 2017) is installed.

Use Cases

Launcher is commonly used in situations where search or analysis of a large data space on independent points is required. Launcher is frequently used for: * Sequence alignment scoring for many hundreds or thousands of DNA/RNA sequence pairs * Docking/scoring (protien/ligand dock scoring, drug/immunotherapy development, etc.) * Immune repetoire analysis * Statistical analysis on unknown variables

Workflow

Launcher uses a tree-based parallel startup mechanism (similar to MPI) to scatter jobs to multiple nodes/processors. The filesystem is used to gather data from separate tasks (typically into individual output files). Multiple launcher invocations can be chained together to perform pipelined analysis or to perform chained simulation/analysis workflows.

launcher workflow

launcher workflow

References

“Portable Hardware Locality (Hwloc).” 2017. Accessed June 7. https://www.open-mpi.org/projects/hwloc/.

Wilson, Lucas A. 2016. “Using Managed High Performance Computing Systems for High-Throughput Computing.” In Conquering Big Data with High Performance Computing, edited by Ritu Arora, 61–79. Cham: http://dx.doi.org/10.1007/978-3-319-33742-5_4; Springer International Publishing. doi:10.1007/978-3-319-33742-5_4.

———. 2017. “Using the Launcher for Executing High Throughput Workloads.” Big Data Research 8: 57–64. doi:https://doi.org/10.1016/j.bdr.2017.04.001.

Wilson, Lucas A., and John M. Fonner. 2014. “Launcher: A Shell-Based Framework for Rapid Development of Parallel Parametric Studies.” In Proceedings of the 2014 Annual Conference on Extreme Science and Engineering Discovery Environment, 40:1–40:8. XSEDE ’14. New York, NY, USA: ACM. doi:10.1145/2616498.2616534.