Abstract

Workflows are a key technology for enabling complex scientific applications. They capture the inter-dependencies between processing steps in data analysis and simulation pipelines, as well as the mechanisms to execute those steps reliably and efficiently in a distributed computing environment. They also enable scientists to capture complex processes to promote method sharing and reuse and provide provenance information necessary for the verification of scientific results and scientific reproducibility. We describe a weak-lensing pipeline that is modelled as a Pegasus workflow with pipeline codes available as a Singularity container. This has enabled us to make this analysis widely available and easily replicable to the astronomy community. Using Pegasus, we have executed various steps of pipelines on different compute sites with varying infrastructures, with Pegasus seamless managing the data across the various compute clusters in a transparent manner.

Date

February 8, 2026

Authors

Karan Vahi, Michael H Wang, Chihway Chang, Scott Dodelson, Mats Rynge, Ewa Deelman

Journal

Proceedings of the 28th Astronomical Data Analysis Software and Systems (ADASS’18)

Volume

523

Pages

689-692