Abstract

This paper introduces WIRE that manages resources for the DAG-based workflows on IaaS clouds. WIRE predicts and plans resources over the MAPE (Monitor-Analyze-Plan-Execute) loops to: 1) Estimate task performance with online data, 2) Conduct simulations to predict the upcoming loads based on online estimates and workflow DAGs, 3) Apply a resource-steering policy to size cloud instance pools for the maximal parallelism that is consistent with low cost. We implement WIRE on Pegasus WMS/HTCondor and evaluate its performance on the ExoGENI network cloud. The results show that WIRE attains low resource cost with the performance that is typically within a factor of two of optimal.

Date

2021

Authors

Bing Xie, Qiang Cao, Mayuresh Kunjir, Linli Wan, Jeff Chase, Anirban Mandal, Mats Rynge

Conference

2021 IEEE International Conference on Cluster Computing (CLUSTER)

Pages

35-46

Publisher

IEEE