Abstract

Before scientific analyses run on shared infrastructure, such as the Open Science Grid or XSEDE, scientists must often transfer or stage key data sets those resources. Often these datasets consist of many files that may be transferred by multiple clients in parallel. We study two techniques that improve the use of available resources for these large, long-running, multi-file transfers. First, we adapt transfer parameters for multi-file transfers based on recent transfer performance. Second, we use VO and site policies to influence the allocation of system resources for transfers, such as available transfer streams. We describe our system design and summarize its implementation and performance.

Date

February 12, 2014

Authors

Ann L Chervenak, Alex Sim, Junmin Gu, Robert Schuler, Nandan Hirpathak

Conference

2014 22nd Euromicro International Conference on Parallel, Distributed, and Network-Based Processing

Pages

244-247

Publisher

IEEE