Abstract

Many science applications increasingly make use of data-intensive methods that require bulk data movement such as staging of large datasets in preparation for analysis on shared computational resources, remote access to large data sets, and data dissemination. Over the next 5 to 10 years, these datasets are projected to grow to exabytes of data, and continued scientific progress will depend on efficient methods for data movement between high performance computing centers. We study two techniques that improve the use of available resources for large, long-running, multi-file transfers. First, we show the effect of adaptation of transfer parameters for multi-file transfers, where the adaptation is based on recent performance. Second, we use Virtual Organization and site policies to influence the allocation of resources such as available transfer streams to clients. We show that these techniques improve completion …

Date

November 16, 2014

Authors

Ann L Chervenak, Alex Sim, Junmin Gu, Robert E Schuler, Nandan Hirpathak

Conference

2014 Fourth International Workshop on Network-Aware Data Management

Pages

1-8

Publisher

IEEE