Streaming Transfer Optimization for Distributed Science Workflows

Abstract

Driven by advancements in computing and sensing technology, many scientific projects started to generate a huge volume of data that reaches to petabytes in scale. Distributed and collaborative nature of these projects requires produced data to be streamed to geographically distributed locations in a timely manner to enable real-time (or near-realtime) processing. Thus, robust and predictable network performance is key to streamline end-to-end workflow execution of streaming projects. On the other hand, existing high-performance data transfer applications and services support “best-effort” Quality of Service(QoS) model, thus fail to sustain high transfer performance when transfer conditions (e.g.,dataset characteristics, background traffic, etc.) deviate from initial observations., making them ill-suited for streaming workflows with stringent performance requirements.In this thesis, we propose FStream to offer performance guarantees to time-sensitive streaming applications by continuously monitoring transfer performance and re-adjusting transfer settings to adapt to dynamic transfer conditions and sustain high transfer performance throughout the workflow execution. To achieve this goal in real-time,FStream employs online profiling to rapidly explore solution space of transfer settings and find the one that meets QoS requirements with minimal overhead. It also takes advantage of the long-running nature of streaming workflows and keeps track of past online profiling results to greatly reduce convergence time of future online profiling runs. We evaluated the performance of FStream by transferring several synthetic and real-world workloads using high-performance production networks and showed that it offers up to an order of magnitude performance improvement over state-of-the-art solutions.