Building reliable data pipelines for managing community data using scientific workflows

Yogesh Simmhan, Catharine Van Ingen, Alex Szalay, Roger Barga, Jim Heasley

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The growing amount of scientific data from sensors and field observations is posing a challenge to "data valets" responsible for managing them in data repositories. These repositories built on commodity clusters need to reliably ingest data continuously and ensure its availability to a wide user community. Workflows provide several benefits to modeling data-intensive science applications and many of these benefits can help manage the data ingest pipelines too. But using workflows is not panacea in itself and data valets need to consider several issues when designing workflows that behave reliably on fault prone hardware while retaining the consistency of the scientific data. In this paper, we propose workflow designs for reliable data ingest in a distributed environment and identify workflow framework features to support resilience. We illustrate these using the data pipeline for the Pan-STARRS repository, one of the largest digital surveys that accumulates 100TB of data annually to support 300 astronomers.

Original languageEnglish (US)
Title of host publicatione-Science 2009 - 5th IEEE International Conference on e-Science
Pages321-328
Number of pages8
DOIs
StatePublished - 2009
Event5th IEEE International Conference on e-Science, e-Science 2009 - Oxford, United Kingdom
Duration: Dec 9 2009Dec 11 2009

Publication series

Namee-Science 2009 - 5th IEEE International Conference on e-Science

Conference

Conference5th IEEE International Conference on e-Science, e-Science 2009
Country/TerritoryUnited Kingdom
CityOxford
Period12/9/0912/11/09

Keywords

  • Clusters
  • Distributed systems
  • Fault-tolerance
  • Provenance
  • Scientific data management
  • Scientific workflows

ASJC Scopus subject areas

  • General Arts and Humanities
  • General Computer Science
  • General Earth and Planetary Sciences
  • Health Information Management

Fingerprint

Dive into the research topics of 'Building reliable data pipelines for managing community data using scientific workflows'. Together they form a unique fingerprint.

Cite this