Stargazing through a digital veil: Managing a large scale sky survey using distributed databases on HPC clusters

Yogesh Simmhan, Catharine Van Ingen, Jim Heasley, Alex Szalay

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The Sloan Digital Sky Survey established the use of relational databases for the scans and cone searches common to astronomy analyses. The Pan-STARRS project scales up SDSS by melding HPC clusters with hierarchical and spatially partitioned distributed databases to meet the challenge of near realtime handling of the multiple data surveys generated by a GigaPixel telescope. This meld provides job management capabilities on the cluster for scientist query submission as well as the backend data updates and fault management necessary for a system with no traditional backup. This paper describes the Pan-STARRS HPC+database experience, highlights the current focus of our work and where further research is needed.

Original languageEnglish (US)
Title of host publicationHPCDB'11 - Proceedings of the 2011 Workshop on High-Performance Computing Meets Databases, Co-located with SC'11
Pages33-36
Number of pages4
DOIs
StatePublished - 2011
Event1st Annual 2011 Workshop on High-Performance Computing Meets Databases, HPCDB'11, Co-located with Supercomputing, SC'11 - Seattle, WA, United States
Duration: Nov 13 2011Nov 13 2011

Publication series

NameHPCDB'11 - Proceedings of the 2011 Workshop on High-Performance Computing Meets Databases, Co-located with SC'11

Conference

Conference1st Annual 2011 Workshop on High-Performance Computing Meets Databases, HPCDB'11, Co-located with Supercomputing, SC'11
Country/TerritoryUnited States
CitySeattle, WA
Period11/13/1111/13/11

Keywords

  • Distributed databases
  • HPC clusters
  • Scientific data
  • Workflows

ASJC Scopus subject areas

  • Computer Science Applications
  • Software

Fingerprint

Dive into the research topics of 'Stargazing through a digital veil: Managing a large scale sky survey using distributed databases on HPC clusters'. Together they form a unique fingerprint.

Cite this