TY - GEN
T1 - I/O streaming evaluation of batch queries for data-intensive computational turbulence
AU - Kanov, Kalin
AU - Perlman, Eric
AU - Burns, Randal
AU - Ahmad, Yanif
AU - Szalay, Alexander
PY - 2011
Y1 - 2011
N2 - We describe a method for evaluating computational turbulence queries, including Lagrange Polynomial interpolation, based on partial sums that allows the underlying data to be accessed in any order and in parts. We exploit these properties to stream data from disk in a single pass and concurrently evaluate batch queries. The combination of sequential I/O and data sharing improves performance by an order of magnitude when compared with direct evaluation of each query. The technique also supports distributed evaluation of queries in a database cluster, assembling the partial sums from each node at the query mediator. Interpolation is fundamental to computational turbulence, over 95% of queries use these routines, and the partial sums method allows the JHU Turbulence Database Cluster to realize scale and throughput for our scientists'data-intensive workloads.
AB - We describe a method for evaluating computational turbulence queries, including Lagrange Polynomial interpolation, based on partial sums that allows the underlying data to be accessed in any order and in parts. We exploit these properties to stream data from disk in a single pass and concurrently evaluate batch queries. The combination of sequential I/O and data sharing improves performance by an order of magnitude when compared with direct evaluation of each query. The technique also supports distributed evaluation of queries in a database cluster, assembling the partial sums from each node at the query mediator. Interpolation is fundamental to computational turbulence, over 95% of queries use these routines, and the partial sums method allows the JHU Turbulence Database Cluster to realize scale and throughput for our scientists'data-intensive workloads.
KW - Data-intensive computing
KW - Database clusters
KW - I/O streaming
KW - Query evaluation
KW - Query optimization
KW - Software for high-throughput computing
UR - http://www.scopus.com/inward/record.url?scp=83155184554&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=83155184554&partnerID=8YFLogxK
U2 - 10.1145/2063384.2063423
DO - 10.1145/2063384.2063423
M3 - Conference contribution
AN - SCOPUS:83155184554
SN - 9781450307710
T3 - Proceedings of 2011 SC - International Conference for High Performance Computing, Networking, Storage and Analysis
BT - Proceedings of 2011 SC - International Conference for High Performance Computing, Networking, Storage and Analysis
T2 - 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, SC11
Y2 - 12 November 2011 through 18 November 2011
ER -