TY - GEN
T1 - Building NDStore through hierarchical storage management and microservice processing
AU - Lillaney, Kunal
AU - Kleissas, Dean
AU - Eusman, Alexander
AU - Perlman, Eric
AU - Gray Roncal, William
AU - Vogelstein, Joshua T.
AU - Burns, Randal
N1 - Funding Information:
This work was supported by grants from the National Institutes of Health (NINDS 1R01NS092474 1U01NS090449) and the National Science Foundation (IOS-1707298 ACI-1649880 OCE-1633124).
Funding Information:
This work was supported by grants from the National Institutes of Health (NINDS 1R01NS092474 1U01NS090449) and the National Science Foundation (IOS-1707298 ACI-1649880 OCE-1633124). The authors would like to thank additional members of the Neurodata Project that contributed to this work. We would also like to thank our collaborators Narayanan Kasthuri, Davi Bock, Forrest Collman, Wei Chung Lee, and Dan Bumbarger.
Publisher Copyright:
© 2018 IEEE.
PY - 2018/12/24
Y1 - 2018/12/24
N2 - We describe NDStore, a scalable multi-hierarchical data storage deployment for spatial analysis of neuroscience data on the AWS cloud. The system design is inspired by the requirement to maintain high I/O throughput for workloads that build neural connectivity maps of the brain from peta-scale imaging data using computer vision algorithms. We store all our data on the AWS object store S3 to limit our deployment costs. S3 serves as our base-tier of storage. Redis, an in-memory key-value engine, is used as our caching tier. The data is dynamically moved between the different storage tiers based on user access. All programming interfaces to this system are RESTful web-services. We include a performance evaluation that shows that our production system provides good performance for a variety of workloads by combining the assets of multiple cloud services.
AB - We describe NDStore, a scalable multi-hierarchical data storage deployment for spatial analysis of neuroscience data on the AWS cloud. The system design is inspired by the requirement to maintain high I/O throughput for workloads that build neural connectivity maps of the brain from peta-scale imaging data using computer vision algorithms. We store all our data on the AWS object store S3 to limit our deployment costs. S3 serves as our base-tier of storage. Redis, an in-memory key-value engine, is used as our caching tier. The data is dynamically moved between the different storage tiers based on user access. All programming interfaces to this system are RESTful web-services. We include a performance evaluation that shows that our production system provides good performance for a variety of workloads by combining the assets of multiple cloud services.
KW - Big Data
KW - Cloud Computing
KW - Neuroscience
KW - Object Storage
KW - Spatial Data
UR - https://www.scopus.com/pages/publications/85061373633
UR - https://www.scopus.com/pages/publications/85061373633#tab=citedBy
U2 - 10.1109/eScience.2018.00037
DO - 10.1109/eScience.2018.00037
M3 - Conference contribution
AN - SCOPUS:85061373633
T3 - Proceedings - IEEE 14th International Conference on eScience, e-Science 2018
SP - 223
EP - 233
BT - Proceedings - IEEE 14th International Conference on eScience, e-Science 2018
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 14th IEEE International Conference on eScience, e-Science 2018
Y2 - 29 October 2018 through 1 November 2018
ER -