Cited 0 time in webofscience Cited 0 time in scopus

A distributed in-situ analysis method for large-scale scientific data

Title
A distributed in-situ analysis method for large-scale scientific data
Authors
Han, DonghyoungNam, Yoon-MinKim, Min-Soo
DGIST Authors
Han, Donghyoung; Nam, Yoon-Min; Kim, Min-Soo
Issue Date
2017
Citation
2017 IEEE International Conference on Big Data and Smart Computing, BigComp 2017, 69-75
Type
Conference
Article Type
Conference Paper
ISBN
9781510000000
Abstract
Recently, a massive amount of data is generated in a wide range of scientific applications such as NASA's satellite, the large hadron collider, and large synoptic survey telescope. Most of scientific data follows the array model, and there are various kinds of standard array formats such as HDF, NetCDF, MDSplus, and ROOT. SciDB is the most well-known DBMS that stores the array-based scientific data and processes queries on it. SciDB is a distributed DBMS, and so, is scalable in terms of query performance. However, it has a severe drawback that takes a huge amount of time for loading a massive amount of scientific data into DBMS. That is, it is not scalable in terms of data loading. To overcome that problem, we propose a distributed in-situ analysis method that allows processing queries on raw scientific data in a distributed manner without explicit data loading. In detail, we propose the in-situ scan operator that scans necessary data of the array format and passes it to upper operators of the pipeline of a query plan. It also performs repartitioning during in-situ scanning, which is required for correct query results. Through experiments using real datasets, we have shown that the SciDB system using our method significantly outperforms the original SciDB system by orders of magnitude in terms of the performance of the first query. © 2017 IEEE.
URI
http://hdl.handle.net/20.500.11750/4322
DOI
10.1109/BIGCOMP.2017.7881718
Publisher
Institute of Electrical and Electronics Engineers Inc.
Related Researcher
Files:
There are no files associated with this item.
Collection:
Information and Communication EngineeringETC2. Conference Papers


qrcode mendeley

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

BROWSE