Cited 0 time in webofscience Cited 0 time in scopus

Resource-aware multi-way join processing using MapReduce

Resource-aware multi-way join processing using MapReduce
Translated Title
자원 활용을 고려한 맵리듀스 기반 멀티웨이 조인 처리
Nam, Yoon Min
DGIST Authors
Nam, Yoon Min; Kim, Min SooChoi, Jihwan P.
Kim, Min Soo
Choi, Jihwan P.
Issue Date
Available Date
Degree Date
2015. 2
multi-way joinresource-awareMapReducestreamingbalanced workload멀티웨이 조인자원고려맵리듀스스트리밍균등한 작업량
With a growing demand of hidden insights from the large scale of data, multi-way join operations become the key of many OLAP-style data analytic tasks for not only relational data analysis, but also various scientific applications. To process OLAP-style data analytic tasks with cost-efficiency, shared-nothing distributed system, such as MapReduce, gets its popularity in both academic field and enterprises. However, due to various lacks in support of MapReduce such as processing data from multiple sources and data skew handling, join operation is inefficient operation in MapReduce. Specifically, generation of query execution plan in MapReduce does not consider the dominantly utilized resources that affect the performance of a query processing significantly. Our work is based on a counter observations from traditional wisdoms of query processing in MapReduce: reducing the number of MapReduce job does not guarantee the performance benefit, and growing intermediate data does not always triggers the performance degradation. In this work, we propose efficient resource-aware multi-way join processing method by taking not only algorithmic approach, but also systemic approach. As an algorithmic approach, we propose in-memory streaming hash join method with careful consideration of memory constraint in a computing machine and a balanced workload of each join task. As a systemic approach, we propose a generation of efficient multi-way join query execution plan. In experimental results, our method improves the performance of multi-way join query processing, especially the latest version of Apache Hive[5], and AQUA[11]. In addition, our method shows better performance even if the aggregated intermediate data is larger than other method by exploiting major resources very efficiently. ⓒ 2015 DGIST
Table Of Contents
Ⅰ. INTRODUCTION -- Ⅱ. BACKGROUND -- 2.1 MapReduce -- 2.2 SQL-on-Hadoop -- Ⅲ. JOIN PROCESSING USING MAPREDUCE -- 3.1 Basic join algorithms in MapReduce -- 3.2 Replicated join -- 3.3 1-Bucket-Theta -- Ⅳ. RESOURCE-AWARE MULTI-WAY JOIN PROCESSING -- 4.1 Problem description -- 4.2 Cost model of operator pipeline in MapReduce -- 4.3 In-memory streaming hash join -- 4.4 Finding multi-way join group -- Ⅴ. EXPERIMENTS -- Ⅵ. RELATED WORK -- Ⅶ. CONCLUSIONS -- Ⅷ. REFERENCE
Information and Communication Engineering
Related Researcher
  • Author Kim, Min-Soo InfoLab
  • Research Interests Big Data Systems; Big Data Mining & Machine Learning; Big Data Bioinformatics; 데이터 마이닝 및 빅데이터 분석; 바이오인포메틱스 및 뉴로인포메틱스; 뇌-기계 인터페이스(BMI)
Department of Information and Communication EngineeringThesesMaster

qrcode mendeley

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.