# Performance Evaluation of Query Plan Recommendation with Apache Hadoop and Apache Spark

**Authors:** Elham Azhir, Mehdi Hosseinzadeh, Faheem Khan, Amir Mosavi  
**arXiv:** [2210.07143](https://arxiv.org/abs/2210.07143)  
**Date:** 2022-09-17  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.07143v1/

## Key points

- Study focuses on evaluating query plan recommendation performance using Apache Hadoop and Apache Spark
- Approach involves executing new queries based on previously created query execution plans (QEPs) and clustering the query space for optimization
- Researchers leveraged MapReduce distributed computing model to address time-consuming nature of traditional clustering algorithms for large datasets
- Methodology included software development, validation, resource management, data curation, original draft preparation, review and editing tasks by different authors
- Algorithms such as Mapper and Reducer classes used for calculating total terms in each query and determining weights for features through term frequency calculations
- Similarity measurement using MapReduce discussed to assess similarities between queries efficiently
- Parallel query clustering significantly enhances scalability in query optimization processes
- Apache Spark outperformed Apache Hadoop in performance metrics with an average speedup of 2x
- Research highlights benefits of leveraging distributed computing frameworks like Apache Spark and Apache Hadoop for effective query execution plan optimization

## Summary

The study focuses on evaluating the performance of query plan recommendation using Apache Hadoop and Apache Spark. The approach involves executing new queries based on previously created query execution plans (QEPs) and clustering the query space for optimization. To address the time-consuming nature of traditional clustering algorithms when dealing with large datasets, the researchers leveraged the MapReduce distributed computing model. By utilizing Apache Spark and Apache Hadoop frameworks, they aimed to cluster various sizes of query datasets within the MapReduce-based access plan recommendation method. The methodology included software development, validation, resource management, data curation, original draft preparation, review and editing tasks handled by different authors. Despite receiving no external funding for their research and having no conflicts of interest related to their work, all authors have approved the final version of the manuscript for publication. The study delves into algorithms such as Mapper and Reducer classes for calculating total terms in each query and determining weights for each feature through term frequency calculations. Additionally, a similarity measurement using MapReduce is discussed in detail to assess similarities between queries efficiently. Overall, parallel query clustering was found to significantly enhance scalability in query optimization processes. Furthermore, Apache Spark outperformed Apache Hadoop in terms of performance metrics with an average speedup of 2x. This research highlights the potential benefits of leveraging distributed computing frameworks like Apache Spark and Apache Hadoop for effective query execution plan optimization.