# Benchmarking Apache Spark and Hadoop MapReduce on Big Data Classification

**Authors:** Taha Tekdogan, Ali Cakmak  
**arXiv:** [2209.10637](https://arxiv.org/abs/2209.10637)  
**Date:** 2022-09-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2209.10637v1/

## Key points

- The study by Taha Tekdogan and Ali Cakmak explores the evolution of Big Data analytics tools, focusing on Apache Spark and Hadoop MapReduce.
- "Big Data Mining" involves using data mining techniques to extract insights from vast amounts of unstructured data.
- Challenges arise in managing the shift from small, structured data to large volumes of unstructured and rapidly changing data.
- Apache Spark outperforms Hadoop MapReduce by being five times faster in training models but may experience performance deterioration with larger workloads.
- Hadoop MapReduce shows better accuracy scores in machine learning utility compared to Spark, especially in small datasets, but does not exhibit similar performance improvements when scaled.
- Task-specific concerns should be considered when selecting a data management framework based on tailored performance metrics for classification tasks.

## Summary

In the study "Benchmarking Apache Spark and Hadoop MapReduce on Big Data Classification" by Taha Tekdogan and Ali Cakmak, the authors explore the evolution of Big Data analytics tools and their ability to handle vast amounts of unstructured data. The term "Big Data Mining" arises as data mining techniques are utilized to extract valuable insights from this massive pool of information. However, the shift from small, structured, and stable data to large volumes of unstructured and rapidly changing data presents numerous challenges in data management. To address this issue, the study focuses on two prominent Big Data analytics tools - Apache Spark and Hadoop MapReduce - and conducts a comprehensive benchmarking analysis on a common task in data mining: classification. Various evaluation metrics such as execution time, accuracy, and scalability are used to compare the performance of these frameworks specifically for classification tasks. The research highlights that Spark outperforms MapReduce by being five times faster in training models. However, it is noted that Spark's performance deteriorates with larger workloads but can be significantly enhanced by scaling the environment with additional clusters. On the other hand, while MapReduce exhibits better accuracy scores in machine learning utility compared to Spark (around 3% higher even in small datasets), it does not show similar performance improvements when scaled like Spark does. The authors emphasize the importance of considering task-specific concerns when selecting a suitable data management framework based on performance metrics tailored for classification tasks. This research provides valuable insights into the comparative analysis of Apache Spark and Hadoop MapReduce in handling Big Data classification tasks, shedding light on their strengths and limitations under varying workload conditions.