# Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning

**Authors:** Sebastian Raschka  
**arXiv:** [1811.12808](https://arxiv.org/abs/1811.12808)  
**Date:** 2018-11-13  
**Summary page:** https://www.summarizepaper.com/en/paper/1811.12808v3/

## Key points

- Importance of using appropriate techniques for model evaluation, model selection, and algorithm selection in machine learning research
- Review of various methods for each task and their advantages and disadvantages
- Recommendations to promote best practices in academic research and real-world applications
- Limitations of common methods like holdout method for model evaluation and selection, especially with small datasets
- Introduction of different flavors of bootstrap technique as an alternative for estimating performance uncertainty
- Exploration of popular cross-validation techniques like leave-one-out cross-validation and k-fold cross-validation
- Discussion on bias-variance trade-off in choosing an appropriate value for k in k-fold cross-validation
- Practical tips for making optimal choices for k based on empirical evidence
- Coverage of statistical tests for comparing different algorithms and strategies for handling multiple comparisons
- Recommendation of alternative methods like combined F-test 5x2 cross validation and nested cross validation when working with small datasets
- Overall comprehensive review serving as a valuable resource for researchers and practitioners in machine learning
- Aim to enhance the quality of research outcomes while considering practical constraints such as dataset size.

## Summary

The article "Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning" by Sebastian Raschka emphasizes the importance of using appropriate techniques for model evaluation, model selection, and algorithm selection in machine learning research. The author reviews various methods that can be employed for each of these tasks and discusses their advantages and disadvantages based on theoretical and empirical studies. The article provides recommendations to promote best practices in both academic research and real-world applications of machine learning. It highlights the limitations of common methods like the holdout method for model evaluation and selection, particularly when dealing with small datasets. Instead, the author introduces different flavors of the bootstrap technique as an alternative for estimating performance uncertainty. This approach proves useful when bootstrapping is computationally feasible. Additionally, the article explores popular cross-validation techniques such as leave-one-out cross-validation and k-fold cross-validation. It delves into the bias-variance trade-off associated with choosing an appropriate value for k in k-fold cross-validation. Practical tips are provided to aid researchers in making optimal choices for k based on empirical evidence. The article also covers statistical tests for comparing different algorithms and addresses strategies for handling multiple comparisons, including omnibus tests and multiple comparison corrections. Furthermore, it recommends alternative methods like the combined F-test 5x2 cross validation and nested cross validation when working with small datasets to facilitate algorithm selection. Overall, this comprehensive review serves as a valuable resource for researchers and practitioners in machine learning by providing insights into effective techniques for model evaluation, model selection, and algorithm selection. The recommendations presented aim to enhance the quality of research outcomes while considering practical constraints such as dataset size.