Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning
Auteurs : Sebastian Raschka
Résumé scientifique
The correct use of model evaluation, model selection, and algorithm selection techniques is vital in academic machine learning research as well as in many industrial settings. This article reviews different techniques that can be used for each of these three subtasks and discusses the main advantages and disadvantages of each technique with references to theoretical and empirical studies. Further, recommendations are given to encourage best yet feasible practices in research and applications of machine learning. Common methods such as the holdout method for model evaluation and selection are covered, which are not recommended when working with small datasets. Different flavors of the bootstrap technique are introduced for estimating the uncertainty of performance estimates, as an alternative to confidence intervals via normal approximation if bootstrapping is computationally feasible. Common cross-validation techniques such as leave-one-out cross-validation and k-fold cross-validation are reviewed, the bias-variance trade-off for choosing k is discussed, and practical tips for the optimal choice of k are given based on empirical evidence. Different statistical tests for algorithm comparisons are presented, and strategies for dealing with multiple comparisons such as omnibus tests and multiple-comparison corrections are discussed. Finally, alternative methods for algorithm selection, such as the combined F-test 5x2 cross-validation and nested cross-validation, are recommended for comparing machine learning algorithms when datasets are small.
Les articles autour de lui, par date et par proximité
Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article
Les articles autour de celui-ci
Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.
Les articles les plus proches
-
Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric SelectionXuan-Yan Liu, I. C. Martin, M. Trovati et al. · 2026 · arXiv.org · preprint, non relu par les pairs · Pas encore résumé
Résumé scientifique
The evaluation of supervised machine learning models is a critical stage in the development of reliable predictive systems. Despite the widespread availability of machine learning libraries and automated workflows, model assessment is often reduced to the reporting of a small set of aggregate metrics, which can lead to misleading conclusions about real-world performance. This paper examines the principles, challenges, and practical considerations involved in evaluating supervised learning algorithms across classification and regression tasks. In particular, it discusses how evaluation outcomes are influenced by dataset characteristics, validation design, class imbalance, asymmetric error cos…
-
A Comprehensive Review of Cross-Validation Techniques in Machine LearningMeenu Bhagat, Brijesh Bakariya · 2025 · International Journal for Sciences and Technology · cité 46 fois · Pas encore résumé
Résumé scientifique
In order to make sure that machine learning models are reliable and broadly applicable, cross-validation approaches are essential. They offer a methodical approach for adjusting hyperparameters, assessing model performance, and resolving issues with overfitting, unbalanced data, and temporal dependencies. This review article provides a thorough analysis of the many cross-validation strategies used in machine learning, from conventional techniques like k-fold cross-validation to more specialized strategies for particular kinds of data and learning objectives. In addition to current developments and best practices in cross-validation methodology, we go over the fundamentals, uses, benefits, an…
-
Testing Of Hypothesis for Model SelectionR.Srilatha, C. Kumar, A. Reddy et al. · 2025 · International Journal For Multidisciplinary Research · Pas encore résumé
Résumé scientifique
This paper examines the application of hypothesis testing in machine learning model selection, focusing on distinguishing between statistically significant performance differences and random variations. We demonstrate how statistical tests such as t-tests and ANOVA can be effectively combined with traditional evaluation metrics including accuracy, F1-score, and precision to validate model performance. This integration, along with cross-validation techniques, helps ensure model generalization while mitigating overfitting risks.
-
Performance of Machine Learning Algorithms with Different K Values in K-fold CrossValidationIsaac Kofi Nti, Owusu N yarko-Boateng, Justice Aning · 2021 · International Journal of Information Technology and Computer Science · cité 220 fois · Pas encore résumé
Résumé scientifique
The numerical value of k in a k-fold cross-validation training technique of machine learning predictive models is an essential element that impacts the model’s performance. A right choice of k results in better accuracy, while a poorly chosen value for k might affect the model’s performance. In literature, the most commonly used values of k are five (5) or ten (10), as these two values are believed to give test error rate estimates that suffer neither from extremely high bias nor very high variance. However, there is no formal rule. To the best of our knowledge, few experimental studies attempted to investigate the effect of diverse k values in training different machine learning models. Thi…
-
Machine Learning Revolutionizing Performance Evaluation: Recent Developments and BreakthroughsV. Raja, D. M, Gautam Solaimalai et al. · 2024 · 2024 2nd International Conference on Sustainable Computing and Smart Systems (ICSCSS) · cité 20 fois · Pas encore résumé
Résumé scientifique
Machine learning is used in many fields because of its rapid development. To determine their efficacy and suitability for certain activities, these models must be evaluated. This study maps the knowledge domain of machine learning approaches to evaluate performance, revealing its important components. Performance metrics-quantitative measurements of machine learning model performance are examined first. Predictive and generalization metrics include accuracy, precision, recall, F1-score, and area under the ROC curve. Performance evaluation requires understanding these indicators' calculation and interpretation. Cross-validation, holdout validation, k-fold cross-validation, and bootstrapping a…
-
Evaluation of Regression Models: Model Assessment, Model Selection and Generalization ErrorF. Emmert-Streib, Matthias Dehmer · 2019 · Machine Learning and Knowledge Extraction · cité 133 fois · Pas encore résumé
Résumé scientifique
When performing a regression or classification analysis, one needs to specify a statistical model. This model should avoid the overfitting and underfitting of data, and achieve a low generalization error that characterizes its prediction performance. In order to identify such a model, one needs to decide which model to select from candidate model families based on performance evaluations. In this paper, we review the theoretical framework of model selection and model assessment, including error-complexity curves, the bias-variance tradeoff, and learning curves for evaluating statistical models. We discuss criterion-based, step-wise selection procedures and resampling methods for model select…
-
An Analysis on Performance Scale and Model Validation in Machine LearningKrishna Gangu, Shikha Tayal, Richa Mishra et al. · 2024 · International Conferences on Contemporary Computing and Informatics · cité 2 fois · Pas encore résumé
Résumé scientifique
This research examines the performance and validation techniques of various machine learning models, including Support Vector Machines (SVM) and Linear Regression Non-Linear Regression, Neural Networks, Random Forest, and K-Nearest Neighbours (KNN), with a focus on their effectiveness with and without model validation. The study follows a thorough process that includes data collecting from many sources, diligent dataset compilation, and rigorous model validation. Data cleansing, missing value management, feature scaling, categorical variable encoding, and cross-validation implementation are all critical aspects of data preparation. The performance criteria examined were accuracy, precision, …
-
Avoiding common machine learning pitfallsM. Lones · 2021 · Patterns · cité 211 fois · Pas encore résumé
Résumé scientifique
Summary Mistakes in machine learning practice are commonplace and can result in loss of confidence in the findings and products of machine learning. This tutorial outlines common mistakes that occur when using machine learning and what can be done to avoid them. While it should be accessible to anyone with a basic understanding of machine learning techniques, it focuses on issues that are of particular concern within academic research, such as the need to make rigorous comparisons and reach valid conclusions. It covers five stages of the machine learning process: what to do before model building, how to reliably build models, how to robustly evaluate models, how to compare models fairly, and…
Les travaux antérieurs sur lesquels il s’appuie
-
Bias in error estimation when using cross-validation for model selectionSudhir Varma, R. Simon · 2006 · BMC Bioinformatics · cité 2 523 fois · Pas encore résumé
Résumé scientifique
BackgroundCross-validation (CV) is an effective method for estimating the prediction error of a classifier. Some recent articles have proposed methods for optimizing classifiers by choosing classifier parameter values that minimize the CV error estimate. We have evaluated the validity of using the CV error estimate of the optimized classifier as an estimate of the true error expected on independent data.ResultsWe used CV to optimize the classification parameters for two kinds of classifiers; Shrunken Centroids and Support Vector Machines (SVM). Random training datasets were created, with no difference in the distribution of the features between the two classes. Using these "null" datasets, w…
-
Approximate Statistical Tests for Comparing Supervised Classification Learning AlgorithmsThomas G. Dietterich · 1998 · Neural Computation · cité 4 342 fois · Pas encore résumé
-
Cross‐validation failure: Small sample sizes lead to large error barsG. Varoquaux · 2017 · NeuroImage · cité 903 fois · Pas encore résumé
Résumé scientifique
ABSTRACT Predictive models ground many state‐of‐the‐art developments in statistical brain image analysis: decoding, MVPA, searchlight, or extraction of biomarkers. The principled approach to establish their validity and usefulness is cross‐validation, testing prediction on unseen data. Here, I would like to raise awareness on error bars of cross‐validation, which are often underestimated. Simple experiments show that sample sizes of many neuroimaging studies inherently lead to large error bars, eg Symbol for 100 samples. The standard error across folds strongly underestimates them. These large error bars compromise the reliability of conclusions drawn with predictive models, such as biomarke…
-
Scikit-learn: Machine Learning in PythonFabian Pedregosa, G. Varoquaux, Alexandre Gramfort et al. · 2011 · Journal of machine learning research · cité 97 935 fois · Pas encore résumé
Résumé scientifique
Scikit-learn is a Python module integrating a wide range of state-of-the-art machine learning algorithms for medium-scale supervised and unsupervised problems. This package focuses on bringing machine learning to non-specialists using a general-purpose high-level language. Emphasis is put on ease of use, performance, documentation, and API consistency. It has minimal dependencies and is distributed under the simplified BSD license, encouraging its use in both academic and commercial settings. Source code, binaries, and documentation can be downloaded from http://scikit-learn.sourceforge.net.
-
Bootstrap Methods: Another Look at the JackknifeD. Hinkley · 2008 · cité 10 037 fois · Pas encore résumé
Les articles qui le citent
-
The use of cross validation in the analysis of designed experimentsMaria L. Weese, Byran J. Smucker, David J. Edwards · 2025 · cité 3 fois · Pas encore résumé
Résumé scientifique
Cross-validation (CV) is a common method to tune machine learning methods and can be used for model selection in regression as well. Because of the structured nature of small, traditional experimental designs, the literature has warned against using CV in their analysis. The striking increase in the use of machine learning, and thus CV, in the analysis of experimental designs, has led us to empirically study the effectiveness of CV compared to other methods of selecting models in designed experiments, including the little bootstrap. We consider both response surface settings where prediction is of primary interest, as well as screening where factor selection is most important. Overall, we pr…
-
Personalized Treatment Planning using Machine Learning Models: A ReviewMohit Singh, Richa Sharma, Deepika · 2025 · 2025 4th International Conference on Automation, Computing and Renewable Systems (ICACRS) · Pas encore résumé
Résumé scientifique
Data preprocessing is a critical determinant of machine learning (ML) model performance, yet systematic synthesis of its techniques and outcomes remains limited. This review adheres to the PRISMA guidelines for evaluating normalization, standardization, missing value imputation, categorical encoding, and data augmentation. The research screened 1,250 studies from IEEE Xplore, PubMed, and Scopus (2015–2023), selecting 29 peer-reviewed articles meeting inclusion criteria (empirical validation, comparative analysis, and reproducibility). The findings reveal that normalization and standardization improve the accuracy of the neural network by up to 15%, while advanced imputation (e.g., MICE) incr…
-
The Impact of Feature Scaling in Machine Learning: Effects on Regression and Classification TasksJ. M. Pinheiro, Suzana Vilas Boas de Oliveira, T. H. Silva et al. · 2025 · IEEE Access · cité 86 fois · Pas encore résumé
Résumé scientifique
This study addresses the lack of comprehensive evaluations of feature scaling by systematically assessing 12 techniques, including less common methods such as VAST and Pareto, in 14 machine learning algorithms and 16 datasets covering both classification and regression tasks. The impact of feature scaling was evaluated in terms of predictive performance (accuracy, MAE, MSE, R2) and computational efficiency (training time, inference time, memory usage). The results show that the ensemble methods (Random Forest, XGBoost, CatBoost, LightGBM) remain robust regardless of scaling, while models such as Logistic Regression, SVM, TabNet, and MLP are highly sensitive to the chosen scaler. By making al…
-
A new intelligent model selection approachCagatay Bal, Ç. Aladag · 2025 · Soft Computing - A Fusion of Foundations, Methodologies and Applications · Pas encore résumé
Résumé scientifique
Modelling spans various scientific fields, including statistics, mathematics, machine learning, artificial intelligence, deep learning, engineering, and bioinformatics. This interdisciplinary nature necessitates the creation of simple representations for modelling results. Performance measures play a crucial role in evaluating these models, indicating their effectiveness through scalar values, ratios, vectors, diagrams, or more complex indicators. These measures serve as the identities of models, highlighting their strengths and weaknesses. Thus, selecting the most appropriate performance measure is pivotal in model identification and comparison. Performance evaluation aids in selecting the …
-
Rigorous Viability Assessment of Machine Learning ProjectsDomenique Zipperling, Lorenz Ott, Michael Vössing et al. · 2026 · Business & Information Systems Engineering · cité 1 fois · Pas encore résumé
Résumé scientifique
Machine learning offers significant potential for organizations, yet transitioning models from development to deployment remains challenging. Frameworks such as CRISP-ML(Q) and MLOps emphasize the need to integrate business, economic, and machine learning perspectives. However, a systematic literature review reveals a lack of methods that link machine learning perspectives with business objectives. To address this gap, the authors introduce a metric – called profit-per-decision (ppd) – for binary classification that incorporates both model performance and economic impacts. Further, the Viability Assessment Framework is proposed, which utilizes the metric and enables organizations to assess v…
Aller plus loin
Comment ces articles sont-ils trouvés ?
Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.
Mis à jour le 5 novembre 2026.