A Guide to Failure in Machine Learning: Reliability and Robustness from Foundations to Practice
Auteurs : Eric Heim, Oren Wright, David Shriver
Résumé scientifique
One of the main barriers to adoption of Machine Learning (ML) is that ML models can fail unexpectedly. In this work, we aim to provide practitioners a guide to better understand why ML models fail and equip them with techniques they can use to reason about failure. Specifically, we discuss failure as either being caused by lack of reliability or lack of robustness. Differentiating the causes of failure in this way allows us to formally define why models fail from first principles and tie these definitions to engineering concepts and real-world deployment settings. Throughout the document we provide 1) a summary of important theoretic concepts in reliability and robustness, 2) a sampling current techniques that practitioners can utilize to reason about ML model reliability and robustness, and 3) examples that show how these concepts and techniques can apply to real-world settings.
Les articles autour de lui, par date et par proximité
Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article
Les articles autour de celui-ci
Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.
Les articles les plus proches
-
Machine Learning Robustness: A PrimerHoussem Ben Braiek, F. Khomh · 2024 · arXiv.org · preprint, non relu par les pairs · cité 33 fois · Pas encore résumé
Résumé scientifique
This chapter explores the foundational concept of robustness in Machine Learning (ML) and its integral role in establishing trustworthiness in Artificial Intelligence (AI) systems. The discussion begins with a detailed definition of robustness, portraying it as the ability of ML models to maintain stable performance across varied and unexpected environmental conditions. ML robustness is dissected through several lenses: its complementarity with generalizability; its status as a requirement for trustworthy AI; its adversarial vs non-adversarial aspects; its quantitative metrics; and its indicators such as reproducibility and explainability. The chapter delves into the factors that impede robu…
-
On Misbehaviour and Fault Tolerance in Machine Learning SystemsLalli Myllyaho, M. Raatikainen, T. Männistö et al. · 2021 · Journal of Systems and Software · cité 29 fois · Pas encore résumé
Résumé scientifique
Abstract Machine learning (ML) provides us with numerous opportunities, allowing ML systems to adapt to new situations and contexts. At the same time, this adaptability raises uncertainties concerning the run-time product quality or dependability, such as reliability and security, of these systems. Systems can be tested and monitored, but this does not provide protection against faults and failures in adapted ML systems themselves. We studied software designs that aim at introducing fault tolerance in ML systems so that possible problems in ML components of the systems can be avoided. The research was conducted as a case study, and its data was collected through five semi-structured intervie…
-
Trustworthy Machine LearningB'alint Mucs'anyi, Michael Kirchhof, Elisa Nguyen et al. · 2023 · arXiv.org · preprint, non relu par les pairs · Pas encore résumé
Résumé scientifique
As machine learning technology gets applied to actual products and solutions, new challenges have emerged. Models unexpectedly fail to generalize to small changes in the distribution, tend to be confident on novel data they have never seen, or cannot communicate the rationale behind their decisions effectively with the end users. Collectively, we face a trustworthiness issue with the current machine learning technology. This textbook on Trustworthy Machine Learning (TML) covers a theoretical and technical background of four key topics in TML: Out-of-Distribution Generalization, Explainability, Uncertainty Quantification, and Evaluation of Trustworthiness. We discuss important classical and c…
-
A Holistic Assessment of the Reliability of Machine Learning SystemsAnthony Corso, David Karamadian, R. Valentin et al. · 2023 · arXiv.org · preprint, non relu par les pairs · cité 13 fois · Pas encore résumé
Résumé scientifique
As machine learning (ML) systems increasingly permeate high-stakes settings such as healthcare, transportation, military, and national security, concerns regarding their reliability have emerged. Despite notable progress, the performance of these systems can significantly diminish due to adversarial attacks or environmental changes, leading to overconfident predictions, failures to detect input faults, and an inability to generalize in unexpected scenarios. This paper proposes a holistic assessment methodology for the reliability of ML systems. Our framework evaluates five key properties: in-distribution accuracy, distribution-shift robustness, adversarial robustness, calibration, and out-of…
-
Taxonomy of Machine Learning Safety: A Survey and PrimerSina Mohseni, Haotao Wang, Chaowei Xiao et al. · 2021 · ACM Computing Surveys · cité 68 fois · Pas encore résumé
Résumé scientifique
The open-world deployment of Machine Learning (ML) algorithms in safety-critical applications such as autonomous vehicles needs to address a variety of ML vulnerabilities such as interpretability, verifiability, and performance limitations. Research explores different approaches to improve ML dependability by proposing new models and training techniques to reduce generalization error, achieve domain adaptation, and detect outlier examples and adversarial attacks. However, there is a missing connection between ongoing ML research and well-established safety principles. In this article, we present a structured and comprehensive review of ML techniques to improve the dependability of ML algorit…
-
Robustness Attributes to Safeguard Machine Learning Models in ProductionHala Abdelkader, Mohamed Abdelrazek, Jean-Guy Schneider et al. · 2023 · 2023 IEEE Engineering Informatics · cité 2 fois · Pas encore résumé
Résumé scientifique
Machine learning (ML) has revolutionized various industries by enabling the development of complex models that learn from data and make accurate predictions. However, moving from prototyping ML models to production software systems poses robustness challenges due to the lack of standardization around ML tools and processes. To tackle these challenges in deployed ML models, different elements of robustness must be addressed, including transparency, safety, and security along with adopting a unified language and terminologies. This paper aims to highlight the key robustness challenges that ML models encounter when deployed in production environments, and to emphasize the significance of proact…
-
Towards a Robust and Trustworthy Machine Learning System DevelopmentPu-Lei Xiong, Scott Buffett, Shahrear Iqbal et al. · 2021 · Journal of Information Security and Applications · cité 22 fois · Pas encore résumé
Résumé scientifique
While Machine Learning (ML) technologies are widely adopted in many mission critical fields to support intelligent decision-making, concerns remain about system resilience against ML-specific security attacks and privacy breaches as well as the trust that users have in these systems. In this article, we present our recent systematic and comprehensive survey on the state-of-the-art ML robustness and trustworthiness from a security engineering perspective, focusing on the problems in system threat analysis, design and evaluation faced in developing practical machine learning applications, in terms of robustness and user trust. Accordingly, we organize the presentation of this survey intended t…
-
Avoiding common machine learning pitfallsM. Lones · 2021 · Patterns · cité 207 fois · Pas encore résumé
Résumé scientifique
Summary Mistakes in machine learning practice are commonplace and can result in loss of confidence in the findings and products of machine learning. This tutorial outlines common mistakes that occur when using machine learning and what can be done to avoid them. While it should be accessible to anyone with a basic understanding of machine learning techniques, it focuses on issues that are of particular concern within academic research, such as the need to make rigorous comparisons and reach valid conclusions. It covers five stages of the machine learning process: what to do before model building, how to reliably build models, how to robustly evaluate models, how to compare models fairly, and…
Les travaux antérieurs sur lesquels il s’appuie
-
A Survey on the Explainability of Supervised Machine LearningNadia Burkart, Marco F. Huber · 2020 · Journal of Artificial Intelligence Research · cité 1 016 fois · Pas encore résumé
Résumé scientifique
Predictions obtained by, e.g., artificial neural networks have a high accuracy but humans often perceive the models as black boxes. Insights about the decision making are mostly opaque for humans. Particularly understanding the decision making in highly sensitive areas such as healthcare or finance, is of paramount importance. The decision-making behind the black boxes requires it to be more transparent, accountable, and understandable for humans. This survey paper provides essential definitions, an overview of the different principles and methodologies of explainable Supervised Machine Learning (SML). We conduct a state-of-the-art survey that reviews past and recent explainable SML approach…
-
Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)R. Hawkins, Colin Paterson, Chiara Picardi et al. · 2021 · arXiv.org · preprint, non relu par les pairs · cité 134 fois · Pas encore résumé
Résumé scientifique
Machine Learning (ML) is now used in a range of systems with results that are reported to exceed, under certain conditions, human performance. Many of these systems, in domains such as healthcare , automotive and manufacturing, exhibit high degrees of autonomy and are safety critical. Establishing justified confidence in ML forms a core part of the safety case for these systems. In this document we introduce a methodology for the Assurance of Machine Learning for use in Autonomous Systems (AMLAS). AMLAS comprises a set of safety case patterns and a process for (1) systematically integrating safety assurance into the development of ML components and (2) for generating the evidence base for ex…
-
Test & Evaluation Best Practices for Machine Learning-Enabled SystemsJaganmohan Chandrasekaran, Tyler Cody, Nicola McCarthy et al. · 2023 · arXiv.org · preprint, non relu par les pairs · cité 10 fois · Pas encore résumé
Résumé scientifique
Machine learning (ML) - based software systems are rapidly gaining adoption across various domains, making it increasingly essential to ensure they perform as intended. This report presents best practices for the Test and Evaluation (T&E) of ML-enabled software systems across its lifecycle. We categorize the lifecycle of ML-enabled software systems into three stages: component, integration and deployment, and post-deployment. At the component level, the primary objective is to test and evaluate the ML model as a standalone component. Next, in the integration and deployment stage, the goal is to evaluate an integrated ML-enabled system consisting of both ML and non-ML components. Finally, onc…
-
Designing Data: Proactive Data Collection and Iteration for Machine LearningAspen K. Hopkins, Fred Hohman, L. Zappella et al. · 2023 · arXiv.org · preprint, non relu par les pairs · cité 7 fois · Pas encore résumé
Résumé scientifique
Lack of diversity in data collection has caused significant failures in machine learning (ML) applications. While ML developers perform post-collection interventions, these are time intensive and rarely comprehensive. Thus, new methods to track&manage data collection, iteration, and model training are necessary for evaluating whether datasets reflect real world variability. We present designing data, an iterative approach to data collection connecting HCI concepts with ML techniques. Our process includes (1) Pre-Collection Planning, to reflexively prompt and document expected data distributions; (2) Collection Monitoring, to systematically encourage sampling diversity; and (3) Data Familiari…
-
Requirements Engineering for Machine Learning: Perspectives from Data ScientistsAndreas Vogelsang, Markus Borg · 2019 · 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW) · cité 216 fois · Pas encore résumé
Résumé scientifique
Machine learning (ML) is used increasingly in real-world applications. In this paper, we describe our ongoing endeavor to define characteristics and challenges unique to Requirements Engineering (RE) for ML-based systems. As a first step, we interviewed four data scientists to understand how ML experts approach elicitation, specification, and assurance of requirements and expectations. The results show that changes in the development paradigm, i.e., from coding to training, also demands changes in RE. We conclude that development of ML systems demands requirements engineers to: (1) understand ML performance measures to state good functional requirements, (2) be aware of new quality requireme…
Les articles qui le citent
-
A Robustness Metric and Evaluation Approach for Machine Learning Components in AviationCasidhe Hutchison, T. Swierzewski, J. Neighbor et al. · 2026 · 2026 56th Annual IEEE International Conference on Dependable Systems and Networks - Supplemental Volume (DSN-S) · Pas encore résumé
Résumé scientifique
We present a stochastic approach for estimating the robustness of machine learning components in safety-critical systems using perturbation-based testing, and how to use random sampling of the perturbation parameters to get estimates of system performance across the full range of parameter space. This empirical robustness estimation approach enables Test and Evaluation/Verification and Validation of AI enabled systems in high integrity applications. We demonstrate this approach on an open dataset and show that the approach produces results that are equivalent to substantially more involved testing.
Aller plus loin
Comment ces articles sont-ils trouvés ?
Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.
Mis à jour le 3 novembre 2026.