A Guide to Failure in Machine Learning: Reliability and Robustness from Foundations to Practice

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

1. Very close Review Not relevant

   Machine Learning Robustness: A Primer

   Houssem Ben Braiek, F. Khomh · 2024 · arXiv.org · preprint, not peer reviewed · cited 33 times · Not summarized yet

   Machine Learning Robustness: A Primer explains robustness, its relation to generalization, and ways to assess it in trustworthy AI.

   Abstract

   This chapter explores the foundational concept of robustness in Machine Learning (ML) and its integral role in establishing trustworthiness in Artificial Intelligence (AI) systems. The discussion begins with a detailed definition of robustness, portraying it as the ability of ML models to maintain stable performance across varied and unexpected environmental conditions. ML robustness is dissected through several lenses: its complementarity with generalizability; its status as a requirement for trustworthy AI; its adversarial vs non-adversarial aspects; its quantitative metrics; and its indicators such as reproducibility and explainability. The chapter delves into the factors that impede robu…

2. Close Broader context Not relevant

   On Misbehaviour and Fault Tolerance in Machine Learning Systems

   Lalli Myllyaho, M. Raatikainen, T. Männistö et al. · 2021 · Journal of Systems and Software · cited 29 times · Not summarized yet

   On Misbehaviour and Fault Tolerance in Machine Learning Systems studies design strategies for preventing faults in adaptive ML systems.

   Abstract

   Abstract Machine learning (ML) provides us with numerous opportunities, allowing ML systems to adapt to new situations and contexts. At the same time, this adaptability raises uncertainties concerning the run-time product quality or dependability, such as reliability and security, of these systems. Systems can be tested and monitored, but this does not provide protection against faults and failures in adapted ML systems themselves. We studied software designs that aim at introducing fault tolerance in ML systems so that possible problems in ML components of the systems can be avoided. The research was conducted as a case study, and its data was collected through five semi-structured intervie…

3. Close Review Not relevant

   Trustworthy Machine Learning

   B'alint Mucs'anyi, Michael Kirchhof, Elisa Nguyen et al. · 2023 · arXiv.org · preprint, not peer reviewed · Not summarized yet

   Trustworthy Machine Learning covers out-of-distribution generalization, uncertainty, explainability, and evaluation of trustworthiness.

   Abstract

   As machine learning technology gets applied to actual products and solutions, new challenges have emerged. Models unexpectedly fail to generalize to small changes in the distribution, tend to be confident on novel data they have never seen, or cannot communicate the rationale behind their decisions effectively with the end users. Collectively, we face a trustworthiness issue with the current machine learning technology. This textbook on Trustworthy Machine Learning (TML) covers a theoretical and technical background of four key topics in TML: Out-of-Distribution Generalization, Explainability, Uncertainty Quantification, and Evaluation of Trustworthiness. We discuss important classical and c…

4. Close Another approach to the problem Not relevant

   A Holistic Assessment of the Reliability of Machine Learning Systems

   Anthony Corso, David Karamadian, R. Valentin et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 13 times · Not summarized yet

   A Holistic Assessment of the Reliability of Machine Learning Systems proposes a framework measuring reliability properties including accuracy, robustness, and calibration.

   Abstract

   As machine learning (ML) systems increasingly permeate high-stakes settings such as healthcare, transportation, military, and national security, concerns regarding their reliability have emerged. Despite notable progress, the performance of these systems can significantly diminish due to adversarial attacks or environmental changes, leading to overconfident predictions, failures to detect input faults, and an inability to generalize in unexpected scenarios. This paper proposes a holistic assessment methodology for the reliability of ML systems. Our framework evaluates five key properties: in-distribution accuracy, distribution-shift robustness, adversarial robustness, calibration, and out-of…

5. Close Review Not relevant

   Taxonomy of Machine Learning Safety: A Survey and Primer

   Sina Mohseni, Haotao Wang, Chaowei Xiao et al. · 2021 · ACM Computing Surveys · cited 68 times · Not summarized yet

   Taxonomy of Machine Learning Safety surveys techniques for ML dependability and connects them with established safety principles.

   Abstract

   The open-world deployment of Machine Learning (ML) algorithms in safety-critical applications such as autonomous vehicles needs to address a variety of ML vulnerabilities such as interpretability, verifiability, and performance limitations. Research explores different approaches to improve ML dependability by proposing new models and training techniques to reduce generalization error, achieve domain adaptation, and detect outlier examples and adversarial attacks. However, there is a missing connection between ongoing ML research and well-established safety principles. In this article, we present a structured and comprehensive review of ML techniques to improve the dependability of ML algorit…

6. Close Broader context Not relevant

   Robustness Attributes to Safeguard Machine Learning Models in Production

   Hala Abdelkader, Mohamed Abdelrazek, Jean-Guy Schneider et al. · 2023 · 2023 IEEE Engineering Informatics · cited 2 times · Not summarized yet

   Robustness Attributes to Safeguard Machine Learning Models in Production discusses robustness challenges and safeguards for deployed ML models.

   Abstract

   Machine learning (ML) has revolutionized various industries by enabling the development of complex models that learn from data and make accurate predictions. However, moving from prototyping ML models to production software systems poses robustness challenges due to the lack of standardization around ML tools and processes. To tackle these challenges in deployed ML models, different elements of robustness must be addressed, including transparency, safety, and security along with adopting a unified language and terminologies. This paper aims to highlight the key robustness challenges that ML models encounter when deployed in production environments, and to emphasize the significance of proact…

7. Close Review Not relevant

   Towards a Robust and Trustworthy Machine Learning System Development

   Pu-Lei Xiong, Scott Buffett, Shahrear Iqbal et al. · 2021 · Journal of Information Security and Applications · cited 22 times · Not summarized yet

   Towards a Robust and Trustworthy Machine Learning System Development surveys robustness and user trust from a security-engineering perspective.

   Abstract

   While Machine Learning (ML) technologies are widely adopted in many mission critical fields to support intelligent decision-making, concerns remain about system resilience against ML-specific security attacks and privacy breaches as well as the trust that users have in these systems. In this article, we present our recent systematic and comprehensive survey on the state-of-the-art ML robustness and trustworthiness from a security engineering perspective, focusing on the problems in system threat analysis, design and evaluation faced in developing practical machine learning applications, in terms of robustness and user trust. Accordingly, we organize the presentation of this survey intended t…

8. Close Broader context Not relevant

   Avoiding common machine learning pitfalls

   M. Lones · 2021 · Patterns · cited 207 times · Not summarized yet

   Avoiding common machine learning pitfalls gives practical guidance on building and evaluating models to prevent mistakes in research and applications.

   Abstract

   Summary Mistakes in machine learning practice are commonplace and can result in loss of confidence in the findings and products of machine learning. This tutorial outlines common mistakes that occur when using machine learning and what can be done to avoid them. While it should be accessible to anyone with a basic understanding of machine learning techniques, it focuses on issues that are of particular concern within academic research, such as the need to make rigorous comparisons and reach valid conclusions. It covers five stages of the machine learning process: what to do before model building, how to reliably build models, how to robustly evaluate models, how to compare models fairly, and…

Prior work it builds on

1. Related Broader context Used in depth by this paper Not relevant

   A Survey on the Explainability of Supervised Machine Learning

   Nadia Burkart, Marco F. Huber · 2020 · Journal of Artificial Intelligence Research · cited 1,016 times · Not summarized yet

   A Survey on the Explainability of Supervised Machine Learning reviews methods for making supervised models' decisions understandable to people.

   Abstract

   Predictions obtained by, e.g., artificial neural networks have a high accuracy but humans often perceive the models as black boxes. Insights about the decision making are mostly opaque for humans. Particularly understanding the decision making in highly sensitive areas such as healthcare or finance, is of paramount importance. The decision-making behind the black boxes requires it to be more transparent, accountable, and understandable for humans. This survey paper provides essential definitions, an overview of the different principles and methodologies of explainable Supervised Machine Learning (SML). We conduct a state-of-the-art survey that reviews past and recent explainable SML approach…

2. Related Broader context Not relevant

   Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)

   R. Hawkins, Colin Paterson, Chiara Picardi et al. · 2021 · arXiv.org · preprint, not peer reviewed · cited 134 times · Not summarized yet

   Guidance on the Assurance of Machine Learning in Autonomous Systems introduces safety-case processes for assuring ML components in autonomous systems.

   Abstract

   Machine Learning (ML) is now used in a range of systems with results that are reported to exceed, under certain conditions, human performance. Many of these systems, in domains such as healthcare , automotive and manufacturing, exhibit high degrees of autonomy and are safety critical. Establishing justified confidence in ML forms a core part of the safety case for these systems. In this document we introduce a methodology for the Assurance of Machine Learning for use in Autonomous Systems (AMLAS). AMLAS comprises a set of safety case patterns and a process for (1) systematically integrating safety assurance into the development of ML components and (2) for generating the evidence base for ex…

3. Related Broader context Not relevant

   Test & Evaluation Best Practices for Machine Learning-Enabled Systems

   Jaganmohan Chandrasekaran, Tyler Cody, Nicola McCarthy et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 10 times · Not summarized yet

   Test & Evaluation Best Practices for Machine Learning-Enabled Systems sets out lifecycle testing practices for ML-enabled software.

   Abstract

   Machine learning (ML) - based software systems are rapidly gaining adoption across various domains, making it increasingly essential to ensure they perform as intended. This report presents best practices for the Test and Evaluation (T&E) of ML-enabled software systems across its lifecycle. We categorize the lifecycle of ML-enabled software systems into three stages: component, integration and deployment, and post-deployment. At the component level, the primary objective is to test and evaluate the ML model as a standalone component. Next, in the integration and deployment stage, the goal is to evaluate an integrated ML-enabled system consisting of both ML and non-ML components. Finally, onc…

4. Related Broader context Not relevant

   Designing Data: Proactive Data Collection and Iteration for Machine Learning

   Aspen K. Hopkins, Fred Hohman, L. Zappella et al. · 2023 · arXiv.org · preprint, not peer reviewed · cited 7 times · Not summarized yet

   Designing Data: Proactive Data Collection and Iteration for Machine Learning offers an iterative process for collecting diverse data that reflect real-world variability.

   Abstract

   Lack of diversity in data collection has caused significant failures in machine learning (ML) applications. While ML developers perform post-collection interventions, these are time intensive and rarely comprehensive. Thus, new methods to track&manage data collection, iteration, and model training are necessary for evaluating whether datasets reflect real world variability. We present designing data, an iterative approach to data collection connecting HCI concepts with ML techniques. Our process includes (1) Pre-Collection Planning, to reflexively prompt and document expected data distributions; (2) Collection Monitoring, to systematically encourage sampling diversity; and (3) Data Familiari…

5. Related Broader context Not relevant

   Requirements Engineering for Machine Learning: Perspectives from Data Scientists

   Andreas Vogelsang, Markus Borg · 2019 · 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW) · cited 216 times · Not summarized yet

   Requirements Engineering for Machine Learning: Perspectives from Data Scientists examines how ML-specific performance and quality requirements are elicited and assured.

   Abstract

   Machine learning (ML) is used increasingly in real-world applications. In this paper, we describe our ongoing endeavor to define characteristics and challenges unique to Requirements Engineering (RE) for ML-based systems. As a first step, we interviewed four data scientists to understand how ML experts approach elicitation, specification, and assurance of requirements and expectations. The results show that changes in the development paradigm, i.e., from coding to training, also demands changes in RE. We conclude that development of ML systems demands requirements engineers to: (1) understand ML performance measures to state good functional requirements, (2) be aware of new quality requireme…

Papers that cite it

1. Close Another approach to the problem Not relevant

   A Robustness Metric and Evaluation Approach for Machine Learning Components in Aviation

   Casidhe Hutchison, T. Swierzewski, J. Neighbor et al. · 2026 · 2026 56th Annual IEEE International Conference on Dependable Systems and Networks - Supplemental Volume (DSN-S) · Not summarized yet

   A Robustness Metric and Evaluation Approach for Machine Learning Components in Aviation estimates robustness through perturbation-based testing and sampling.

   Abstract

   We present a stochastic approach for estimating the robustness of machine learning components in safety-critical systems using perturbation-based testing, and how to use random sampling of the perturbation parameters to get estimates of system performance across the full range of parameter space. This empirical robustness estimation approach enables Test and Evaluation/Verification and Validation of AI enabled systems in high integrity applications. We demonstrate this approach on an open dataset and show that the approach produces results that are equivalent to substantially more involved testing.