Model-based Clustering with Missing Not At Random Data

AI-generated keywords: Technological advancements Model-based clustering Missing values MNAR data Expectation-Maximization

AI-generated Key Points

The license of the paper does not allow us to build upon its content and the key points are generated using the paper metadata rather than the full article.

  • Technological advancements have led to the collection of vast amounts of data
  • Model-based clustering is a popular methodology for exploring data within a statistical framework
  • Missing values in large datasets pose a challenge for traditional approaches
  • The paper addresses Missing Not At Random (MNAR) data, where missingness depends on observed and unobserved data values
  • Authors propose an approach that integrates MNAR data into model-based clustering algorithms using a selection model
  • The study investigates identifiability of parameters for various MNAR sub-models
  • Estimation is done using EM and Stochastic EM algorithms to handle complexities introduced by missing not at random data
  • Empirical evaluations on synthetic data sets validate the effectiveness of the proposed method
  • Application on real-world medical data from TraumaBase dataset demonstrates potential impact in practical settings
Also access our AI generated: Comprehensive summary, Lay summary, Blog-like article; or ask questions about this paper to our AI assistant.

Authors: Aude Sportisse (CRISAM, UCA), Christophe Biernacki (CNRS), Claire Boyer (SU), Julie Josse (CRISAM, IDESP), Matthieu Marbac Lourdelle (UNIV-RENNES, ENSAI, CNRS, CREST), Gilles Celeux (UCO), Fabien Laporte (UCO)

Abstract: In recent decades, technological advances have made it possible to collect large data sets. In this context, the model-based clustering is a very popular, flexible and interpretable methodology for data exploration in a well-defined statistical framework. One of the ironies of the increase of large datasets is that missing values are more frequent. However, traditional ways (as discarding observations with missing values or imputation methods) are not designed for the clustering purpose. In addition, they rarely apply to the general case, though frequent in practice, of Missing Not At Random (MNAR) values, i.e. when the missingness depends on the unobserved data values and possibly on the observed data values. The goal of this paper is to propose a novel approach by embedding MNAR data directly within model-based clustering algorithms. We introduce a selection model for the joint distribution of data and missing-data indicator. It corresponds to a mixture model for the data distribution and a general MNAR model for the missing-data mechanism, which may depend on the underlying classes (unknown) and/or the values of the missing variables themselves. A large set of meaningful MNAR sub-models is derived and the identifiability of the parameters is studied for each of the sub-models, which is usually a key issue for any MNAR proposals. The EM and Stochastic EM algorithms are considered for estimation. Finally, we perform empirical evaluations for the proposed submodels on synthetic data and we illustrate the relevance of our method on a medical register, the TraumaBase (R) dataset.

Submitted to arXiv on 20 Dec. 2021

Ask questions about this paper to our AI assistant

You can also chat with multiple papers at once here.

The license of the paper does not allow us to build upon its content and the AI assistant only knows about the paper metadata rather than the full article.

AI assistant instructions?

Results of the summarizing process for the arXiv paper: 2112.10425v1

This paper's license doesn't allow us to build upon its content and the summarizing process is here made with the paper's metadata rather than the article.

In recent years, the exponential growth of technological advancements has enabled the collection of vast amounts of data. Model-based clustering has emerged as a popular and versatile methodology for exploring data within a well-defined statistical framework. However, with the increase in large datasets, missing values have become more prevalent, posing a challenge for traditional approaches such as discarding observations with missing values or using imputation methods. This paper addresses the issue of Missing Not At Random (MNAR) data, where the missingness depends on both observed and unobserved data values. The authors propose a novel approach that integrates MNAR data directly into model-based clustering algorithms. They introduce a selection model that combines a mixture model for the data distribution with a general MNAR model for the missing-data mechanism. This model allows for dependencies on underlying classes and the values of missing variables, offering flexibility in handling complex missing data scenarios. The study derives a range of meaningful MNAR sub-models and investigates the identifiability of parameters for each sub-model—a crucial aspect in addressing MNAR scenarios. The estimation is carried out using EM (Expectation-Maximization) and Stochastic EM algorithms to handle the complexities introduced by missing not at random data. To validate their approach, empirical evaluations are conducted on synthetic data sets, demonstrating the effectiveness and relevance of the proposed method. Furthermore, the authors showcase its application on real-world medical data from the TraumaBase dataset, highlighting its potential impact in practical settings. Overall, this research contributes to advancing methodologies for handling missing not at random data within model-based clustering frameworks, offering insights into addressing challenges posed by incomplete datasets in modern statistical analysis.
Created on 11 Oct. 2024

Assess the quality of the AI-generated content by voting

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Similar papers summarized with our AI tools

Navigate through even more similar papers through a

tree representation

Look for similar papers (in beta version)

By clicking on the button above, our algorithm will scan all papers in our database to find the closest based on the contents of the full papers and not just on metadata. Please note that it only works for papers that we have generated summaries for and you can rerun it from time to time to get a more accurate result while our database grows.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.