Search Results for « CLIP-based models »

Abstract In this study, we present a comprehensive evaluation of Contrastive Language-Image Pre-training (CLIP) for zero-shot electrocardiogram (ECG) classification across multiple datasets and diagnostic classes. Traditional ECG classification models require significant labeled training data for every diagnostic class, limiting their adaptability to new or unseen classes. To address this limitation, we trained and evaluated 24 CLIP-based models, composed of one image encoder (CNN Base, CNN V2, CNN V3, RNN, ISIBrno) and one text encoder (BioBERT, Bio+ClinicalBERT, bert-case-uncased) on 27 seen classes on three datasets (PTB-XL, Ningbo, and Gerogia) and evaluated zero-shot classification performance on 11 unseen classes. We investigate the impact of training dataset size, encoder architectures, and pretraining effects on both in-distribution and out-of-distribution generalization performance. Our experiments included internal evaluation (Experiment A) and external validation on two independent datasets (SPH and CODE-15%, Experiment B). The top-performing models achieves a macro-averaged ROC-AUC of 0.70 for zero-shot out-of-distribution classification, 0.70 for zero-shot in-distribution classification, and 0.83 for out-of-distribution classification with classic training. These results demonstrate that CLIP-based models can meaningfully classify ECG conditions beyond the scope of their training, offering a flexible alternative to current ECG diagnostic systems where clinical needs are constantly evolving.

The effectiveness of the ImageNet diffusion model and CLIP models for image generation based on textual descriptions was investigated. Two experiments were conducted using various textual inputs and different parameters to determine the optimal settings for generating images from text descriptions. The results showed that while ImageNet performed well in generating images, CLIP demonstrated better alignment between textual prompts and relevant images. The obtained results highlight the high potential of combining these mentioned models for creating high-quality and contextually relevant images based on textual descriptions.