Résultats de recherche pour « Multimodal Language Model »

The “noise resistance” of an AI model determines its usability under non-ideal real-world conditions—in technology products, scientific research, etc.—where a few characters in text or pixels in images are often inaccurate. Prior to this study, the noise resistance of classification models and text-based large language models (LLMs) had been investigated, but the noise resistance of multimodal LLMs (MLLMs) had not. Thus, I studied MLLMs’ noise resistance against both textual noise (misspellings) and image noise (Gaussian, salt-pepper, and speckle). I also employed two denoising algorithms, spell-check (“aspell”) for textual prompts and OpenCV’s “Fast NL Means” for image prompts, to see if such pre-processing improves MLLM accuracy. I developed 10 textual prompts and 30 image-based prompts, each then noised and then denoised. I tested two MLLMs (LLaVA and GPT-4o), alongside a traditional LLM (GPT-3.5) given the textual prompts for comparison. I hypothesized that MLLMs would have poor noise resistance (even worse than traditional LLMs) and be helped by denoising algorithms. The first hypothesis was supported by the data, but the second hypothesis was refuted—traditional denoising algorithms generally hurt model performance. I also predicted, though not central to my study, that lower-parameter models would fare worse, which the data supported; however, as it was not a factor I set out to measure, future controlled studies should confirm this. Future studies should employ larger sample sizes to reduce variability and experiment with using smaller AI models as denoisers. MLLM users should put effort into crafting clean prompts and avoid traditional algorithmic denoisers.

The development of multimodal large language models (MLLMs) enables the evaluation of image quality through natural language descriptions. This advancement allows for more detailed assessments. However, these MLLM-based IQA methods primarily rely on general contextual descriptions, sometimes limiting fine-grained quality assessment. To address this limitation, we introduce a new image quality assessment (IQA) task paradigm, GROUNDING-IQA. This paradigm integrates multimodal referring and grounding with IQA to realize more fine-grained quality perception. Specifically, grounding-IQA comprises two subtasks: grounding-IQA-description (GIQA-DES) and visual question answering (GIQA-VQA). GIQA-DES involves detailed descriptions with precise locations (_e.g._, bounding boxes), while GIQA-VQA focuses on quality QA for local regions. To realize grounding-IQA, we construct a corresponding dataset, GIQA-160K, through our proposed automated annotation pipeline. Furthermore, we develop a well-designed benchmark, GIQA-Bench. The benchmark comprehensively evaluates the model grounding-IQA performance from three perspectives: description quality, VQA accuracy, and grounding precision. Experiments demonstrate that our proposed task paradigm, dataset, and benchmark facilitate the more fine-grained IQA application. Code: https://github.com/zhengchen1999/Grounding-IQA.

Evaluating the improvisational capabilities of large language models (LLMs) like ChatGPT-4, Mistral, and Anthropic Claude across textual, visual, and psychological domains provides critical insights into their functionality and potential applications. The research demonstrates significant variances in the ability of these models to generate creative, contextually appropriate responses, visually coherent images from textual descriptions, and emotionally nuanced interactions. ChatGPT-4 excelled in textual improvisation, showcasing its capacity to produce linguistically rich and innovative content that pushes the boundaries of traditional text-based AI applications. Mistral distinguished itself in the generation of visual content, effectively translating abstract textual prompts into detailed and contextually relevant images, indicating its utility in creative and design fields. Anthropic Claude performed exceptionally well in psychological adaptability, interpreting and responding to emotional cues with a high degree of empathy and accuracy, suitable for customer service and therapeutic applications. The findings underscore the diverse capabilities of these LLMs, highlighting their potential to transform industries that require nuanced understanding and generation of complex content. Future research should focus on enhancing the reliability of these models across varied scenarios, improving their ethical deployment, and exploring hybrid approaches to leverage their unique strengths.

Abstract Thyroid carcinoma is one of the most prevalent endocrine malignancies worldwide, and accurate preoperative differentiation between benign and malignant thyroid nodules remains clinically challenging. Diagnostic methods that medical practitioners use at present depend on their personal judgment to evaluate both imaging results and separate clinical tests, which creates inconsistency that leads to incorrect medical evaluations. The combination of radiological imaging with clinical information systems enables healthcare providers to enhance their capacity to make reliable predictions about patient outcomes while improving their decision-making abilities. The study introduces a deep learning framework that utilizes multiple data sources by combining magnetic resonance imaging (MRI) data with clinical text to predict thyroid cancer. The system uses a Vision Transformer (ViT) to obtain advanced MRI scan features, while a domain-adapted language model processes clinical documents that contain patient medical history and symptoms and laboratory results. The cross-modal attention system enables the system to merge imaging data with textual information from different sources, which helps to identify how the two types of data are interconnected. The system uses a classification layer to classify the fused features, which allows it to determine the probability of cancerous tumors. The experimental results show that the proposed multimodal system achieves better results than the unimodal base systems because it has higher accuracy, sensitivity, specificity, and AUC values, which help medical personnel to make better preoperative decisions.

Organic synthesis relies heavily on the effective selection and optimization of catalysts, a process often limited by empirical and time-consuming trial-and-error approaches. This work introduces ChemReactLLM, a multimodal large language model designed to transform catalyst-driven organic reaction prediction and optimization. ChemReactLLM integrates a foundation language model with a specialized ReactAdapter for molecular structure embedding and a Text–Structure Cross-Attention Block that fuses textual experimental conditions with graph-based structural representations. Trained on a large multimodal dataset, the model predicts reaction outcomes, infers catalyst structure–activity relationships, and supports inverse catalyst design. Experiments demonstrate superior performance in yield prediction and robust generalization to unseen reactions. ChemReactLLM also provides mechanistic interpretability and chemically consistent reasoning, highlighting its potential to accelerate catalyst discovery and advance data-driven organic synthesis.

In such cases, the trust that can be placed in the system by the human operator is more important than the accuracy of the system used in combination with human actions, which are referred to as human and artificial intelligence (AI) systems. These are assessed in terms of their effect on joint performance (overtrust and undertrust), and their risk to developing safety-critical failures. This review concentrates on an emerging technique of trust calibration which integrates two complementary types of data: the continuous involuntary physiological data (EEG, EDA, ET, HRV and fNIRS) capturing the tacit psychological processes of a user (cognition and affective state), and the natural-language explanations generated by the large language model (LLM) reflecting the model's reasoning and uncertainty. We curate 52 peer-reviewed recently published papers from 2019 to 2025 and seminal papers on the subject of trust-in-automation, which are usually developed separately, to summarise the current state of the art of physiological sensing, and outline concepts and technical solutions for an actual multimodal closed-loop calibration architecture. The review establishes a conceptual framework that connects physiological sensing aspects with behavioural signals in domain and LLM explanation generation, summarizes comparative evidence on physiological trust-classification accuracy along with uncertainty-communication strategies for LLM, and points out methodological gaps, such as the lack of ecological validation, the degree of obtrusiveness in physiological measurements, cross-domain generalisability and absence of standard measures for calibration in LLm-based explanation generation. Lastly, we explain some implications with regard to the generation of dynamic, real-time trust calibration as a potentially promising, but largely unexamined avenue of attraction value calibration, and suggest some priority areas for empirical testing, including mandated minimum physical equipment and acceptable calibration error in "critical application" and possible longitudinal field studies.

Structure elucidation is crucial for identifying unknown chemical compounds, yet traditional spectroscopic analysis remains labour-intensive and challenging, particularly when applied to a large number of spectra. Although machine learning models have successfully predicted chemical structures from individual spectroscopic modalities, they typically fail to integrate multiple modalities concurrently, as expert chemists usually do. Here, we introduce a multimodal multitask transformer model capable of accurately predicting molecular structures from integrated spectroscopic data, including Nuclear Magnetic Resonance (NMR) and Infrared (IR) spectroscopy. Trained initially on extensive simulated datasets and subsequently finetuned on experimental spectra, our model achieves Top-1 prediction accuracies up to 96%. We demonstrate the model's capability to leverage synergistic information from different spectroscopic techniques and show that it performs on par with expert human chemists, significantly outperforming traditional computational methods. Our model represents a major advancement toward fully automated chemical analysis, offering substantial improvements in efficiency and accuracy for chemical research and discovery.