Do Multimodal Models Decide Consistently on Equivalent Optical Spectra?
Auteurs : Ziyuan Wang, Genliang Zhu, Xiaobo Chen
Résumé scientifique
A spectrum can be given to a model as a table in nanometers or as a plot in photon energy, and the decision should not depend on which one it sees. When the data needed for an answer are missing, the model should say so. We test representation invariance and abstention on measured optical spectra of silicon metasurfaces (MOCLIP) and gold nanoparticles, with labels computed from the data. Each spectrum is shown as four tables (nm, $μ$m, scientific notation, eV) and two plots (nm, eV), and partial spectra remove data inside or outside the queried range. We evaluate the decision models d3-nano and Jev 1.13, three GPT-6 Luna configurations and the Qwen3.5-2B base model. Changing only the representation moves accuracy by up to 41 percentage points within one model, mostly on photon-energy tables and plots. With the same evidence, asking in photon energy instead of wavelength helps some model-view pairs by 16-50 points and hurts others by up to 25. When the relevant extremum is removed, unadapted models rarely answer "undetermined". LoRA fine-tuning of d3-nano improves comparisons on the trained table formats, but much of its gain on unanswerable questions comes from a shortcut: it abstains whenever the input is truncated. Training with answerable truncations avoids the shortcut in all six seeds, with a 90-100 point gap between abstention after inside and outside cuts on held-out structures; with the same amount of extra original-task data, five of six seeds keep the shortcut. On gold spectra the two training sets perform similarly. Multimodal language models used for scientific decisions should be tested on equivalent representations and on missing evidence, not only on average accuracy. We release the tasks, per-response results and analysis code.
Les articles autour de lui, par date et par proximité
Plus haut, c’est plus proche de cet article ; à gauche, c’est plus ancien. Les traits relient les articles proches entre eux : les groupes de traits sont des sous-thèmes. Sélectionnez un point pour voir pourquoi il est là. Retour au résumé de cet article
Les articles autour de celui-ci
Classés selon la proximité de leur titre et de leur résumé avec ceux de cet article. Sélectionnez-en pour les lire ensemble.
Les articles les plus proches
-
Evaluating large language models on multimodal chemistry olympiad examsYiming Cui, Xin Yao, Yuxuan Qin et al. · 2025 · Communications Chemistry · cité 17 fois · Pas encore résumé
«Evaluating large language models on multimodal chemistry olympiad exams» teste le raisonnement visuel et textuel de modèles multimodaux sur des questions de chimie, plutôt que sur des spectres optiques équivalents.
Résumé scientifique
Multimodal scientific reasoning remains a significant challenge for large language models (LLMs), particularly in chemistry, where problem-solving relies on symbolic diagrams, molecular structures, and structured visual data. Here, we systematically evaluate 40 proprietary and open-source multimodal LLMs, including GPT-5, o3, Gemini-2.5-Pro, and Qwen2.5-VL, on a curated benchmark of Olympiad-style chemistry questions drawn from over two decades of U.S. National Chemistry Olympiad (USNCO) exams. These questions require integrated visual and textual reasoning across diverse modalities. We find that many models struggle with modality fusion, where, in some cases, removing the image even improve…
-
Unified neural network model for predicting optical responses in gold nanostructuresXiao-Bo Wang, Yu-Chen Cao, Feng Feng et al. · 2025 · Journal of Applied Physics · cité 6 fois · Pas encore résumé
«Unified neural network model for predicting optical responses in gold nanostructures» prédit les spectres de nanostructures d’or à partir de leur géométrie, tandis que cet article évalue des décisions sur des spectres mesurés.
Résumé scientifique
In this paper, we present a deep neural network model capable of simultaneously predicting the optical transmittance spectra of three distinct gold nanostructures: nanodisks, truncated cones, and nanocones. Our model achieved a mean squared error loss as low as 0.0015. By systematically varying geometric parameters, we generated a comprehensive dataset to train the model, which captures complex dependencies between structure and optical response. A key finding is the redshift in resonance dip associated with increasing sidewall tilt. We further validated the model by comparing its predictions with experimental results, showing excellent agreement. This work demonstrates a powerful, data-driv…
-
SpecVQA: A Benchmark for Spectral Understanding and Visual Question Answering in Scientific ImagesJialun Shen, Han Lyu, Su-Yang Zhong et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 1 fois · Pas encore résumé
SpecVQA évalue la compréhension de spectres et les réponses à des questions scientifiques sur des images, comme cet article évalue les décisions de modèles multimodaux sur des spectres.
Résumé scientifique
Spectra are a prevalent yet highly information-dense form of scientific imagery, presenting substantial challenges to multimodal large language models (MLLMs) due to their unstructured and domain-specific characteristics. Here we introduce SpecVQA, a professional scientific-image benchmark for evaluating multimodal models on scientific spectral understanding, covering 7 representative spectrum types with expert-annotated question-answer pairs. The aim comprises two aspects: spectra scientific QA evaluation and corresponding underlying task evaluation. SpecVQA contains 620 figures and 3100 QA pairs curated from peer-reviewed literature, targeting both direct information extraction and domain-…
-
Probing the limitations of multimodal language models for chemistry and materials researchNawaf Alampara, Mara Schilling-Wilhelmi, Martiño Ríos-García et al. · 2024 · Nature Computational Science · cité 89 fois · Pas encore résumé
MaCBench évalue des modèles vision-langage sur des tâches de chimie et de science des matériaux, dont l’interprétation de résultats, sans se limiter à l’équivalence des formats spectraux.
Résumé scientifique
Recent advancements in artificial intelligence have sparked interest in scientific assistants that could support researchers across the full spectrum of scientific workflows, from literature review to experimental design and data analysis. A key capability for such systems is the ability to process and reason about scientific information in both visual and textual forms—from interpreting spectroscopic data to understanding laboratory set-ups. Here we introduce MaCBench, a comprehensive benchmark for evaluating how vision language models handle real-world chemistry and materials science tasks across three core aspects: data extraction, experimental execution and results interpretation. Throug…
-
Reading and steering representations of materials-science mechanisms in an open-weight language modelMarkus J. Buehler · 2026 · AI for Science · cité 1 fois · Pas encore résumé
«Reading and steering representations of materials-science mechanisms in an open-weight language model» étudie les représentations internes et le raisonnement en science des matériaux, plutôt que la cohérence des décisions sur des spectres.
Résumé scientifique
Large language models can answer scientific questions, yet a correct output does not reveal whether the model represents or uses the governing physics. Here, using three open-weight Gemma 4 instruction-tuned models (google/gemma-4-E4B-it, google/gemma-4-12B-it, and google/gemma-4-31B-it) we identify three experimentally separable signatures of materials-science mechanism information: selective concept readability, relational encoding of qualitative constitutive orientation, and causal, context-dependent control of constrained engineering answers. Across 50 held-out descriptions, three independently fitted Jacobian lenses produced nearly identical concept ranks, but 36 prompts recovered no de…
-
Good Scores, Bad Data: A Metric for Multimodal CoherenceV. Srinivasan · 2026 · arXiv.org · preprint, non relu par les pairs · Pas encore résumé
«Good Scores, Bad Data» mesure la cohérence multimodale sur des images Visual Genome et COCO, tandis que cet article teste la stabilité des décisions sur des spectres.
Résumé scientifique
Multimodal AI systems are evaluated by downstream task accuracy, but high accuracy does not mean the underlying data is coherent. A model can score well on Visual Question Answering (VQA) while its inputs contradict each other. We introduce the Multimodal Coherence Score (MCS), a metric that evaluates fusion quality independent of any downstream model. MCS decomposes coherence into four dimensions, identity, spatial, semantic, and decision, with weights learned via Nelder-Mead optimization. We evaluate on 1,000 Visual Genome images using DETR, CLIP, and ViLT, and validate on 150 COCO images with no retraining. Across three fusion architectures, MCS discriminates quality with higher sensitivi…
-
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language ModelsChaoyou Fu, Peixian Chen, Yunhang Shen et al. · 2023 · Computer Vision and Pattern Recognition · cité 1 927 fois · Résumé IA disponible
MME propose un benchmark général de perception et cognition multimodales, alors que cet article cible la cohérence des réponses à des spectres représentés de plusieurs façons.
Résumé scientifique
Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks, showing amazing emergent abilities in recent studies, such as writing poems based on an image. However, it is difficult for these case studies to fully reflect the performance of MLLM, lacking a comprehensive evaluation. In this paper, we fill in this blank, presenting the first MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks. In order to avoid data leakage that may arise from direct use of public datasets for evaluation, the annotations of instruction-answer pairs are all manually designed. The concise instruction design allows us to fai…
-
Same Answer, Different Representations: Hidden instability in VLMsFarooq Ahmad Wani, Alessandro Suglia, Rohit Saxena et al. · 2026 · Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing · cité 4 fois · Pas encore résumé
«Same Answer, Different Representations» étudie la stabilité des modèles vision-langage sous des représentations différentes, question également testée ici avec des formats équivalents de spectres.
Résumé scientifique
The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial intern…
Les travaux antérieurs sur lesquels il s’appuie
-
Large Language Models Are Not Robust Multiple Choice SelectorsChujie Zheng, Hao Zhou, Fan-Dong Meng et al. · 2023 · International Conference on Learning Representations · cité 725 fois · Pas encore résumé
«Large Language Models Are Not Robust Multiple Choice Selectors» teste les biais de position dans les choix multiples, non les variations de représentation des données scientifiques.
Résumé scientifique
Multiple choice questions (MCQs) serve as a common yet important task format in the evaluation of large language models (LLMs). This work shows that modern LLMs are vulnerable to option position changes in MCQs due to their inherent"selection bias", namely, they prefer to select specific option IDs as answers (like"Option A"). Through extensive empirical analyses with 20 LLMs on three benchmarks, we pinpoint that this behavioral bias primarily stems from LLMs' token bias, where the model a priori assigns more probabilistic mass to specific option ID tokens (e.g., A/B/C/D) when predicting answers from the option IDs. To mitigate selection bias, we propose a label-free, inference-time debiasin…
-
Sufficient Context: A New Lens on Retrieval Augmented Generation SystemsHailey Joren, Jian-Yi Zhang, C. Ferng et al. · 2024 · International Conference on Learning Representations · cité 141 fois · Pas encore résumé
«Sufficient Context» distingue les questions répondables de celles privées d’informations suffisantes, comme cet article teste l’abstention lorsque la portion pertinente du spectre manque.
Résumé scientifique
Augmenting LLMs with context leads to improved performance across many applications. Despite much research on Retrieval Augmented Generation (RAG) systems, an open question is whether errors arise because LLMs fail to utilize the context from retrieval or the context itself is insufficient to answer the query. To shed light on this, we develop a new notion of sufficient context, along with a method to classify instances that have enough information to answer the query. We then use sufficient context to analyze several models and datasets. By stratifying errors based on context sufficiency, we find that larger models with higher baseline performance (Gemini 1.5 Pro, GPT 4o, Claude 3.5) excel …
-
Jev for Scientific Decisions: Evaluating Semantic Choices and Their ConsequencesBo-Yuan Deng, Shu-Yi Fan, Hong-Yang Zhang et al. · 2026 · arXiv.org · preprint, non relu par les pairs · cité 13 fois · Pas encore résumé
«Jev for Scientific Decisions» évalue Jev sur des choix sémantiques scientifiques, tandis que cet article inclut Jev 1.13 pour décider à partir de spectres.
Résumé scientifique
Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture, treatment or reference standard can change the scientific meaning of the resulting count or comparison. Once the evidence is prepared and the calculation is written as code, this choice reduces to one semantic judgment among listed candidates. We evaluate Jev on this step, using a harness that follows its documented guidance. The study compares twelve model configurations on twenty Choices in ten scientific cases built from published sources, each repeated five times. We measure semantic selections, downstream outputs and final claim labels s…
-
From Text Decisions to Pixels: An Study of Jev-Style Visual Choice ModelXun-Lan Zhou, Xian-Liang Yang, Li Zhao · 2026 · arXiv.org · preprint, non relu par les pairs · cité 7 fois · Pas encore résumé
PixelJev adapte une interface de choix Jev aux images et évalue des modèles multimodaux, tandis que cet article teste Jev sur des représentations graphiques de spectres.
Résumé scientifique
Visual software often needs a decision over supplied alternatives rather than a generated explanation. We present PixelJev, a native-image decision interface that maps an image, a task instruction, and a runtime candidate set to a structured choice and candidate-conditioned probabilities using small open multimodal models. Its initial realization unifies recognition and multiplechoice visual question answering through an existing language-model readout, with separately evaluated options for frozen inference, language-side adaptation, and held-out calibration. Across seven benchmark evaluations, 64-shot source adaptation raises Pets accuracy from 60.13% to 92.40% across optimization seeds and…
-
MOCLIP: a foundation model for large-scale nanophotonic inverse designS. Rodionov, A. Burguete-Lopez, M. Makarenko et al. · 2025 · Nature Communications · cité 5 fois · Pas encore résumé
MOCLIP fournit les données expérimentales de métasurfaces dont cet article évalue les spectres et les décisions multimodales.
Résumé scientifique
Foundation models are transforming artificial intelligence by enabling generalizable, data-efficient solutions across diverse domains and applications. However, the lack of large and diverse datasets remains a key barrier to their development in nanophotonics. This work presents MOCLIP (Metasurface Optics Contrastive Learning Pretrained), a nanophotonic foundation model that encodes metasurfaces’ structural and spectral information into a shared latent space via contrastive learning, using an experimentally acquired dataset with sample density approaching the scale of ImageNet-1K. The study demonstrates MOCLIP’s inverse design capabilities, including high-throughput zero-shot prediction at 2…
Aller plus loin
Comment ces articles sont-ils trouvés ?
Nous comparons le titre et le résumé de cet article avec ceux de tous les articles résumés sur ce site, et avec les articles que Semantic Scholar recommande, ceux qu’il cite et ceux qui le citent (pour un article très cité, ses citations les plus récentes et, depuis OpenAlex, les plus citées). Parmi les travaux antérieurs et les suites, les articles qui s’utilisent en profondeur, une méthode qu’il utilise et les suites très citées remontent un peu. Les niveaux (très proche, proche, apparenté) viennent de la similarité des textes, pas d’un jugement de qualité ; * signale un article sans résumé, comparé par son titre. La phrase sous un article est écrite par une IA à partir des deux résumés : vérifiez-la dans les articles. Un article jugé hors sujet par trois lecteurs quitte la liste.
Mis à jour le 9 novembre 2026.