Do Multimodal Models Decide Consistently on Equivalent Optical Spectra?
Authors: Ziyuan Wang, Genliang Zhu, Xiaobo Chen
Abstract
A spectrum can be given to a model as a table in nanometers or as a plot in photon energy, and the decision should not depend on which one it sees. When the data needed for an answer are missing, the model should say so. We test representation invariance and abstention on measured optical spectra of silicon metasurfaces (MOCLIP) and gold nanoparticles, with labels computed from the data. Each spectrum is shown as four tables (nm, $μ$m, scientific notation, eV) and two plots (nm, eV), and partial spectra remove data inside or outside the queried range. We evaluate the decision models d3-nano and Jev 1.13, three GPT-6 Luna configurations and the Qwen3.5-2B base model. Changing only the representation moves accuracy by up to 41 percentage points within one model, mostly on photon-energy tables and plots. With the same evidence, asking in photon energy instead of wavelength helps some model-view pairs by 16-50 points and hurts others by up to 25. When the relevant extremum is removed, unadapted models rarely answer "undetermined". LoRA fine-tuning of d3-nano improves comparisons on the trained table formats, but much of its gain on unanswerable questions comes from a shortcut: it abstains whenever the input is truncated. Training with answerable truncations avoids the shortcut in all six seeds, with a 90-100 point gap between abstention after inside and outside cuts on held-out structures; with the same amount of extra original-task data, five of six seeds keep the shortcut. On gold spectra the two training sets perform similarly. Multimodal language models used for scientific decisions should be tested on equivalent representations and on missing evidence, not only on average accuracy. We release the tasks, per-response results and analysis code.
The papers around it, by date and closeness
Higher is closer to this paper; left is older. Lines join papers close to each other: groups of lines are sub-topics. Select a point to see why it is there. Back to the summary of this paper
Papers around this one
Ranked by how close their title and abstract are to this paper’s. Select some to read them together.
Closest papers
-
Evaluating large language models on multimodal chemistry olympiad examsYiming Cui, Xin Yao, Yuxuan Qin et al. · 2025 · Communications Chemistry · cited 17 times · Not summarized yet
Evaluates multimodal models on Olympiad chemistry questions requiring visual and textual reasoning, rather than equivalent presentations of measured optical spectra.
Abstract
Multimodal scientific reasoning remains a significant challenge for large language models (LLMs), particularly in chemistry, where problem-solving relies on symbolic diagrams, molecular structures, and structured visual data. Here, we systematically evaluate 40 proprietary and open-source multimodal LLMs, including GPT-5, o3, Gemini-2.5-Pro, and Qwen2.5-VL, on a curated benchmark of Olympiad-style chemistry questions drawn from over two decades of U.S. National Chemistry Olympiad (USNCO) exams. These questions require integrated visual and textual reasoning across diverse modalities. We find that many models struggle with modality fusion, where, in some cases, removing the image even improve…
-
Unified neural network model for predicting optical responses in gold nanostructuresXiao-Bo Wang, Yu-Chen Cao, Feng Feng et al. · 2025 · Journal of Applied Physics · cited 6 times · Not summarized yet
Predicts optical spectra of gold nanostructures and compares predictions with experiments, while this paper tests model decisions on measured gold-nanoparticle spectra.
Abstract
In this paper, we present a deep neural network model capable of simultaneously predicting the optical transmittance spectra of three distinct gold nanostructures: nanodisks, truncated cones, and nanocones. Our model achieved a mean squared error loss as low as 0.0015. By systematically varying geometric parameters, we generated a comprehensive dataset to train the model, which captures complex dependencies between structure and optical response. A key finding is the redshift in resonance dip associated with increasing sidewall tilt. We further validated the model by comparing its predictions with experimental results, showing excellent agreement. This work demonstrates a powerful, data-driv…
-
SpecVQA: A Benchmark for Spectral Understanding and Visual Question Answering in Scientific ImagesJialun Shen, Han Lyu, Su-Yang Zhong et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet
Introduces a benchmark for multimodal models answering questions about scientific spectra, including extraction from spectral figures.
Abstract
Spectra are a prevalent yet highly information-dense form of scientific imagery, presenting substantial challenges to multimodal large language models (MLLMs) due to their unstructured and domain-specific characteristics. Here we introduce SpecVQA, a professional scientific-image benchmark for evaluating multimodal models on scientific spectral understanding, covering 7 representative spectrum types with expert-annotated question-answer pairs. The aim comprises two aspects: spectra scientific QA evaluation and corresponding underlying task evaluation. SpecVQA contains 620 figures and 3100 QA pairs curated from peer-reviewed literature, targeting both direct information extraction and domain-…
-
Probing the limitations of multimodal language models for chemistry and materials researchNawaf Alampara, Mara Schilling-Wilhelmi, Martiño Ríos-García et al. · 2024 · Nature Computational Science · cited 89 times · Not summarized yet
Benchmarks vision-language models on chemistry and materials tasks including data extraction and results interpretation, broader than optical-spectrum decision consistency.
Abstract
Recent advancements in artificial intelligence have sparked interest in scientific assistants that could support researchers across the full spectrum of scientific workflows, from literature review to experimental design and data analysis. A key capability for such systems is the ability to process and reason about scientific information in both visual and textual forms—from interpreting spectroscopic data to understanding laboratory set-ups. Here we introduce MaCBench, a comprehensive benchmark for evaluating how vision language models handle real-world chemistry and materials science tasks across three core aspects: data extraction, experimental execution and results interpretation. Throug…
-
Reading and steering representations of materials-science mechanisms in an open-weight language modelMarkus J. Buehler · 2026 · AI for Science · cited 1 time · Not summarized yet
Studies whether language models encode and use materials-science mechanisms, rather than whether decisions remain stable across spectrum formats.
Abstract
Large language models can answer scientific questions, yet a correct output does not reveal whether the model represents or uses the governing physics. Here, using three open-weight Gemma 4 instruction-tuned models (google/gemma-4-E4B-it, google/gemma-4-12B-it, and google/gemma-4-31B-it) we identify three experimentally separable signatures of materials-science mechanism information: selective concept readability, relational encoding of qualitative constitutive orientation, and causal, context-dependent control of constrained engineering answers. Across 50 held-out descriptions, three independently fitted Jacobian lenses produced nearly identical concept ranks, but 36 prompts recovered no de…
-
Good Scores, Bad Data: A Metric for Multimodal CoherenceV. Srinivasan · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet
Proposes a multimodal coherence metric for image-text inputs, instead of testing model decisions on equivalent scientific-spectrum representations.
Abstract
Multimodal AI systems are evaluated by downstream task accuracy, but high accuracy does not mean the underlying data is coherent. A model can score well on Visual Question Answering (VQA) while its inputs contradict each other. We introduce the Multimodal Coherence Score (MCS), a metric that evaluates fusion quality independent of any downstream model. MCS decomposes coherence into four dimensions, identity, spatial, semantic, and decision, with weights learned via Nelder-Mead optimization. We evaluate on 1,000 Visual Genome images using DETR, CLIP, and ViLT, and validate on 150 COCO images with no retraining. Across three fusion architectures, MCS discriminates quality with higher sensitivi…
-
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language ModelsChaoyou Fu, Peixian Chen, Yunhang Shen et al. · 2023 · Computer Vision and Pattern Recognition · cited 1,927 times · AI summary available
Introduces a broad benchmark of multimodal perception and cognition tasks, rather than focused tests of optical-spectrum invariance and abstention.
Abstract
Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks, showing amazing emergent abilities in recent studies, such as writing poems based on an image. However, it is difficult for these case studies to fully reflect the performance of MLLM, lacking a comprehensive evaluation. In this paper, we fill in this blank, presenting the first MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks. In order to avoid data leakage that may arise from direct use of public datasets for evaluation, the annotations of instruction-answer pairs are all manually designed. The concise instruction design allows us to fai…
-
Same Answer, Different Representations: Hidden instability in VLMsFarooq Ahmad Wani, Alessandro Suglia, Rohit Saxena et al. · 2026 · Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing · cited 4 times · Not summarized yet
Tests whether vision-language models preserve answers across changed representations, paralleling this paper’s tests of decision stability across equivalent spectrum formats.
Abstract
The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial intern…
Prior work it builds on
-
Large Language Models Are Not Robust Multiple Choice SelectorsChujie Zheng, Hao Zhou, Fan-Dong Meng et al. · 2023 · International Conference on Learning Representations · cited 725 times · Not summarized yet
Studies answer changes caused by multiple-choice option order, whereas this paper varies spectrum representation and evidence availability.
Abstract
Multiple choice questions (MCQs) serve as a common yet important task format in the evaluation of large language models (LLMs). This work shows that modern LLMs are vulnerable to option position changes in MCQs due to their inherent"selection bias", namely, they prefer to select specific option IDs as answers (like"Option A"). Through extensive empirical analyses with 20 LLMs on three benchmarks, we pinpoint that this behavioral bias primarily stems from LLMs' token bias, where the model a priori assigns more probabilistic mass to specific option ID tokens (e.g., A/B/C/D) when predicting answers from the option IDs. To mitigate selection bias, we propose a label-free, inference-time debiasin…
-
Sufficient Context: A New Lens on Retrieval Augmented Generation SystemsHailey Joren, Jian-Yi Zhang, C. Ferng et al. · 2024 · International Conference on Learning Representations · cited 141 times · Not summarized yet
Separates model errors from insufficient retrieved evidence, closely paralleling this paper’s tests of abstention when queried spectral evidence is missing.
Abstract
Augmenting LLMs with context leads to improved performance across many applications. Despite much research on Retrieval Augmented Generation (RAG) systems, an open question is whether errors arise because LLMs fail to utilize the context from retrieval or the context itself is insufficient to answer the query. To shed light on this, we develop a new notion of sufficient context, along with a method to classify instances that have enough information to answer the query. We then use sufficient context to analyze several models and datasets. By stratifying errors based on context sufficiency, we find that larger models with higher baseline performance (Gemini 1.5 Pro, GPT 4o, Claude 3.5) excel …
-
Jev for Scientific Decisions: Evaluating Semantic Choices and Their ConsequencesBo-Yuan Deng, Shu-Yi Fan, Hong-Yang Zhang et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 13 times · Not summarized yet
Evaluates Jev for structured scientific choices, and this paper also tests Jev as a decision model on optical-spectrum tasks.
Abstract
Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture, treatment or reference standard can change the scientific meaning of the resulting count or comparison. Once the evidence is prepared and the calculation is written as code, this choice reduces to one semantic judgment among listed candidates. We evaluate Jev on this step, using a harness that follows its documented guidance. The study compares twelve model configurations on twenty Choices in ten scientific cases built from published sources, each repeated five times. We measure semantic selections, downstream outputs and final claim labels s…
-
From Text Decisions to Pixels: An Study of Jev-Style Visual Choice ModelXun-Lan Zhou, Xian-Liang Yang, Li Zhao · 2026 · arXiv.org · preprint, not peer reviewed · cited 7 times · Not summarized yet
Introduces PixelJev for choosing among visual candidates with multimodal models, extending Jev-style structured decisions to images.
Abstract
Visual software often needs a decision over supplied alternatives rather than a generated explanation. We present PixelJev, a native-image decision interface that maps an image, a task instruction, and a runtime candidate set to a structured choice and candidate-conditioned probabilities using small open multimodal models. Its initial realization unifies recognition and multiplechoice visual question answering through an existing language-model readout, with separately evaluated options for frozen inference, language-side adaptation, and held-out calibration. Across seven benchmark evaluations, 64-shot source adaptation raises Pets accuracy from 60.13% to 92.40% across optimization seeds and…
-
MOCLIP: a foundation model for large-scale nanophotonic inverse designS. Rodionov, A. Burguete-Lopez, M. Makarenko et al. · 2025 · Nature Communications · cited 5 times · Not summarized yet
Introduces MOCLIP and its experimentally acquired metasurface dataset, which this paper uses for optical-spectrum decision tasks.
Abstract
Foundation models are transforming artificial intelligence by enabling generalizable, data-efficient solutions across diverse domains and applications. However, the lack of large and diverse datasets remains a key barrier to their development in nanophotonics. This work presents MOCLIP (Metasurface Optics Contrastive Learning Pretrained), a nanophotonic foundation model that encodes metasurfaces’ structural and spectral information into a shared latent space via contrastive learning, using an experimentally acquired dataset with sample density approaching the scale of ImageNet-1K. The study demonstrates MOCLIP’s inverse design capabilities, including high-throughput zero-shot prediction at 2…
Go further
How are these papers found?
We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.
Updated on 9 November 2026.