Summarize Paper logo SummarizePaper.com

AI summary · arXiv: 2611.00428v1 · cs.LG

Do Multimodal Models Decide Consistently on Equivalent Optical Spectra?

Ziyuan Wang, Genliang Zhu, Xiaobo Chen
Original paper Submitted to arXiv on 30 Oct. 2026 20 pages, 4 figures, 11 tables. Code and data: https://github.com/zwang-devv/spectra-decision-audit License: CC BY 4.0 Also in: fr

How to read this summary? Illustrated guide

The essentials

Equivalent optical spectra can produce inconsistent model decisions, and models often fail to abstain when the queried evidence is missing.

Question

Do multimodal and decision models preserve their answers when the same measured optical spectrum is presented in equivalent table or plot formats and coordinate systems? Can they distinguish an answerable partial spectrum from one missing the evidence required for the decision?

Method

The study tested models on measured spectra from 10,000 silicon metasurfaces and 448 gold-nanoparticle syntheses, using filtered subsets and labels computed directly from the data. Each core variant was rendered in four table formats and two plot formats; paired tasks also varied question wording and removed data inside or outside the queried interval. The authors evaluated decision models, three GPT-6 Luna configurations, and Qwen3.5-2B, then fine-tuned d3-nano with LoRA across training arms, generally using three seeds per arm.

Findings

  • Result Equivalent representations changed accuracy substantially, with the largest reported drop being 41 percentage points for Jev on gold eV tables relative to nm tables.
    See the passage in the paper
    On gold, eV tables lower accuracy by 41 points for Jev, 21 for Luna gen. none, 20 for d3-nano, and 13 for Luna Decisions.
  • Result Changing the question from wavelength to photon-energy wording on unchanged eV evidence helped some model-task combinations but hurt others, including a 25-point decline for d3-nano on gold comparisons.
    See the passage in the paper
    d3-nano drops from 59% to 34% on gold compare tables ($-25$), and low-reasoning Luna drops from 100% to 90% on gold threshold tables ($-10$).
  • Result Unadapted models generally failed to distinguish missing relevant evidence from answerable truncations, with low scope discrimination for several configurations.
    See the passage in the paper
    Unadapted API configurations almost never abstain on outside cuts, and their accuracy changes by at most seven points.
  • Result Training with answerable truncations produced more consistent discrimination between unanswerable and answerable cuts on MOCLIP than adding a similar amount of original-task data.
    See the passage in the paper
    Across both training durations, all six seeds trained with answerable truncations avoid the shortcut. Five of the six seeds trained with a similar amount of original-task data lose 27–99 points on outside cuts.
  • Authors’ interpretation The authors conclude that adaptation gains were limited, with weak or variable transfer to plots, unseen tasks, and the second optical dataset.
    See the passage in the paper
    Task-specific adaptation improves the cases covered by training, but transfer to other representations, tasks, and optical data remains limited.

Limitations

  • The datasets cover a narrow range of optical systems and select spectra with clear extrema, limiting generalization to other spectra and narrow resonances.
    See the passage in the paper
    Both datasets cover a narrow range of optical systems, and we select spectra with clear extrema.
  • The evaluation relies on fixed multiple-choice templates and only two query intervals per scope set; abstention was available only for sufficiency tasks.
    See the passage in the paper
    The tasks use fixed multiple-choice templates. Each scope set has two query intervals, and their boundaries are partly associated with answer bands.
  • The adaptation conclusions are constrained by the use of one base model and three seeds per arm, alongside training and hardware differences across comparisons.
    See the passage in the paper
    The adaptation study uses one base model, a frozen vision tower, and three seeds per arm.

Why it matters

The authors argue that average accuracy alone can conceal representation-sensitive errors and inappropriate responses to missing evidence. They recommend evaluating scientific decision models on paired equivalent representations and on both answerable and unanswerable partial inputs.

Written by AI from the paper. Passages are checked word for word against the text we analysed; the rest may contain errors.

AI-generated keywords: multimodal models optical spectra representation sensitivity evidence sufficiency abstention

Read further

  • Comprehensive Summary
  • Layman's Summary
  • Blog article
  • Original abstract
The paper investigates whether multimodal models make consistent decisions when the same optical spectrum is presented in equivalent forms, and whether they abstain when the evidence needed to answer a question is missing. The authors evaluate tasks involving measured spectra from silicon metasurfaces (MOCLIP) and gold nanoparticles. They derive labels directly from the data and filter for spectra with sufficiently clear extrema. Tasks include locating extrema within specified wavelength bands, comparing dip positions, selecting candidates that meet stated conditions, and determining whether a partial spectrum contains enough information to answer a question. For each task, the researchers vary the representation while keeping the underlying samples fixed. Representations include tables in nanometers, micrometers, scientific notation, or photon energy, as well as plots in wavelength or energy coordinates. Additional tests vary question wording, evidence order, and row order in energy tables. To test whether models recognize insufficient evidence, the researchers compare complete spectra with two types of partial views. One removes data outside the queried interval, leaving the answer recoverable; the other removes the relevant extremum, making the answer indeterminate. The models include d3-nano, Jev 1.13, three GPT-6 Luna configurations, and the Qwen3.5-2B base model. The researchers also fine-tune d3-nano with LoRA, varying the training representations and testing whether examples of answerable truncations improve abstention. Changing the representation often affected accuracy, particularly for photon-energy inputs. On gold spectra, eV tables reduced accuracy relative to nanometer tables by 41 percentage points for Jev, 21 for Luna’s no-reasoning generative configuration, 20 for d3-nano, and 13 for Luna Decisions. Energy plots also caused substantial declines for several configurations; for example, Luna gen. none fell by 34 points on gold spectra. Changes to wavelength number formats generally had smaller effects. Asking about photon energy rather than wavelength while keeping the eV evidence unchanged substantially improved performance in some model–task combinations. Jev improved by 48 points on gold comparisons, while d3-nano improved by 50 points on gold threshold tables. However, wording changes could also hurt performance: d3-nano’s accuracy on gold-table comparisons fell by 25 points. The results therefore do not identify a single phrasing conversion that consistently helps. Unadapted models seldom recognized when relevant evidence had been removed. In the main MOCLIP sufficiency test, d3-nano, Jev, and Luna Decisions never abstained. In query-relative scope tests, d3-nano showed no discrimination between unanswerable inside cuts and answerable outside cuts for tables. Other models varied, with Luna gen. low reaching 55 points on MOCLIP tables and 82 points on gold tables. These findings matter because abstaining on every truncated input is not sufficient: outside cuts leave the answer available. LoRA adaptation improved some trained table tasks but transferred unevenly to plots, other task types, and gold spectra. Training with answerable truncations produced strong MOCLIP table scope discrimination across all six reported seeds and both training-duration setups, generally around 90–100 points, while preserving outside-cut accuracy much better. In contrast, five of six seeds trained with a similar quantity of additional original-task data showed large losses on answerable outside cuts. The two training approaches performed more similarly on gold scope tests. The authors conclude that average accuracy alone can conceal representation sensitivity, direction errors, and failures to recognize missing evidence. They recommend evaluating equivalent representations alongside answerable and unanswerable partial inputs. The study is limited to selected spectra with clear extrema, fixed multiple-choice templates, a small number of query intervals, one adapted model, and three seeds per training arm.
Created on 09 Nov. 2026

Was this synthesis useful?

Score: 0

Why do we need votes?

Votes are used to determine whether we need to re-run our summarizing tools. If the count reaches -10, our tools can be restarted.

Ask questions about this paper to our AI assistant

How to chat with this paper? Illustrated guide

Compare with the PDF

You can also chat with multiple papers at once here.

Figures and plots

Ask about a figure by number or use its question button to request visual analysis. Up to two figures and four image panels per question. Values read from plots are approximate; check the original data.
AI assistant instructions?

Tools for this paper

Explain it differently 1 free rewrite left today
Save to a collection Group papers in collections to compare them, chat with them and export them. Continue with Google Create a free account Log in
Export BibTeX · RIS (Zotero, Mendeley, EndNote) · Obsidian (Markdown) · Notion (Markdown)

Advanced tools Premium

For researchers who follow a field closely: citation alerts, journal-club slides, version comparison and Zotero export. Discover Premium

Zotero
Citation alerts Citation alerts guide
Who works on it? The people and institutions who publish on its topic
Weekly letter Presented in "Artificial intelligence this week" of 9 November 2026. Artificial intelligence this week: When better scores miss the point
A rising topic This paper is part of "How AI makes decisions", rising in Computation and Language (cs.CL) (×3.0 compared with usual). The rising topics of that week · The history of the term
Journal club Premium+ Slides to present it Propose for the journal club
Versions What changed between versions
Talk about it in public Premium+ Press kit: release, questions, hype check Press kit guide

Papers around this one

See them on a map

Ranked by how close their title and abstract are to this paper’s. Select some to read them together.

Closest papers

  1. Close Broader context
    Evaluating large language models on multimodal chemistry olympiad exams
    Yiming Cui, Xin Yao, Yuxuan Qin et al. · 2025 · Communications Chemistry · cited 17 times · Not summarized yet

    Evaluates multimodal models on Olympiad chemistry questions requiring visual and textual reasoning, rather than equivalent presentations of measured optical spectra.

    Abstract

    Multimodal scientific reasoning remains a significant challenge for large language models (LLMs), particularly in chemistry, where problem-solving relies on symbolic diagrams, molecular structures, and structured visual data. Here, we systematically evaluate 40 proprietary and open-source multimodal LLMs, including GPT-5, o3, Gemini-2.5-Pro, and Qwen2.5-VL, on a curated benchmark of Olympiad-style chemistry questions drawn from over two decades of U.S. National Chemistry Olympiad (USNCO) exams. These questions require integrated visual and textual reasoning across diverse modalities. We find that many models struggle with modality fusion, where, in some cases, removing the image even improve…

  2. Related Same question Cited by this paper
    Unified neural network model for predicting optical responses in gold nanostructures
    Xiao-Bo Wang, Yu-Chen Cao, Feng Feng et al. · 2025 · Journal of Applied Physics · cited 6 times · Not summarized yet

    Predicts optical spectra of gold nanostructures and compares predictions with experiments, while this paper tests model decisions on measured gold-nanoparticle spectra.

    Abstract

    In this paper, we present a deep neural network model capable of simultaneously predicting the optical transmittance spectra of three distinct gold nanostructures: nanodisks, truncated cones, and nanocones. Our model achieved a mean squared error loss as low as 0.0015. By systematically varying geometric parameters, we generated a comprehensive dataset to train the model, which captures complex dependencies between structure and optical response. A key finding is the redshift in resonance dip associated with increasing sidewall tilt. We further validated the model by comparing its predictions with experimental results, showing excellent agreement. This work demonstrates a powerful, data-driv…

  3. Related Same question Cited by this paper
    SpecVQA: A Benchmark for Spectral Understanding and Visual Question Answering in Scientific Images
    Jialun Shen, Han Lyu, Su-Yang Zhong et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 1 time · Not summarized yet

    Introduces a benchmark for multimodal models answering questions about scientific spectra, including extraction from spectral figures.

    Abstract

    Spectra are a prevalent yet highly information-dense form of scientific imagery, presenting substantial challenges to multimodal large language models (MLLMs) due to their unstructured and domain-specific characteristics. Here we introduce SpecVQA, a professional scientific-image benchmark for evaluating multimodal models on scientific spectral understanding, covering 7 representative spectrum types with expert-annotated question-answer pairs. The aim comprises two aspects: spectra scientific QA evaluation and corresponding underlying task evaluation. SpecVQA contains 620 figures and 3100 QA pairs curated from peer-reviewed literature, targeting both direct information extraction and domain-…

  4. Related Broader context Cited by this paper
    Probing the limitations of multimodal language models for chemistry and materials research
    Nawaf Alampara, Mara Schilling-Wilhelmi, Martiño Ríos-García et al. · 2024 · Nature Computational Science · cited 89 times · Not summarized yet

    Benchmarks vision-language models on chemistry and materials tasks including data extraction and results interpretation, broader than optical-spectrum decision consistency.

    Abstract

    Recent advancements in artificial intelligence have sparked interest in scientific assistants that could support researchers across the full spectrum of scientific workflows, from literature review to experimental design and data analysis. A key capability for such systems is the ability to process and reason about scientific information in both visual and textual forms—from interpreting spectroscopic data to understanding laboratory set-ups. Here we introduce MaCBench, a comprehensive benchmark for evaluating how vision language models handle real-world chemistry and materials science tasks across three core aspects: data extraction, experimental execution and results interpretation. Throug…

  5. Related Broader context
    Reading and steering representations of materials-science mechanisms in an open-weight language model
    Markus J. Buehler · 2026 · AI for Science · cited 1 time · Not summarized yet

    Studies whether language models encode and use materials-science mechanisms, rather than whether decisions remain stable across spectrum formats.

    Abstract

    Large language models can answer scientific questions, yet a correct output does not reveal whether the model represents or uses the governing physics. Here, using three open-weight Gemma 4 instruction-tuned models (google/gemma-4-E4B-it, google/gemma-4-12B-it, and google/gemma-4-31B-it) we identify three experimentally separable signatures of materials-science mechanism information: selective concept readability, relational encoding of qualitative constitutive orientation, and causal, context-dependent control of constrained engineering answers. Across 50 held-out descriptions, three independently fitted Jacobian lenses produced nearly identical concept ranks, but 36 prompts recovered no de…

  6. Related Broader context
    Good Scores, Bad Data: A Metric for Multimodal Coherence
    V. Srinivasan · 2026 · arXiv.org · preprint, not peer reviewed · Not summarized yet

    Proposes a multimodal coherence metric for image-text inputs, instead of testing model decisions on equivalent scientific-spectrum representations.

    Abstract

    Multimodal AI systems are evaluated by downstream task accuracy, but high accuracy does not mean the underlying data is coherent. A model can score well on Visual Question Answering (VQA) while its inputs contradict each other. We introduce the Multimodal Coherence Score (MCS), a metric that evaluates fusion quality independent of any downstream model. MCS decomposes coherence into four dimensions, identity, spatial, semantic, and decision, with weights learned via Nelder-Mead optimization. We evaluate on 1,000 Visual Genome images using DETR, CLIP, and ViLT, and validate on 150 COCO images with no retraining. Across three fusion architectures, MCS discriminates quality with higher sensitivi…

  7. Related Broader context
    MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
    Chaoyou Fu, Peixian Chen, Yunhang Shen et al. · 2023 · Computer Vision and Pattern Recognition · cited 1,927 times · AI summary available

    Introduces a broad benchmark of multimodal perception and cognition tasks, rather than focused tests of optical-spectrum invariance and abstention.

    Abstract

    Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks, showing amazing emergent abilities in recent studies, such as writing poems based on an image. However, it is difficult for these case studies to fully reflect the performance of MLLM, lacking a comprehensive evaluation. In this paper, we fill in this blank, presenting the first MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks. In order to avoid data leakage that may arise from direct use of public datasets for evaluation, the annotations of instruction-answer pairs are all manually designed. The concise instruction design allows us to fai…

  8. Related Same question
    Same Answer, Different Representations: Hidden instability in VLMs
    Farooq Ahmad Wani, Alessandro Suglia, Rohit Saxena et al. · 2026 · Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing · cited 4 times · Not summarized yet

    Tests whether vision-language models preserve answers across changed representations, paralleling this paper’s tests of decision stability across equivalent spectrum formats.

    Abstract

    The robustness of Vision Language Models (VLMs) is commonly assessed through output-level invariance, implicitly assuming that stable predictions reflect stable multimodal processing. In this work, we argue that this assumption is insufficient. We introduce a representation-aware and frequency-aware evaluation framework that measures internal embedding drift, spectral sensitivity, and structural smoothness (spatial consistency of vision tokens), alongside standard label-based metrics. Applying this framework to modern VLMs across the SEEDBench, MMMU, and POPE datasets reveals three distinct failure modes. First, models frequently preserve predicted answers while undergoing substantial intern…

Prior work it builds on

  1. Related Broader context
    Large Language Models Are Not Robust Multiple Choice Selectors
    Chujie Zheng, Hao Zhou, Fan-Dong Meng et al. · 2023 · International Conference on Learning Representations · cited 725 times · Not summarized yet

    Studies answer changes caused by multiple-choice option order, whereas this paper varies spectrum representation and evidence availability.

    Abstract

    Multiple choice questions (MCQs) serve as a common yet important task format in the evaluation of large language models (LLMs). This work shows that modern LLMs are vulnerable to option position changes in MCQs due to their inherent"selection bias", namely, they prefer to select specific option IDs as answers (like"Option A"). Through extensive empirical analyses with 20 LLMs on three benchmarks, we pinpoint that this behavioral bias primarily stems from LLMs' token bias, where the model a priori assigns more probabilistic mass to specific option ID tokens (e.g., A/B/C/D) when predicting answers from the option IDs. To mitigate selection bias, we propose a label-free, inference-time debiasin…

  2. Related Same question
    Sufficient Context: A New Lens on Retrieval Augmented Generation Systems
    Hailey Joren, Jian-Yi Zhang, C. Ferng et al. · 2024 · International Conference on Learning Representations · cited 141 times · Not summarized yet

    Separates model errors from insufficient retrieved evidence, closely paralleling this paper’s tests of abstention when queried spectral evidence is missing.

    Abstract

    Augmenting LLMs with context leads to improved performance across many applications. Despite much research on Retrieval Augmented Generation (RAG) systems, an open question is whether errors arise because LLMs fail to utilize the context from retrieval or the context itself is insufficient to answer the query. To shed light on this, we develop a new notion of sufficient context, along with a method to classify instances that have enough information to answer the query. We then use sufficient context to analyze several models and datasets. By stratifying errors based on context sufficiency, we find that larger models with higher baseline performance (Gemini 1.5 Pro, GPT 4o, Claude 3.5) excel …

  3. Related Same method, another problem
    Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences
    Bo-Yuan Deng, Shu-Yi Fan, Hong-Yang Zhang et al. · 2026 · arXiv.org · preprint, not peer reviewed · cited 13 times · Not summarized yet

    Evaluates Jev for structured scientific choices, and this paper also tests Jev as a decision model on optical-spectrum tasks.

    Abstract

    Scientific workflows often require choosing among known relations before a deterministic calculation can proceed. Whether observations share a culture, treatment or reference standard can change the scientific meaning of the resulting count or comparison. Once the evidence is prepared and the calculation is written as code, this choice reduces to one semantic judgment among listed candidates. We evaluate Jev on this step, using a harness that follows its documented guidance. The study compares twelve model configurations on twenty Choices in ten scientific cases built from published sources, each repeated five times. We measure semantic selections, downstream outputs and final claim labels s…

  4. Related Same method, another problem
    From Text Decisions to Pixels: An Study of Jev-Style Visual Choice Model
    Xun-Lan Zhou, Xian-Liang Yang, Li Zhao · 2026 · arXiv.org · preprint, not peer reviewed · cited 7 times · Not summarized yet

    Introduces PixelJev for choosing among visual candidates with multimodal models, extending Jev-style structured decisions to images.

    Abstract

    Visual software often needs a decision over supplied alternatives rather than a generated explanation. We present PixelJev, a native-image decision interface that maps an image, a task instruction, and a runtime candidate set to a structured choice and candidate-conditioned probabilities using small open multimodal models. Its initial realization unifies recognition and multiplechoice visual question answering through an existing language-model readout, with separately evaluated options for frozen inference, language-side adaptation, and held-out calibration. Across seven benchmark evaluations, 64-shot source adaptation raises Pets accuracy from 60.13% to 92.40% across optimization seeds and…

  5. Related Groundwork for this paper
    MOCLIP: a foundation model for large-scale nanophotonic inverse design
    S. Rodionov, A. Burguete-Lopez, M. Makarenko et al. · 2025 · Nature Communications · cited 5 times · Not summarized yet

    Introduces MOCLIP and its experimentally acquired metasurface dataset, which this paper uses for optical-spectrum decision tasks.

    Abstract

    Foundation models are transforming artificial intelligence by enabling generalizable, data-efficient solutions across diverse domains and applications. However, the lack of large and diverse datasets remains a key barrier to their development in nanophotonics. This work presents MOCLIP (Metasurface Optics Contrastive Learning Pretrained), a nanophotonic foundation model that encodes metasurfaces’ structural and spectral information into a shared latent space via contrastive learning, using an experimentally acquired dataset with sample density approaching the scale of ImageNet-1K. The study demonstrates MOCLIP’s inverse design capabilities, including high-throughput zero-shot prediction at 2…

Go further

How are these papers found?

We compare the title and abstract of this paper with those of every paper summarized on this site, and with the papers Semantic Scholar recommends, those it cites and those citing it (for a much-cited paper, its most recent citations and, from OpenAlex, the most cited ones). Among the prior work and the follow-ups, the papers that use each other in depth, a method it uses and the much-cited follow-ups come a little higher. The levels (very close, close, related) come from the similarity of the texts, not from a judgment of quality; * marks a paper without an abstract, compared by its title. The sentence under a paper is written by an AI from the two abstracts: check it in the papers. A paper marked not relevant by three readers leaves the list.

Updated on 9 November 2026.

Disclaimer: The AI-based summarization tool and virtual assistant provided on this website may not always provide accurate and complete summaries or responses. We encourage you to carefully review and evaluate the generated content to ensure its quality and relevance to your needs.