MLLM Perception of Color-Encoded Scalar Fields Reveals Model-Dependent Inversion of Chain-of-Thought
Authors
Minyi Liu (Shandong University), Yue Zhao (Shandong Second Medical University), Xiaoyang Song (Shandong University), Shufan Qian (Shandong University), Yulong Bian (Shandong University), Qiong Zeng (Shandong University)
Presentation
- Session
- Is the Model Even Thinking?
- Time
- Thursday, Nov 12, 08:45 – 08:54 (US/Eastern) · session 08:00 – 09:30
- Location
- Hall America south
Keywords
Continuous colormaps, color-encoded scalar field visualization, graphical perception, multi-modal large language models
Abstract
Color-encoded scalar field visualizations are fundamental to scientific data analysis, yet their interpretation depends critically on both data characteristics and colormap design. While multimodal large language models (MLLMs) have emerged as promising tools for visualization tasks, their ability to comprehend and interpret scalar field visualizations remains largely unexplored. We investigate how effectively MLLMs perceive these visualizations without direct access to the underlying data values, focusing on two graphical perception tasks: value identification and gradient comparison. Building on the well-established two-step human perception workflow—legend understanding and linking legend to visualization—we develop a structured prompting framework for MLLM perception workflows and implement it through both decomposed-step and chain-of-thought (CoT) prompting strategies across five state-of-the-art MLLMs, alongside fine-tuning techniques for model adaptation. Our results reveal a model-dependent interaction between prompting and model capacity: CoT prompting benefits weaker models (e.g., reducing Task 2 error by up to 21.5%) but degrades stronger ones (relative error increases of up to 84.76% on Task 1 for the strongest models we tested), while fine-tuning yields consistent improvements (up to a 43.80% relative error reduction across both tasks). Our work bridges human and machine perception workflows, demonstrating the promising potential of leveraging MLLMs to understand graphical perception while highlighting the need for model-aware prompting strategies in visualization perception.
For Practitioners
Practitioners building MLLM tools on scientific visualizations (visualization researchers, tool developers, and simulation scientists) can use the paper's findings to choose prompting strategies per model and to fine-tune smaller models when prompting alone underperforms.