Automatic Transfer Function Design via MLLM-Assisted 2D Semantic Decomposition

Authors

Haejin Jeong (Korea University), Won-Ki Jeong (Korea University)

Presentation

Session: Modern talking-visually
Time: Tuesday, Nov 10, 13:36 – 13:48 (US/Eastern) · session 13:00 – 14:30
Location: Hall America center

Keywords

Direct volume rendering, transfer function, multimodal large language model

Abstract

Designing transfer functions (TFs) for direct volume rendering is a fundamental yet labor-intensive task, as users need to carefully explore the relationship between voxel attributes and visual outcomes to reveal structures of interest. Despite advances in learning-based approaches, many existing methods depend on repeated user input. Furthermore, many methods relying on conventional data value-based TFs, which use voxel intensity or gradient information, often struggle to separate voxels into meaningful semantic groups. To address these problems, we present a framework that enables automatic TF design via multimodal large language model (MLLM)-assisted 2D semantic decomposition, requiring only minimal user input, namely the volume name and a projection axis. Starting from a single maximum intensity projection of the volume, our method transforms it into a semantic segmentation mask using an MLLM, which is then lifted into 3D as a sparse set of reliable voxel cues. To convert these sparse cues into a complete semantic representation, we introduce a voxel classification network trained with a confidence-aware propagation loss, resulting in dense voxel-level semantic assignments that are readily translated into TFs. Despite relying solely on single-image segmentation, our method achieves precise part-level separation. Evaluation on a range of volume datasets shows that ours produces better semantic renderings while significantly reducing the effort and time required for TF design compared to state-of-the-art techniques.

For Practitioners

Designing transfer functions for volumetric data traditionally forces practitioners into repeated trial-and-error—manually exploring intensity and gradient ranges to find each structure of interest—and even recent learning-based or MLLM-assisted methods still require iterative user interaction or remain restricted to 1D intensity mappings that cannot separate structures sharing similar intensity values. This is directly relevant to practitioners who need to interpret volumetric scans but lack the time or visualization expertise for manual TF tuning—such as radiologists and medical imaging specialists, biologists examining scanned specimens, and engineers inspecting industrial CT data. Our framework addresses this by requiring only a volume name and a projection axis: an MLLM semantically decomposes a single 2D projection of the volume into labeled parts, which are then automatically lifted into a dense, part-level 3D semantic representation without any iterative interaction. As a result, practitioners can obtain a meaningful, part-separated visualization of an unfamiliar dataset fast and can then further refine it by adjusting the color and opacity of individual semantic parts as a practical starting point for deeper analysis.