Scalable Semantic Steering of Embedding Projections
Authors
Wei Liu (Computer Science, Virginia Tech), Eric Krokos (Dod), Kirsten Whitley (Department of Defense), Rebecca Faust (Tulane University), Chris North (Virginia Tech)
Presentation
- Session
- Lost in Dimensions
- Time
- Thursday, Nov 12, 15:36 – 15:45 (US/Eastern) · session 15:00 – 16:30
- Location
- Hall Essex north
Keywords
Semantic Steering, Semantic Interaction, Embedding Projections, Large Language Models, Semantic Prototypes.
Abstract
Low-dimensional projections support interactive visual analysis of high-dimensional data embeddings, but their structure often does not align with analyst-defined semantic relationships. Recent LLM-augmented semantic steering methods address this gap by externalizing analyst intent from user-defined groups of seed examples, but they propagate intent through per-item LLM reasoning, causing LLM calls and cost to grow linearly with collection size. We propose a scalable semantic steering method that shifts semantic computation from individual items to user-defined groups. A single LLM call generates structured profiles for all groups, which are embedded and combined with seed centroids to form hybrid semantic prototypes. The method then propagates intent without retraining, using embedding-space soft assignment, abstention, and alignment-scaled updates before reprojection. On a 5K-document LitCovid corpus, our method achieves global alignment comparable to per-item LLM steering while reducing LLM calls by over three orders of magnitude. An image case study shows that the same prototype-based mechanism extends to multimodal embeddings. These results suggest that group-level representations can make semantic steering more practical for larger embedding collections.
For Practitioners
Practitioners who analyze large text or image embedding collections, such as visual analytics researchers, data scientists, biomedical literature analysts, intelligence analysts, and machine learning practitioners, may be interested in this paper. They can apply the proposed group-level semantic steering approach to reorganize embedding projections around analyst-defined concepts using only a small number of seed examples, reducing the cost of LLM-augmented steering for larger collections.