AttentionRadViz: Token-Anchored Attention Profile Visualization for Transformer Head Analysis

Authors

Presentation

Session: Is the Model Even Thinking?
Time: Thursday, Nov 12, 08:54 – 09:03 (US/Eastern) · session 08:00 – 09:30
Location: Hall America south

Keywords

Transformer interpretability, attention visualization, visual analytics, RadViz, machine learning visualization.

Abstract

While attention heatmaps are the standard interface for inspect- ing multi-head attention, they primarily expose pairwise token-to- token weights rather than supporting comparison of how multiple tokens attend over the full context or how those attention profiles evolve across layers. We present AttentionRadViz, a visual analyt- ics tool that maps each token’s attention distribution to a point in a RadViz layout whose anchors are the input tokens. In this token- anchored projection, spatial proximity indicates similarity between tokens as attenders, while anchor positions preserve a semantic ref- erence to the input sequence. The system combines single-head inspection with layer-by-layer small multiples and radial token tra- jectories, helping researchers compare attention profiles across to- kens, follow how those profiles change across depth, and triage heads by profile structure, readily distinguishing structured heads from collapsed or sink-like ones. We demonstrate AttentionRadViz on Pythia-410M attention exports with focused examples of inter- active coreference-profile inspection, attention-sink detection, and transient cross-layer profile emergence. We also report a projection sanity check showing that RadViz preserves local attention-profile neighborhoods while retaining interpretable token anchors. Atten- tionRadViz is released as an open-source library.

For Practitioners

What type of practitioners would be interested in reading this paper? ML interpretability researchers and engineers who analyze transformer internals, such as mechanistic interpretability practitioners, LLM evaluation teams, and researchers debugging attention behavior. Secondarily, visualization tool builders working on model understanding, and data scientists who inspect models in notebook workflows. How could practitioners apply what they learn from this paper to their work? Practitioners can adopt the paper's workflow framing of treating attention head analysis as search andtriage rather than exhaustive inspection. With the released library, they can reproduce it directly. The core design lesson, anchoring projections to semantically meaningful entities to obtain a stable reference frame across layers or conditions, transfers to visualizing any evolving set of distributions over a fixed item set.