OTaT — Attending to Multimodal Generation One Token at a Time • OTaT — Attending to Multimodal Generation One Token at a Time

Understanding how Multimodal Large Language Models dynamically route information during autoregressive generation.

Varun Gupta · Vineet Gandhi · Makarand Tapaswi

CVIT, IIIT Hyderabad

Abstract

Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context. Prior work on interpretability has focused on individual layers and circuits (where), leaving the token-level dynamics of multimodal computation during generation (when) underexplored. We address this gap and study attention shifts as per semantic role; tracking model attention to image, text, instruction, and previously generated tokens, One Token at a Time (OTaT). We introduce multimodal tasks that require explicit switching between visual and textual context within a single response. Across two mainstream model families and four open-weight MLLMs of varying sizes, we establish consistent patterns: attention to image peaks at tokens requiring image-derived information, instruction tokens are revisited during task transitions, and attention to previously generated tokens increases as the generation progresses. Causal attention blocking interventions validate the functional role of these trends. We profile model behavior under disrupted attention and observe responses falling back to language priors, or exhibiting cross-modal leakage, denial, or recovery. Finally, informed of the attention dynamics through our novel analysis, we propose a simple test-time intervention to boost attention to the relevant modality at the right time, significantly improving multimodal task performance.

OTaT Process

Motivation

Why Token-Level Attention Dynamics?

The mechanistic interpretability community has made remarkable progress examining individual attention heads, layers, and isolated circuits — revealing where particular behaviors are implemented. But MLLMs are dynamic generative systems that iteratively construct outputs one token at a time. Prior work on interpretability has focused on individual layers and circuits (where), leaving the token-level dynamics of multimodal computation during generation (when) underexplored. We address this gap and study attention shifts as per semantic role; tracking model attention to image, text, instruction, and previously generated tokens, One Token at a Time (OTaT). Beyond where models attend, we study how attention patterns evolve as decoding progresses.

OTaT Process

Method

Task Setup

Samples from multimodal tasks. Fruit-Math (left) shows an image of a fruit along with an unrelated math puzzle. VSR (middle) features an image with a paired caption that describes the scene, but with a conflicting spatial relationship. ChartQA (right) shows a diagram followed by two questions related to it. The model is instructed to (respectively): identify the fruit in the image and solve the math puzzle, identify the spatial discrepancy between image and text, and answer both questions.

Extracting Attention Patterns — OTaT

Our approach to analyze attention patterns in autoregressive MLLMs. A. Example from Fruit-Math. B. Information flow from context to CGT — chunks are grouped into static and dynamic context. C. Tagging and grouping of output tokens. D. Raw attention scores from CGT to chunks. E. Normalized attention scores (mean subtracted) make trends easier to interpret.

We treat the full input context as a sequence of semantically grouped chunks: Image, Text, Instruction, and Previously generated tokens. At each decoding step, we track attention from the Currently Generating Token (CGT) to each chunk, then aggregate over semantic output roles to reveal dataset-level trends.

Main Findings

Attention Orchestrates Autoregression

Across two model families (LLaVA-OneVision and Qwen2.5-VL) and four open-weight MLLMs, we find consistent patterns in how attention is dynamically allocated during generation.

Finding 01

MLLMs dynamically reallocate attention to the relevant modality during generation.

Finding 02

MLLMs revisit instruction tokens when switching tasks (e.g. from visual perception to arithmetic). Blocking this interaction, leads to incomplete task solving.

Finding 03

Attention scores are influenced by what the model is decoding (y_t) and the input (y_t−1).

Finding 04

Raw attention scores may be high for possibly unimportant tokens. However, via normalization, studying whether attention scores change across timesteps is helpful to gain insights.

Finding 05

MLLMs fall back to language priors, leak cross-modal information, deny visual content, or sometimes, exhibit recovery behaviors when attention to the image is blocked at the critical stage.

All the above finding are backed by causal interventions, wherein we block the attention to the relevant modality at critical generation steps and notice corresponding disruptions.

Causal Interventions

Effects of Blocking Information Flow

To validate that observed attention patterns are functionally important, we intervene by blocking attention to specific chunks at critical generation steps.

Blocking strategies in QK attention (gray is the causal mask, red is blocked). Left: Lazy blocking only affects CGT (most existing works implement this). Right: Total blocking prevents information flow to any future token.

LLaVA-OneVision 0.5B — Cross-modal Leakage. When image attention is blocked, the model leaks visual information through intermediate tokens rather than directly attending to the image.

LLaVA-OneVision 7B — Previous Token Reliance. Blocking image attention at a critical step causes the model to fall back on previously generated tokens, producing responses anchored in prior output rather than visual content.

Qwen2.5-VL 3B — Task Handoff Disruption. Blocking instruction attention during task transition prevents proper handoff between visual perception and the subsequent reasoning step.

Qwen2.5-VL 3B — Recovery Behavior. Despite image attention being blocked at a critical stage, this model exhibits a recovery pattern — rerouting information through alternate paths to produce a partially correct response.

Qwen2.5-VL 7B — Visual Denial. With image attention blocked, the model explicitly denies the presence of visual content rather than hallucinating or guessing — a distinct failure mode unique to larger models.

Application

Token-Level Attention Guidance Improves Performance

Informed by our analysis of attention dynamics, we propose a simple test-time attention boosting strategy: selectively amplify attention to the relevant modality at critical decoding steps.

Attention Boosting Results (hover to zoom · click to expand)

Conclusion

Summary

We present OTaT, a framework for studying token-level attention dynamics in MLLMs during autoregressive generation. Our analysis reveals consistent, interpretable routing patterns that generalize across model families and benchmarks.

Models attend to relevant context — image attention peaks when generating image-derived content.
Instruction tokens are revisited during task handoff, enabling multi-task solving.
Previous-token attention progressively increases, reflecting growing self-context reliance.
Causal blocking validates functional importance of observed attention patterns.
Attention boosting at critical steps significantly improves multimodal task performance.