Foundation Models

Multi-Head Attention

Multi-head attention is the core operation of the transformer, introduced by Vaswani et al. in 2017, in which several attention heads run in parallel, each projecting inputs to its own query, key, and value subspaces, attending over the sequence, and concatenating results. Separate heads learn complementary relational patterns, and the mechanism's content-based, order-agnostic routing is what lets transformers fuse tokens from arbitrary modalities.

Why it matters for physical AI

Attention over interleaved image patches, language tokens, and proprioceptive states is the architectural glue of vision-language-action models, letting one network bind instructions to percepts and motor outputs.