Foundation Models

Attention Mechanism

An attention mechanism is a neural network operation that computes a weighted combination of a set of value vectors, with weights derived from the compatibility between a query and corresponding keys, letting the model dynamically focus on relevant inputs. Scaled dot-product self-attention, introduced with the transformer by Vaswani et al. in 2017, replaced recurrence as the dominant sequence-modeling primitive and underlies essentially all modern language, vision, and robot policy backbones.

Why it matters for physical AI

Attention is the mechanism through which vision-language-action models bind language instructions to image regions and past observations to current actions, and its quadratic cost drives the efficiency work needed for onboard deployment.