Foundation Models

Positional Encoding

Positional encoding is the mechanism by which transformers, whose attention operation is inherently permutation-invariant, receive information about token order or spatial location. The original sinusoidal scheme of Vaswani et al. (2017) has been joined by learned embeddings and relative schemes such as rotary position embeddings (RoPE). Related Fourier-feature encodings let coordinate networks like NeRF represent high-frequency spatial detail.

Why it matters for physical AI

Robot policies built on transformers must encode temporal order of observations, spatial structure of image patches, and action-chunk indices, and encoding choices affect how models generalize across sequence lengths and camera configurations.