Robot Learning

Latent Action Representation

A latent action representation is a learned, compact encoding of actions or action-like transitions, often inferred from data without ground-truth control labels. Systems such as LAPA and Genie learn discrete latent actions directly from videos by modeling frame-to-frame changes, enabling pretraining on action-free web video. The latent actions are later decoded or aligned to real robot commands with a small amount of labeled data.

Why it matters for physical AI

Human video vastly outnumbers robot demonstration data; latent actions let foundation models mine that video for transferable manipulation knowledge, then bridge to executable controls with modest robot-specific fine-tuning.