Foundation Models

Multimodal Learning

Multimodal learning is the training of models that jointly process multiple input modalities, such as vision, language, audio, proprioception, and touch, by aligning them in shared representation spaces or fusing them within a single architecture. Contrastive alignment as in CLIP and unified transformer architectures over tokenized modalities are dominant patterns. Robotics extends the modality set to depth, force-torque, and tactile signals.

Why it matters for physical AI

Robots inherently sense across modalities, and skills like insertion demand fusing vision with force and touch; multimodal architectures are how foundation models absorb these complementary signals into one policy.