Perception

Panoptic Segmentation

Panoptic segmentation is a computer vision task that assigns every pixel in an image both a semantic class and, for countable objects, an instance identity, unifying semantic segmentation of background regions with instance segmentation of discrete objects. Formalized by Kirillov et al. in 2019 along with the panoptic quality metric, it produces a single coherent scene decomposition rather than two separate outputs.

Why it matters for physical AI

Robots acting in clutter need to know both what surfaces are (floor, table, wall) and which pixels belong to which manipulable object, making panoptic output a natural scene representation for manipulation and navigation.