Data & Benchmarks

Task Success Rate

Task success rate is the fraction of evaluation episodes in which a robot policy achieves a task's success criterion, and it is the primary headline metric in manipulation and locomotion research. Reported rates depend heavily on evaluation protocol: the number of trials, object and scene randomization, distractor placement, initial-state distribution, and how success is judged, whether automatically in simulation or by a human. Small trial counts produce wide confidence intervals that are frequently omitted.

Why it matters for physical AI

Comparing robot foundation models rigorously requires standardized, statistically sound success-rate evaluation, and the gap between papers reporting on curated scenes and performance in uncontrolled deployments is a persistent source of overestimated capability.