MMSI-Video-Bench: A Holistic Benchmark for Video-based Spatial Intelligence

Jingli Lin^1,2*, Runsen Xu^1,3*^†, Shaohao Zhu^1,4, Sihan Yang¹, Peizhou Cao^1,5, Yunlong Ran^1,4, Miao Hu⁶,
Chenming Zhu^1,7, Yiman Xie^1,4, Yilin Long^1,8, Wenbo Hu^1,9, Dahua Lin^1,3, Tai Wang^1✉, Jiangmiao Pang^1✉

1. Shanghai AI Laboratory 2. Shanghai Jiaotong University 3. The Chinese University of Hong Kong 4. Zhejiang University 5. Beihang University
6. Xi'an Jiaotong University 7. University of Hong Kong 8. Fudan University 9. University of California, Los Angeles

^*Equal Contribution ^†Project Lead

Paper

Code

Datasets

Diverse Scenarios

25 public + 1 in-house datasets covering indoor, outdoor, tabletop, and movie scenes.

High Data Quality

All manually annotated by multiple 3DV experts under strict quality control.

Comprehensive Tasks

Spatial layout reasoning, motion understanding, planning, prediction, and cross-video reasoning.

Challenging Benchmark

Most models perform poorly on MMSI-Video-Bench, highlighting a ~60% human-AI gap.

Multifaceted Perspectives

🏠 Indoor Scene Perception Bench

🤖 Robot Bench

📍 Grounding Bench

MMSI-Video-Bench covers spatial layout reasoning, motion understanding, decision-making, and cross-video reasoning, providing a holistic evaluation of video-based spatial intelligence.

Overview

llustrative examples of different subtypes in MMSI-Video-Bench. Rel., Inst., and Cam. stand for Relationship, Instance, and Camera.

(Top) The data collection, annotation, and verification pipeline of MMSI-Video-Bench. (Bottom) Dataset statistics of MMSI-Video-Bench, including the proportion of each subtype, the distribution of video durations, and the word cloud.

BibTeX

@misc{lin2025mmsivideobenchholisticbenchmarkvideobased,
        title={MMSI-Video-Bench: A Holistic Benchmark for Video-Based Spatial Intelligence}, 
        author={Jingli Lin and Runsen Xu and Shaohao Zhu and Sihan Yang and Peizhou Cao and Yunlong Ran and Miao Hu and Chenming Zhu and Yiman Xie and Yilin Long and Wenbo Hu and Dahua Lin and Tai Wang and Jiangmiao Pang},
        year={2025},
        eprint={2512.10863},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2512.10863}, 
  }