MVBench
mvBench is an innovative dataset that transforms static tasks into dynamic tasks to assess temporal capabilities via videos. It contains annotations that are transformed into multiple choice questions, promoting accurate and fair evaluation of video models. Some video files (320) require an external manual download.
Description
MVBench offers a video benchmark focused on various temporal tasks. By converting static tasks into dynamic ones, it makes it possible to test skills ranging from simple perception to complex cognitive functions via multiple choice questions associated with videos.
What is this dataset for?
- Evaluate the temporal and dynamic understanding of video models
- Test temporal reasoning skills in complex tasks
- Facilitate the automatic creation of reliable video benchmarks
Can it be enriched or improved?
This dataset can be enriched by integrating more annotated videos and by diversifying the types of temporal tasks. The addition of additional video data (including the 320 to be downloaded separately) will improve the robustness of the assessment.
🔎 In summary
🧠 Recommended for
- Video vision researchers
- Video model developers
- AI laboratories
🔧 Compatible tools
- Hugging Face Datasets
- PyTorch Video
- TensorFlow Video
- Video QA frameworks
💡 Tip
Be sure to manually fetch all 320 external videos for a complete evaluation.
Frequently Asked Questions
Why do some video files need to be downloaded separately?
Due to NTU RGB+D license restrictions, 320 videos cannot be distributed directly and must be picked up through the official website.
What type of time tasks does this dataset evaluate?
Tasks ranging from simple perception to complex cognition, transformed into multiple choice questions for accurate evaluation.
Is this dataset suitable for fine-tuning video models?
Yes, it can be used for both evaluating and fine-tuning video models on time-based tasks.
