Cosmos Reason1 SFT Dataset
A massive data set for embodied reasoning, combining videos (robotics, AV, human gestures) and detailed text annotations.
Approximately 1.7 million video-text pairs (mp4 + annotations), JSON and video formats, 300+ GB
CC-BY 4.0
Description
Cosmos-Reason1 SFT Dataset is a multimodal dataset published by NVIDIA, combining over 1 million video-text pairs annotated to train embodied reasoning models. It covers several areas: robotics (RoboVQA, BridgeDataV2, Agibot), human gestures (HoloAssist), and autonomous driving (AV). Annotations include explicit reasoning, thought chains, and detailed metadata.
What is this dataset for?
- Train models capable of understanding actions in videos and deducing complex reasoning
- Training embedded AIs for robotics or human assistance in a real environment
- Evaluate the ability of LLMs to integrate visual perception and logical reasoning
Can it be enriched or improved?
Yes, it is possible to add custom annotations, extend the corpus to other areas, or create localized versions. Source videos are accessible or replaceable via the identifiers provided, allowing for semi-automated enrichments.
🔎 In summary
🧠 Recommended for
- Robotics laboratories
- Researchers in embodied cognition
- Development of autonomous AI agents
🔧 Compatible tools
- PyTorch
- Hugging Face Transformers
- VideoLM
- DeepSpeed
- LoRa
💡 Tip
To effectively use the dataset, start with the subsets (RoboVQA or BridgeDataV2) before processing the complete corpus.
Frequently Asked Questions
What types of tasks can this dataset be used to process?
It covers embodied reasoning tasks, video QA, action recognition, and human interaction comprehension tasks.
Can this dataset be used for commercial applications?
Yes, the CC-BY 4.0 license allows it, provided the source is credited in accordance with the terms.
Are there video access limits?
Only some sets (RoboVQA, AV) include videos. The others can be found using the original file names.




