By clicking "Accept", you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. See our Privacy Policy for more information
Open Datasets
Cosmos Reason1 SFT Dataset
Multimodal

Cosmos Reason1 SFT Dataset

A massive data set for embodied reasoning, combining videos (robotics, AV, human gestures) and detailed text annotations.

Download dataset
Size

Approximately 1.7 million video-text pairs (mp4 + annotations), JSON and video formats, 300+ GB

Licence

CC-BY 4.0

Description

‍

Cosmos-Reason1 SFT Dataset is a multimodal dataset published by NVIDIA, combining over 1 million video-text pairs annotated to train embodied reasoning models. It covers several areas: robotics (RoboVQA, BridgeDataV2, Agibot), human gestures (HoloAssist), and autonomous driving (AV). Annotations include explicit reasoning, thought chains, and detailed metadata.

‍

‍

What is this dataset for?

‍

  • Train models capable of understanding actions in videos and deducing complex reasoning
  • Training embedded AIs for robotics or human assistance in a real environment
  • Evaluate the ability of LLMs to integrate visual perception and logical reasoning

‍

‍

Can it be enriched or improved?

‍

Yes, it is possible to add custom annotations, extend the corpus to other areas, or create localized versions. Source videos are accessible or replaceable via the identifiers provided, allowing for semi-automated enrichments.

‍

‍

🔎 In summary

Criterion Evaluation
🧩 Ease of use⭐⭐⭐⭐✩ (Well-structured files, but large)
🧼 Need for cleaning⭐⭐⭐⭐⭐ (Low – coherent annotations ready-to-use)
🏷️ Annotation richness⭐⭐⭐⭐⭐ (Very rich: reasoning, actions, chains of thought)
📜 Commercial license✅ Yes (CC-BY 4.0)
👨‍💻 Beginner friendly⚠️ No – recommended for advanced users in multimodal AI
🔁 Fine-tuning ready🎯 Excellent base for fine-tuning visuo-cognitive LLMs
🌍 Cultural diversity⚠️ Needs improvement – data focused on technical scenarios

‍

‍

🧠 Recommended for

  • Robotics laboratories
  • Researchers in embodied cognition
  • Development of autonomous AI agents

‍

‍

🔧 Compatible tools

  • PyTorch
  • Hugging Face Transformers
  • VideoLM
  • DeepSpeed
  • LoRa

‍

‍

💡 Tip

To effectively use the dataset, start with the subsets (RoboVQA or BridgeDataV2) before processing the complete corpus.

Frequently Asked Questions

What types of tasks can this dataset be used to process?

It covers embodied reasoning tasks, video QA, action recognition, and human interaction comprehension tasks.

Can this dataset be used for commercial applications?

Yes, the CC-BY 4.0 license allows it, provided the source is credited in accordance with the terms.

Are there video access limits?

Only some sets (RoboVQA, AV) include videos. The others can be found using the original file names.

Similar datasets

See more
Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.

Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.

Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.