By clicking "Accept", you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. See our Privacy Policy for more information
Open Datasets
Pixmo-cap
Multimodal

Pixmo-cap

A unique multimodal dataset combining images, audio transcripts and long captions from oral annotations transformed by an LLM. Useful for training vision-language models.

Download dataset
Size

Approximately 717,000 items, 1.1 GB in Parquet format, including images (URLs), audio transcripts, and long captions (~200 words)

Licence

ODC-BY-1.0

Description

‍

The dataset Pixmo-cap contains more than 700,000 multimodal examples, combining images (via URLs), audio transcripts of oral annotations, and long captions generated by a language model. Each element includes an oral recording of 60-90 seconds described in text by an LLM, offering significant semantic richness for vision-language tasks.

‍

‍

What is this dataset for?

‍

  • Pre-train or refine vision-language models with detailed legends
  • Look for multimodal generation or comprehension methods including text, image, and audio
  • Testing image analysis systems coupled with speech and natural language processing

‍

‍

Can it be enriched or improved?

‍

Yes, it is possible to add additional annotations to the images, to enrich the audio transcripts with time labels, or to rework the captions to adapt them to specific cases. The collection can be remixed to diversify languages or styles of descriptions.

‍

‍

🔎 In summary

Criterion Evaluation
🧩 Ease of use⭐⭐⭐⭐✩ (Requires URL management and multimodal processing)
🧼 Need for cleaning⭐⭐⭐⭐✩ (Moderate: check quality of generated transcriptions and captions)
🏷️ Annotation richness⭐⭐⭐⭐✩ (Long captions and associated audio transcriptions)
📜 Commercial license✅ Yes, with attribution (ODC-BY-1.0)
👨‍💻 Beginner friendly⚠️ Moderate – complex multimodal format
🔁 Fine-tuning ready🎨 Perfect for vision-language and multimodal models
🌍 Cultural diversity⚠️ Mainly English, various annotators

‍

‍

🧠 Recommended for

  • Computer vision researchers
  • Multimodal LLM developers
  • Audio-visual recognition projects

‍

‍

🔧 Compatible tools

  • Hugging Face Datasets
  • PyTorch
  • TensorFlow
  • Multimodal frameworks (e.g. CLIP, Flamingo)

‍

‍

💡 Tip

To fully exploit this dataset, provide a pipeline to download and manage images via URLs and synchronize with audio transcripts.

Frequently Asked Questions

Does this dataset include the original audio files for the annotations?

Yes, annotator audio transcripts are included, but raw audio files need to be checked separately.

Can this dataset be used to train models in languages other than English?

The dataset is mainly in English, but it is possible to complete it by translation or to create multilingual extensions.

Do I have to download the images separately?

Yes, images are provided as URLs and must be uploaded independently for training.

Similar datasets

See more
Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.

Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.

Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.