Pixmo-cap
A unique multimodal dataset combining images, audio transcripts and long captions from oral annotations transformed by an LLM. Useful for training vision-language models.
Approximately 717,000 items, 1.1 GB in Parquet format, including images (URLs), audio transcripts, and long captions (~200 words)
ODC-BY-1.0
Description
The dataset Pixmo-cap contains more than 700,000 multimodal examples, combining images (via URLs), audio transcripts of oral annotations, and long captions generated by a language model. Each element includes an oral recording of 60-90 seconds described in text by an LLM, offering significant semantic richness for vision-language tasks.
What is this dataset for?
- Pre-train or refine vision-language models with detailed legends
- Look for multimodal generation or comprehension methods including text, image, and audio
- Testing image analysis systems coupled with speech and natural language processing
Can it be enriched or improved?
Yes, it is possible to add additional annotations to the images, to enrich the audio transcripts with time labels, or to rework the captions to adapt them to specific cases. The collection can be remixed to diversify languages or styles of descriptions.
🔎 In summary
🧠 Recommended for
- Computer vision researchers
- Multimodal LLM developers
- Audio-visual recognition projects
🔧 Compatible tools
- Hugging Face Datasets
- PyTorch
- TensorFlow
- Multimodal frameworks (e.g. CLIP, Flamingo)
💡 Tip
To fully exploit this dataset, provide a pipeline to download and manage images via URLs and synchronize with audio transcripts.
Frequently Asked Questions
Does this dataset include the original audio files for the annotations?
Yes, annotator audio transcripts are included, but raw audio files need to be checked separately.
Can this dataset be used to train models in languages other than English?
The dataset is mainly in English, but it is possible to complete it by translation or to create multilingual extensions.
Do I have to download the images separately?
Yes, images are provided as URLs and must be uploaded independently for training.




