CharXiv: Understanding Scientific Graphics
Multimodal graphical comprehension dataset, composed of images of real graphics from arXiv and open questions.
2,323 high resolution images with 11,000+ questions, Parquet format
CC BY-SA 4.0
Description
CharXIV is a multimodal benchmark designed to test the comprehension of graphics by large visual language models. It contains 2,323 graphics taken from scientific articles, accompanied by 5 questions per graph (descriptive questions, reasoning questions and one deliberately unanswered). Each example challenges a model's ability to read, interpret, and respond openly based on complex visual elements.
What is this dataset for?
- Evaluate the visual understanding of multimodal models (LLMs + images)
- Test robustness in open QA on realistic scientific data
- Train or adjust models for graph reading tasks
Can it be enriched or improved?
Yes. It is possible to augment this dataset with other types of graphs (e.g. non-scientific), or to generate linguistic variants of questions for a multilingual approach. You can also add finer annotations such as the complexity of the graph or the type of data represented.
🔎 In summary
🧠 Recommended for
- Multimodal LLM developers
- Visual QA researchers
- Educational projects on reading charts
🔧 Compatible tools
- PyTorch
- Hugging Face Datasets
- Transformers
- OpenFlamingo
- MMGpt
💡 Tip
Use visual embeddings from pre-trained models to combine image and text effectively during training.
Frequently Asked Questions
Does this dataset contain accurate or open answers?
It contains open, but verifiable responses that assess the ability of a model to formulate accurate answers without multiple choices.
Can this dataset be used for non-multimodal models?
Not directly, because the image is essential to understanding. It is designed for architectures that can process text + image.
What type of graph do we find in this dataset?
These are scientific graphs taken from arXiv articles, covering various types such as bars, lines, scatter plots, etc.
