MedXpertqa

MedXpertQA is an expert medical benchmark combining question-answer tasks in text alone or with clinical images. It focuses on advanced reasoning, diagnosis, and understanding of body systems.

Download dataset

Size

4,460 questions (text + images), JSONL and JPEG formats, divided into dev/test for textual and multimodal medical QA

Licence

MIT

Description

‍

MedXpertqa is a medical dataset designed to assess the ability of artificial intelligence models to understand and reason about complex clinical cases. It consists of two subsets: one based on text (medical QA), and the other multimodal including images (radiologies, skin exams, etc.) and structured clinical pictures. The dataset covers various specialties such as dermatology, oncology, internal medicine, etc.

‍

What is this dataset for?

‍

Test the ability of models to perform high-level medical reasoning
Train multimodal models to understand medical cases illustrated by images and tabular data
Serve as a benchmark to compare different LLMs approaches in medicine

‍

Can it be enriched or improved?

‍

Yes. It is possible to add new clinical cases, in particular from poorly represented specialties or from varied geographical regions. Annotations can also be enriched by adding justifications to answers or detailed clinical explanations. The dataset is structured to facilitate expansion and reuse.

‍

🔎 In summary

Criterion	Evaluation
🧩 Ease of use	⭐⭐⭐⭐⭐ (Well-structured with two clear subsets)
🧼 Need for cleaning	⭐⭐⭐⭐⭐ (Low – data already filtered and reviewed)
🏷️ Annotation richness	⭐⭐⭐⭐⭐ (High – precise tasks, question types, medical specialties)
📜 Commercial license	✅ Yes (MIT)
👨‍💻 Beginner friendly	⚠️ Moderate – better for medical NLP researchers
🔁 Fine-tuning ready	🎯 Excellent for training medical QA models
🌍 Cultural diversity	⚠️ Medium – global medical questions, but not localized