CODAH - Commonsense Dataset Adversarially Created by Humans
The <strong>CODAH</strong> dataset contains difficult commonsense comprehension questions, created by humans who used feedback from a pre-trained model to design adversarial questions. Used to assess the ability of models to reason beyond traditional automatic data.
16,656 question-answer pairs in JSON format, weight ~2.25 MB
Open Data Commons Attribution License (ODC-BY)
Description
CODAH is an evaluation corpus in natural language understanding, specifically oriented to commonsense reasoning. Each example includes a question and several answer options, designed by humans to be difficult for classical models to predict.
What is this dataset for?
- Test and evaluate NLP models on commonsense understanding
- Develop robust systems in the face of adversarial issues
- Improving models of contextual understanding and reasoning
Can it be enriched or improved?
This dataset can be enriched by creating new adversarial questions in other fields or languages, or by annotating questions according to difficulty categories or types of reasoning.
🔎 In summary
🧠 Recommended for
- NLP researchers
- AI developers in QA
- Computational linguistics teachers
🔧 Compatible tools
- PyTorch
- TensorFlow
- Hugging Face Transformers
- QA assessment tools
💡 Tip
Integrate this dataset into a suite of adversarial evaluations to test the robustness of your models.
Frequently Asked Questions
What is the specificity of the questions in CODAH?
The questions are designed by humans to be difficult, using feedback from models to create pitfalls and adversarial cases.
How many examples does the CODAH dataset contain?
The dataset contains 16,656 question-answer pairs.
Can this dataset be used to train models?
Yes, it is suitable for evaluating and fine-tuning natural language comprehension models.




