By clicking "Accept", you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. See our Privacy Policy for more information
Open Datasets
CODAH - Commonsense Dataset Adversarially Created by Humans
Text

CODAH - Commonsense Dataset Adversarially Created by Humans

The <strong>CODAH</strong> dataset contains difficult commonsense comprehension questions, created by humans who used feedback from a pre-trained model to design adversarial questions. Used to assess the ability of models to reason beyond traditional automatic data.

Download dataset
Size

16,656 question-answer pairs in JSON format, weight ~2.25 MB

Licence

Open Data Commons Attribution License (ODC-BY)

Description

‍

CODAH is an evaluation corpus in natural language understanding, specifically oriented to commonsense reasoning. Each example includes a question and several answer options, designed by humans to be difficult for classical models to predict.

‍

‍

What is this dataset for?

‍

  • Test and evaluate NLP models on commonsense understanding
  • Develop robust systems in the face of adversarial issues
  • Improving models of contextual understanding and reasoning

‍

‍

Can it be enriched or improved?

‍

This dataset can be enriched by creating new adversarial questions in other fields or languages, or by annotating questions according to difficulty categories or types of reasoning.

‍

‍

🔎 In summary

Criterion Evaluation
🧩 Ease of use⭐⭐⭐⭐✩ (Simple JSON format, clear examples)
🧼 Need for cleaning⭐⭐⭐⭐⭐ (Low, ready-to-use data)
🏷️ Annotation richness⭐⭐⭐✩✩ (Medium, QA commonsense type annotations)
📜 Commercial license⚡ Yes (ODC-BY)
👨‍💻 Beginner friendly🌟 Yes, accessible even to beginners
🔁 Fine-tuning ready🎯 Perfect for evaluation and commonsense fine-tuning
🌍 Cultural diversity🌏 English, international general knowledge questions

‍

‍

🧠 Recommended for

  • NLP researchers
  • AI developers in QA
  • Computational linguistics teachers

‍

‍

🔧 Compatible tools

  • PyTorch
  • TensorFlow
  • Hugging Face Transformers
  • QA assessment tools

‍

‍

💡 Tip

Integrate this dataset into a suite of adversarial evaluations to test the robustness of your models.

Frequently Asked Questions

What is the specificity of the questions in CODAH?

The questions are designed by humans to be difficult, using feedback from models to create pitfalls and adversarial cases.

How many examples does the CODAH dataset contain?

The dataset contains 16,656 question-answer pairs.

Can this dataset be used to train models?

Yes, it is suitable for evaluating and fine-tuning natural language comprehension models.

Similar datasets

See more
Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.

Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.

Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.