By clicking "Accept", you agree to the storing of cookies on your device to enhance site navigation, analyze site usage, and assist in our marketing efforts. See our Privacy Policy for more information
Text

X-CSR

The X-CSR dataset makes it possible to evaluate multilingual models based on common sense reasoning via a zero-shot transfer protocol. It contains two automatically translated sub-datasets (X-CSQA and X-CODAH) covering 16 languages, including French, English, Chinese, etc. This data comes from automatic translations of the original English versions, providing a multilingual benchmark.

Download dataset
Size

Contains examples translated into 16 languages, JSON format, variable size depending on splits

Licence

MIT

Description

‍

X-CSR is a multilingual benchmark designed to assess the common-sense reasoning skills of language models in a context of cross-language zero-shot transfer. It includes automatically translated versions of the CSQA and CODAH datasets in 16 languages, including French.

‍

‍

What is this dataset for?

‍

  • Test the generalization of multilingual models using common sense reasoning
  • Evaluate the zero-shot transfer from English to other languages
  • Provide a unified benchmark to compare different multilingual models

‍

‍

Can it be enriched or improved?

‍

Yes, by replacing machine translations with more accurate human translations and by extending the number of languages or examples for a more robust benchmark.

‍

‍

🔎 In summary

Criterion Evaluation
🧩 Ease of use⭐⭐⭐⭐✩ (Standard format, but requires multilingual handling)
🧼 Need for cleaning⭐⭐⭐✩✩ (Moderate due to automatic translations which may contain errors)
🏷️ Annotation richness⭐⭐⭐⭐✩ (Good, with labels from original datasets)
📜 Commercial license✅ Yes (MIT)
👨‍💻 Beginner friendly⚠️ Moderate, recommended for users with multilingual experience
🔁 Fine-tuning ready✅ Useful for fine-tuning and multilingual evaluation
🌍 Cultural diversity🈳 Wide coverage of 16 languages including several language families

‍

‍

🧠 Recommended for

  • Multilingual NLP researchers
  • Multilingual model developers
  • Assessment teams

‍

‍

🔧 Compatible tools

  • Hugging Face Transformers
  • PyTorch
  • TensorFlow
  • Multilingual translation and assessment tools

‍

‍

💡 Tip

For a more accurate benchmark, combine this data with human translations as soon as possible.

Frequently Asked Questions

How many languages are covered by the X-CSR dataset?

The dataset covers 16 languages, including French, English, English, Chinese, German, Spanish, and more.

Are the translations done by humans?

No, the available versions are the result of automatic translations, which can introduce noise.

Can this dataset be used for commercial use?

Yes, the MIT license allows free commercial use.

Similar datasets

See more
Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.

Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.

Category

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Suspendisse varius enim in eros elementum tristique.