bLiMP - LLM Grammatical Assessment
Structured corpus to test the grammatical abilities of language models in English, covering 67 syntactic, morphological, and semantic phenomena.
Description
The dataset Blimp (Benchmark of Linguistic Minimal Pairs) is designed to test the grammatical skills of language models. It contains approximately 67,000 minimum pairs of sentences, divided into 67 grammatical categories such as syntactic islands, subject-verb agreement, negation, or even verbal morphology. Each pair is automatically generated according to rigorous linguistic rules defined by experts.
What is this dataset for?
- Evaluate language models on specific grammatical phenomena
- Compare language performance between different models (GPT, BERT, etc.)
- Analyze the syntactic or semantic sensitivity of trained models
Can it be enriched or improved?
Yes, bLimp can be enriched by multi-lingual pairs or by additional manual annotations. It is also possible to adapt grammars for other languages or to add metadata on the difficulty or frequency of linguistic constructions. This corpus can also be used as a basis for generating similar challenges in French or in specialized contexts.
🔎 In summary
🧠 Recommended for
- Researchers in computational linguistics
- LLM Assessment
- NLP developers
🔧 Compatible tools
- PyTorch
- Transformers
- Evaluate (HF)
- Jupyter Notebooks
💡 Tip
To assess the robustness of a model, test several grammatical subcategories separately and compare the results.
Frequently Asked Questions
Is the dataset suitable for training models?
No, it's primarily designed for evaluation. It is best to use other datasets for initial training.
Are all examples in English?
Yes, all of the examples are about English grammar. However, it can be used as a basis for creating multilingual versions.
Are the sentences annotated manually?
No, they are automatically generated from grammars written by experts, which ensures high accuracy.
