English Wikipedia Articles 2017 (SQLite)
Massive corpus of Wikipedia articles in English dated 2017, structured in an SQLite database to facilitate large-scale textual analysis.
Description
This dataset offers a complete dump of the English version of Wikipedia dated August 2017. It contains nearly 5 million items divided into over 23 million sections. Everything is stored in an SQLite database, allowing for fast queries and seamless integration into industrial NLP pipelines.
What is this dataset for?
- Train language models (LLMs) on a rich encyclopedic corpus
- Test automatic summarization or thematic classification techniques
- Building search engines or question-answer systems based on general content
Can it be enriched or improved?
Yes. This corpus can be enriched by linguistic annotations, semantic vectors, or crossed with databases of internal links. It can also be updated with newer versions of Wikipedia or adapted for multilingual translation and automated summary generation.
🔎 In summary
🧠 Recommended for
- NLP researchers
- Generalist data scientists
- LLMs developers
🔧 Compatible tools
- Gensim
- Hugging Face
- SQLite
- SpacY
- LangChain
💡 Tip
Filter articles by length or theme via SQL for targeted sub-corpora adapted to your NLP tasks.
Frequently Asked Questions
Does the dataset include Wikipedia images, links, or metadata?
No, only the texts of the articles are present. All internal links and other HTML elements have been removed.
Is it easy to access content without SQL knowledge?
A basic knowledge of the SQL language is recommended to fully exploit the structure of the SQLite file.
Is this corpus suitable for training large language models?
Yes, its size and richness make it an excellent basis for pre-training or refining large-scale language models.
