TLDR-17
Massive dataset of Reddit posts in English with content and associated summary, designed to train and evaluate automatic summarization models.
Approximately 3.8 million posts with summary, JSON files, total size ~19 GB
CC-BY 4.0
Description
The dataset TLDR-17 contains more than 3.8 million Reddit posts in English, each with a short summary (on average 28 words). Data is preprocessed and provided in JSON format with multiple fields such as author, content, summary, and original subreddit.
What is this dataset for?
- Train automatic summary models (abstractive summarization)
- Evaluate the performance of NLP models on real social content
- Develop content synthesis systems for forums and social networks
Can it be enriched or improved?
This dataset can be enriched by adding qualitative annotations on the quality of the summaries, or by creating multi-language versions. Selecting and cleaning can improve model performance.
🔎 In summary
🧠 Recommended for
- NLP researchers
- AI social media teams
- Chatbot developers and assistants
🔧 Compatible tools
- Hugging Face Transformers
- TensorFlow
- PyTorch
- SpacY
- Jupyter notebooks
💡 Tip
Pre-filter the data to limit the noise associated with certain subreddits or off-topic posts.
Frequently Asked Questions
Does this dataset include metadata about the authors or the date of the posts?
Yes, fields like author, subreddit, and subreddit_id are included, but specific dates are not always present.
Can this dataset be used for tasks other than the automatic summary?
Mostly summary-oriented, but the content can be reused for other NLP tasks such as text classification.
What is the total size of the downloaded dataset?
Approximately 19 GB of data generated, with 3.14 GB of compressed files to download.
