How Large Language Models Work: A Visual Deep Dive into Training Data Collection
By
ynarwal__
A respectable bake. You'd come back tomorrow for another.
Summary
This article provides a visual deep dive into how Large Language Models (LLMs) work, starting with the data collection process. It explains that organizations like Common Crawl have been indexing the web since 2007, amassing billions of pages by 2024. The raw data is filtered into high-quality datasets like FineWeb, with the goal of obtaining a large quantity of diverse, high-quality documents. After aggressive filtering, the dataset amounts to about 44 terabytes — roughly 15 trillion tokens — fitting on a single hard drive. The key insight emphasized is that the quality and diversity of training data has the most significant impact on the final model's performance.
Key quotes
· 4 pulledThe first step is collecting an enormous amount of text.
Organizations like Common Crawl have been crawling the web since 2007 — indexing 2.7 billion pages by 2024.
The quality and diversity of this training data has more impact on the final model.
After aggressive filtering, you end up with about 44 terabytes — roughly what fits on a single hard drive — representing ~15 trillion tokens.
You might also wanna read
RTP-LLM: Alibaba's High-Performance Inference Engine for Large Language Model Deployment
This paper presents RTP-LLM, a high-performance inference engine developed by Alibaba for industrial-scale deployment of Large Language Mode
arxiv.org·1d ago
What pretraining on unlabeled text teaches large language models about language structure
Pretraining on unlabeled text teaches large language models to model the statistical structure of language by optimizing next-token predicti
sebastianraschka.com·1d ago