TX · Utils

LLM Training Data Calculator

How many tokens, words, and Wikipedia-scale articles common GPT-style models need — Chinchilla (~20×) and legacy (~80×) scaling heuristics.

data.tokens(N)

Model	Params	BF16	Tokens · 20×	Tokens · 80×	Words · 20×	Wiki arts · 20×	EnWiki · 20×	Books · 20×
Loading…

Sorted by parameter count. Simple rule: tokens ≈ ratio × parameters. Chinchilla (~20×) is compute-optimal; GPT-2 era models used far more data relative to size (~80×). Word / article / book columns use the 20× regime with ÷ 1.33 tokens/word, ÷ 600 words/article, ~4B English Wikipedia words, and ÷ 80k words/book. BF16 is weights only (2 bytes/param).

References

Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models (Chinchilla)
Kaplan et al. (2020) — Scaling Laws for Neural Language Models
Radford et al. (2019) — Language Models are Unsupervised Multitask Learners (GPT-2)
Wikipedia: Size of Wikipedia — article and word counts
Transformer GPU Memory Calculator — BF16 weight memory