TX · Utils

LLM Training Data Calculator

If your model has N parameters, how much text does it need? A quick estimate for GPT-2 style decoder models using Chinchilla scaling heuristics.

data.tokens(N)

Model

Preset model

Parameter count

124M parameters

Training regime

GPU weight memory

FP32 · 4 B/param

—

FP16 · 2 B/param

—

BF16 · 2 B/param

—

INT8 · 1 B/param

—

INT4 · 0.5 B/param

—

Weights only (params × bytes/param). Excludes activations, optimizer state, and KV cache. Architecture-based calculator →

Estimate

Training tokens needed

—

Words

—

÷ 1.33 tokens/word

Wikipedia articles

—

÷ 600 words/article

English Wikipedia

—

of ~4B words total

Books (avg.)

—

÷ 80k words/book

Simple rule: tokens ≈ ratio × parameters. Chinchilla found ~20× is compute-optimal; GPT-2 era models used far more data relative to size. Word and article counts are rough conversions — see references for nuance.

Tokens vs. model size (log–log)

Chinchilla (~20×) Legacy (~80×) Selected model

References

Hoffmann et al. (2022) — Training Compute-Optimal Large Language Models (Chinchilla)
Kaplan et al. (2020) — Scaling Laws for Neural Language Models
Radford et al. (2019) — Language Models are Unsupervised Multitask Learners (GPT-2)
Wikipedia: Size of Wikipedia — article and word counts