Transformer GPU Memory Calculator

Sorted by BF16 weight memory (2 bytes/param), highest to lowest. Weights only — activations, optimizer state, and KV cache are not included. GPT / GPT-2 / GPT-3 presets use tied embeddings and learned absolute positional embeddings; GPT-NeoX / Pythia use RoPE and untied embeddings. Counts assume pre-norm decoder blocks with FFN expansion ×4.