TX · Utils

Transformer GPU Memory Calculator

Estimate parameter count and weight memory for a GPT-style decoder-only transformer on GPU. Select a historical preset or dial in your own architecture.

model.size()

Architecture

Preset model

Layers

Attention heads

Hidden size

FFN multiplier

Vocabulary size

Context length

KV heads (GQA) — blank for MHA

Weight precision

Tie token embedding & LM head Learned positional embeddings Linear bias terms (GPT-2 style)

Results

GPU weight memory

—

— parameters

—

Counts a standard pre-norm decoder block: multi-head self-attention, feed-forward network (d × ffn × d), layer norms, token + optional position embeddings, and an untied LM head when disabled. Inference activations and KV cache are not included.