Practice. Another from-scratch training of a GPT-2 architecture model with about 124M parameters. This time, instead of 4 epochs on a small subset of data, it was trained on a stream of a subset of Wikipedia (English). It trained for 50k steps, which was about 1.64B tokens. This took about 38.5 hours on my NVIDIA GeForce RTX 3070.