2026
July – September
-
It's data time. I've been getting the practice and mechanics of training models off the ground, with just basic eval (slice of generic pre-training data). My intentions are to pre-train and fine-tune toward more specific tasks. So it's time to get a first round of some simple ev…
-
Did some light optimization work for a 30% decrease in the time to train my baseline model. Notes in my logbook: https://codycollier.com/lx/2026/2026-08-01-a-model-training-perf-baseline-and-some-optimizations.html
-
So much capital and engineering work has been put into scaling transformer based LLMs. It seems like there are lots of opportunities left in the wake of that focused effort.
-
Latest from scratch pre-training. Gemma architecture (~164M parameters), C4 dataset, 50k steps. Almost 47 hours on my RTX 3070.
-
Overnight HVAC lends a hand for GPU temp on a long training run :)
-
I wonder what proportion of the training tokens come from scanned books versus the Internet at the frontier training shops.
-
Always Be Training Working out some problems with pre-training a GPT-2 model with C4 dataset on a Google TPU v6e1.
-
Having fun with some of the output of my alien ink library :)
-
One way to reason about applied LLMs (and agents), is to think of them as bridges between natural language and structured data and interfaces.
-
If you don't like the weather in AI, wait 15 months.
-
I've been revisiting older podcast episodes for LLM fine-tuning, gpt-3 release, etc. It's been helpful for refreshing memory but it's also interesting to hear the ideas and thinking from not that long ago.
-
Practice. Another from-scratch training of a GPT-2 architecture model with about 124M parameters. This time, instead of 4 epochs on a small subset of data, it was trained on a stream of a subset of Wikipedia (English). It trained for 50k steps, which was about 1.64B tokens. This…
-
6.5 hours, 4 epochs, gpt-2 architecture with English Wikipedia data
-
Watching learning curves is satisfying. Kind of like watching htop or logs while analyzing a linux machine under load.
-
Heads up on a new paper. A. Use BM-25 to gather candidates. B. Use direct interaction (grep etc) on that smaller set. This is deceptively simple, and the intuition is better aligned when it comes to recall then precision. With a little query expansion on the front end, there may…