Recent
-
I've been revisiting older podcast episodes for LLM fine-tuning, gpt-3 release, etc. It's been helpful for refreshing memory but it's also interesting to hear the ideas and thinking from not that long ago.
-
Practice. Another from-scratch training of a GPT-2 architecture model with about 124M parameters. This time, instead of 4 epochs on a small subset of data, it was trained on a stream of a subset of Wikipedia (English). It trained for 50k steps, which was about 1.64B tokens. This…
-
6.5 hours, 4 epochs, gpt-2 architecture with English Wikipedia data
-
Watching learning curves is satisfying. Kind of like watching htop or logs while analyzing a linux machine under load.
-
Heads up on a new paper. A. Use BM-25 to gather candidates. B. Use direct interaction (grep etc) on that smaller set. This is deceptively simple, and the intuition is better aligned when it comes to recall then precision. With a little query expansion on the front end, there may…
-
Search, agents, and fine-tuning models are an interesting triangle.
-
Frontier LLMs sucked all the air out of the room. Recently though, the market is encountering frictions. Fine-tuning small and reasonably sized language models will get more breathing room again.
-
It feels like several things are happening which will support a resurgence of interest in LLM fine tuning for local model use.