I thought fine-tuning a small language model would be a weekend project... it took three full days

I wanted to make a basic AI that could write simple, clear summaries of tech news articles. I had a small dataset of about 500 examples I'd made myself. I figured I'd just load it into a model like Mistral 7B, run the training script, and be done by Sunday night. The first run failed because my formatting was wrong, and the output was just garbled text. The second try used too much memory and crashed my local machine after 8 hours. I finally got it running on a cloud service, but then I spent a whole day just tweaking the learning rate and batch size over and over to stop it from giving the same answer every time. What I thought would take maybe 15 hours ended up being closer to 30, with most of that just fixing my own setup mistakes and bad guesses. Has anyone else had a simple training job spiral because of basic config issues?

3 comments

3 Comments

matthewdixon21d ago

I read a blog post where someone spent a week just trying to get the right data format for a fine-tuning job. They kept getting weird repeated phrases and had to rebuild their dataset three times from scratch. The config file parameters were the easy part compared to that.

piper77921d ago

Ugh, what learning rate did you end up using? I tried to fine-tune a tiny model on my own chat logs and it was a mess. Totally get what @matthewdixon said about the data being a trap, my first batch was full of weird typos so the model just learned to be a worse speller. I lost a full day just cleaning that up before training even started.

sage30821d agoOG Member

Feel your pain. Spent a whole afternoon because I saved my data in the wrong type of text file. The model just learned to copy my file paths. @matthewdixon is right, the data part is a trap. You think you're being so careful, then the output is just the same weird sentence fragment 500 times. My weekend project turned into a week of me arguing with my own laptop.