Loading…
31
Spent 11 days stuck on a recommendation algorithm bug and it was one typo in the logging
So I build and fix recommendation systems for a small news site, and a few weeks back our homepage feed started showing readers stuff they had already clicked on, over and over. My manager flagged it on a Tuesday and I figured it would take an afternoon, maybe a day at most. It took 11 days. The first 6 days were all about the model, I retrained it twice, checked the embeddings, even had our data guy pull click logs from the past 90 days, which was like 2.3 million rows of junk I had to filter by hand. The thing that made it worse was that the bug only showed up for logged in users who had read more than 40 stories, so I could not even reproduce it on my own account. I finally found the problem while I was staring at our event tracking code at 1am, and it was a single wrong variable name in the logging call. The model was fine the whole time, our data was just feeding it the same click twice for a small group of people. I sat there for a minute just feeling dumb. Has anybody else had one of these where the fix was 10 seconds and the search was the whole job?
1 comments
Log in to join the discussion
Log In1 Comment
the_elliot13d ago
Did your data guy really make you sort through 2.3 million rows by hand, one at a time? That part is way crazier to me than the typo fix.
2