Loading…
14
Appreciation post: That one server crash that taught me about backups
Last Tuesday our e-commerce site went down at 2 PM during a flash sale in Austin. Turns out our database backup script had been failing for 6 months without anyone checking. I spent 14 hours rebuilding customer orders from email receipts and PayPal logs.
2 comments
Log in to join the discussion
Log In2 Comments
jana_shah2mo ago
6 months of failing" - IT guys make great horror movie villains.
4
jana_jones2mo ago
Wait, nobody checked the backup logs for 6 months? Oof. That's the kind of thing that only gets noticed when everything blows up. I had a similar close call with my small shop's database last year. Turns out the backup size was zero bytes for 4 months because I renamed a folder. Just never looked at the reports. Now I have a weekly calendar reminder to actually open the backup file and make sure it has data in it. Lesson learned hard way I guess.
3