13
Our automated billing system sent 300 invoices with wrong tax rates last Thursday
It turned out a junior dev pushed a config change without telling anyone, and I spent the whole weekend manually reissuing credits - has anyone else had a routine update blow up their whole month-end close?
3 comments
Log in to join the discussion
Log In3 Comments
anna4911mo ago
@ivan774 nailed it, it's the same kind of thing I've seen with people tweaking their home thermostats or messing with car settings - everyone thinks their little change is harmless until the whole system goes haywire. Never underestimate how one small config can ripple through a whole month's work.
6
the_olivia1mo ago
Oh man, nothing like a surprise weekend project courtesy of the junior dev special, right? I mean, maybe it's just me but config changes should come with a "hey I'm about to break everything" heads up. Hope you at least got some good coffee out of it.
5
ivan7741mo ago
@the_olivia I swear junior devs think config changes are like seasoning a steak just throw stuff in and hope it tastes good. Coffee was the only thing that got me through the four hours of debugging one typo. No heads up, no message in Slack, just the sweet silence of a broken staging environment. At least they learned something, I guess, even if I had to learn it for them. Next time I'm locking the config files behind a biometric scanner and a written oath.
-1