The Tuesday our billing sync failed and I spent 6 hours untangling a $14k mess
Last Tuesday we hit a sync error between Stripe and our internal tool at around 9am. 38 customers got double charged, totaling about $14,000, and support started getting tickets by lunch. I spent the rest of the day in spreadsheets and webhook logs, and honestly it was the most useful disaster I've had all year. Now I'm wondering how many of you run a daily reconciliation check, and if so, what tool actually catches this stuff before customers do?
Been there and it sucks so bad, we had almost the exact thing happen last spring when a retry loop kept firing and nobody noticed for two days. What finally saved us was a cheap script that pulls both sides every morning and drops any mismatch into a Slack channel before anyone's coffee gets cold. It won't fix the mess, but it sure beats finding out from angry customers.