Rant: our Stripe billing integration went down for 3 hours during a renewal push last Tuesday

We had 47 customers hit payment failures before our ops guy caught it, and I spent the whole afternoon manually re-running invoices through the Stripe dashboard because the webhook just silently stopped firing.

3 comments

3 Comments

leo_black761mo agoProlific Poster

oh man, i feel your pain. what i started doing after a similar incident is setting up a separate heartbeat check on the webhook endpoint itself using a simple cron job that pings it every 5 minutes and alerts if it doesn't respond. also, i made a backup script that cross-references stripe's recent charges against our db every hour so we catch gaps before they pile up. that three hour window is brutal but at least now you know exactly where the weak link is.

dixon.james1mo ago

Lol yeah it really does feel like we're building a whole surveillance state just to make sure our own code works. Next step is probably a cron job that emails me if my other cron jobs haven't emailed me lately. At some point you gotta wonder if the monitoring is actually more fragile than the thing it's watching lmao.

thomas_sanchez1mo ago

Oh great, so now I need a whole second system just to watch the first system? Might as well hook up a camera to watch the camera watching the webhook. At this rate I'll end up with more monitoring scripts than actual application code, and my server will be running an existential crisis instead of a business. But yeah, that hourly cross-reference script is smart, I should probably add one before I end up with a whole months worth of missing payments and a very angry accountant.