A probe that pings the chat only when the checkout breaks

A checkout page went down one night and nobody noticed until the morning. The ads kept running, the clicks kept landing on a page that showed a spinner, and a day of traffic bought nothing. The hosting provider's status page was green the whole time, because the server was up; it was the page that was broken. After that I wanted a probe that looks at the page the way a customer does.

What it checks

Not "does the server answer". It fetches the checkout URL with a normal browser user agent, a ten-second timeout, and asserts three things: status 200, response time under a threshold, and a string that must be present in the HTML. The string is the label on the pay button. If the button is gone, the page is down as far as a customer is concerned, whatever the status code says.

It runs every ten minutes from a cron line on a machine that is always on.

Why state change only

The first version posted a line to the chat on every failed check. Within a week everybody had muted the channel. A report every ten minutes is noise even when the news is bad, and noise trains people to ignore the one message that matters.

The second version keeps a tiny state file with the last known state, up or down, and a counter of consecutive failures. It posts only when the state flips.

fails = fails + 1 if not ok else 0
new_state = "down" if fails >= 2 else "up"
if new_state == state:
    return
post(f"checkout {new_state}: {url} ({reason})")
state = new_state

Two consecutive failures before calling it down, so a single slow response or a blip at the CDN does not page anyone. One check back up before calling it recovered, with a short note saying how long it was out. Between those two messages the channel is silent, which means a message in the channel is always news.

It never acts on its own

The probe does not pause anything, does not touch the ads, does not restart a server. It says what it saw and leaves the decision to a person. I keep that boundary on purpose: a monitor that is wrong once and only talks costs nothing, a monitor that is wrong once and acts costs a day of revenue in the other direction. Every message it posts ends with the word ADVISORY so nobody mistakes it for an action already taken.

Small things that turned out to matter

The timeout has to be shorter than the cron interval, or runs stack up. The user agent has to look like a browser, or some hosts serve a lighter page without the button. The string check has to be something a human would miss if it were gone, not a comment in the HTML that a redesign can remove without anyone noticing. And the state file has to survive a reboot, so it lives on disk, not in memory.

The whole thing is about sixty lines. It has fired four times in two months, three of them real, and the one false alarm was a redesign that renamed the button, which is a fair thing to be told about too.