Automationsmonitoringalerts

Alerts that post once, not every hour

Christian Drexler

I had an hourly check that looked at a handful of running things and posted to a chat room when one of them was not healthy. It worked exactly as written. One item had a real problem that nobody fixed for days, so the check reported it every hour. Several other items were switched off on purpose, which the check also considered unhealthy, also every hour.

After a few days the room was full of the same handful of facts, repeated. The one that needed a person was buried under the ones that did not, and under itself.

The check was correct each time it posted. It was also useless by then, because a channel full of repeats is a channel people mute.

The mistake in one sentence

The check posted whenever a condition was true. Conditions that change slowly stay true for a long time. So the number of messages measured how long a problem had existed, not how many problems there were.

What I wanted was a message when something changes: a new problem, a problem that went away, and an occasional reminder for the ones that stay.

A fingerprint for each finding

To know whether a finding is new, it needs an identity that is the same from one run to the next. The obvious candidate, the message text, does not work, because the text contains numbers that move.

checkout page: 3 failed probes, last success 4 h ago
checkout page: 4 failed probes, last success 5 h ago

Same problem, different string. So before hashing, the finding is reduced to the parts that define it: what it is about and what kind of problem it is. Counts, durations and amounts are stripped out.

import hashlib, re

def fingerprint(entity_id: str, kind: str, detail: str = "") -> str:
    # drop everything that drifts between runs: numbers, units, timestamps
    stable = re.sub(r"\d+([.,]\d+)?\s*(h|min|s|%|\$|€)?", "#", detail.lower())
    return hashlib.sha1(f"{entity_id}|{kind}|{stable}".encode()).hexdigest()[:12]

The entity identifier stays in, untouched. Two different pages with the same problem are two findings. One page with the same problem an hour later is one.

It is worth being generous with what you strip. If the fingerprint changes when it should not, you are back to a post per hour. If two slightly different states of the same problem collapse into one, you lose nothing that matters.

The state file and the three rules

Between runs the check keeps a small JSON file: for each fingerprint, when it was first seen and when it was last posted.

import json, time, pathlib

STATE = pathlib.Path("~/.cache/healthcheck_state.json").expanduser()
REPEAT_HOURS = 24

def decide(findings: dict[str, str]) -> list[str]:
    """findings: fingerprint -> human text. Returns the lines to post."""
    state = json.loads(STATE.read_text()) if STATE.exists() else {}
    now, out = time.time(), []

    for fp, text in findings.items():
        seen = state.get(fp)
        if seen is None:
            out.append(f"NEW  {text}")
            state[fp] = {"first": now, "posted": now}
        elif now - seen["posted"] >= REPEAT_HOURS * 3600:
            days = int((now - seen["first"]) // 86400)
            out.append(f"STILL OPEN {days}d  {text}")
            seen["posted"] = now

    for fp in [f for f in state if f not in findings]:
        out.append(f"RESOLVED  {state[fp].get('text', fp)}")
        del state[fp]

    for fp, text in findings.items():
        state[fp]["text"] = text
    STATE.write_text(json.dumps(state))
    return out

That is the whole mechanism. The rules it implements:

  1. A finding that was not there last time is posted once, marked NEW.
  2. A finding that is still there is repeated at most once every 24 hours, marked STILL OPEN with its age. The age is the useful part: "still open 3d" reads very differently from the first report.
  3. A finding that disappeared is posted once as RESOLVED, and forgotten.

If the list of lines is empty, nothing is posted. Most runs post nothing, which is the goal.

Do not report on things that are switched off

The second half of the noise had a different cause. The check looked at everything it knew about, including items that had been paused deliberately. A paused thing is not delivering, so it failed the health test, so it was reported.

A finding about something that is off cannot be acted on. The filter belongs before the fingerprinting: only things that are supposed to be running get checked at all. It sounds too obvious to write down. It was responsible for half of the messages.

Seeding, so the first run is quiet

One practical detail. On the day the deduplication goes live, the state file is empty, so everything currently wrong counts as new and gets posted in one burst. If the room has already seen all of it a hundred times, that burst is the last thing anyone wants.

So the script has a flag that fills the state from the current findings without posting. Run it once with the flag, and from the next run on only real changes come through.

What changed

The same check, with the same thresholds, went from a message per finding per hour to a few lines a day, and on a normal day to none. A NEW line now gets read, because it is the only line there.

I have since put the same three rules into every recurring job that talks to a chat room. They add about thirty lines to a script, and they are the difference between a check that gets read and one that gets muted.

monitoringalertspythonchat