The worst failure an uptime monitor can have is not going down. It is staying up and going quiet: probes stop running, dashboards stay green, and the customer finds out their site was down from a customer of their own. The monitor did not lie exactly. It just stopped telling the truth and told nobody.
We have had two brushes with this ourselves. On 2026-08-10, a wedged process held a region's lock for nine hours while every machine reported healthy. On 2026-08-13, a deploy race meant nine minutes where nothing probed at all. Neither incident broke any code path. In both cases the system noticed something and told no one.
So we started keeping a register: every way we know the probe can fail, what the system does about it today, and, the column that actually matters, whether anyone finds out. This post is a tour of the mindset and the mechanisms that came out of it.
Why does a monitor fail silently?
A monitor fails silently because the thing that would report the failure is the thing that failed. Crashes are the easy case. A crashed process stops writing heartbeats, heartbeats go stale within two minutes, and staleness pages us on a single failing poll. The process that would have noticed is dead, so something outside it has to do the noticing. That outside thing exists and is itself monitored.
The hard cases are the ones where everything keeps running.
A region that stops probing while its machines report healthy. Each of our regions is owned by exactly one process at a time, via a database lock. The lock is a Postgres session-level advisory lock, and the Postgres documentation says such a lock is held until it is released or the session ends. If a process dies without releasing that lock, the survivors politely wait, and nothing probes the region.
Today a passive process watches its region's heartbeat. If the supposed owner goes quiet, it records an operational error naming the runbook, within about a minute. The lock table also names who we believe holds each region and when they last renewed, so a human can check the claim directly.
A check that errors on our side of the wire. A database blip while recording a result is our failure, not the customer's outage. Those used to be swallowed by a pool's catch block and counted nowhere. They are now tallied separately from real check failures. Confusing "we could not look" with "we looked and it was down" is exactly the kind of lie the whole product exists to prevent.
An interval that quietly drifts. We advertise 60 second checks. For a while, production actually delivered them every 75 seconds, an artifact of when a timestamp was stamped relative to a 15 second scheduling grid. There was no error and no crash, just a product doing 25% less than its own pricing page said.
The fix was straightforward. The interesting part is the guard, which now measures the achieved gap directly and fails our own health checks if it drifts past 1.2x, so the same regression cannot happen silently twice.
Who watches the watcher?
A second, much smaller set of watchers covers what the product's own monitors cannot. The public monitors already watch the site, the API, MCP and the status page from the probe regions, and email on a real outage. The extra watchers exist only for failures that would silently lose data or stop shipping: backups going stale, disks filling, a deploy pipeline that stops ticking, a job that stopped running for any reason.
An internal health endpoint is polled once a minute. A non-answer is re-probed three times, ten seconds apart before it counts. A signal that could be a rollout blip needs two failing polls in a row, while the ones that are already confirmed conditions, such as a probe region's heartbeat going stale, count on the first.
Scheduled jobs report in with dead-man heartbeats, the same pattern Healthchecks documents: the job pings on success, and the absence of a ping is the alert. That catches the case no error handler can, where the job never ran at all.
How does an alert get delivered without becoming noise?
Alerts follow one rule everywhere. A check that starts failing sends one email. A check that passes again sends one recovery email, with how long it was down. Nothing is sent in between, and a failure that was never mailed ends silently.
The rule has a guard against the mail path failing itself. While any alert on a host has not been delivered by any transport, that host's dead-man ping goes to a failure endpoint with the list, and that alerts through a separate email route. A broken mail path therefore cannot hide itself.
What we deliberately do not defend
Honesty also means saying what the register does not cover. We do not self-heal a stranded region lock, because forcibly terminating a database session on the strength of a two minute old timestamp is a destructive action, and a human confirms the signature first. And a target that returns a cheerful 200 on a broken page is out of scope for most check types: we can tell you your server answered, not that the answer was any good.
Why the register is public
The phrase "we monitor the monitor" is a claim every vendor makes, and it is unfalsifiable as marketing copy. A register of specific failure modes, with the embarrassing incidents that motivated them, is checkable. It also keeps us honest: the rule in our codebase is that any change to the probe updates the register in the same pull request, so the document cannot rot into aspiration.
If you run infrastructure, we would gently suggest the exercise: list the ways your monitoring can fail, and for each one write down who finds out, and how fast. Google's SRE book makes the related point that monitoring should tell you what is broken and why, and a monitor that cannot say it is broken fails the first half. The rows where the answer is "nobody" are the outages you have not had yet.