Production fell a day behind and nothing said a word

Deploy tooling is built around events. A build started. A build succeeded. A build failed. Those are the moments something can fire a webhook, and every one of them is a thing that happened. The failure that burned me was the opposite kind. It was a thing that stopped happening. Auto-deploy had quietly quit promoting commits, so there was no failed build to alert on and no error to catch, only a widening quiet where new deploys used to land. You cannot subscribe to an event that never fires. The one signal I needed was the exact signal an event stream can never send.

So the radar does not wait to be told. It reads three signals and folds them with one small pure function. First, what is actually live, the commit the running deployment reports at the site's version endpoint. Second, the event stream, for the deploys that do happen. Third, the tip of main, read straight from GitHub, so a paused pipeline still shows up as behind. The verdict falls out of comparing what is running against where the code really is. That is what makes a stall visible even when nothing at all is deploying. I stopped asking whether a deploy had failed. I started asking whether one was missing.

One move matters more than it looks. The worst possible moment for a deploy pipeline is the day the platform running it is having its own bad day, and a stall detector that lives on that same platform goes down right alongside it, exactly when you need it awake. So the alerting half runs somewhere else entirely. It sits on a schedule on separate infrastructure, off the platform it watches. It reads the live version, compares it to main, classifies with the same function the dashboard uses, and pushes a message the instant it sees a stall. If the host is down, the watcher is still up. That is the whole reason it lives out of band.

This is not a hypothetical. The first stall the radar ever flagged was the one that made me build it in the first place. I opened the dashboard one morning and the verdict was red. Production was twenty-eight commits behind main, and the oldest of those commits had been waiting to ship for roughly nineteen hours. The threshold that trips a stall is twenty minutes. A commit that lands on main should be live within that window, and past it, with nothing building, the pipeline is stuck rather than slow. Nineteen hours is that same window missed more than fifty times over, in total silence. Auto-deploy had quietly paused, and the last deploy that did go out was still sitting there green, which is the whole reason nothing had told me. The fix took a few minutes once I could see it. Seeing it was the part that had been impossible.

Now the honest limits, because they carry the most weight. The reader is fail-soft. It reports only SHAs, states, and timestamps, never a word about what any deploy contains, and the whole page is admin-only. The webhook half accepts signed deliveries alone and fails closed the moment its secret goes missing, so nothing can spoof it into a false all-clear. And it tells me production is behind. It does not tell me why the pipeline stopped. That part of the diagnosis stays mine. This is the same shape as the heartbeat I built for scheduled jobs, turned now on the one system every other green quietly depends on. A deploy that silently did nothing is just a cron that silently did nothing, with the entire site sitting downstream. That is why the two of them share a screen on the burn dashboard.