A scheduled job that does nothing still looks green

Alerting is built for errors: something threw, something returned non-zero, something timed out. The jobs that hurt me were doing none of that. A queue drainer that pulled zero items because the queue was mis-keyed and always empty. A notifier that exited clean after sending nothing. A health check that dutifully probed a URL that no longer existed. Each one was green. Each one was also completely inert, and the greenness was actively lying about it.

The first piece is a catalog: every scheduled job, what it is, and how often it should fire. The obvious risk is that the catalog drifts from reality the moment someone adds a cron and forgets to list it. So the catalog is not documentation, it is a test. A check compares the catalog against the actual workflow files and fails if a job lives in one but not the other. You cannot add a scheduled job without the dashboard learning it should have run, because the suite goes red until you do. The invariant is encoded, not remembered.

With a trustworthy list of what should run, a probe classifies what actually happened in each job's window. Fired means it ran and did work. Skipped means it ran and correctly had nothing to do, which is fine. Silent means it did not run when it should have, or ran and produced nothing where nothing is the wrong answer. That third bucket is the whole point. It is the state no error-based alert will ever show you, and it is the one the dashboard paints red.

Turning it on was uncomfortable in the good way. Eleven jobs came up silent. The queue drainer had been writing its key in the wrong case long enough that the queue it fed had never once filled. The notifier had been exiting clean while sending nothing, because a base URL it needed was unset in the scheduler. The health check had been probing a path a rename removed. None of these had ever thrown. All of them had been green for weeks. I fixed several that afternoon, and each fix was small; the hard part was never the repair, it was seeing the failure at all.

While I was there I put month-to-date model spend against its ceiling on the same screen, because it is the same class of question. A budget that silently runs past its limit is a silent failure with a bill attached. One dashboard now answers both did every job that should have fired actually do something, and are we under the money line, which are the two things about this system I most want to be unable to not-know.

The honest limits. The reader is fail-soft and reports only booleans and timestamps, never a job's contents, and the whole thing is admin-only. It tells me a job went silent; it does not diagnose why. And it can only watch jobs the catalog knows about, which is exactly why the drift test, not the dashboard, is the load-bearing part. A pretty dashboard over a catalog that silently missed half the crons would just be a nicer way to be wrong. The test is what makes the green mean something.