A scheduled job that does nothing still looks green

A cron that crashes will page you. A cron that quietly stops firing, or fires and does nothing, stays silent, and silence looks exactly like success on the dashboard. So I built a heartbeat check for the silent failure mode: every job has to check in on schedule, and the absence of a check-in is itself the alert. Green now means "it ran," not "nothing threw an error."

A crashing cron pages you. A cron that quietly stops firing says nothing, and nothing looks identical to success. I wired up heartbeats: every job has to check in on time. A missing check-in is the alert. Green finally means "it ran."

How I built it: https://www.jakelawrence.xyz/blog/did-the-cron-actually-fire

The scary cron failure isn't the one that crashes. It's the one that stops firing and tells you nothing, because silence looks exactly like a healthy green check. My fix: every scheduled job has to check in on time. If the check-in doesn't arrive, that absence is the alert. Now green means "it actually ran," not "nothing threw."

A cron that crashes will page you. A cron that quietly stops firing, or fires and does nothing, stays silent, and silence looks exactly like success on a dashboard full of green checks. So I built a heartbeat for the silent failure mode. Every scheduled job has to check in on time. The absence of a check-in is itself the alert. Green now means "it ran," not just "nothing errored." https://www.jakelawrence.xyz/blog/did-the-cron-actually-fire

The dangerous cron isn't the one that crashes and pages you. It's the one that quietly stops firing and says nothing. Silence looks exactly like success. Fix: every job has to check in on time. A missing check-in is the alert.

Wrote up the heartbeat approach: https://www.jakelawrence.xyz/blog/did-the-cron-actually-fire

A cron that crashes pages you. A cron that quietly stops firing says nothing, and silence looks identical to a green check. So I flipped it: every job has to check in on schedule, and a missing check-in is the alert. Green means "it ran."