Silent failure: when a scheduled job stops and nobody notices
A silent failure is when an automated job stops producing its expected output without generating an error, a log entry, or an alert of any kind. The job either never runs, exits with a swallowed error, or hangs indefinitely — and none of these conditions become visible until someone notices the missing result. Silent failures accumulate in the dark while all your monitoring dashboards show green. The fix is to invert the problem: instead of waiting for a failure signal, expect a success signal — and alert on its absence.
Why silence is the worst kind of failure
Error monitoring, crash detection, and log alerts all depend on the broken system producing a signal. Silent failures produce nothing — which means your entire monitoring stack is blind to them.
The job never ran
Crontab lost in a server rebuild. GitHub Actions schedule disabled after 60 days without repo activity. EventBridge rule pointing at a deleted Lambda. The scheduler fires nothing — and nothing reports the absence.
The job ran but failed silently
Exit code non-zero, but stderr goes to a local mail spool nobody reads. A Python
exception caught by a broad except: pass.
A backup that writes zero bytes without error because the disk was full.
The job hung
Stuck on a database lock, a full write buffer, or a dead network mount. The job is technically still running — so it hasn't failed yet and sends no alert. It occupies the slot, prevents the next run, and produces nothing.
Why error monitoring can't detect silent failures
Exception trackers (Sentry, Rollbar), log alert rules (CloudWatch, Datadog), and uptime checkers all work by detecting a signal the broken system emits. A silent failure emits nothing — so these tools see exactly what a healthy system produces: silence. From an error monitor's perspective, a job that never runs is indistinguishable from a job that ran perfectly. Only a tool that expects a signal on a schedule — and treats its absence as the alert — can detect silent failures reliably.
Which systems produce silent failures most often
Some schedulers are architecturally silent by default. Knowing which is which helps you prioritise what to instrument first.
-
Unix cron — failed job output goes to
MAILTO, typically a local mail spool nobody reads. A job that doesn't run at all produces no output, nothing to mail, and nothing in any log except a cron daemon line that most people never check. - GitHub Actions scheduled workflows — automatically disabled after 60 days of repository inactivity, with no notification. A workflow that hangs never reaches failure status, so GitHub's failure email never fires either.
-
AWS EventBridge → Lambda — if the Lambda function was deleted, the IAM role
revoked, or the function throws an uncaught error, EventBridge retries twice and then
drops the invocation. The CloudWatch metric
FailedInvocationsexists but is not alarmed by default. -
Kubernetes CronJobs — a failed job does not prevent the next scheduled run,
but failed runs accumulate silently unless you watch
kubectl get jobs. A CronJob suspended by an operator also fires nothing. - Heroku Scheduler — Heroku documents it as "best effort" — it is not guaranteed to fire. A missed run produces no log entry and no notification; only the absence of your expected output reveals it.
- Cloud functions on timer triggers — Google Cloud Scheduler, Azure Timer Functions, Cloudflare Workers cron triggers — all have the same property: a function that throws silently (logs the error but doesn't alert) or a trigger misconfiguration (wrong timezone, deleted target) produces complete silence from the outside.
The fix: invert the signal
Dead-man's-switch monitoring makes the expected success signal the alert trigger — not the absence of an error signal.
The principle is simple. Instead of asking "did the job report an error?", you ask "did the job report success in the window I expect it?" If the answer is no — for any reason — an incident opens. It doesn't matter whether the failure was silent (never ran), noisy (crashed), or ambiguous (hung): if the ping doesn't arrive, you hear about it.
The same pattern works for every system: a ping at the end of a Lambda handler,
a fetch
at the end of a Cloudflare Worker, a requests.get
at the end of a Cloud Scheduler function, a
workflow_run
webhook for GitHub Actions (which also catches the 60-day silent disable).
For AI agents, the LastPing MCP server exposes the same
dead-man's-switch as a tool the agent calls directly.
See the platform-specific setup guides: cron, GitHub Actions, AWS Lambda, Google Cloud Scheduler, Azure Functions, Cloudflare Workers, or the full guide index.
Questions people ask
Front-loaded answers — the most important fact first.
-
What is a silent failure in a scheduled job?
A silent failure is when a scheduled or automated job stops producing its expected result without generating an error message, log entry, or alert. The job either doesn't run at all, runs but exits with a swallowed error, or hangs indefinitely — and none of these conditions are visible until someone notices the missing output, often days or weeks later.
-
Why can't error monitoring detect silent failures?
Exception trackers, log alert rules, and uptime checkers all detect signals the broken system emits. A silent failure emits nothing. From an error monitor's perspective, a job that never runs is indistinguishable from a job that ran perfectly. Only dead-man's-switch monitoring — which expects a signal and alerts on its absence — can catch silent failures.
-
What is the difference between a silent failure and a hung job?
A silent failure is any failure that produces no observable signal. A hung job is one specific variety: the job started, is technically still running, but will never complete. Dead-man's-switch monitoring catches hung jobs through overrun detection: send a
/startping when the job begins; if no completion arrives within the grace window, an incident opens even though the job process is still alive. -
Which backup-specific silent failure should I be most worried about?
The most dangerous backup silent failure is a zero-byte output with no error — the backup "ran" but wrote nothing. This happens on full disks, revoked credentials, or unreachable targets. The backup log shows success; the backup file shows 0 bytes. Always verify both the ping (job ran) and the output size (job produced something). See the dedicated backup silent failure page for patterns.
-
Is LastPing free?
Yes. LastPing is free and fully hosted with no monitor cap and no paid tier — unlike Healthchecks.io's 20-check free plan or Cronitor's 5-monitor free plan. See comparisons: vs Healthchecks.io and vs Cronitor.
Silent failures are invisible until they're expensive.
One ping per job makes every silent failure loud. Set up dead-man's-switch monitoring in minutes — and LastPing monitors itself the same way.