Silent failure: when a scheduled job stops and nobody notices

Why silence is the worst kind of failure

Error monitoring, crash detection, and log alerts all depend on the broken system producing a signal. Silent failures produce nothing, which means your entire monitoring stack is blind to them.

Which systems produce silent failures most often

Some schedulers are architecturally silent by default. Knowing which is which helps you prioritise what to instrument first.

• ! Unix cron: failed job output goes to MAILTO, typically a local mail spool nobody reads. A job that doesn't run at all produces no output, nothing to mail, and nothing in any log except a cron daemon line that most people never check.
• ! GitHub Actions scheduled workflows: automatically disabled after 60 days of inactivity in public repositories, with no notification when it happens. A workflow that hangs never reaches failure status, so GitHub's failure email never fires either.
• ! AWS EventBridge → Lambda: if the Lambda function was deleted, the IAM role revoked, or the function throws an uncaught error, EventBridge retries twice and then drops the invocation. The CloudWatch metric FailedInvocations exists but is not alarmed by default.
• ! Kubernetes CronJobs: a failed job does not prevent the next scheduled run, but failed runs accumulate silently unless you watch kubectl get jobs. A CronJob suspended by an operator also fires nothing.
• ! Heroku Scheduler: Heroku documents it as "best effort"; it is not guaranteed to fire. A missed run produces no log entry and no notification; only the absence of your expected output reveals it.
• ! Cloud functions on timer triggers (Google Cloud Scheduler, Azure Timer Functions, Cloudflare Workers cron triggers) all have the same property: a function that throws silently (logs the error but doesn't alert) or a trigger misconfiguration (wrong timezone, deleted target) produces complete silence from the outside.

The fix: invert the signal

Dead-man's-switch monitoring makes the expected success signal the alert trigger, not the absence of an error signal.

The principle is simple. Instead of asking "did the job report an error?", you ask "did the job report success in the window I expect it?" If the answer is no (for any reason), an incident opens. It doesn't matter whether the failure was silent (never ran), noisy (crashed), or ambiguous (hung): if the ping doesn't arrive, you hear about it.

# The entire integration for a cron job:
0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://ping.lastping.dev/<your-monitor-id>

# For an immediate failure alert too:
0 3 * * * /usr/local/bin/backup.sh \
  && curl -fsS -m 10 --retry 3 -o /dev/null https://ping.lastping.dev/<id> \
  || curl -fsS -m 10 --retry 3 -o /dev/null https://ping.lastping.dev/<id>/fail

The same pattern works for every system: a ping at the end of a Lambda handler, a fetch at the end of a Cloudflare Worker, a requests.get at the end of a Cloud Scheduler function, a workflow_run webhook for GitHub Actions (which also catches the 60-day silent disable). For AI agents, the LastPing MCP server exposes the same dead-man's-switch as a tool the agent calls directly.

See the platform-specific setup guides: cron, GitHub Actions, AWS Lambda, Google Cloud Scheduler, Azure Functions, Cloudflare Workers, or the full guide index.

Questions people ask

Front-loaded answers: the most important fact first.

• What is a silent failure in a scheduled job?

  A silent failure is when a scheduled or automated job stops producing its expected result without generating an error message, log entry, or alert of any kind. The job either doesn't run at all, runs but exits with a swallowed error, hangs indefinitely, or keeps running while making no further progress, and none of these conditions are visible until someone notices the missing output, often days or weeks later. Examples include a cron job whose crontab was deleted during a server rebuild, a backup script that silently exits on a full disk, or a GitHub Actions scheduled workflow disabled after 60 days of inactivity in a public repository.

• Why are silent failures so dangerous?

  Silent failures accumulate undetected. A backup that stopped three weeks ago looks fine until the day you need to restore. A certificate renewal job that stopped last month looks fine until the cert expires. A data sync that stopped last week looks fine until a user reports stale data. The damage compounds in the dark while all monitoring dashboards show green.

• Why can't error monitoring detect silent failures?

  Exception trackers, log alert rules, and uptime checkers all detect signals the broken system emits. A silent failure emits nothing. From an error monitor's perspective, a job that never runs is indistinguishable from a job that ran perfectly. Only dead-man's-switch monitoring (which expects a signal and alerts on its absence) can catch silent failures.

• What is the difference between a silent failure and a crash?

  A crash generates an error signal: an exit code, a stack trace, a log line, an exception alert. A silent failure generates nothing. The job simply stops doing its work. Crash detection (error monitors, exception trackers, log alerts) cannot detect silent failures because there is nothing to detect. Only dead-man's-switch monitoring, which expects a signal on a schedule and alerts on its absence, reliably catches silent failures.

• What is the difference between a silent failure and a hung job?

  A silent failure is any failure that produces no observable signal. A hung job is one specific variety: the job started, is technically still running, but will never complete. Dead-man's-switch monitoring catches hung jobs through overrun detection: send a /start ping when the job begins; if no completion arrives within the grace window, an incident opens even though the job process is still alive.

• Which systems are most prone to silent failures?

  Unix cron (errors go to an unread mail spool), GitHub Actions scheduled workflows (disabled after 60 days of inactivity in public repositories, with no notice when it happens), cloud functions on event schedules (EventBridge, Cloud Scheduler, Azure Timer), Kubernetes CronJobs (a failed job does not re-schedule the next run), Heroku Scheduler (best-effort, not guaranteed to fire), and any script run by a task queue where the worker process died.

• How do you detect a silent failure?

  Dead-man's-switch monitoring. The job is instrumented to send a ping (a plain HTTP GET) on successful completion. An external service (LastPing) expects the ping on the same schedule as the job, plus a grace window. If the ping doesn't arrive, an incident opens and alerts are sent. The absence of the signal is the alert, which makes it impossible for a silent failure to remain silent.

• Which backup-specific silent failure should I be most worried about?

  The most dangerous backup silent failure is a zero-byte output with no error: the backup "ran" but wrote nothing. This happens on full disks, revoked credentials, or unreachable targets. The backup log shows success; the backup file shows 0 bytes. Always verify both the ping (job ran) and the output size (job produced something). See the dedicated backup silent failure page for patterns.

• Is LastPing free?

  Yes. LastPing is free for individuals with no monitor cap, unlike Healthchecks.io's 20-check free plan or Cronitor's 5-monitor free plan. See comparisons: vs Healthchecks.io and vs Cronitor.