GUIDE · BACKUPS · SILENT FAILURE

The backup that ran but didn't run: detecting backup silent failure

Backup jobs fail silently more often than any other class of scheduled task. The log says success. The cron job fired. The file exists — it's just zero bytes. Or the crontab was wiped in last month's server rebuild and nobody noticed because the backups were "the kind of thing that just works." The failure accumulates in the dark until the day you need a restore. Here's how to detect it before that day.

Monitor your backup — free Cron monitoring guide

Backup-specific silent failure modes

Each of these produces no error alert. Only an external expectation catches them.

Failure mode 01

Zero-byte output, exit code 0

mysqldump exits 0 if it can connect and start. A dropped connection mid-stream creates a truncated file that appears to have run successfully. Similarly, writing to a full disk often creates a zero-byte file without an error exit code.

Failure mode 02

Crontab lost in server rebuild

The most common: a new server, Docker image rebuild, or EC2 AMI bake-and-replace that didn't include the crontab. The backup job runs zero times per day, the log shows nothing, and last week's backup is now "current."

Failure mode 03

Destination unreachable

S3 bucket deleted. NFS mount stale. SSH credentials rotated. The backup tool connects, gets an auth error, writes the error to stderr, exits 0 anyway, and the backup directory stays frozen at whatever it was last week.

The fix: ping + size check, under a minute to add

Two lines catch both "backup didn't run" and "backup ran but wrote nothing."

Create a heartbeat monitor in LastPing

In the console, create a monitor with the backup job's schedule (e.g., "0 3 * * *", timezone UTC) and a grace window (30–60 minutes for a nightly backup is typical). Copy the ping URL.

https://ping.lastping.dev/<your-monitor-id>

Basic: ping on success only

The && means the ping fires only if the backup exits 0. A crontab that's gone, a job that errors, or a machine that's down produce silence, and silence opens an incident.

# crontab -e 0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://ping.lastping.dev/<your-monitor-id>

Better: validate the output file size too

This catches the dangerous zero-byte case where mysqldump or another tool exits 0 but wrote nothing. The script dumps, checks the file is larger than a minimum size, then pings on success or pings /fail on either failure.

#!/bin/sh # /usr/local/bin/backup-with-check.sh DEST=/backup/db-$(date +%Y%m%d).sql.gz PING_URL="https://ping.lastping.dev/<your-monitor-id>" MIN_BYTES=10240 # 10 KB — adjust to your expected minimum curl -fsS -m 10 --retry 3 -o /dev/null "$PING_URL/start" # Run the backup mysqldump -u backup -p"$DB_PASS" --all-databases | gzip > "$DEST" # Validate: file must exist and exceed minimum size if [ -f "$DEST" ] && [ "$(stat -c%s "$DEST" 2>/dev/null || stat -f%z "$DEST")" -ge "$MIN_BYTES" ]; then curl -fsS -m 10 --retry 3 -o /dev/null "$PING_URL" else curl -fsS -m 10 --retry 3 -o /dev/null "$PING_URL/fail" exit 1 fi
# crontab -e 0 3 * * * /usr/local/bin/backup-with-check.sh

For pg_dump or rsync backups

The same wrapper works for any backup tool. Replace the mysqldump line with your command:

# PostgreSQL pg_dump -Fc mydb > "$DEST" # rsync to S3 via s3cmd or rclone rclone sync /data/uploads s3:my-backup-bucket/uploads --stats-one-line # borg backup borg create --stats /backup/borg::daily-{now:%Y-%m-%d} /data

What this catches

Dead-man's-switch detection + size validation covers every known backup silent failure mode.

  • Crontab lost — the backup never ran; no ping arrives in the window; incident opens.
  • Backup tool exits non-zero — /fail ping fires immediately; incident opens.
  • Zero-byte output (exit 0) — size check fails; /fail ping fires; incident opens.
  • Hung backup — /start arrived but no completion in grace window → incident opens even while the job is stuck.
  • Machine down — no signal of any kind; absence triggers the incident.
  • Recovery — next successful backup closes the incident automatically. No manual reset.

Running backups in Kubernetes? See Kubernetes CronJobs. Running in GitHub Actions? See GitHub Actions monitoring — especially the 60-day silent disable issue. For the broader concept, see silent failure explained and dead-man's switch explained.

Questions people ask

Front-loaded answers — the most important fact first.

  • Why do backup scripts fail silently?

    Several reasons: many backup tools exit 0 even after a partial failure; output goes to an unread mail spool; the crontab was deleted in a server rebuild; the destination was unreachable but the write error went to stderr only. The fix is dead-man's-switch monitoring with a size validation step.

  • Does mysqldump exit non-zero on failure?

    Not reliably. mysqldump exits 0 if it can connect and start dumping, even if the connection drops mid-stream and the resulting file is truncated and unrestorable. Always validate the output file size after dumping.

  • My backup log shows success but the file is empty. Why?

    Most likely causes: (1) destination disk or mount is full — writes create zero-byte files on some filesystems without error; (2) S3 or remote credentials changed — the upload command exits 0 but the PUT failed with an auth error that went to stderr; (3) redirection to a file on a full disk creates a zero-byte append. Always check file size, not just the exit code.

  • How do I catch a hung backup job?

    Send a /start ping at the beginning and a success ping at the end. If the completion doesn't arrive within the grace window, LastPing opens an incident even though the job process is technically still running. See the full hung job detection guide.

  • Is this free?

    Yes. LastPing is free and fully hosted with no monitor cap and no paid tier. See how it compares to Healthchecks.io and Cronitor.

FREE · FULLY HOSTED · NO PAID TIER

Your backup is only as good as the last verified restore.

One wrapper script and one heartbeat monitor. Know the moment a backup fails — not when you need the restore and find it missing.

Monitor your backup — free Full cron monitoring guide