The backup that ran but didn't run: detecting backup silent failure

Backup-specific silent failure modes

Each of these produces no error alert. Only an external expectation catches them.

The fix: ping + size check, under a minute to add

Two lines catch both "backup didn't run" and "backup ran but wrote nothing."

https://ping.lastping.dev/<your-monitor-id>

# crontab -e
0 3 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 --retry 3 -o /dev/null https://ping.lastping.dev/<your-monitor-id>

#!/bin/sh
# /usr/local/bin/backup-with-check.sh
DEST=/backup/db-$(date +%Y%m%d).sql.gz
PING_URL="https://ping.lastping.dev/<your-monitor-id>"
MIN_BYTES=10240   # 10 KB — adjust to your expected minimum

curl -fsS -m 10 --retry 3 -o /dev/null "$PING_URL/start"

# Run the backup
mysqldump -u backup -p"$DB_PASS" --all-databases | gzip > "$DEST"

# Validate: file must exist and exceed minimum size
if [ -f "$DEST" ] && [ "$(stat -c%s "$DEST" 2>/dev/null || stat -f%z "$DEST")" -ge "$MIN_BYTES" ]; then
  curl -fsS -m 10 --retry 3 -o /dev/null "$PING_URL"
else
  curl -fsS -m 10 --retry 3 -o /dev/null "$PING_URL/fail"
  exit 1
fi

# crontab -e
0 3 * * * /usr/local/bin/backup-with-check.sh

# PostgreSQL
pg_dump -Fc mydb > "$DEST"

# rsync to S3 via s3cmd or rclone
rclone sync /data/uploads s3:my-backup-bucket/uploads --stats-one-line

# borg backup
borg create --stats /backup/borg::daily-{now:%Y-%m-%d} /data

What this catches

Dead-man's-switch detection + size validation covers every known backup silent failure mode.

• ✓Crontab lost: the backup never ran; no ping arrives in the window; incident opens.
• ✓Backup tool exits non-zero: /fail ping fires immediately; incident opens.
• ✓Zero-byte output (exit 0): size check fails; /fail ping fires; incident opens.
• ✓Hung backup: /start arrived but no completion in grace window → incident opens even while the job is stuck.
• ✓Stalled mid-backup. Optional: report a /step per stage (dump, compress, upload, verify) and set step_timeout_s. A backup that wedges on the upload opens an incident naming that stage minutes in, instead of at the end of the whole window.
• ✓Machine down: no signal of any kind; absence triggers the incident.
• ✓Recovery: next successful backup closes the incident automatically. No manual reset.

Running backups in Kubernetes? See Kubernetes CronJobs. Running in GitHub Actions? See GitHub Actions monitoring, especially the 60-day silent disable issue. For the broader concept, see silent failure explained and dead-man's switch explained.

Questions people ask

Front-loaded answers: the most important fact first.

• Why do backup scripts fail silently?

  Several reasons: (1) Many backup tools exit 0 (success) even after a partial failure: mysqldump exits 0 after writing an incomplete dump if the connection drops mid-stream. (2) Output goes to a local mail spool nobody reads. (3) The crontab was deleted during a server rebuild. (4) The destination (S3 bucket, NFS mount, remote server) is unreachable, but the script doesn't check that the write succeeded and the write error went to stderr only. (5) In public repositories, GitHub Actions scheduled workflows are disabled after 60 days without activity, so if your backup runs in CI, this silently stops it. The fix is dead-man's-switch monitoring with a size validation step.

• How do I know if my backup is actually working?

  Add two checks: (1) Dead-man's-switch monitoring: append a ping URL to the end of your backup command so LastPing expects a signal on your schedule. If no ping arrives, an incident opens. (2) Size validation: after writing the backup, verify it's non-zero: BACKUP_SIZE=$(stat -c%s backup.sql.gz) && [ $BACKUP_SIZE -gt 1024 ] && curl -fsS ping.lastping.dev/<id> || curl -fsS ping.lastping.dev/<id>/fail. This catches the dangerous case where the backup ran, wrote 0 bytes, and exited 0.

• Does mysqldump exit non-zero on failure?

  Not reliably. mysqldump exits 0 if it can connect and start dumping, even if the connection drops mid-stream, and the resulting file may be truncated and unrestorable. Always validate the output file size after dumping, not just the exit code. Similarly, pg_dump --format=custom exits 0 on a complete dump but may write a valid-looking file that is missing tables if the schema changed mid-dump.

• My backup log shows success but the backup file is empty. Why?

  The most common causes: (1) The destination disk or mount is full: writes silently fail or create zero-byte files on some filesystems, without an error. (2) The S3 bucket permissions or remote credentials changed: the upload command exits 0 but the PUT was rejected with an auth error that went to stderr nobody reads. (3) The backup script uses >> redirection to an existing file on a full disk, creating a zero-byte or truncated append. Always check file size, not just the exit code.

• How do I catch a hung backup job?

  Send a /start ping at the beginning and a success ping at the end. If the completion doesn't arrive within the grace window, LastPing opens an incident even though the job process is technically still running. See the full hung job detection guide.

• Is backup monitoring free in LastPing?

  Yes. LastPing is free for individuals with no monitor cap. You can monitor as many backup jobs as you have (daily database dump, weekly full backup, offsite sync), all without paying anything. See how it compares to Healthchecks.io and Cronitor.