Monitor backups: start → work → success or fail
The classic disaster isn't losing data — it's discovering, mid-restore, that the backups stopped three months ago. Three pings fix that: signal the start, do the work, report the outcome — with the error log attached when it fails. Free, no monitor cap.
Backups are where silent failures go to compound
Every failure mode is invisible until the one day none of them are allowed to be.
Failing quietly for months
An expired credential, a renamed bucket, a full disk — the script errors nightly into a log nobody tails. The gap in your recovery point grows one silent night at a time.
"It ran" is not "it worked"
Pipelines swallow exit codes: pg_dump fails,
gzip shrugs, the upload succeeds — and an empty
archive lands in the bucket looking exactly like a backup.
The hung backup
Stuck on a lock, a dead network mount, a throttled API — the process never exits, never errors, and blocks tomorrow's run too. Two nights lost, zero notifications.
The pattern: three pings around the work
The example dumps Postgres to S3 — swap in restic, borg,
mysqldump, anything with an exit code.
Create a heartbeat monitor
In the console, create a monitor on the backup's schedule. Size the grace window to a generous normal runtime — it doubles as the hung-backup alarm via the start ping below.
Wrap the backup in the three signals
set -o pipefail is what makes "it ran" mean "it
worked": if pg_dump fails mid-pipe, the
if takes the failure branch even though the upload
succeeded.
What each signal does
/start— arms overrun detection: no matching success within the grace window opens an incident, even while the process is still "running".- plain URL — success; with the same
?rid=as the start, the pair records how long the run took. /fail— opens an incident immediately; the POSTed body (up to 10,000 bytes) is stored with the ping event and retrievable via the API, so the failure carries its own evidence.
Monitor the restore too
A backup you've never restored is a hope, not a backup. Give a periodic restore test its own monitor — same pattern, weekly schedule — so "restores work" is a signal, not an assumption.
What this catches
Detection runs on durable deadlines in the database — a crash, restart, or deploy never misses a due monitor.
- The backup failed — the
/failping opens an incident immediately, with the log tail attached to the ping. - The backup hung — the
/startping with no completion inside grace opens an incident while the process is still wedged. - The backup never started — cron gone, machine down, schedule lost. No signal in the expected window → incident.
- Run duration drift — the
rid-paired start/success records each run's duration, so a backup quietly growing from minutes to hours is visible before it hits the grace ceiling. - Recovery — the next successful run closes the incident and sends a recovery notice automatically.
Where does the script run? Wire the schedule with cron, systemd timers, or a Kubernetes CronJob — the pings are identical. Browse all guides.
Questions people ask
Front-loaded answers — the most important fact first.
-
How do I catch a backup that hangs?
Send a
/startping before the work begins. That arms overrun detection: if the matching success doesn't arrive within the grace window, LastPing opens an incident even though the process is still technically running. -
Can I attach the error output to the alert?
Yes. Any ping accepts a POST body of up to 10,000 bytes, stored with the ping event and retrievable via the API. POST the tail of your backup log to
/failso the failure carries its own evidence. -
Does this work with restic, borg, or mysqldump?
Yes — the pattern is tool-agnostic. Anything that runs as a command with an exit code can be wrapped: check the exit status, ping the monitor URL on success, ping
/failon failure. The/<exit-code>endpoint even does the mapping for you:0is success, anything else is a failure. -
Why not just check the bucket for new files?
A file appearing proves an upload happened — not that the dump was complete, non-empty, or restorable, and it needs its own scheduled checker (which itself can fail silently). Exit-code wrapping with
pipefailtests the actual work, and the dead-man's-switch tests the schedule — with no second system to babysit. -
Is this free?
Yes. LastPing is free and fully hosted with no monitor cap and no paid tier. See how it compares to Healthchecks.io and Cronitor.
You'll need exactly one backup. You won't know which.
Three pings tonight beat an unrecoverable morning later. First alert wired up in minutes — and LastPing monitors itself the same way.