Hung job detection: alert when a job starts but never finishes

Why hung jobs are the hardest failure to catch

Every monitoring tool that fires on errors is blind to a hung job. The job hasn't errored: it's just stuck.

The start/success ping pattern

Two pings per job: one at the start, one at the end. The gap between them is your overrun detection window. Optionally a third (a step ping per stage) narrows that window to the gap between stages.

https://ping.lastping.dev/<your-monitor-id>

#!/bin/sh
# /usr/local/bin/lastping-run.sh — universal wrapper
URL="https://ping.lastping.dev/<your-monitor-id>"

# Arm overrun detection
curl -fsS -m 10 --retry 3 -o /dev/null "$URL/start"

# Run the actual job
"$@"
rc=$?

# Report result: /0 = success, /1 (or any non-zero) = failure
curl -fsS -m 10 --retry 3 -o /dev/null "$URL/$rc"
exit $rc

# crontab -e — use the wrapper for any job
0 3 * * * /usr/local/bin/lastping-run.sh /usr/local/bin/backup.sh

#!/bin/sh
URL="https://ping.lastping.dev/<your-monitor-id>"
MAX_SECONDS=3600   # kill after 1 hour

curl -fsS -m 10 --retry 3 -o /dev/null "$URL/start"

timeout "$MAX_SECONDS" "$@"
rc=$?

# exit code 124 means timeout -> report as failure
curl -fsS -m 10 --retry 3 -o /dev/null "$URL/$rc"
exit $rc

# Python
import urllib.request, sys

PING = "https://ping.lastping.dev/<your-monitor-id>"

def run():
    urllib.request.urlopen(f"{PING}/start", timeout=10)
    try:
        do_work()
        urllib.request.urlopen(PING, timeout=10)
    except Exception:
        urllib.request.urlopen(f"{PING}/fail", timeout=10)
        raise

// Node.js
const PING = 'https://ping.lastping.dev/<your-monitor-id>';

async function run() {
  await fetch(`${PING}/start`);
  try {
    await doWork();
    await fetch(PING);
  } catch (err) {
    await fetch(`${PING}/fail`);
    throw err;
  }
}

RID=$(uuidgen)
URL="https://ping.lastping.dev/<your-monitor-id>"

curl -fsS -m 10 --retry 3 -o /dev/null "$URL/start?rid=$RID"

pg_dump mydb > /backup/db.sql
curl -fsS -m 10 --retry 3 -o /dev/null "$URL/step?rid=$RID&step=dump"

aws s3 cp /backup/db.sql s3://backups/
curl -fsS -m 10 --retry 3 -o /dev/null "$URL/step?rid=$RID&step=upload"

curl -fsS -m 10 --retry 3 -o /dev/null "$URL?rid=$RID"

What the start/success pattern catches

Four signals, four failure modes covered, all with a single monitor.

• ✓Job never started: no /start ping in the expected window → incident opens. Crontab lost, machine down, scheduler misconfigured, all caught.
• ✓Job started but is hung: /start arrived; no success within grace → incident opens while the job is still running. Overrun detection fires.
• ✓Job started but stopped making progress: the run reported /step pings and then went quiet for longer than the monitor's step_timeout_s → a stalled incident opens naming the last step that reported. This is the one that tells you where it wedged, and tells you in minutes rather than at the end of the run budget. Opt-in: leave step_timeout_s unset and the monitor behaves exactly as it always has.
• ✓Job failed: /fail or a non-zero exit code ping closes the run with failure → incident opens immediately.
• ✓Recovery: next successful run closes the incident automatically. No manual reset.
• ✓Flap protection: a transient failure that recovers within the damping window doesn't page you repeatedly.

Running GitHub Actions? The workflow_run webhook delivers started and completed events: no YAML changes needed. See GitHub Actions monitoring. For the conceptual overview of why absence is a better signal than failure, see dead-man's switch explained and silent failure in scheduled jobs. For AI agents that need to report their own heartbeat, the LastPing MCP server returns ready-to-run check-in requests for all four signals via get_ping_instructions; the check-ins themselves are plain HTTP.

Questions people ask

Front-loaded answers: the most important fact first.

• How do I detect a hung job?

  Send a /start ping at the beginning of the job and a success ping at the end. Set the grace window to your maximum expected runtime. If the success ping doesn't arrive within the grace window after the /start, LastPing opens an incident even though the job process is technically still running. This is called overrun detection. For a job with distinct stages you can also report a /step ping as each stage finishes and set step_timeout_s on the monitor: LastPing then opens a stalled incident naming the last step that reported, as soon as progress stops, instead of waiting for the whole run budget to expire. Step reporting is opt-in: a monitor that sets no step_timeout_s behaves exactly as it did before.

• What is the difference between a hung job and a failed job?

  A failed job exits with a non-zero exit code, producing a signal (error log, exception) that exception monitoring can detect. A hung job is still running (its process is alive), so it produces no failure signal. The job has not failed yet; it's just not making progress. Error monitoring can catch failures; only overrun detection can catch hung jobs, because it uses time (the gap between start and expected completion) as the trigger. Without overrun detection, a hung job is invisible to all monitoring until someone notices the next scheduled run couldn't start because the previous one is still holding resources. Overrun detection tells you the job did not finish in time; stall detection (step pings plus step_timeout_s) tells you which stage it stopped in, and tells you sooner.

• What causes jobs to hang?

  Common causes: (1) database lock wait: the job is waiting for a lock held by a long-running transaction; (2) full disk: a write blocks indefinitely on some filesystems; (3) dead NFS or network mount: reads and writes block forever on a stale mount; (4) network timeout too long: the job is waiting for a response that will never come, with a default socket timeout measured in minutes or hours; (5) stdin waiting: the job is reading from stdin and blocking because it's not attached to a terminal.

• Why can't I just set a timeout on the cron job?

  You can and should: timeout 3600 /usr/local/bin/backup.sh is a good safety net that kills a stuck job automatically. But a timeout alone doesn't alert you; it just kills the job and exits with code 124. You still need external monitoring to know the job was killed and why. Combine both: a timeout for automatic cleanup and to prevent resource exhaustion, and a /start+success ping pattern so LastPing opens an incident the moment the job exceeds its expected runtime.

• How do I detect a hung GitHub Actions run?

  Point a workflow_run webhook at LastPing. When a run starts, the webhook delivers a started event; when it completes, it delivers a completed event. If started arrives but completed doesn't arrive within the grace window, LastPing opens an incident. This catches hung runs without any YAML changes. See the GitHub Actions monitoring guide for the full setup.

• Is this free?

  Yes. LastPing is free for individuals with no monitor cap. See how it compares to Healthchecks.io and Cronitor.