Monitor Prefect flows: alerts for failed and missed flow runs
Prefect gives you rich state tracking inside the UI, but its worst failure is invisible to anything running inside a flow: if the worker or work pool is down, scheduled flow runs never enter a Running state, so an in-flow notification never fires. Add a dead-man's-switch with two state hooks and LastPing alerts you when a run fails and when a run that should have happened never did.
Your assistant can do this for you
Monitor my Prefect flows with LastPing
Setup: two state hooks on your flow
Ping on completion, ping /fail on failure. A matching monitor schedule catches missed runs.
Create a heartbeat monitor in LastPing
In the console, create a monitor with the same schedule as your Prefect deployment. Set the grace window to your flow's expected runtime plus a margin. Copy the ping URL.
https://ping.lastping.dev/<your-monitor-id>Add on_completion / on_failure hooks
State hooks receive (flow, flow_run, state) and run in the flow process. Ping on success, ping /fail on failure:
from prefect import flow
import httpx
PING = "https://ping.lastping.dev/<your-monitor-id>"
def ping_success(flow, flow_run, state):
httpx.get(PING, timeout=10)
def ping_failure(flow, flow_run, state):
httpx.post(f"{PING}/fail", content=str(state.message)[:10_000], timeout=10)
@flow(on_completion=[ping_success], on_failure=[ping_failure], on_crashed=[ping_failure])
def nightly_etl():
... # your flow logicon_failure runs once retries are exhausted and the run enters a Failed state. A run stopped by its infrastructure (out of memory, an evicted pod) enters Crashed instead, so the same hook is also registered as on_crashed. The state's message is POSTed to /fail (up to 10,000 bytes are stored with the ping). Prefect's docs note that hooks run client side and their execution cannot be guaranteed; the monitor's own schedule in the next step is the backstop.
Catch the "never started" case with the schedule
Because the LastPing monitor has its own schedule, a run that never starts (worker down, deployment paused, work pool empty) simply produces no ping, and the incident opens on its own. No code required for that path; it's the whole point of the dead-man's-switch.
Signal long flows with a start ping
For long ETL flows, ping PING + "/start" from an on_running hook, which Prefect runs when the flow run enters a Running state, and set a tight grace window to detect a hung run before it blocks the next schedule.
def ping_start(flow, flow_run, state):
httpx.get(f"{PING}/start", timeout=10)
@flow(on_running=[ping_start], on_completion=[ping_success],
on_failure=[ping_failure], on_crashed=[ping_failure])
def nightly_etl():
...What this catches
- Flow run fails: the on_failure hook pings
/fail; you're paged immediately. - Worker / work pool down: scheduled runs never start, no ping arrives → incident opens.
- Flow run crashes: out of memory or an evicted pod ends in Crashed; the on_crashed hook pings
/failwhen it can, and the missing success ping covers it when it cannot. - Deployment schedule paused or cleared: Prefect creates no runs, so none go Late and nothing fails; caught by the missing check-in.
- Prefect server or Cloud unreachable: automations cannot fire, but LastPing runs outside Prefect and still notices the missing check-in.
- Recovery: the next successful flow run closes the incident automatically.
Using Airflow instead? See Airflow DAGs. Dagster? See Dagster schedules. Celery? See Celery beat. Hatchet, Inngest or Trigger.dev? See Hatchet, Inngest and Trigger.dev. For the base pattern, see Monitor Python scripts. Comparing orchestrators? See Monitoring failed workflows across orchestrators.
Questions people ask
Front-loaded answers: the most important fact first.
-
Can't Prefect already send failure notifications?
Yes, for runs that start. Automations, in Prefect Cloud and in a Prefect server you run, can notify when a flow run enters a Failed or Crashed state, and a proactive trigger can fire when an expected event does not happen, such as a flow run staying Running for longer than it should. The server also marks a scheduled run Late when no worker picks it up. But the failure mode that hurts most (a worker or work pool that's down, so scheduled runs never enter a Running state at all) can't be reported by anything running inside a flow or by an in-process notification, because no flow runs. Automations run on the Prefect server, so they cannot report that server being down, and a paused or cleared deployment schedule creates no runs, so nothing goes Late. A dead-man's-switch expects a check-in on every scheduled run and opens an incident when one is missing.
-
How do I add a dead-man's-switch to a Prefect flow?
Add
on_completionandon_failurestate-change hooks to the@flowdecorator. Inon_completion, GEThttps://ping.lastping.dev/<id>; inon_failure, GET the/failpath. Create a LastPing monitor with the deployment's schedule so a run that never starts also alerts. -
Does this work with self-hosted Prefect (open source)?
Yes. State hooks are part of open-source Prefect and execute wherever the flow runs, so this works identically on self-hosted Prefect Server and on Prefect Cloud.
-
Should I ping from a hook or from inside the flow?
Prefer hooks.
on_completionfires only when the flow reaches a Completed state, so it won't ping success on a flow that failed partway. Pinging at the end of the flow body would be skipped by an exception, which is fine for the fail path, but hooks keep success and failure cleanly separated. -
Is this free?
Yes. LastPing is free for individuals with no monitor cap. See vs Healthchecks.io and vs Cronitor.
A down worker means silent flows. Catch it.
Two state hooks and a matching schedule. First alert in under a minute, and LastPing monitors itself the same way.