Monitor Hatchet workflows: alerts for failed, stalled and missed runs

Create a heartbeat monitor in LastPing

In the console, create a monitor with the same cron as your Hatchet workflow. Hatchet's cron is the time it enqueues the run, not the time the run starts, so set the grace window to queue time plus the workflow's expected runtime plus a margin. Copy the ping URL.

https://ping.lastping.dev/<your-monitor-id>

Python: on_success_task and on_failure_task

An on-success task runs as the last step of a workflow whose tasks all succeeded; an on-failure task runs as the last step of a workflow with at least one failed task, and ctx.task_run_errors maps each failed task to its error message:

from datetime import timedelta

import httpx
from hatchet_sdk import Context, EmptyModel, Hatchet

hatchet = Hatchet()
PING = "https://ping.lastping.dev/<your-monitor-id>"

nightly = hatchet.workflow(name="nightly-etl", on_crons=["0 3 * * *"])

@nightly.task(execution_timeout=timedelta(minutes=30))
def extract(input: EmptyModel, ctx: Context) -> dict[str, str]:
    ...   # your work
    return {"status": "done"}

@nightly.on_success_task()
def ping_success(input: EmptyModel, ctx: Context) -> None:
    httpx.get(PING, timeout=10)

@nightly.on_failure_task()
def ping_failure(input: EmptyModel, ctx: Context) -> None:
    httpx.post(f"{PING}/fail", content=str(ctx.task_run_errors)[:10_000], timeout=10)

Register nightly on your worker as usual. The POSTed body (up to 10,000 bytes) is stored with the fail ping, so the incident carries the error.

TypeScript: onSuccess and onFailure

The TypeScript SDK has the same pair on a workflow declaration; ctx.errors() returns the upstream errors inside the failure task:

import { hatchet } from './hatchet-client';

const PING = 'https://ping.lastping.dev/<your-monitor-id>';

export const nightly = hatchet.workflow({
  name: 'nightly-etl',
  on: { cron: '0 3 * * *' },
});

nightly.task({
  name: 'extract',
  executionTimeout: '30m',
  fn: async () => {
    // your work
    return { status: 'done' };
  },
});

nightly.onSuccess({
  name: 'ping-success',
  fn: async () => {
    await fetch(PING);
  },
});

nightly.onFailure({
  name: 'ping-failure',
  fn: async (_input, ctx) => {
    await fetch(`${PING}/fail`, {
      method: 'POST',
      body: JSON.stringify(ctx.errors()).slice(0, 10000),
    });
  },
});

The run that never starts needs no code

With no worker running, the cron still enqueues the task, but nothing executes it, and Hatchet cancels it once it has waited longer than its schedule_timeout (five minutes by default). The on-failure task needs a worker too, so it never runs. A workflow paused with cron runs set to drop creates no run at all. In every one of these cases LastPing receives no ping, and the incident opens on its own when the grace window ends.

Bound long tasks with execution_timeout

Hatchet treats a task that exceeds its execution_timeout (60 seconds by default) as failed, so a stuck task reaches the on-failure task and pings /fail. Set the timeout to the longest run you consider healthy. If the worker itself stops responding, the success ping never arrives and the grace window catches it.