How to monitor Apache Airflow jobs

Report success and failure

Create a check and set PING_URL to its private URL. Replace the example command with your actual task. These examples preserve command failure and do not let a failed reporting request turn a successful job into a failure.

#!/bin/sh
# PING_URL is the private URL copied from your check.
: "${PING_URL:?Set PING_URL first}"
rid="$(date +%s)-$$"
report() { curl -fsS -m 10 --retry 3 "$PING_URL/$1?rid=$rid" >/dev/null 2>&1 || :; }
trap 'code=$?; report "$code"; exit "$code"' EXIT
report start
sleep 1
/app/report.sh

Connect it to Apache Airflow

Add this task to an existing DAG, and configure its expected cadence in CronPulse.

from airflow.operators.bash import BashOperator
monitored = BashOperator(
    task_id="monitored_report",
    bash_command="sh /opt/monitored-job.sh",
)

These are integration fragments for an existing project, not a full application. Set the URL in the job’s environment, keep credentials private and ensure the runtime has the dependencies shown.

Things to watch

The shell examples wait one second between start and work to respect the per-check rate guard. For very short tasks, completion-only pings avoid that delay. A killed process may never send a failure ping; the completion deadline catches the missing run instead.

Test the setup

1. Run a successful task and confirm the check turns up.
2. Force a nonzero exit or exception and verify the check turns down.
3. Stop the scheduler after one success and wait for the expected run plus grace.

Set the schedule to your task’s completion cadence, allowing for runtime and queue delay. Use the cron helper to review your calendar schedule.

Questions about Apache Airflow

Platform reference: official Apache Airflow documentation. Adapt fragments to your installed version.