How to monitor cron jobs with heartbeats (a ping URL, one curl line and a grace period)

A job that stops does not fail, it goes quiet

An uptime monitor works because the thing it watches has an address: it connects, and either something answers or nothing does. A cron job has no address. It runs at 02:00, writes a backup, and exits, and the only trace is a file that is a day newer than yesterday's. When it stops running there is no error anywhere. The crontab is intact, the script is intact, the server is up. The daemon was not restarted after an upgrade, the job's user was removed, the disk filled, the job hangs on a lock, the machine was rebuilt and the crontab was not: nothing raises an alarm, because from every side but one, nothing happened.

The side that notices is time. The job used to check in; now it does not. A heartbeat monitor watches exactly that: it does nothing until the job calls, and treats silence past a deadline as the failure. In the words of hostnag's own feature page, "a job that goes quiet becomes an alert, not a surprise weeks later."

The ping URL

Add a heartbeat monitor and hostnag mints a URL for it, of the form https://hostnag.com/api/ping/<token>. The token is a long random string and it is the whole credential: there is no account, key or header to send, and anyone who has the URL can ping, so the monitor page has a control to rotate it if it leaks (the old URL stops working at once). A GET and a POST both count, and the answer is an empty success with nothing to parse. Nothing is installed anywhere: the job's host needs curl, or anything else that can fetch a URL.

One line at the end of the job

Put the call on the job's last line, joined with &&, so it runs only when the job succeeded. A job that fails halfway then goes silent, which is the signal you want; a ping at the top of the script would report a backup that never wrote a byte.

0 2 * * * /usr/local/bin/backup.sh && curl -fsS https://hostnag.com/api/ping/<token> >/dev/null

The three flags matter. -f makes curl exit non-zero on an HTTP error instead of printing the error page and succeeding; -s silences the progress meter, which cron would otherwise mail you every night; -S puts errors back on stderr, so a ping that could not get out still shows in the mail cron sends. The monitor page prints the same line for you, with a timeout and retries added, and a copy button next to it.

Expected every, and grace

Two numbers describe the schedule. Expected every is how often the job should call: the cron interval, in minutes. Grace is how late a call may be before it counts as missed. hostnag judges the monitor by the time since the last ping: within expected plus grace it is up; past that it is down, and the alert says how long overdue the ping is, counted from when it was expected. A monitor that has never been pinged is given one full window from the moment it was created before it is judged at all. The defaults are 60 minutes with 10 minutes of grace, which suits an hourly job.

Set grace to cover everything that makes two pings further apart than the schedule says: how long the job itself runs, and how much that varies. A nightly backup that takes twenty minutes one night and fifty the next pings thirty minutes later on the second night than on the first. With a grace shorter than that, the slow night is a false alarm. The same goes for a scheduler that starts jobs a few minutes late under load. Grace is not a delay on the alert; it is the tolerance the job actually needs, and the feature page's step says it plainly: "an expected interval and a grace period before a missed ping counts as overdue."

The judgement itself runs on the monitor's own interval, as often as every 1 minute on Pro and Business and every 5 minutes on the Free plan, so at the shortest setting a missed deadline is noticed within that many minutes of passing. There is no second location to confirm from, because there is nothing to connect to: the grace period is the confirmation, and one miss past it is down at once.

What missed and back look like

The alert for a missed job names the monitor and says it missed its heartbeat, with how long overdue it is, the name it pings as, and when it went missing. In the feature page's words: "No ping past the deadline raises down, and hostnag says exactly how many minutes overdue it is." While the job stays silent, the group's policy sends reminders on its down-reminder interval, each with the running duration.

When the job runs again, the ping itself pulls the next judgement forward, so the recovery lands within a minute rather than at the next scheduled look. The recovery notice says the job is pinging again and how long it was silent. The pair of alerts is the whole story: quiet at 02:10, back at 09:40 after someone noticed the disk and cleared it.

Windows tasks and the Beacon

On Windows there is no cron. A scheduled task that runs curl works just as well, but there is a shorter route. The Beacon is a plain script, served at https://hostnag.com/beacon.ps1 for Windows and https://hostnag.com/beacon.sh for Linux and macOS, that reads a config file and calls the right URL for every line in it. The line for a heartbeat is the word ping and the token:

ping <token>

Open the Beacon dialog on the group page and it writes the config for every heartbeat, link inside ping, status report and certificate report in the group, one line each with the monitor's name as a comment above it, and prints the download line, where the config goes, a verify line that runs the script by hand with -Verbose (--verbose on Linux), and the schtasks or crontab line that schedules it at the shortest interval any of those monitors expects. The dialog's "Heard from" table shows the moment each one is heard. The script is unsigned, readable end to end, never updates itself, and is removed by deleting it and its config. That is the rule hostnag follows for anything that runs inside your systems: a plain script you can read, schedule and delete yourself.

When silence is not enough

Some jobs know they failed and want to say so. A heartbeat cannot carry that: it is a ping or nothing. For those, the same feature offers a status report, a second kind of URL that takes up, down or degraded and a short message, so a job that finished but found something wrong can say what. The feature page's answer draws the line: a status report can say up, down or degraded with its own message, and silence past its cadence is a "could not check", never a down. In the Beacon config it is a status line with a command: exit code zero reports up, anything else reports down, and the last line of output rides along as the message.

Check it now, then watch it

Check it now: a heartbeat needs a URL minted for you, so there is no public tool for this one. Add a heartbeat monitor, run its curl line by hand, and watch the first ping land on the monitor page.

Watch it for good: heartbeat monitoring mints the URL, holds the job to its schedule and grace, and turns silence into an alert with the minutes overdue in it.