How to monitor cron jobs with heartbeats: one curl line and a grace period

A job that stops goes quiet

An uptime monitor works because the thing it watches has an address: it connects, and either something answers or nothing does. A cron job has no address. It runs at 02:00, writes a backup, and exits, and the only trace is a file that is a day newer than yesterday's. When it stops running there is no error anywhere. The crontab is intact, the script is intact, the server is up. The daemon was not restarted after an upgrade, the job's user was removed, the disk filled, the job hangs on a lock, the machine was rebuilt and the crontab was not: nothing raises an alarm, because from every side but one, nothing happened.

The side that notices is time. The job used to check in; now it does not. A heartbeat monitor watches exactly that. It does nothing until the job calls, and treats silence past a deadline as the failure.

The ping URL

Add a heartbeat monitor and hostnag mints a URL for it, of the form https://hostnag.com/api/ping/<token>. The token is a long random string and it is the whole credential: there is no account, key or header to send, and anyone who has the URL can ping, so the monitor page has a control to rotate it if it leaks (the old URL stops working at once). A GET and a POST both count, and the answer is an empty success with nothing to parse. Nothing is installed anywhere: the job's host needs curl, or anything else that can fetch a URL.

One line at the end of the job

Put the call on the job's last line, joined with &&, so it runs only when the job succeeded. A job that fails halfway then goes silent, which is the signal you want; a ping at the top of the script would report a backup that never wrote a byte.

0 2 * * * /usr/local/bin/backup.sh && curl -fsS https://hostnag.com/api/ping/<token> >/dev/null

The three flags matter. -f makes curl exit non-zero on an HTTP error instead of printing the error page and succeeding; -s silences the progress meter, which cron would otherwise mail you every night; -S puts errors back on stderr, so a ping that could not get out still shows in the mail cron sends. The monitor page prints the same line for you, with a timeout and retries added, and a copy button next to it. Run it once by hand to see the first ping land on the monitor page.

Expected every, and grace

Two numbers describe the schedule. Expected every is how often the job should call: the cron interval, in minutes. Grace is how late a call may be before it counts as missed. hostnag judges the monitor by the time since the last ping: within expected plus grace it is up; past that it is down, and the alert says how long overdue the ping is, counted from when it was expected. A monitor that has never been pinged is given one full window from the moment it was created before it is judged at all. The defaults are 60 minutes with 10 minutes of grace, which suits an hourly job.

Set grace to cover everything that makes two pings further apart than the schedule says: how long the job itself runs, and how much that varies. A nightly backup that takes twenty minutes one night and fifty the next pings thirty minutes later on the second night than on the first. With a grace shorter than that, the slow night is a false alarm. The same goes for a scheduler that starts jobs a few minutes late under load. Set it from the job's slowest night.

The judgement itself runs on the monitor's own interval, as often as every 1 minute on Pro and Business and every 5 minutes on the Free plan, so at the shortest setting a missed deadline is noticed within that many minutes of passing. There is no second location to confirm from, because there is nothing to connect to: the grace period is the confirmation, and one miss past it is down at once.

Windows tasks and the Beacon

On Windows there is no cron. A scheduled task that runs curl works just as well, and a box that already has a Beacon offers something else. The Beacon is a plain script, served at https://hostnag.com/beacon.ps1 for Windows and https://hostnag.com/beacon.sh for Linux and macOS, that one line installs on a box in your network and your own scheduler runs every minute. It is told what to check by hostnag and never handed code; a heartbeat is not one of those things (the job pings for itself), but an office link's internal ping, a status report and a certificate report can each pick a beacon under Checked by, and the box then pings, runs the status command or posts the certificate from its own disk with no URL kept anywhere. The script is readable end to end, never updates itself, and uninstall removes all of it. The beacon guide covers the install and what it may reach.

When silence is not enough

Some jobs know they failed and want to say so. A heartbeat cannot carry that, since it is a ping or nothing. For those, the same feature offers a status report, a second kind of URL that takes up, down or degraded and a short message, so a job that finished but found something wrong can say what. The judging differs in one way: silence past a status report's cadence is a "could not check", because a report that did not arrive says nothing about what it would have said. On a box with a Beacon it is a status line in the beacon's config with a command: exit code zero reports up, anything else reports down, and the last line of output rides along as the message.

What missed and back look like

The alert for a missed job names the monitor and says it missed its heartbeat, with how long overdue it is, the name it pings as, and when it went missing. While the job stays silent, the group's policy sends reminders on its down-reminder interval, each with the running duration.

When the job runs again, the ping itself pulls the next judgement forward, so the recovery lands within a minute rather than at the next scheduled look. The recovery notice says the job is pinging again and how long it was silent. Read together, the two alerts are the night's story: quiet at 02:10, back at 09:40 after someone noticed the disk and cleared it.