Why are you here? Because you just realized your monitoring system has a monitoring system's classic design flaw: it assumes it will be alive to send the alert. Out of the jokes, this is what it took to build a Zabbix → Telegram pipeline that doesn't die quietly, plus the off-site tripwire for the day it does anyway.

We run Zabbix x.X.X in X with MariaDB behind it, watching X hosts and everything else that matters. Dashboards were fine. Restricted users could see what they needed. The part that wasn't fine was the alerting path, and specifically the assumption baked into every monitoring setup by default: the thing that watches everything is also watching itself. It isn't. Nothing watches Zabbix but Zabbix.

The Problem

Two separate gaps, both boring, both the kind of thing that only matters exactly once, at the worst possible time:

• Alert delivery. Email alerts get lost in inboxes, filtered, or just not checked fast enough. We wanted something that lands on a phone and makes noise.
• The watchman problem. If the Zabbix host itself goes down — VM crash, X daemon dies, disk fills — Zabbix cannot alert anyone about its own death. By definition, it's not running to send the alert.

The first one is a features problem. The second one is a design problem, and it's the one people forget until the night it happens.

Building The Telegram Bot

zabbix_telegram_bot.py — a small service that receives Zabbix action triggers and pushes them to Telegram. The obvious version of this is a script with a bot token and a chat ID, running as root, forever. We did not want the obvious version.

The traps

Trap 1: an unscoped API token is a skeleton key. The bot needs to query Zabbix for context (host, severity, problem details) when a trigger fires. Giving it a full-admin API token means that if the bot's environment is ever compromised, so is Zabbix. Scoped the token down to exactly the read permissions the bot needs and nothing else.

Trap 2: one flat Telegram group for everything trains people to ignore it. A single channel that gets every warning, every info-level blip, every "recovered" message becomes noise within a week, and noise gets muted. Built a hardened group model instead — severity and host-group determine which chat a message lands in, so the channel that pages on-call only pages for things that should page on-call.

Trap 3: running the bot as a bare script is one reboot away from silently not running. Wrapped it as a proper systemd unit, sandboxed:

[Service]
DynamicUser=yes
ProtectSystem=strict
ProtectHome=yes
NoNewPrivileges=yes
PrivateTmp=yes

If the bot process dies, systemd restarts it. If the bot is compromised somehow, DynamicUser and ProtectSystem=strict mean there's very little on disk it can actually touch.

Result

Triggers fire, Zabbix calls the bot, the right severity lands in the right Telegram chat, and the token the bot holds can't do anything beyond reading what it needs to read. Good enough for the first problem. Does nothing for the second.

The Watchman Problem

This is the part most monitoring writeups skip, because it's slightly embarrassing to admit: if the box running Zabbix falls over, Zabbix cannot tell you.

The fix has to live somewhere that isn't this infrastructure at all. Deployed Uptime Kuma v2 on a separate Hetzner VPS, configured as a dead-man's-switch:

• Zabbix (or a small cron job alongside it) sends a push heartbeat to Uptime Kuma on a schedule.
• Uptime Kuma expects that heartbeat. If it stops arriving, that's the alert — not "something is wrong," but "the thing that tells you what's wrong has gone silent," which is a strictly scarier and more actionable signal.

*/5 * * * * curl -fsS https://kuma.example.com/api/push/XXXXXXXX?status=up

Simple on purpose. The whole point of a dead-man's-switch is that it must not depend on anything it's supposed to be watching.

Hardening the off-site box

Since this VPS now holds the "alert of last resort," it got the full treatment rather than being another soft target:

None of this is exotic. All of it is the difference between "off-site monitoring box" and "off-site monitoring box that's also an easy target because nobody thought it needed hardening since it's just pinging a heartbeat."

What I Would Tell Past Me

• A monitoring system cannot alert on its own death. Something outside it has to. Build the dead-man's-switch before you need it, not after the first silent outage.
• Scope every token to exactly what the integration needs. A convenience bot with an admin token is a compromise waiting for an excuse.
• One alert channel for everything is the same as no alert channel, eventually. Split by severity before people learn to mute you.
• The box that watches your infrastructure from the outside deserves the same hardening as anything client-facing — it's the one thing that's supposed to still be standing when everything else isn't.

Next failure mode on the list: what happens if Hetzner itself has a bad day and both boxes are affected. That one doesn't have a clean answer yet.