Monitoring and backups for every host
Part 7 of the series that starts with Four servers, three distributions: why I consolidated.
Every host gets the same two things from Ansible: a monitoring agent and a backup. This post shows both. The monitoring part is mostly about the network: how Prometheus at home reaches the VPS without exposing anything to the internet. The backup part is one script template that works for all hosts.
The monitoring network
Prometheus runs on my home server. The VPS are somewhere on the internet. I did not want node_exporter’s port open to the world, and I did not want the VPS to be able to reach my home network.
The solution is a WireGuard tunnel from each VPS to the home server. I call it wg-mon.
vps1 (10.99.0.13) ──┐
vps2 (10.99.0.12) ──┼── WireGuard ──► home1 wg0 (10.99.0.1) ── Prometheus
mail1 (10.99.0.11) ─┘
Two firewall rules make it safe in both directions:
- On each VPS, the tunnel interface is in a firewalld zone of its own, and only port 9100 (node_exporter) is open there.
- On the home server, the VPS tunnel addresses land in firewalld’s
blockzone. Prometheus can open connections to them, and the replies come back. A VPS cannot open a single connection to the home server.
If somebody takes over a VPS, they get a tunnel that leads nowhere.
My own devices (laptop, phone) use the same wg0 interface as “road warriors”. Their addresses go into the trusted zone, so they reach the LAN.
The VPS side
# /etc/wireguard/wg-mon.conf
[Interface]
Address = {{ tunnel_ip }}/32
PostUp = wg set %i private-key /etc/wireguard/wg-mon.key
[Peer]
PublicKey = {{ wg_mon_server_public_key }}
Endpoint = {{ wg_mon_server_endpoint }}
AllowedIPs = {{ wg_mon_server_ip }}/32
PersistentKeepalive = 25
The private key is in a separate file, written from pass with no_log. The config file can then show in a diff without leaking the key. PersistentKeepalive keeps the NAT mapping at home open, so that Prometheus can always reach the VPS.
- name: Tunnel up
ansible.builtin.systemd_service:
name: wg-quick@wg-mon
enabled: true
state: started
- name: firewalld zone for the tunnel
ansible.posix.firewalld:
zone: monitoring
state: present
permanent: true
register: monitoring_zone
# A new zone exists only after a reload
- name: Load the new zone
ansible.builtin.command: firewall-cmd --reload
when: monitoring_zone.changed and not ansible_check_mode
changed_when: true
- name: Tunnel interface in the zone
ansible.posix.firewalld:
zone: monitoring
interface: wg-mon
permanent: true
immediate: true
state: enabled
- name: node_exporter port in the zone
ansible.posix.firewalld:
zone: monitoring
port: 9100/tcp
permanent: true
immediate: true
state: enabled
The hub side, generated from the inventory
Each host’s public key and tunnel address are in the inventory (wg_mon_public_key, tunnel_ip). The hub’s config loops over all hosts, so a new VPS needs no extra step:
# /etc/wireguard/wg0.conf on home1
[Interface]
PrivateKey = {{ lookup('community.general.passwordstore', wg_hub_key_pass) }}
Address = {{ wg_hub_address }}
ListenPort = {{ wg_hub_port }}
{% for p in wg_hub_peers %}
# {{ p.name }}
[Peer]
PublicKey = {{ p.public_key }}
PresharedKey = {{ lookup('community.general.passwordstore', p.psk_pass) }}
AllowedIPs = {{ p.ip }}/32
{% endfor %}
{% for h in groups['all'] if h != inventory_hostname and hostvars[h].wg_mon_public_key is defined %}
# {{ h }} (monitoring)
[Peer]
PublicKey = {{ hostvars[h].wg_mon_public_key }}
AllowedIPs = {{ hostvars[h].tunnel_ip }}/32
{% endfor %}
This file contains the private and preshared keys, so its task has no_log: true and diff: false. The handler reloads the interface with wg syncconf (systemctl reload wg-quick@wg0). Peers change without dropping the open tunnels.
The firewall on the hub:
- name: Road-warrior peers are trusted
ansible.posix.firewalld:
zone: trusted
source: "{{ item.ip }}/32"
state: enabled
permanent: true
immediate: true
loop: "{{ wg_hub_peers }}"
loop_control:
label: "{{ item.name }}"
- name: All other wg0 traffic is blocked
ansible.posix.firewalld:
zone: block
interface: wg0
state: enabled
permanent: true
immediate: true
firewalld picks a zone by source address first, and by interface second. A packet from a road warrior’s address matches trusted. Everything else on wg0 falls through to block.
A sender on the LAN could fake a road warrior’s address. The base role’s sysctl settings turn on strict reverse path filtering (net.ipv4.conf.all.rp_filter = 1), and the kernel drops such packets.
Sometimes one VPS must reach exactly one service at home. My CI runner needs one port for a later part of this series. For that the role has a list of exceptions, as firewalld rich rules in the block zone:
# host_vars/home1.yml
wg_hub_block_exceptions:
- {source: 10.99.0.11, port: 8443/tcp}
- name: Exceptions in the block zone
ansible.posix.firewalld:
zone: block
rich_rule: >-
rule family="ipv4" source address="{{ item.source }}"
port port="{{ item.port.split('/')[0] }}" protocol="{{ item.port.split('/')[1] }}" accept
state: enabled
permanent: true
immediate: true
loop: "{{ wg_hub_block_exceptions }}"
node_exporter
node_exporter runs as a root quadlet (/etc/containers/systemd/), because it needs to see the whole host. It listens only on the tunnel address:
# /etc/containers/systemd/node-exporter.container
[Unit]
Description=Prometheus node_exporter (monitoring tunnel address only)
Wants=wg-quick@wg-mon.service
After=wg-quick@wg-mon.service
[Container]
ContainerName=node-exporter
Image=quay.io/prometheus/node-exporter:latest
AutoUpdate=registry
Network=host
PodmanArgs=--pid=host
Volume=/:/host:ro,rslave
Volume=/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket:ro
# Confined: its own SELinux type (post 4)
SecurityLabelType=node_exporter.process
Exec=--path.rootfs=/host --web.listen-address={{ tunnel_ip }}:9100 --collector.systemd --collector.textfile.directory=/host/var/lib/node_exporter/textfile
[Service]
Restart=always
[Install]
WantedBy=multi-user.target
The textfile directory is how scripts report to Prometheus. A script writes a .prom file, and node_exporter serves its content. My backup script and a daily update check use it.
Prometheus
On the home server the targets are a short static list:
# prometheus.yml (part)
scrape_configs:
- job_name: node-remote
static_configs:
- targets: [10.99.0.11:9100]
labels: {host: mail1}
- targets: [10.99.0.12:9100]
labels: {host: vps2}
- targets: [10.99.0.13:9100]
labels: {host: vps1}
The tunnel addresses come from the inventory, so you could render this list from hostvars too. With four hosts I kept it static and readable.
Alertmanager sends alerts to a self-hosted ntfy. That covers “a host is down”. It does not cover “the home server, where Prometheus and Alertmanager run, is down”. For that, the next section uses an external dead man’s switch.
Backups: one script for every host
Each host backs up with restic to its own repository on an SFTP backup box. The hosts differ in what they must dump first (databases, Mailcow) and in what they exclude. Everything else is the same. So there is one script template, with the host-specific parts as variables.
The settings
# host_vars/vps1.yml
backup_repository: sftp:backup-box:vps1
backup_password_pass: vps1/restic
backup_time: "03:15"
backup_shell_vars: STORAGE=$APP_HOME/.local/share/containers/storage
backup_args:
- --exclude "$STORAGE/overlay"
- --exclude "$STORAGE/overlay-images"
- /etc
- $APP_HOME
- $DUMP_DIR
The container image layers are excluded. They can be pulled again and are the largest part of the storage. The volumes stay in.
The script
This is the template, shortened a little:
#!/bin/bash
# /usr/local/sbin/backup-{{ backup_name }}.sh, from the backup role
set -uo pipefail
export RESTIC_REPOSITORY={{ backup_repository }}
export RESTIC_PASSWORD_FILE=/root/.config/restic/{{ backup_name }}.pass
# systemd services have no $HOME, so restic needs an explicit cache dir
export RESTIC_CACHE_DIR=/var/cache/restic
HC_PING_URL=$(cat /root/.config/restic/{{ backup_name }}.hc 2>/dev/null || true)
APP_USER={{ app_user }}
APP_HOME={{ app_home }}
{{ backup_shell_vars }}
DUMP_DIR=/var/backups/{{ backup_name }}
failed=()
START=$(date +%s)
hc_ping() {
[ -n "$HC_PING_URL" ] || return 0
curl -fsS -m 10 --retry 3 -o /dev/null --data-binary @<(tail -n 100 "$RUN_LOG") \
"$HC_PING_URL$1" || echo "healthchecks ping failed"
}
# Run a step, note a failure, go on with the next step
step() {
local name=$1 rc
shift
echo "=== $(date '+%F %T') $name"
"$@"
rc=$?
if [ $rc -ne 0 ]; then
echo "!!! $name failed (exit $rc)"
failed+=("$name")
fi
return $rc
}
app_podman() {
runuser -u "$APP_USER" -- env XDG_RUNTIME_DIR=/run/user/$(id -u "$APP_USER") podman "$@"
}
# Podman secrets are in no volume and not in Git: keep them in the encrypted backup
secrets_dump() {
local s
for s in $(app_podman secret ls --format '{{.Name}}'); do
app_podman secret inspect --showsecret --format '{{.SecretData}}' "$s" > "$DUMP_DIR/secrets/$s" || return 1
done
}
exec 9>/run/backup-{{ backup_name }}.lock
if ! flock -n 9; then
echo "previous backup run still in progress"
hc_ping /fail
exit 1
fi
hc_ping /start
umask 077
mkdir -p "$DUMP_DIR/secrets"
{{ backup_pre_steps }}
step "podman secrets" secrets_dump
step "restic unlock" restic unlock
if step "restic backup" restic backup --verbose=0 --exclude-caches {{ backup_args | join(' ') }}; then
step "restic forget/prune" restic forget --prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6
fi
step "restic check" restic check
write_metrics $(( ${#failed[@]} == 0 ))
if [ ${#failed[@]} -gt 0 ]; then
hc_ping /fail
exit 1
fi
hc_ping ""
The Go template syntax in secrets_dump ({{.Name}}) clashes with Jinja. The real template therefore starts with a header that changes Jinja’s comment markers and wraps those lines in {% raw %}. I left that out here to keep the script readable.
Some decisions in it:
- The script does not stop at the first error. A failed database dump should not prevent the file backup. Each failed step is collected, and the end result is “failed” if any step failed.
flockprevents two runs at the same time, for example when a run takes longer than the interval.restic unlockremoves stale locks from a run that was killed.restic checkat the end verifies the repository structure. It does not read all data, which would be slow over SFTP.restic check --read-data-subset=5%reads a sample, if you want more.
Host-specific steps
A host adds dump functions and steps as shell code in its variables:
# host_vars/mail1.yml (part)
backup_functions: |
forgejo_dump() {
app_podman exec forgejo-postgres pg_dumpall -U forgejo | gzip > "$DUMP_DIR/db/forgejo.sql.gz"
}
backup_pre_steps: |
step "forgejo db" forgejo_dump
healthchecks.io: the dead man’s switch
The script pings healthchecks.io at the start, at the end, and with /fail on errors. The service alerts me when a ping is late. So I also hear about a backup that never started, because the host is down or the timer broke. Monitoring on the same host could never report that. The last 100 lines of the run’s log go with the ping, so the alert e-mail shows what failed.
Metrics for Prometheus
At the end the script writes /var/lib/node_exporter/textfile/backup.prom:
backup_last_run_timestamp_seconds 1761246000
backup_last_success_timestamp_seconds 1761246000
backup_last_run_success 1
backup_last_duration_seconds 312
It writes to a temporary file first and then renames it. node_exporter never reads a half-written file. A Grafana overview shows the age of the last good backup per host. A failed run also makes the systemd unit fail, and an alert rule for failed units catches that.
The timer
# /etc/systemd/system/backup-vps1.timer
[Unit]
Description=Daily backup of vps1
[Timer]
OnCalendar=*-*-* 03:15
RandomizedDelaySec=15min
Persistent=true
[Install]
WantedBy=timers.target
Persistent=true runs a missed backup after the host was off at 03:15.
Restore, at least once
A backup that was never restored is a hope. I restored each host completely during the rehearsals in the next post. The restore from the old Arch host reported errors, because restic tried to set SELinux labels that the backup never had (see post 4). The files were fine.
R="restic -r sftp:backup-box:vps1 --password-file /root/.config/restic/vps1.pass --no-lock"
$R snapshots
$R restore latest --target /var/tmp/restore --include /home/svc/traefik/acme --exclude-xattr security.selinux
--no-lock lets a test VM read the live repository without writing a lock into it.
Check
sudo systemctl start backup-vps1.service
sudo tail -20 /var/log/backup-vps1.log
curl -s http://10.99.0.13:9100/metrics | grep ^backup_ # from the hub
Next: Rehearse before you reinstall.
Further reading
- WireGuard quick start: keys, peers and
wg-quick. - firewalld documentation: zones and policies, for the hub’s
blockzone. - Prometheus documentation: scrape configs, rules and alerts.
- node_exporter: the collectors and the textfile directory.
- blackbox_exporter: HTTP, TCP and ICMP probes.
- restic documentation: repositories, retention with
forget --prune, checks and restore. - healthchecks.io documentation: pings, start and fail signals, and grace times.