Linux Lunatix

Monitoring and backups for every host

Stefan Marx
| 9 minutes

Part 7 of the series that starts with Four servers, three distributions: why I consolidated.

Every host gets the same two things from Ansible: a monitoring agent and a backup. This post shows both. The monitoring part is mostly about the network: how Prometheus at home reaches the VPS without exposing anything to the internet. The backup part is one script template that works for all hosts.

The monitoring network

Prometheus runs on my home server. The VPS are somewhere on the internet. I did not want node_exporter’s port open to the world, and I did not want the VPS to be able to reach my home network.

The solution is a WireGuard tunnel from each VPS to the home server. I call it wg-mon.

vps1 (10.99.0.13) ──┐
vps2 (10.99.0.12) ──┼── WireGuard ──► home1 wg0 (10.99.0.1) ── Prometheus
mail1 (10.99.0.11) ─┘

Two firewall rules make it safe in both directions:

  1. On each VPS, the tunnel interface is in a firewalld zone of its own, and only port 9100 (node_exporter) is open there.
  2. On the home server, the VPS tunnel addresses land in firewalld’s block zone. Prometheus can open connections to them, and the replies come back. A VPS cannot open a single connection to the home server.

If somebody takes over a VPS, they get a tunnel that leads nowhere.

My own devices (laptop, phone) use the same wg0 interface as “road warriors”. Their addresses go into the trusted zone, so they reach the LAN.

The VPS side

# /etc/wireguard/wg-mon.conf
[Interface]
Address = {{ tunnel_ip }}/32
PostUp = wg set %i private-key /etc/wireguard/wg-mon.key

[Peer]
PublicKey = {{ wg_mon_server_public_key }}
Endpoint = {{ wg_mon_server_endpoint }}
AllowedIPs = {{ wg_mon_server_ip }}/32
PersistentKeepalive = 25

The private key is in a separate file, written from pass with no_log. The config file can then show in a diff without leaking the key. PersistentKeepalive keeps the NAT mapping at home open, so that Prometheus can always reach the VPS.

- name: Tunnel up
  ansible.builtin.systemd_service:
    name: wg-quick@wg-mon
    enabled: true
    state: started

- name: firewalld zone for the tunnel
  ansible.posix.firewalld:
    zone: monitoring
    state: present
    permanent: true
  register: monitoring_zone

# A new zone exists only after a reload
- name: Load the new zone
  ansible.builtin.command: firewall-cmd --reload
  when: monitoring_zone.changed and not ansible_check_mode
  changed_when: true

- name: Tunnel interface in the zone
  ansible.posix.firewalld:
    zone: monitoring
    interface: wg-mon
    permanent: true
    immediate: true
    state: enabled

- name: node_exporter port in the zone
  ansible.posix.firewalld:
    zone: monitoring
    port: 9100/tcp
    permanent: true
    immediate: true
    state: enabled

The hub side, generated from the inventory

Each host’s public key and tunnel address are in the inventory (wg_mon_public_key, tunnel_ip). The hub’s config loops over all hosts, so a new VPS needs no extra step:

# /etc/wireguard/wg0.conf on home1
[Interface]
PrivateKey = {{ lookup('community.general.passwordstore', wg_hub_key_pass) }}
Address = {{ wg_hub_address }}
ListenPort = {{ wg_hub_port }}
{% for p in wg_hub_peers %}

# {{ p.name }}
[Peer]
PublicKey = {{ p.public_key }}
PresharedKey = {{ lookup('community.general.passwordstore', p.psk_pass) }}
AllowedIPs = {{ p.ip }}/32
{% endfor %}
{% for h in groups['all'] if h != inventory_hostname and hostvars[h].wg_mon_public_key is defined %}

# {{ h }} (monitoring)
[Peer]
PublicKey = {{ hostvars[h].wg_mon_public_key }}
AllowedIPs = {{ hostvars[h].tunnel_ip }}/32
{% endfor %}

This file contains the private and preshared keys, so its task has no_log: true and diff: false. The handler reloads the interface with wg syncconf (systemctl reload wg-quick@wg0). Peers change without dropping the open tunnels.

The firewall on the hub:

- name: Road-warrior peers are trusted
  ansible.posix.firewalld:
    zone: trusted
    source: "{{ item.ip }}/32"
    state: enabled
    permanent: true
    immediate: true
  loop: "{{ wg_hub_peers }}"
  loop_control:
    label: "{{ item.name }}"

- name: All other wg0 traffic is blocked
  ansible.posix.firewalld:
    zone: block
    interface: wg0
    state: enabled
    permanent: true
    immediate: true

firewalld picks a zone by source address first, and by interface second. A packet from a road warrior’s address matches trusted. Everything else on wg0 falls through to block.

A sender on the LAN could fake a road warrior’s address. The base role’s sysctl settings turn on strict reverse path filtering (net.ipv4.conf.all.rp_filter = 1), and the kernel drops such packets.

Sometimes one VPS must reach exactly one service at home. My CI runner needs one port for a later part of this series. For that the role has a list of exceptions, as firewalld rich rules in the block zone:

# host_vars/home1.yml
wg_hub_block_exceptions:
  - {source: 10.99.0.11, port: 8443/tcp}
- name: Exceptions in the block zone
  ansible.posix.firewalld:
    zone: block
    rich_rule: >-
      rule family="ipv4" source address="{{ item.source }}"
      port port="{{ item.port.split('/')[0] }}" protocol="{{ item.port.split('/')[1] }}" accept
    state: enabled
    permanent: true
    immediate: true
  loop: "{{ wg_hub_block_exceptions }}"

node_exporter

node_exporter runs as a root quadlet (/etc/containers/systemd/), because it needs to see the whole host. It listens only on the tunnel address:

# /etc/containers/systemd/node-exporter.container
[Unit]
Description=Prometheus node_exporter (monitoring tunnel address only)
Wants=wg-quick@wg-mon.service
After=wg-quick@wg-mon.service

[Container]
ContainerName=node-exporter
Image=quay.io/prometheus/node-exporter:latest
AutoUpdate=registry
Network=host
PodmanArgs=--pid=host
Volume=/:/host:ro,rslave
Volume=/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket:ro
# Confined: its own SELinux type (post 4)
SecurityLabelType=node_exporter.process
Exec=--path.rootfs=/host --web.listen-address={{ tunnel_ip }}:9100 --collector.systemd --collector.textfile.directory=/host/var/lib/node_exporter/textfile

[Service]
Restart=always

[Install]
WantedBy=multi-user.target

The textfile directory is how scripts report to Prometheus. A script writes a .prom file, and node_exporter serves its content. My backup script and a daily update check use it.

Prometheus

On the home server the targets are a short static list:

# prometheus.yml (part)
scrape_configs:
  - job_name: node-remote
    static_configs:
      - targets: [10.99.0.11:9100]
        labels: {host: mail1}
      - targets: [10.99.0.12:9100]
        labels: {host: vps2}
      - targets: [10.99.0.13:9100]
        labels: {host: vps1}

The tunnel addresses come from the inventory, so you could render this list from hostvars too. With four hosts I kept it static and readable.

Alertmanager sends alerts to a self-hosted ntfy. That covers “a host is down”. It does not cover “the home server, where Prometheus and Alertmanager run, is down”. For that, the next section uses an external dead man’s switch.

Backups: one script for every host

Each host backs up with restic to its own repository on an SFTP backup box. The hosts differ in what they must dump first (databases, Mailcow) and in what they exclude. Everything else is the same. So there is one script template, with the host-specific parts as variables.

The settings

# host_vars/vps1.yml
backup_repository: sftp:backup-box:vps1
backup_password_pass: vps1/restic
backup_time: "03:15"
backup_shell_vars: STORAGE=$APP_HOME/.local/share/containers/storage
backup_args:
  - --exclude "$STORAGE/overlay"
  - --exclude "$STORAGE/overlay-images"
  - /etc
  - $APP_HOME
  - $DUMP_DIR

The container image layers are excluded. They can be pulled again and are the largest part of the storage. The volumes stay in.

The script

This is the template, shortened a little:

#!/bin/bash
# /usr/local/sbin/backup-{{ backup_name }}.sh, from the backup role
set -uo pipefail

export RESTIC_REPOSITORY={{ backup_repository }}
export RESTIC_PASSWORD_FILE=/root/.config/restic/{{ backup_name }}.pass
# systemd services have no $HOME, so restic needs an explicit cache dir
export RESTIC_CACHE_DIR=/var/cache/restic
HC_PING_URL=$(cat /root/.config/restic/{{ backup_name }}.hc 2>/dev/null || true)

APP_USER={{ app_user }}
APP_HOME={{ app_home }}
{{ backup_shell_vars }}
DUMP_DIR=/var/backups/{{ backup_name }}

failed=()
START=$(date +%s)

hc_ping() {
    [ -n "$HC_PING_URL" ] || return 0
    curl -fsS -m 10 --retry 3 -o /dev/null --data-binary @<(tail -n 100 "$RUN_LOG") \
        "$HC_PING_URL$1" || echo "healthchecks ping failed"
}

# Run a step, note a failure, go on with the next step
step() {
    local name=$1 rc
    shift
    echo "=== $(date '+%F %T') $name"
    "$@"
    rc=$?
    if [ $rc -ne 0 ]; then
        echo "!!! $name failed (exit $rc)"
        failed+=("$name")
    fi
    return $rc
}

app_podman() {
    runuser -u "$APP_USER" -- env XDG_RUNTIME_DIR=/run/user/$(id -u "$APP_USER") podman "$@"
}

# Podman secrets are in no volume and not in Git: keep them in the encrypted backup
secrets_dump() {
    local s
    for s in $(app_podman secret ls --format '{{.Name}}'); do
        app_podman secret inspect --showsecret --format '{{.SecretData}}' "$s" > "$DUMP_DIR/secrets/$s" || return 1
    done
}

exec 9>/run/backup-{{ backup_name }}.lock
if ! flock -n 9; then
    echo "previous backup run still in progress"
    hc_ping /fail
    exit 1
fi

hc_ping /start
umask 077
mkdir -p "$DUMP_DIR/secrets"

{{ backup_pre_steps }}
step "podman secrets" secrets_dump
step "restic unlock" restic unlock

if step "restic backup" restic backup --verbose=0 --exclude-caches {{ backup_args | join(' ') }}; then
    step "restic forget/prune" restic forget --prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6
fi
step "restic check" restic check

write_metrics $(( ${#failed[@]} == 0 ))
if [ ${#failed[@]} -gt 0 ]; then
    hc_ping /fail
    exit 1
fi
hc_ping ""

The Go template syntax in secrets_dump ({{.Name}}) clashes with Jinja. The real template therefore starts with a header that changes Jinja’s comment markers and wraps those lines in {% raw %}. I left that out here to keep the script readable.

Some decisions in it:

Host-specific steps

A host adds dump functions and steps as shell code in its variables:

# host_vars/mail1.yml (part)
backup_functions: |
  forgejo_dump() {
      app_podman exec forgejo-postgres pg_dumpall -U forgejo | gzip > "$DUMP_DIR/db/forgejo.sql.gz"
  }
backup_pre_steps: |
  step "forgejo db" forgejo_dump

healthchecks.io: the dead man’s switch

The script pings healthchecks.io at the start, at the end, and with /fail on errors. The service alerts me when a ping is late. So I also hear about a backup that never started, because the host is down or the timer broke. Monitoring on the same host could never report that. The last 100 lines of the run’s log go with the ping, so the alert e-mail shows what failed.

Metrics for Prometheus

At the end the script writes /var/lib/node_exporter/textfile/backup.prom:

backup_last_run_timestamp_seconds 1761246000
backup_last_success_timestamp_seconds 1761246000
backup_last_run_success 1
backup_last_duration_seconds 312

It writes to a temporary file first and then renames it. node_exporter never reads a half-written file. A Grafana overview shows the age of the last good backup per host. A failed run also makes the systemd unit fail, and an alert rule for failed units catches that.

The timer

# /etc/systemd/system/backup-vps1.timer
[Unit]
Description=Daily backup of vps1

[Timer]
OnCalendar=*-*-* 03:15
RandomizedDelaySec=15min
Persistent=true

[Install]
WantedBy=timers.target

Persistent=true runs a missed backup after the host was off at 03:15.

Restore, at least once

A backup that was never restored is a hope. I restored each host completely during the rehearsals in the next post. The restore from the old Arch host reported errors, because restic tried to set SELinux labels that the backup never had (see post 4). The files were fine.

R="restic -r sftp:backup-box:vps1 --password-file /root/.config/restic/vps1.pass --no-lock"
$R snapshots
$R restore latest --target /var/tmp/restore --include /home/svc/traefik/acme --exclude-xattr security.selinux

--no-lock lets a test VM read the live repository without writing a lock into it.

Check

sudo systemctl start backup-vps1.service
sudo tail -20 /var/log/backup-vps1.log
curl -s http://10.99.0.13:9100/metrics | grep ^backup_     # from the hub

Next: Rehearse before you reinstall.

Further reading

Previous: One generic Ansible role for every app Next: Rehearse before you reinstall

Related content

Rehearse before you reinstall
One generic Ansible role for every app
Ansible groundwork: inventory, bootstrap and a base role