<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Ansible on Linux Lunatix</title><link>https://www.linux-lunatix.org/categories/ansible/</link><description>Recent content in Ansible on Linux Lunatix</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><copyright>2025 Stefan Marx :: All written work is licensed under CC BY-SA 4.0 unless otherwise stated. Images are taken by me unless otherwise stated and need written permission to be used elsewhere.</copyright><lastBuildDate>Sat, 21 Nov 2026 10:15:00 +0100</lastBuildDate><atom:link href="https://www.linux-lunatix.org/categories/ansible/index.xml" rel="self" type="application/rss+xml"/><item><title>Removing the plain SSH keys</title><link>https://www.linux-lunatix.org/blog/ssh-05-plain-keys/</link><pubDate>Sat, 21 Nov 2026 10:15:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/ssh-05-plain-keys/</guid><description>&lt;p>&lt;em>Part 5 of the SSH series that starts with &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/em>&lt;/p>
&lt;p>Certificates work on every host now. The old plain keys are still in the &lt;code>authorized_keys&lt;/code> files, and while they are there, the certificates change nothing about security. A stolen laptop key would still open every door. This post removes the plain keys. Only one break-glass key per account stays, for the day when Bao is gone.&lt;/p>
&lt;h2 id="the-target">The target&lt;/h2>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Account&lt;/th>
&lt;th>Certificate principals&lt;/th>
&lt;th>Plain keys&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>the admin account (&lt;code>fedora&lt;/code>)&lt;/td>
&lt;td>&lt;code>alice&lt;/code>&lt;/td>
&lt;td>break-glass&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>svc&lt;/code> (the apps)&lt;/td>
&lt;td>&lt;code>alice&lt;/code> (+ &lt;code>deploy-websites&lt;/code> on the web host)&lt;/td>
&lt;td>none&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>ansible&lt;/code>&lt;/td>
&lt;td>&lt;code>ansible&lt;/code>&lt;/td>
&lt;td>break-glass&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>&lt;code>svc&lt;/code> gets no plain key at all. If I need it without Bao, I log in as the admin account and use &lt;code>sudo&lt;/code>.&lt;/p>
&lt;h2 id="step-1-the-break-glass-key">Step 1: the break-glass key&lt;/h2>
&lt;p>The break-glass key is a normal Ed25519 key. Its private part lives only in pass, never in &lt;code>~/.ssh&lt;/code>. Create it in memory and move it straight into pass:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">d&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>mktemp -d -p /dev/shm&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -q -t ed25519 -N &lt;span class="s1">&amp;#39;&amp;#39;&lt;/span> -C &lt;span class="s1">&amp;#39;break-glass@infra&amp;#39;&lt;/span> -f &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/key&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">pass insert -m infra/ssh-break-glass &amp;lt; &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/key&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cp &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/key.pub&amp;#34;&lt;/span> ~/ops/client/ssh/break-glass.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">shred -u &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/key&amp;#34;&lt;/span> &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> rm -r &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The public part goes into the inventory:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># group_vars/all.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_break_glass_key&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;ssh-ed25519 AAAA...break-glass-public-key... break-glass@infra&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>To use it, a small script loads the key from pass into ssh-agent for 15 minutes. It never writes it to disk:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># ssh-break-glass: only when Bao is gone and the certificates have expired&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">#&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-break-glass vps1 # any ssh arguments&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-break-glass -- ansible-playbook ... # a command: Ansible with the key&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">pub&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>dirname &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">$(&lt;/span>readlink -f &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$0&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span>/../ssh/break-glass.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">pass show infra/ssh-break-glass &lt;span class="p">|&lt;/span> ssh-add -q -t 15m -
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> &lt;span class="o">[&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">1&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;--&amp;#34;&lt;/span> &lt;span class="o">]&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">shift&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">exec&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$@&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --private-key &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$pub&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">exec&lt;/span> ssh -i &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$pub&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$@&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>ssh-add -&lt;/code> reads the private key from stdin. &lt;code>-t 15m&lt;/code> removes it from the agent after 15 minutes.&lt;/p>
&lt;p>&lt;code>-i&lt;/code> with a &lt;code>.pub&lt;/code> file looks odd. ssh then selects the matching key in the agent and offers it first. That matters with &lt;code>IdentitiesOnly yes&lt;/code> and &lt;code>MaxAuthTries 3&lt;/code>. Ansible&amp;rsquo;s &lt;code>--private-key&lt;/code> works the same way.&lt;/p>
&lt;p>Test it now, while everything works, and not on the day you need it.&lt;/p>
&lt;h2 id="step-2-authorized-keys-from-the-inventory">Step 2: authorized_keys from the inventory&lt;/h2>
&lt;p>Each host lists its plain keys per account:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/vps1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_authorized_keys&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fedora&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_break_glass_key }}&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">svc&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_break_glass_key }}&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The rules: a listed account gets exactly these keys, and nothing else stays in its file. An empty list removes the file. An account that is not listed is left alone, so the role cannot touch accounts it does not know.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># roles/base/tasks/ssh_keys.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Accounts&amp;#39; home directories&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.getent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">database&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">passwd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">authorized_keys with exactly the listed keys&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.authorized_key&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.key }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">key&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.value | join(&amp;#39;\n&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">exclusive&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_authorized_keys | dict2items | selectattr(&amp;#39;value&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.key }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">No&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">authorized_keys for cert-only accounts&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ansible_facts.getent_passwd[item.key][4] }}/.ssh/authorized_keys&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">absent&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_authorized_keys | dict2items | rejectattr(&amp;#39;value&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.key }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>exclusive: true&lt;/code> is what removes the old keys. The home directory comes from &lt;code>getent&lt;/code>, because &lt;code>svc&lt;/code> and the admin account do not have to live under &lt;code>/home/&amp;lt;name&amp;gt;&lt;/code>.&lt;/p>
&lt;p>In the role&amp;rsquo;s &lt;code>main.yml&lt;/code> these tasks come after the &lt;code>flush_handlers&lt;/code> from &lt;a href="../../blog/ssh-03-hosts/">the previous post&lt;/a>. On a host&amp;rsquo;s first run, sshd already trusts the CA when the keys go away.&lt;/p>
&lt;p>&lt;code>bootstrap.yml&lt;/code> from the Ansible post still installs the control nodes&amp;rsquo; plain keys for &lt;code>ansible&lt;/code>. That is for the first contact with a new host, before it trusts the CA. The first &lt;code>site.yml&lt;/code> run then replaces them with the break-glass key.&lt;/p>
&lt;h2 id="step-3-roll-it-out-one-host-at-a-time">Step 3: roll it out, one host at a time&lt;/h2>
&lt;p>I did this first on a throwaway VM, then host by host.&lt;/p>
&lt;ol>
&lt;li>Get fresh certificates: &lt;code>ssh-cert&lt;/code>. Check that both are valid: &lt;code>ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub&lt;/code> and the same for &lt;code>ansible_ed25519-cert.pub&lt;/code>.&lt;/li>
&lt;li>Open a standby session as the admin account and leave it open:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh -M -S ~/.ssh/cm/standby-vps1 -fN fedora@vps1.example.org
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>Close Ansible&amp;rsquo;s reused connections, then run the base role:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="k">for&lt;/span> s in ~/.ansible/cp/*&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span> ssh -O &lt;span class="nb">exit&lt;/span> -o &lt;span class="nv">ControlPath&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$s&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> x 2&amp;gt;/dev/null&lt;span class="p">;&lt;/span> &lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t base --check
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t base
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>Test with fresh connections. Each of these must work:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">F&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;-o ControlMaster=no -o ControlPath=none&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh &lt;span class="nv">$F&lt;/span> fedora@vps1.example.org &lt;span class="nb">true&lt;/span> &lt;span class="c1"># certificate&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh &lt;span class="nv">$F&lt;/span> svc@vps1.example.org &lt;span class="nb">true&lt;/span> &lt;span class="c1"># certificate&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible vps1 -m ping &lt;span class="c1"># the ansible certificate&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-break-glass &lt;span class="nv">$F&lt;/span> fedora@vps1.example.org &lt;span class="nb">true&lt;/span> &lt;span class="c1"># break-glass key&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>And this must fail:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">d&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>mktemp -d&lt;span class="k">)&lt;/span> &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> cp ~/.ssh/id_ed25519 &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh &lt;span class="nv">$F&lt;/span> -o &lt;span class="nv">IdentityAgent&lt;/span>&lt;span class="o">=&lt;/span>none -o &lt;span class="nv">IdentitiesOnly&lt;/span>&lt;span class="o">=&lt;/span>yes -i &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/id_ed25519&amp;#34;&lt;/span> fedora@vps1.example.org &lt;span class="nb">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">rm -r &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>The copy has no &lt;code>-cert.pub&lt;/code> next to it, and &lt;code>IdentityAgent=none&lt;/code> keeps the agent out, so ssh offers only the plain key. Before this step it was in &lt;code>authorized_keys&lt;/code>. Now it is refused.&lt;/li>
&lt;li>The second run must report &lt;code>changed=0&lt;/code>. Then close the standby session.&lt;/li>
&lt;/ol>
&lt;h2 id="devices-that-cannot-talk-to-bao">Devices that cannot talk to Bao&lt;/h2>
&lt;p>My tablet runs &lt;a href="https://blink.sh">Blink&lt;/a>, an SSH client for iOS. Blink cannot log in to Bao, but it can use a certificate. So the tablet gets a certificate for 30 days, which I sign on the laptop.&lt;/p>
&lt;ol>
&lt;li>In Blink, create an Ed25519 key and copy its public part to the laptop.&lt;/li>
&lt;li>Sign it with the &lt;code>device&lt;/code> role:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-cert device tablet.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>In Blink: Keys, the key, Certificate, paste.&lt;/li>
&lt;li>Put a reminder into the calendar a few days before the 30 days end. An expired certificate is simply refused.&lt;/li>
&lt;/ol>
&lt;p>Blink cannot check host certificates. It asks about each host key once. Compare the fingerprint with the host&amp;rsquo;s real key before you accept it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh vps1.example.org ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A second Ansible control node that is not always on gets a 30-day certificate for its &lt;code>ansible&lt;/code> key the same way, with a role of its own.&lt;/p>
&lt;h2 id="a-lost-device">A lost device&lt;/h2>
&lt;p>For the laptop, the certificates expire within 16 hours. But its Bao token can sign new ones until the token expires. Revoke it at once from another machine:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">bao login -no-print -method&lt;span class="o">=&lt;/span>userpass &lt;span class="nv">username&lt;/span>&lt;span class="o">=&lt;/span>alice
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao list auth/token/accessors
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao token lookup -accessor &amp;lt;accessor&amp;gt; &lt;span class="c1"># find the oidc-alice tokens&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao token revoke -accessor &amp;lt;accessor&amp;gt;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>For a device with a 30-day certificate, put its public key into the revoked list and roll it out:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># group_vars/all.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_revoked_keys&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;ssh-ed25519 AAAA...the-tablet&amp;#39;s-public-key...&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -t base
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>sshd then refuses that key with any certificate. The device&amp;rsquo;s other keys (GitHub, an SFTP backup box) are plain keys and must be removed there, as before.&lt;/p>
&lt;h2 id="when-bao-is-gone">When Bao is gone&lt;/h2>
&lt;ul>
&lt;li>Certificates that are already issued keep working until they expire. Hosts check them offline.&lt;/li>
&lt;li>After that: &lt;code>ssh-break-glass &amp;lt;host&amp;gt;&lt;/code> for a login, and &lt;code>ssh-break-glass -- ansible-playbook ...&lt;/code> for Ansible.&lt;/li>
&lt;li>Without pass, too: the provider&amp;rsquo;s VNC console for the VPS, the local console at home.&lt;/li>
&lt;li>Then bring Bao back, or restore it on another host from its Raft snapshot (&lt;a href="../../blog/ssh-02-openbao/">the OpenBao post&lt;/a>).&lt;/li>
&lt;/ul>
&lt;h2 id="how-i-nearly-locked-myself-out">How I nearly locked myself out&lt;/h2>
&lt;p>Two things went wrong during this work. Each one taught me a rule.&lt;/p>
&lt;h3 id="fail2ban-banned-my-home-ip">fail2ban banned my home IP&lt;/h3>
&lt;p>During a negative test I connected to a host by its bare FQDN, without the &lt;code>User&lt;/code> that my SSH config sets for its short name. ssh therefore used my local user name. That account is not in the host&amp;rsquo;s &lt;code>AllowUsers&lt;/code>, and fail2ban counts such a login as a failure. Together with my negative tests that made three failures within ten minutes, and fail2ban banned my home IP for an hour.&lt;/p>
&lt;p>Interestingly, a refused key alone (&amp;ldquo;Failed publickey&amp;rdquo;) does not count in fail2ban&amp;rsquo;s sshd filter. A login as an account that &lt;code>AllowUsers&lt;/code> does not allow does count.&lt;/p>
&lt;p>I got in again through another of my hosts, which still had a working connection, and unbanned the IP there:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh -J other-host vps1.example.org &lt;span class="s1">&amp;#39;sudo fail2ban-client set sshd unbanip 203.0.113.50&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The rule: test negative cases sparingly, with explicit user names, and know how you get in from another IP.&lt;/p>
&lt;h3 id="ansible-s-tests-passed-without-a-certificate">Ansible&amp;rsquo;s tests passed without a certificate&lt;/h3>
&lt;p>After the first host, all Ansible tests passed. Then I noticed that my laptop had no Ansible certificate at all. The &lt;code>ssh-cert&lt;/code> script signed only the login key at that point. Ansible had passed because it reused its SSH connections from before the change (&lt;code>ControlPersist&lt;/code>). A fresh connection would have failed.&lt;/p>
&lt;p>Since then &lt;code>ssh-cert&lt;/code> signs both keys by default, and I close &lt;code>~/.ansible/cp/*&lt;/code> before every test.&lt;/p>
&lt;h2 id="the-rules-in-short">The rules, in short&lt;/h2>
&lt;ul>
&lt;li>Keep a standby session open while you change anything that controls SSH access.&lt;/li>
&lt;li>Test with fresh connections: &lt;code>-o ControlMaster=no -o ControlPath=none&lt;/code>, and close Ansible&amp;rsquo;s connections.&lt;/li>
&lt;li>Test the break-glass path before you need it.&lt;/li>
&lt;li>Keep the principals and the keys in the inventory, and make each list exclusive.&lt;/li>
&lt;li>Short certificates beat revocation. Revoke the token that can sign, not the certificate.&lt;/li>
&lt;/ul>
&lt;h2 id="where-this-leaves-me">Where this leaves me&lt;/h2>
&lt;p>Every host trusts one CA key and lists, per account, which principals may log in. My laptop gets certificates for the day with one command and needs no &lt;code>known_hosts&lt;/code> entries for my hosts. CI jobs get a certificate for five minutes and can run one command. A reinstalled host gets a new host certificate during its first Ansible run, and nobody has to check a fingerprint over a console. The only plain key left lives in pass.&lt;/p>
&lt;p>That was the last post of the series. Part 1, about consolidating the servers, starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/collections/ansible/posix/authorized_key_module.html">ansible.posix.authorized_key&lt;/a>: &lt;code>exclusive&lt;/code> and the other options.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/ssh-add">ssh-add(1)&lt;/a>: &lt;code>-t&lt;/code> for a lifetime and &lt;code>-&lt;/code> for a key from stdin.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/ssh_config#ControlMaster">ssh_config(5), ControlMaster&lt;/a>: connection reuse, and why it hides a changed login.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/collections/ansible/builtin/ssh_connection.html">Ansible ssh connection plugin&lt;/a>: &lt;code>control_path&lt;/code> and &lt;code>ssh_args&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/commands/token/revoke/">bao token revoke&lt;/a>: revoke by accessor, for a lost laptop.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd_config#RevokedKeys">sshd_config(5), RevokedKeys&lt;/a>: plain key lists and KRLs.&lt;/li>
&lt;li>&lt;a href="https://docs.blink.sh/">Blink Shell documentation&lt;/a>: keys and certificates on iOS.&lt;/li>
&lt;li>&lt;a href="https://github.com/fail2ban/fail2ban">fail2ban&lt;/a>: what the sshd filter counts.&lt;/li>
&lt;/ul></description></item><item><title>Hosts that trust the SSH CA</title><link>https://www.linux-lunatix.org/blog/ssh-03-hosts/</link><pubDate>Tue, 10 Nov 2026 19:20:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/ssh-03-hosts/</guid><description>&lt;p>&lt;em>Part 3 of the SSH series that starts with &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/em>&lt;/p>
&lt;p>Bao from &lt;a href="../../blog/ssh-02-openbao/">the previous post&lt;/a> can sign certificates. Now the hosts must trust them, and the laptop must use them. At the end of this post:&lt;/p>
&lt;ul>
&lt;li>every host accepts user certificates for the accounts and principals listed in the inventory,&lt;/li>
&lt;li>every host has a host certificate, renewed by Ansible,&lt;/li>
&lt;li>the laptop gets its certificates with one command and trusts the hosts without &lt;code>known_hosts&lt;/code> entries.&lt;/li>
&lt;/ul>
&lt;p>The plain keys stay in place during this step. They go away in the last post of the series, when everything else works.&lt;/p>
&lt;h2 id="the-inventory">The inventory&lt;/h2>
&lt;p>The CA public keys go into &lt;code>group_vars/all.yml&lt;/code>. They are public, and a rollout needs no Bao for them:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># group_vars/all.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># bao read -field=public_key ssh-user/config/ca&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_user_ca&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;ssh-ed25519 AAAA...user-ca-public-key... openbao ssh-user&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># bao read -field=public_key ssh-host/config/ca&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_host_ca&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;ssh-ed25519 AAAA...host-ca-public-key... openbao ssh-host&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">bao_addr&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">https://bao.example.net&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Each host says who may log in as which account:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/vps1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_principals&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fedora&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">alice]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">svc&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">alice]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">ansible]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># group_vars/websites.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_principals&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fedora&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">alice]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">svc&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">alice, &amp;#39;restrict,command=&amp;#34;podman auto-update&amp;#34; deploy-websites&amp;#39;]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">ansible]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A line may start with options, in the same syntax as in &lt;code>authorized_keys&lt;/code>. The host enforces them, whatever the certificate says. &lt;a href="../../blog/ssh-01-why-certificates/">The first post&lt;/a> explains why that matters.&lt;/p>
&lt;h2 id="the-base-role-trust-the-ca">The base role: trust the CA&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># roles/base/tasks/ssh_certs.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SSH CA directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/ca&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0755&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">User CA&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/ca/user_ca.pub&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_user_ca }}\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ssh_user_ca | length &amp;gt; 0&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Principals directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/principals&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0755&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Principals per account&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/principals/{{ item.key }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.value | join(&amp;#39;\n&amp;#39;) }}\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_principals | dict2items }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.key }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Principals files in place&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.find&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">paths&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/principals&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">principals_files&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># Exclusive: an account no longer listed has no principals file, so no cert logs in as it&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Principals files of accounts no longer listed&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">absent&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ principals_files.files | rejectattr(&amp;#39;path&amp;#39;, &amp;#39;in&amp;#39;, ssh_principals.keys() | map(&amp;#39;regex_replace&amp;#39;, &amp;#39;^&amp;#39;, &amp;#39;/etc/ssh/principals/&amp;#39;) | list) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Revoked keys&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/revoked_keys&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ (ssh_revoked_keys | join(&amp;#39;\n&amp;#39;) ~ &amp;#39;\n&amp;#39;) if ssh_revoked_keys else &amp;#39;&amp;#39; }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The principals files need no reload. sshd reads them at each login.&lt;/p>
&lt;p>The &amp;ldquo;no longer listed&amp;rdquo; task makes the list exclusive. If I remove an account from &lt;code>ssh_principals&lt;/code>, its file goes away, and no certificate can log in as that account any more. A list that only adds is a list that only grows.&lt;/p>
&lt;p>&lt;code>RevokedKeys&lt;/code> accepts a plain file of public keys. A key in it is refused even with a valid certificate. That covers a lost device whose certificate is still valid for weeks (the last post of the series has the details). The file must exist, even empty. If &lt;code>RevokedKeys&lt;/code> points to a missing file, sshd refuses all public key logins.&lt;/p>
&lt;h3 id="sshd-s-config">sshd&amp;rsquo;s config&lt;/h3>
&lt;p>The hardening drop-in from &lt;a href="../../blog/infra-05-ansible-base/">the Ansible groundwork post&lt;/a> gets these lines:&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2">HostKey /etc/ssh/ssh_host_ed25519_key
{% if ssh_host_cert_installed | default(false) %}
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
{% endif %}
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
PubkeyAcceptedAlgorithms ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com
# The CA must sign with Ed25519, too
CASignatureAlgorithms ssh-ed25519
{% if ssh_user_ca %}
# Who may log in as an account is the account&amp;#39;s principals file (with options),
# not the certificate alone
TrustedUserCAKeys /etc/ssh/ca/user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/principals/%u
{% endif %}
RevokedKeys /etc/ssh/revoked_keys
&lt;/code>&lt;/pre>&lt;p>&lt;code>HostKeyAlgorithms&lt;/code> must list the certificate type, or sshd never offers the host certificate. The same applies to &lt;code>PubkeyAcceptedAlgorithms&lt;/code> for user certificates.&lt;/p>
&lt;p>The &lt;code>HostCertificate&lt;/code> line appears only once the certificate exists, so sshd never points to a missing file.&lt;/p>
&lt;h2 id="host-certificates-from-the-ansible-run">Host certificates from the Ansible run&lt;/h2>
&lt;p>The host&amp;rsquo;s key is on the host. The CA is in Bao. The control node has a Bao login. So the playbook reads the host&amp;rsquo;s public key, has it signed on the control node and installs the certificate on the host.&lt;/p>
&lt;p>First the role decides whether a new certificate is needed. A shell task describes the current one as JSON:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Host cert state&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.shell&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> set -o pipefail
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> c=/etc/ssh/ssh_host_ed25519_key-cert.pub
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> k=/etc/ssh/ssh_host_ed25519_key.pub
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> if [ ! -s $c ]; then echo &amp;#39;{&amp;#34;present&amp;#34;: false}&amp;#39;; exit 0; fi
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> info=$(ssh-keygen -Lf $c)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> to=$(sed -n &amp;#39;s/^ *Valid: from .* to \(.*\)$/\1/p&amp;#39; &amp;lt;&amp;lt;&amp;lt;&amp;#34;$info&amp;#34;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> left=$(( ($(date -d &amp;#34;$to&amp;#34; +%s 2&amp;gt;/dev/null || date +%s) - $(date +%s)) / 86400 ))
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> ca=$(sed -n &amp;#39;s/^ *Signing CA: [A-Z0-9-]* \(SHA256:[^ ]*\).*/\1/p&amp;#39; &amp;lt;&amp;lt;&amp;lt;&amp;#34;$info&amp;#34;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> cert_key=$(sed -n &amp;#39;s/^ *Public key: [A-Z0-9-]* \(SHA256:[^ ]*\).*/\1/p&amp;#39; &amp;lt;&amp;lt;&amp;lt;&amp;#34;$info&amp;#34;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> principals=$(sed -n &amp;#39;/^ *Principals:/,/^ *Critical Options:/p&amp;#39; &amp;lt;&amp;lt;&amp;lt;&amp;#34;$info&amp;#34; | sed &amp;#39;1d;$d&amp;#39; | tr -d &amp;#39; &amp;#39; | sort | paste -sd,)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> printf &amp;#39;{&amp;#34;present&amp;#34;: true, &amp;#34;days_left&amp;#34;: %d, &amp;#34;ca&amp;#34;: &amp;#34;%s&amp;#34;, &amp;#34;want_ca&amp;#34;: &amp;#34;%s&amp;#34;, &amp;#34;cert_key&amp;#34;: &amp;#34;%s&amp;#34;, &amp;#34;host_key&amp;#34;: &amp;#34;%s&amp;#34;, &amp;#34;principals&amp;#34;: &amp;#34;%s&amp;#34;}\n&amp;#39; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> &amp;#34;$left&amp;#34; &amp;#34;$ca&amp;#34; &amp;#34;$(ssh-keygen -lf /dev/stdin &amp;lt;&amp;lt;&amp;lt;&amp;#34;$HOST_CA&amp;#34; | awk &amp;#39;{print $2}&amp;#39;)&amp;#34; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> &amp;#34;$cert_key&amp;#34; &amp;#34;$(ssh-keygen -lf $k | awk &amp;#39;{print $2}&amp;#39;)&amp;#34; &amp;#34;$principals&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">args&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">executable&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/bin/bash&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">HOST_CA&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_host_ca }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">host_cert_state&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">check_mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Host cert needed?&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.set_fact&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">host_cert_sign&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">&amp;gt;-&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> {{ not state.present
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> or state.days_left &amp;lt; 60
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> or state.ca != state.want_ca
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> or state.cert_key != state.host_key
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> or state.principals != (ssh_host_cert_principals | sort | join(&amp;#39;,&amp;#39;)) }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vars&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ host_cert_state.stdout | from_json }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A new certificate is signed when there is none, when less than 60 days are left, or when it is from another CA (after a CA rotation). It is also signed when it is for another key (after a reinstall) or for other names. Otherwise the run leaves it alone and stays at &lt;code>changed=0&lt;/code>.&lt;/p>
&lt;p>Then the signing:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Host cert&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">host_cert_sign&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">block&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Host public key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.slurp&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/ssh_host_ed25519_key.pub&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">host_pubkey&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># On the control node, with its Bao login (the admin policy may sign host certs)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Sign the host key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">argv&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">bao&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">write&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- -&lt;span class="l">field=signed_key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">ssh-host/sign/hosts&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">cert_type=host&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">valid_principals={{ ssh_host_cert_principals | join(&amp;#39;,&amp;#39;) }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">public_key=-&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">stdin&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ host_pubkey.content | b64decode }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">BAO_ADDR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ bao_addr }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">delegate_to&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">localhost&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">become&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">host_cert_signed&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">not ansible_check_mode&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Install the host cert&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/ssh_host_ed25519_key-cert.pub&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ host_cert_signed.stdout }}\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">not ansible_check_mode&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">rescue&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">No&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Bao login on the control node?&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.fail&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">msg&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">&amp;gt;-&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> Signing the host cert failed ({{ host_cert_signed.stderr | default(&amp;#39;&amp;#39;) | trim }}).
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> Log in on the control node first (bao login -no-print -method=oidc),
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> or skip host certs with -e ssh_host_cert=false.&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>delegate_to: localhost&lt;/code> with &lt;code>become: false&lt;/code> runs &lt;code>bao&lt;/code> on the control node, as me, with my Bao token. The host needs no access to Bao at all. The &lt;code>rescue&lt;/code> block turns Bao&amp;rsquo;s error into a message that says what to do.&lt;/p>
&lt;p>&lt;code>ssh_host_cert_principals&lt;/code> defaults to &lt;code>[ansible_host]&lt;/code>, the FQDN that I use to connect. A certificate is valid only for the names in it. If you also connect by a short name or an IP address, either put those into the certificate or make the client use the FQDN (see &lt;code>HostKeyAlias&lt;/code> below).&lt;/p>
&lt;h3 id="order-matters-flush-the-handlers">Order matters: flush the handlers&lt;/h3>
&lt;p>In the role, the CA trust and the sshd config come first. Then a &lt;code>meta: flush_handlers&lt;/code> reloads sshd at once, before anything else touches SSH access. In the last post of the series the next tasks remove the plain keys. A host&amp;rsquo;s first run must not take away the key it came in with while certificates do not work there yet.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SSH certificates&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.include_tasks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ssh_certs.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SSH hardening&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.template&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd-hardening.conf.j2&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/sshd_config.d/00-hardening.conf&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">validate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd -t -f %s&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd reloaded before the keys change&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.meta&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">flush_handlers&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="the-laptop">The laptop&lt;/h2>
&lt;h3 id="ssh-cert-certificates-for-the-day">ssh-cert: certificates for the day&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># ssh-cert: SSH certificates from OpenBao. Logs in through Forgejo (OIDC) when there&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># is no valid Bao token, then signs a public key:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">#&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-cert both of these (the daily call: logins and Ansible)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-cert admin ~/.ssh/id_ed25519 -&amp;gt; ~/.ssh/id_ed25519-cert.pub (16 h)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-cert ansible ~/.ssh/ansible_ed25519 -&amp;gt; ~/.ssh/ansible_ed25519-cert.pub (16 h)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-cert device &amp;lt;key.pub&amp;gt; prints a 30-day cert for a device without Bao&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="nv">https&lt;/span>&lt;span class="p">://bao.example.net&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sign&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> &lt;span class="c1"># role public-key output (- for stdout)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> &lt;span class="nv">role&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nv">$1&lt;/span> &lt;span class="nv">pub&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nv">$2&lt;/span> &lt;span class="nv">out&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nv">$3&lt;/span> cert
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">cert&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>bao write -field&lt;span class="o">=&lt;/span>signed_key &lt;span class="s2">&amp;#34;ssh-user/sign/&lt;/span>&lt;span class="nv">$role&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="nv">public_key&lt;/span>&lt;span class="o">=&lt;/span>- &amp;lt; &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$pub&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="o">[&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$out&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">=&lt;/span> - &lt;span class="o">]&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">printf&lt;/span> &lt;span class="s1">&amp;#39;%s\n&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$cert&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">else&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">printf&lt;/span> &lt;span class="s1">&amp;#39;%s\n&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$cert&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt; &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$out&lt;/span>&lt;span class="s2">.tmp&amp;#34;&lt;/span> &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> mv &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$out&lt;/span>&lt;span class="s2">.tmp&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$out&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">printf&lt;/span> &lt;span class="s1">&amp;#39;%s\n&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$cert&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="p">|&lt;/span> ssh-keygen -Lf /dev/stdin &lt;span class="p">|&lt;/span> sed -n &lt;span class="s1">&amp;#39;s/^ *\(Valid: .*\)/\1/p&amp;#39;&lt;/span> &lt;span class="p">|&lt;/span> sed &lt;span class="s2">&amp;#34;s|^|&lt;/span>&lt;span class="nv">$role&lt;/span>&lt;span class="s2">: |&amp;#34;&lt;/span> &amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">2&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">what&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">1&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="nv">all&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">case&lt;/span> &lt;span class="nv">$what&lt;/span> in
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> all&lt;span class="p">|&lt;/span>admin&lt;span class="p">|&lt;/span>ansible&lt;span class="p">|&lt;/span>device&lt;span class="o">)&lt;/span> &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> *&lt;span class="o">)&lt;/span> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;usage: ssh-cert [admin|ansible|device &amp;lt;key.pub&amp;gt;]&amp;#34;&lt;/span> &amp;gt;&lt;span class="p">&amp;amp;&lt;/span>2&lt;span class="p">;&lt;/span> &lt;span class="nb">exit&lt;/span> &lt;span class="m">2&lt;/span> &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">esac&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao token lookup &amp;gt;/dev/null 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">1&lt;/span> &lt;span class="o">||&lt;/span> bao login -no-print -method&lt;span class="o">=&lt;/span>oidc
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">case&lt;/span> &lt;span class="nv">$what&lt;/span> in
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> all&lt;span class="o">)&lt;/span> sign admin ~/.ssh/id_ed25519.pub ~/.ssh/id_ed25519-cert.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> sign ansible ~/.ssh/ansible_ed25519.pub ~/.ssh/ansible_ed25519-cert.pub &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> admin&lt;span class="o">)&lt;/span> sign admin ~/.ssh/id_ed25519.pub ~/.ssh/id_ed25519-cert.pub &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ansible&lt;span class="o">)&lt;/span> sign ansible ~/.ssh/ansible_ed25519.pub ~/.ssh/ansible_ed25519-cert.pub &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> device&lt;span class="o">)&lt;/span> sign device &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">2&lt;/span>&lt;span class="p">:?public key file&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> - &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">esac&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo dnf install openbao &lt;span class="c1"># the bao CLI&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ln -s ~/ops/client/bin/ssh-cert ~/.local/bin/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-cert
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># admin: Valid: from 2026-11-10T08:01:00 to 2026-11-11T00:02:00&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ansible: Valid: from 2026-11-10T08:01:00 to 2026-11-11T00:02:00&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The Bao token from the OIDC login is also valid for 16 hours. Run &lt;code>ssh-cert&lt;/code> again during the day, and it signs without the browser.&lt;/p>
&lt;p>ssh picks up &lt;code>id_ed25519-cert.pub&lt;/code> next to &lt;code>id_ed25519&lt;/code> by itself. Nothing else changes in the SSH config. With &lt;code>IdentitiesOnly yes&lt;/code>, ssh offers the plain key first and the certificate second. That is two of the three attempts that &lt;code>MaxAuthTries 3&lt;/code> allows, so do not add more keys to the same host entry.&lt;/p>
&lt;p>Ansible uses its own key, &lt;code>~/.ssh/ansible_ed25519&lt;/code>, and its certificate is picked up the same way.&lt;/p>
&lt;h3 id="one-line-instead-of-known-hosts">One line instead of known_hosts&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl"># ~/.ssh/known_hosts.ca
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">@cert-authority *.example.org,*.example.net ssh-ed25519 AAAA...host-ca-public-key... openbao ssh-host
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl"># ~/.ssh/config
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Host *
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> # The first file is where ssh writes new keys
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> UserKnownHostsFile ~/.ssh/known_hosts ~/.ssh/known_hosts.ca
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The order matters. ssh writes new host keys into the first file of the list. Keep your normal &lt;code>known_hosts&lt;/code> first, so that ssh never writes into the CA file.&lt;/p>
&lt;p>Now remove the old entries of your managed hosts from &lt;code>known_hosts&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">cp ~/.ssh/known_hosts ~/.ssh/known_hosts.bak
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">for&lt;/span> h in vps1.example.org vps2.example.org home1.example.net&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span> ssh-keygen -R &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$h&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>What happens with other hosts? Nothing changes for them. A server at a friend&amp;rsquo;s place that is not in &lt;code>*.example.org&lt;/code> has no certificate from my CA. ssh asks about its key once, as before, and writes it into &lt;code>~/.ssh/known_hosts&lt;/code>. And the plain key is still offered to hosts that do not trust my user CA.&lt;/p>
&lt;p>An expired or invalid host certificate does not lock you out either. ssh falls back to the plain host key, finds no entry for it and asks, as for a new host. In that situation, do not just say yes: find out why the certificate is not valid.&lt;/p>
&lt;h3 id="short-names-and-hostkeyalias">Short names and HostKeyAlias&lt;/h3>
&lt;p>A host certificate is valid for the names in it. If you connect with &lt;code>ssh v2&lt;/code> or by IP address, the name does not match. In my SSH config every alias of a host sets the FQDN as the name to check:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Host v2 vps2
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> HostName vps2.example.org
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> HostKeyAlias vps2.example.org
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>HostKeyAlias&lt;/code> also helps when you reach a host over different paths: at home by its LAN address, away through a jump host. The check is always against the same name.&lt;/p>
&lt;h2 id="roll-it-out">Roll it out&lt;/h2>
&lt;ol>
&lt;li>Run Bao&amp;rsquo;s configuration (previous post), and add both CA keys to &lt;code>group_vars/all.yml&lt;/code>.&lt;/li>
&lt;li>Pick one host. Keep a second SSH session open to it, as an emergency way back in:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh -M -S ~/.ssh/cm/standby-vps1 -fN fedora@vps1.example.org
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>&lt;code>bao login -no-print -method=oidc&lt;/code>, then &lt;code>ansible-playbook site.yml -l vps1 -t base --check&lt;/code>, then the same without &lt;code>--check&lt;/code>.&lt;/li>
&lt;li>Test with a fresh connection, not a reused one:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh -o &lt;span class="nv">ControlMaster&lt;/span>&lt;span class="o">=&lt;/span>no -o &lt;span class="nv">ControlPath&lt;/span>&lt;span class="o">=&lt;/span>none -v fedora@vps1.example.org &lt;span class="nb">true&lt;/span> 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">1&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> grep -E &lt;span class="s1">&amp;#39;Server host certificate|Authenticated&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>Expect &amp;ldquo;Server host certificate: &lt;a href="mailto:ssh-ed25519-cert-v01@openssh.com">ssh-ed25519-cert-v01@openssh.com&lt;/a> &amp;hellip; ID &amp;hellip;-host&amp;rdquo; and &amp;ldquo;Authenticated &amp;hellip; using publickey with ED25519-CERT&amp;rdquo;.&lt;/li>
&lt;li>On the host, the log says which certificate logged in:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo journalctl -u sshd -n &lt;span class="m">20&lt;/span> &lt;span class="p">|&lt;/span> grep &lt;span class="s1">&amp;#39;ED25519-CERT&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Accepted publickey for fedora from 198.51.100.7 ... ED25519-CERT SHA256:... ID oidc-alice-admin (serial ...) CA ED25519 SHA256:...&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>Close the standby session (&lt;code>ssh -S ~/.ssh/cm/standby-vps1 -O exit x&lt;/code>), then the next host.&lt;/li>
&lt;/ol>
&lt;h2 id="negative-tests">Negative tests&lt;/h2>
&lt;p>Test that the host refuses what it should refuse:&lt;/p>
&lt;ul>
&lt;li>A certificate for &lt;code>alice&lt;/code> as &lt;code>ansible&lt;/code> or &lt;code>root&lt;/code>: refused (not in the principals file).&lt;/li>
&lt;li>An expired certificate: sign one with a short TTL in a test role, wait, try. Refused.&lt;/li>
&lt;li>A key from &lt;code>ssh_revoked_keys&lt;/code>: refused, even with a valid certificate.&lt;/li>
&lt;li>The host under a name that is not in its certificate, with only the CA line in &lt;code>known_hosts&lt;/code>: ssh asks or refuses.&lt;/li>
&lt;/ul>
&lt;p>Do these tests sparingly from your own IP. fail2ban counts some of these failures, for example a login as an account that &lt;code>AllowUsers&lt;/code> does not list. Three of them in ten minutes ban you for an hour. I learned that the hard way, more in the last post.&lt;/p>
&lt;p>Next: &lt;a href="../../blog/ssh-04-ci-oidc/">CI deploys without a deploy key&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://man.openbsd.org/sshd_config">sshd_config(5)&lt;/a>: &lt;code>TrustedUserCAKeys&lt;/code>, &lt;code>AuthorizedPrincipalsFile&lt;/code>, &lt;code>HostCertificate&lt;/code> and &lt;code>RevokedKeys&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/ssh_config">ssh_config(5)&lt;/a>: &lt;code>CertificateFile&lt;/code>, &lt;code>HostKeyAlias&lt;/code>, &lt;code>UserKnownHostsFile&lt;/code> and &lt;code>ControlMaster&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd#SSH_KNOWN_HOSTS_FILE_FORMAT">sshd(8), known_hosts format&lt;/a>: the &lt;code>@cert-authority&lt;/code> line on the client.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/ssh-keygen#CERTIFICATES">ssh-keygen(1), section CERTIFICATES&lt;/a>: &lt;code>-L&lt;/code> to inspect a certificate, &lt;code>-h&lt;/code> for host certificates.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/secrets/ssh/signed-ssh-certificates/">OpenBao signed SSH certificates&lt;/a>: the section on host key signing.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_delegation.html">Ansible delegation&lt;/a>: &lt;code>delegate_to: localhost&lt;/code> and &lt;code>run_once&lt;/code>.&lt;/li>
&lt;/ul></description></item><item><title>Rehearse before you reinstall</title><link>https://www.linux-lunatix.org/blog/infra-08-rehearsal/</link><pubDate>Tue, 27 Oct 2026 19:45:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/infra-08-rehearsal/</guid><description>&lt;p>&lt;em>Part 8 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>Two of my hosts had to be reinstalled: the Arch VPS with the websites, and the Ubuntu host with mail and Git. Before I touched either of them, I ran the whole reinstall on a local VM: the install, the playbook, the restore of the real backups and the tests. This post shows how. The same approach works for any host you are afraid to touch.&lt;/p>
&lt;h2 id="the-idea">The idea&lt;/h2>
&lt;p>A rehearsal VM is the real host with three differences:&lt;/p>
&lt;ol>
&lt;li>It runs on a machine at home, on libvirt&amp;rsquo;s NAT network.&lt;/li>
&lt;li>It cannot talk to the world in ways that would hurt: no outgoing mail, no Matrix federation, no certificate requests, no CI jobs.&lt;/li>
&lt;li>It never runs the real host&amp;rsquo;s backup timer or monitoring tunnel.&lt;/li>
&lt;/ol>
&lt;p>Everything else is real: the real inventory settings, the real secrets from pass and the real backup repository (read-only). If the rehearsal passes, the cut-over is a repeat of known steps.&lt;/p>
&lt;h2 id="step-1-a-vm-from-the-fedora-cloud-image">Step 1: a VM from the Fedora Cloud image&lt;/h2>
&lt;p>A provider&amp;rsquo;s Fedora image is the Fedora Cloud image with cloud-init. So I start from the same image. Download it and check its signature first:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">cd&lt;/span> /var/lib/libvirt/images/base
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-44-1.7-x86_64-CHECKSUM
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -O https://fedoraproject.org/fedora.gpg
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">gpgv --keyring ./fedora.gpg Fedora-Cloud-44-1.7-x86_64-CHECKSUM
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sha256sum -c --ignore-missing Fedora-Cloud-44-1.7-x86_64-CHECKSUM
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then a small script creates a VM from it. Each VM gets a thin qcow2 overlay on the base image. cloud-init creates the user &lt;code>fedora&lt;/code> with my key and passwordless sudo, the same starting point as a fresh VPS:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># rehearsal-vm.sh create &amp;lt;name&amp;gt; &amp;lt;memory MiB&amp;gt; &amp;lt;vcpus&amp;gt; &amp;lt;disk GB&amp;gt; &amp;lt;ssh pubkey file&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># rehearsal-vm.sh destroy &amp;lt;name&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># rehearsal-vm.sh ip &amp;lt;name&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">BASE_DIR&lt;/span>&lt;span class="o">=&lt;/span>/var/lib/libvirt/images/base
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">IMAGE&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">IMAGE&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="nv">$BASE_DIR&lt;/span>&lt;span class="p">/Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">VM_DIR&lt;/span>&lt;span class="o">=&lt;/span>/var/lib/libvirt/images
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">LIBVIRT_DEFAULT_URI&lt;/span>&lt;span class="o">=&lt;/span>qemu:///system
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">cmd&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">1&lt;/span>&lt;span class="p">:?command&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nv">name&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">2&lt;/span>&lt;span class="p">:?name&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">case&lt;/span> &lt;span class="nv">$cmd&lt;/span> in
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">create&lt;span class="o">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">mem&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">3&lt;/span>&lt;span class="p">:?memory MiB&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nv">cpus&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">4&lt;/span>&lt;span class="p">:?vcpus&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nv">disk&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">5&lt;/span>&lt;span class="p">:?disk GB&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nv">key&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>cat &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">6&lt;/span>&lt;span class="p">:?ssh pubkey file&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">tmp&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>mktemp -d&lt;span class="k">)&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nb">trap&lt;/span> &lt;span class="s1">&amp;#39;rm -rf &amp;#34;$tmp&amp;#34;&amp;#39;&lt;/span> EXIT
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> cat &amp;gt;&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$tmp&lt;/span>&lt;span class="s2">/user-data&amp;#34;&lt;/span> &lt;span class="s">&amp;lt;&amp;lt;EOF
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">#cloud-config
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">hostname: $name
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">users:
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> - name: fedora
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> groups: [wheel]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> sudo: &amp;#34;ALL=(ALL) NOPASSWD:ALL&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> ssh_authorized_keys: [&amp;#34;$key&amp;#34;]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> sudo qemu-img create -q -f qcow2 -F qcow2 -b &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$IMAGE&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$VM_DIR&lt;/span>&lt;span class="s2">/&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">.qcow2&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">disk&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">G&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> virt-install --name &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --memory &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$mem&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --vcpus &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$cpus&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --disk &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$VM_DIR&lt;/span>&lt;span class="s2">/&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">.qcow2,bus=virtio&amp;#34;&lt;/span> --import &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --os-variant fedora-unknown --network &lt;span class="nv">network&lt;/span>&lt;span class="o">=&lt;/span>default &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --cloud-init user-data&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$tmp&lt;/span>&lt;span class="s2">/user-data&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --graphics none --noautoconsole
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">destroy&lt;span class="o">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> virsh destroy &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> 2&amp;gt;/dev/null &lt;span class="o">||&lt;/span> &lt;span class="nb">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> virsh undefine &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --remove-all-storage
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ip&lt;span class="o">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> virsh domifaddr &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --source lease &lt;span class="p">|&lt;/span> awk &lt;span class="s1">&amp;#39;/ipv4/ {sub(&amp;#34;/.*&amp;#34;, &amp;#34;&amp;#34;, $4); print $4}&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">esac&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">rehearsal-vm.sh create mail1-rehearsal &lt;span class="m">8192&lt;/span> &lt;span class="m">4&lt;/span> &lt;span class="m">100&lt;/span> ~/.ssh/id_ed25519.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">rehearsal-vm.sh ip mail1-rehearsal &lt;span class="c1"># e.g. 192.168.122.76&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>virt-install --cloud-init&lt;/code> builds the seed ISO by itself. You need no separate &lt;code>cloud-localds&lt;/code> step.&lt;/p>
&lt;h2 id="step-2-an-inventory-for-the-vms">Step 2: an inventory for the VMs&lt;/h2>
&lt;p>The VMs are in a separate inventory file, which I pass only when I mean it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># inventory/rehearsal.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">all&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mail1-rehearsal&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">192.168.122.76&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_ssh_common_args&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">&amp;gt;-&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> -J me@vmhost.example.net
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> -o UserKnownHostsFile=~/.ssh/known_hosts.rehearsal -o StrictHostKeyChecking=accept-new&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bootstrap_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fedora&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hostname&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">mail1-rehearsal&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># use the real host&amp;#39;s secrets from pass&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">pass_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">mail1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">backup_timer&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">monitoring_tunnel&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># the CI runner would register under the real name and take real jobs&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">docker_compose_projects&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">children&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fedora&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">mail1-rehearsal&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mail&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">mail1-rehearsal&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The trick is where the real host&amp;rsquo;s settings live. They are not in &lt;code>host_vars/mail1.yml&lt;/code>, but in &lt;code>group_vars/mail.yml&lt;/code>, for a group with one member. The VM joins the same group and gets all of them. The few variables in &lt;code>rehearsal.yml&lt;/code> are host variables, and those beat group variables. So the VM can switch off the backup timer, the tunnel and the runner.&lt;/p>
&lt;p>The VMs are recreated often. Each new VM has new host keys, so the VMs get their own &lt;code>known_hosts&lt;/code> file. My real &lt;code>known_hosts&lt;/code> stays clean. &lt;code>accept-new&lt;/code> is acceptable here, and only here: the VM is on a NAT network behind my own machine.&lt;/p>
&lt;h2 id="step-3-guards">Step 3: guards&lt;/h2>
&lt;p>The rehearsal VM holds the real data and the real credentials. Left alone, it would:&lt;/p>
&lt;ul>
&lt;li>deliver mail that is waiting in the restored queue,&lt;/li>
&lt;li>federate with other Matrix servers under the real server name,&lt;/li>
&lt;li>ask Let&amp;rsquo;s Encrypt for certificates of the real domains,&lt;/li>
&lt;li>take real CI jobs as the real runner.&lt;/li>
&lt;/ul>
&lt;p>The inventory handles the runner. For the rest I put an nftables table on the VM that drops the dangerous outgoing traffic. Something like this:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">table inet rehearsal_guard {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> chain output {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> type filter hook output priority 0; policy accept;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> oifname &amp;#34;lo&amp;#34; accept
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> tcp dport { 25, 465, 587, 8448 } drop
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> # once the images are pulled: no new connections from the app user
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> meta skuid &amp;#34;svc&amp;#34; ct state new drop
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> }
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The app user&amp;rsquo;s rule goes in only after the first full run, because Podman must pull the images first. DNS needs one more step: unbound marks the root servers as down while the block is on. Restart it after you lift the block for a test.&lt;/p>
&lt;h2 id="step-4-the-rollout-in-this-order">Step 4: the rollout, in this order&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">I&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;-i inventory/hosts.yml -i inventory/rehearsal.yml&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 1. the ansible user&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook &lt;span class="nv">$I&lt;/span> playbooks/bootstrap.yml -l mail1-rehearsal
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 2. update and reboot at once (see post 5: dnf-automatic would do it mid-run)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible &lt;span class="nv">$I&lt;/span> mail1-rehearsal -b -m shell -a &lt;span class="s1">&amp;#39;dnf -y upgrade &amp;amp;&amp;amp; systemctl reboot&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 3. everything except the apps&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook &lt;span class="nv">$I&lt;/span> playbooks/site.yml -l mail1-rehearsal -t base,monitoring,backup
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 4. restore the state (below), before Traefik starts&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 5. everything&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook &lt;span class="nv">$I&lt;/span> playbooks/site.yml -l mail1-rehearsal
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 6. once more, with fresh connections: changed=0&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook &lt;span class="nv">$I&lt;/span> playbooks/site.yml -l mail1-rehearsal &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -e &lt;span class="nv">ansible_ssh_common_args&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;-J me@vmhost.example.net -o ControlPath=none -o UserKnownHostsFile=~/.ssh/known_hosts.rehearsal&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Step 3 installs restic and the backup credentials, but no apps. The restore can then put the volumes in place before any container starts on empty data.&lt;/p>
&lt;h2 id="step-5-the-restore">Step 5: the restore&lt;/h2>
&lt;p>The restore script runs on the new host as root and reads the old host&amp;rsquo;s restic repository with &lt;code>--no-lock&lt;/code>. It has three parts, because the order matters:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">restore.sh files # volumes and Traefik&amp;#39;s acme.json: before the apps start
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">restore.sh dbs # database dumps: after the full run (the databases must run)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">restore.sh mailcow # Mailcow&amp;#39;s own backup and its local config: after the full run
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>files&lt;/code> part copies each old bind mount into a new Podman volume with &lt;code>podman unshare tar&lt;/code> (shown in &lt;a href="../../blog/infra-02-quadlets/">post 2&lt;/a>). The &lt;code>dbs&lt;/code> part drops the empty databases that the containers created at their first start, then loads the dumps. The &lt;code>mailcow&lt;/code> part restores Mailcow&amp;rsquo;s own config files over the checkout. Then it runs Mailcow&amp;rsquo;s &lt;code>backup_and_restore.sh restore&lt;/code>, which asks four questions. A &lt;code>printf&lt;/code> answers them:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">printf&lt;/span> &lt;span class="s1">&amp;#39;1\n0\nY\ny\n&amp;#39;&lt;/span> &lt;span class="p">|&lt;/span> &lt;span class="nv">MAILCOW_BACKUP_LOCATION&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$tmp&lt;/span>&lt;span class="s2">/backup&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> /opt/mailcow-dockerized/helper-scripts/backup_and_restore.sh restore
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="step-6-tests">Step 6: tests&lt;/h2>
&lt;p>Test the services against the VM without DNS changes:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">ip&lt;/span>&lt;span class="o">=&lt;/span>192.168.122.76
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">for&lt;/span> h in git.example.org chat.example.org paste.example.org mail1.example.org&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> curl -s -o /dev/null -w &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$h&lt;/span>&lt;span class="s2"> %{http_code} verify=%{ssl_verify_result}\n&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --resolve &lt;span class="nv">$h&lt;/span>:443:&lt;span class="nv">$ip&lt;/span> https://&lt;span class="nv">$h&lt;/span>/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>verify=0&lt;/code> means that the restored certificate is valid for the name. I also compared counts from the old host with the VM: mail domains, mailboxes, aliases, DKIM keys, the number of messages in one INBOX, and the repositories in Forgejo. They must be identical.&lt;/p>
&lt;p>On the VM itself:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo ausearch -m avc -ts boot &lt;span class="c1"># no SELinux denials&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ps -eZ &lt;span class="p">|&lt;/span> grep -c container_t &lt;span class="c1"># every container confined&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Mailcow on Docker with SELinux enforcing gave no denials at all, which surprised me.&lt;/p>
&lt;p>Things a VM cannot test: real mail delivery, certificates from the HTTP challenge (no inbound port 80), the CI runner. I wrote those down as checks for the real cut-over.&lt;/p>
&lt;h2 id="a-provider-without-a-fedora-image">A provider without a Fedora image&lt;/h2>
&lt;p>My VPS provider for the website host offers no Fedora image. It does accept an uploaded qcow2 image, but it has no cloud metadata service. A plain Fedora Cloud image would boot there and wait for cloud-init data that never comes, so there would be no user and no key.&lt;/p>
&lt;p>The fix is to bake the cloud-init data into the image, as a NoCloud seed. I built it from a rehearsal VM:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">rehearsal-vm.sh create image-build &lt;span class="m">2048&lt;/span> &lt;span class="m">2&lt;/span> &lt;span class="m">6&lt;/span> ~/.ssh/id_ed25519.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh fedora@&amp;lt;vm&amp;gt; &lt;span class="s1">&amp;#39;sudo dnf -y upgrade &amp;amp;&amp;amp; sudo systemctl reboot&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh fedora@&amp;lt;vm&amp;gt; &lt;span class="s1">&amp;#39;sudo bash -s&amp;#39;&lt;/span> -- vps2.example.org &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">$(&lt;/span>cat ~/.ssh/id_ed25519.pub&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;lt; seal-image.sh
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The script, slightly shortened:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># seal-image.sh &amp;lt;fqdn&amp;gt; &amp;lt;ssh public key&amp;gt;: run as root inside the build VM&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">fqdn&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">1&lt;/span>&lt;span class="p">:?fqdn&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nv">key&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">2&lt;/span>&lt;span class="p">:?ssh public key&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">install -d -m &lt;span class="m">0755&lt;/span> /var/lib/cloud/seed/nocloud
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cat &amp;gt; /var/lib/cloud/seed/nocloud/meta-data &lt;span class="s">&amp;lt;&amp;lt;EOF
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">instance-id: ${fqdn%%.*}-$(date +%Y%m%d)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">local-hostname: ${fqdn%%.*}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cat &amp;gt; /var/lib/cloud/seed/nocloud/user-data &lt;span class="s">&amp;lt;&amp;lt;EOF
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">#cloud-config
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">ssh_pwauth: false
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">disable_root: true
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">users:
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> - name: fedora
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> groups: [wheel]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> sudo: &amp;#34;ALL=(ALL) NOPASSWD:ALL&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> lock_passwd: true
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> ssh_authorized_keys: [&amp;#34;$key&amp;#34;]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cat &amp;gt; /etc/cloud/cloud.cfg.d/90-datasource.cfg &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;EOF&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"># No metadata service: only the seed in /var/lib/cloud/seed/nocloud
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">datasource_list: [ NoCloud, None ]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">preserve_hostname: true
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">restorecon -R /var/lib/cloud /etc/cloud
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># The provider&amp;#39;s VMs have no serial port: drop console=ttyS0, keep GRUB on the VNC console&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">grubby --update-kernel&lt;span class="o">=&lt;/span>ALL --remove-args&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;console=ttyS0,115200n8&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sed -i -e &lt;span class="s1">&amp;#39;s/ console=ttyS0,115200n8//&amp;#39;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -e &lt;span class="s1">&amp;#39;s/^GRUB_TERMINAL_INPUT=.*/GRUB_TERMINAL_INPUT=&amp;#34;console&amp;#34;/&amp;#39;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -e &lt;span class="s1">&amp;#39;s/^GRUB_TERMINAL_OUTPUT=.*/GRUB_TERMINAL_OUTPUT=&amp;#34;console&amp;#34;/&amp;#39;&lt;/span> /etc/default/grub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sed -i &lt;span class="s1">&amp;#39;/^GRUB_SERIAL_COMMAND=/d&amp;#39;&lt;/span> /etc/default/grub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">grub2-mkconfig -o /boot/grub2/grub.cfg 2&amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># First-boot state: new machine-id, new host keys, no network profile bound to this MAC&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">rm -f /etc/NetworkManager/system-connections/cloud-init-*
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cloud-init clean --logs --machine-id
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">rm -f /etc/ssh/ssh_host_* /home/fedora/.bash_history /root/.bash_history
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$fqdn&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt; /etc/hostname
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">restorecon /etc/hostname
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">dnf clean all -q
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --rotate -q
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --vacuum-time&lt;span class="o">=&lt;/span>1s -q
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">fstrim -a
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> SEALED
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then power the VM off and compress its disk into the image:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo qemu-img convert -c -O qcow2 /var/lib/libvirt/images/image-build.qcow2 vps2-fedora44.qcow2
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sha256sum vps2-fedora44.qcow2 &amp;gt; vps2-fedora44.qcow2.sha256
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Some details matter here:&lt;/p>
&lt;ul>
&lt;li>&lt;code>preserve_hostname: true&lt;/code>. cloud-init sets the hostname with &lt;code>hostnamectl&lt;/code> so early that D-Bus is not up yet. That fails, and &lt;code>cloud-init status&lt;/code> then reports &amp;ldquo;degraded&amp;rdquo;. The hostname is already in &lt;code>/etc/hostname&lt;/code>, and the base role sets it again anyway.&lt;/li>
&lt;li>The serial console. The provider&amp;rsquo;s VMs have no serial port. With &lt;code>console=ttyS0&lt;/code> on the kernel command line, &lt;code>serial-getty&lt;/code> restarts every few seconds, and GRUB shows its menu only on the serial line. You would never see it in the VNC console.&lt;/li>
&lt;li>&lt;code>cloud-init clean --machine-id&lt;/code>. Without a new machine-id, every host made from the image would have the same one, and the same DHCP client ID too.&lt;/li>
&lt;li>The image contains only the user &lt;code>fedora&lt;/code> with my public key. No secret is in it.&lt;/li>
&lt;/ul>
&lt;p>Test the image the way the provider runs it. Mine boots with BIOS, a virtio-scsi disk and no cloud-init ISO:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">cp vps2-fedora44.qcow2 test.qcow2 &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> qemu-img resize test.qcow2 117G
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">virt-install --name image-test --memory &lt;span class="m">2048&lt;/span> --vcpus &lt;span class="m">2&lt;/span> --import &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --controller &lt;span class="nv">type&lt;/span>&lt;span class="o">=&lt;/span>scsi,model&lt;span class="o">=&lt;/span>virtio-scsi --disk test.qcow2,bus&lt;span class="o">=&lt;/span>scsi &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --os-variant fedora-unknown --network &lt;span class="nv">network&lt;/span>&lt;span class="o">=&lt;/span>default --graphics none --noautoconsole
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh fedora@&amp;lt;&lt;span class="nb">test&lt;/span> vm&amp;gt; cloud-init status --long &lt;span class="c1"># status: done, DataSourceNoCloud&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>cloud-init grows the root filesystem to the full disk at the first boot.&lt;/p>
&lt;h2 id="the-cut-over">The cut-over&lt;/h2>
&lt;p>With a passed rehearsal, the cut-over is a list:&lt;/p>
&lt;ol>
&lt;li>On the old host: stop what writes (the web proxy, the mail daemons), run a last backup, check its log and note the snapshot ID.&lt;/li>
&lt;li>Reinstall the host in the provider&amp;rsquo;s panel, with the provider&amp;rsquo;s Fedora image or the uploaded one. The IP address stays.&lt;/li>
&lt;li>Check the host key over the console (below).&lt;/li>
&lt;li>Inventory: the host&amp;rsquo;s &lt;code>bootstrap_user&lt;/code>, and the host into the &lt;code>fedora&lt;/code> group.&lt;/li>
&lt;li>The same six rollout steps as on the VM, with &lt;code>-l mail1&lt;/code>, and with the snapshot ID for the restore.&lt;/li>
&lt;li>The checks that the VM could not do: real mail in both directions, Mailcow&amp;rsquo;s certificate, the runner online, a deploy of a site, the monitoring targets, the first backup run.&lt;/li>
&lt;/ol>
&lt;p>The provider&amp;rsquo;s reinstall offered Fedora 43 only. Upgrading to 44 right after the bootstrap took under two minutes on the VM:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo dnf -y upgrade --refresh
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo dnf -y system-upgrade download --releasever&lt;span class="o">=&lt;/span>&lt;span class="m">44&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo dnf -y offline reboot
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="checking-the-new-host-key">Checking the new host key&lt;/h2>
&lt;p>After a reinstall, ssh warns that the host key changed. That is expected, and it is exactly the moment an attacker in the middle would use. So I check the new key over a channel that the attacker does not control: the provider&amp;rsquo;s VNC console.&lt;/p>
&lt;p>cloud-init prints the host key fingerprints during the first boot, but they scroll by quickly. A trick that always works:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># keep the new key apart from the real known_hosts&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh -o &lt;span class="nv">UserKnownHostsFile&lt;/span>&lt;span class="o">=&lt;/span>/tmp/new_host -o &lt;span class="nv">StrictHostKeyChecking&lt;/span>&lt;span class="o">=&lt;/span>accept-new fedora@vps2.example.org &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="s2">&amp;#34;printf &amp;#39;%s\n&amp;#39; \&amp;#34;code 7341: \$(ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub)\&amp;#34; | sudo tee /dev/tty1&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Pick a new random code each time. If the code appears in the VNC console with the fingerprint, the connection reached the real host and not some machine in between. Then compare the fingerprint with &lt;code>ssh-keygen -lf /tmp/new_host&lt;/code>, and only then put the key into your real &lt;code>known_hosts&lt;/code>.&lt;/p>
&lt;p>The second part of this series removes this step: with host certificates, the host proves its identity by itself.&lt;/p>
&lt;h2 id="what-the-rehearsals-found">What the rehearsals found&lt;/h2>
&lt;ul>
&lt;li>Pi-hole binds its DNS port to an address on the WireGuard interface. The WireGuard role therefore has to run before the apps, not after them.&lt;/li>
&lt;li>Fedora&amp;rsquo;s &lt;code>dockerd&lt;/code> already runs with &lt;code>--selinux-enabled&lt;/code>. Adding it to &lt;code>daemon.json&lt;/code> as well stops the daemon. Also, &lt;code>moby-engine&lt;/code> creates no &lt;code>/etc/docker&lt;/code> directory.&lt;/li>
&lt;li>Element&amp;rsquo;s nginx cannot bind port 80 as a rootless user. It has a setting for another port.&lt;/li>
&lt;li>Mailcow&amp;rsquo;s &lt;code>update.sh&lt;/code> makes local Git commits in its checkout, so my role checks out the release tag and not the branch.&lt;/li>
&lt;li>The first &lt;code>base&lt;/code> run locked out the &lt;code>ansible&lt;/code> user on a fresh host (the &lt;code>AllowUsers&lt;/code> issue in post 5). A reused SSH connection had hidden it during the first rehearsal.&lt;/li>
&lt;/ul>
&lt;p>Each of these would have cost me time during a real cut-over, with mail down.&lt;/p>
&lt;p>This was the last post of part 1. Part 2 is about SSH: &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://fedoraproject.org/cloud/">Fedora Cloud&lt;/a>: the qcow2 images used for the rehearsal VMs.&lt;/li>
&lt;li>&lt;a href="https://cloudinit.readthedocs.io/en/latest/reference/datasources/nocloud.html">cloud-init NoCloud datasource&lt;/a>: the seed that the custom image carries.&lt;/li>
&lt;li>&lt;a href="https://libvirt.org/docs.html">libvirt documentation&lt;/a>: domains, networks and storage.&lt;/li>
&lt;li>&lt;a href="https://github.com/virt-manager/virt-manager/blob/main/man/virt-install.rst">virt-install&lt;/a>: all options, including &lt;code>--cloud-init&lt;/code> and &lt;code>--import&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://libguestfs.org/virt-customize.1.html">virt-customize&lt;/a>: changes a disk image without booting it. An alternative to a build VM.&lt;/li>
&lt;li>&lt;a href="https://docs.fedoraproject.org/en-US/quick-docs/upgrading-fedora-offline/">Upgrading Fedora using the DNF system upgrade&lt;/a>: the upgrade path from the end of this post.&lt;/li>
&lt;/ul></description></item><item><title>Monitoring and backups for every host</title><link>https://www.linux-lunatix.org/blog/infra-07-monitoring-backup/</link><pubDate>Fri, 23 Oct 2026 20:20:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-07-monitoring-backup/</guid><description>&lt;p>&lt;em>Part 7 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>Every host gets the same two things from Ansible: a monitoring agent and a backup. This post shows both. The monitoring part is mostly about the network: how Prometheus at home reaches the VPS without exposing anything to the internet. The backup part is one script template that works for all hosts.&lt;/p>
&lt;h2 id="the-monitoring-network">The monitoring network&lt;/h2>
&lt;p>Prometheus runs on my home server. The VPS are somewhere on the internet. I did not want node_exporter&amp;rsquo;s port open to the world, and I did not want the VPS to be able to reach my home network.&lt;/p>
&lt;p>The solution is a WireGuard tunnel from each VPS to the home server. I call it &lt;code>wg-mon&lt;/code>.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">vps1 (10.99.0.13) ──┐
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">vps2 (10.99.0.12) ──┼── WireGuard ──► home1 wg0 (10.99.0.1) ── Prometheus
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">mail1 (10.99.0.11) ─┘
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Two firewall rules make it safe in both directions:&lt;/p>
&lt;ol>
&lt;li>On each VPS, the tunnel interface is in a firewalld zone of its own, and only port 9100 (node_exporter) is open there.&lt;/li>
&lt;li>On the home server, the VPS tunnel addresses land in firewalld&amp;rsquo;s &lt;code>block&lt;/code> zone. Prometheus can open connections to them, and the replies come back. A VPS cannot open a single connection to the home server.&lt;/li>
&lt;/ol>
&lt;p>If somebody takes over a VPS, they get a tunnel that leads nowhere.&lt;/p>
&lt;p>My own devices (laptop, phone) use the same &lt;code>wg0&lt;/code> interface as &amp;ldquo;road warriors&amp;rdquo;. Their addresses go into the &lt;code>trusted&lt;/code> zone, so they reach the LAN.&lt;/p>
&lt;h3 id="the-vps-side">The VPS side&lt;/h3>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2"># /etc/wireguard/wg-mon.conf
[Interface]
Address = {{ tunnel_ip }}/32
PostUp = wg set %i private-key /etc/wireguard/wg-mon.key
[Peer]
PublicKey = {{ wg_mon_server_public_key }}
Endpoint = {{ wg_mon_server_endpoint }}
AllowedIPs = {{ wg_mon_server_ip }}/32
PersistentKeepalive = 25
&lt;/code>&lt;/pre>&lt;p>The private key is in a separate file, written from pass with &lt;code>no_log&lt;/code>. The config file can then show in a diff without leaking the key. &lt;code>PersistentKeepalive&lt;/code> keeps the NAT mapping at home open, so that Prometheus can always reach the VPS.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Tunnel up&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">wg-quick@wg-mon&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">enabled&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">started&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">firewalld zone for the tunnel&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">present&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring_zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># A new zone exists only after a reload&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Load the new zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">firewall-cmd --reload&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring_zone.changed and not ansible_check_mode&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Tunnel interface in the zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">interface&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">wg-mon&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">node_exporter port in the zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">port&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">9100&lt;/span>&lt;span class="l">/tcp&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="the-hub-side-generated-from-the-inventory">The hub side, generated from the inventory&lt;/h3>
&lt;p>Each host&amp;rsquo;s public key and tunnel address are in the inventory (&lt;code>wg_mon_public_key&lt;/code>, &lt;code>tunnel_ip&lt;/code>). The hub&amp;rsquo;s config loops over all hosts, so a new VPS needs no extra step:&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2"># /etc/wireguard/wg0.conf on home1
[Interface]
PrivateKey = {{ lookup(&amp;#39;community.general.passwordstore&amp;#39;, wg_hub_key_pass) }}
Address = {{ wg_hub_address }}
ListenPort = {{ wg_hub_port }}
{% for p in wg_hub_peers %}
# {{ p.name }}
[Peer]
PublicKey = {{ p.public_key }}
PresharedKey = {{ lookup(&amp;#39;community.general.passwordstore&amp;#39;, p.psk_pass) }}
AllowedIPs = {{ p.ip }}/32
{% endfor %}
{% for h in groups[&amp;#39;all&amp;#39;] if h != inventory_hostname and hostvars[h].wg_mon_public_key is defined %}
# {{ h }} (monitoring)
[Peer]
PublicKey = {{ hostvars[h].wg_mon_public_key }}
AllowedIPs = {{ hostvars[h].tunnel_ip }}/32
{% endfor %}
&lt;/code>&lt;/pre>&lt;p>This file contains the private and preshared keys, so its task has &lt;code>no_log: true&lt;/code> and &lt;code>diff: false&lt;/code>. The handler reloads the interface with &lt;code>wg syncconf&lt;/code> (&lt;code>systemctl reload wg-quick@wg0&lt;/code>). Peers change without dropping the open tunnels.&lt;/p>
&lt;p>The firewall on the hub:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Road-warrior peers are trusted&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">trusted&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">source&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.ip }}/32&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ wg_hub_peers }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.name }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">All other wg0 traffic is blocked&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">block&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">interface&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">wg0&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>firewalld picks a zone by source address first, and by interface second. A packet from a road warrior&amp;rsquo;s address matches &lt;code>trusted&lt;/code>. Everything else on &lt;code>wg0&lt;/code> falls through to &lt;code>block&lt;/code>.&lt;/p>
&lt;p>A sender on the LAN could fake a road warrior&amp;rsquo;s address. The base role&amp;rsquo;s sysctl settings turn on strict reverse path filtering (&lt;code>net.ipv4.conf.all.rp_filter = 1&lt;/code>), and the kernel drops such packets.&lt;/p>
&lt;p>Sometimes one VPS must reach exactly one service at home. My CI runner needs one port for a later part of this series. For that the role has a list of exceptions, as firewalld rich rules in the &lt;code>block&lt;/code> zone:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/home1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">wg_hub_block_exceptions&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">source: 10.99.0.11, port&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">8443&lt;/span>&lt;span class="l">/tcp}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Exceptions in the block zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">block&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">rich_rule&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">&amp;gt;-&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> rule family=&amp;#34;ipv4&amp;#34; source address=&amp;#34;{{ item.source }}&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> port port=&amp;#34;{{ item.port.split(&amp;#39;/&amp;#39;)[0] }}&amp;#34; protocol=&amp;#34;{{ item.port.split(&amp;#39;/&amp;#39;)[1] }}&amp;#34; accept&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ wg_hub_block_exceptions }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="node-exporter">node_exporter&lt;/h3>
&lt;p>node_exporter runs as a root quadlet (&lt;code>/etc/containers/systemd/&lt;/code>), because it needs to see the whole host. It listens only on the tunnel address:&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2"># /etc/containers/systemd/node-exporter.container
[Unit]
Description=Prometheus node_exporter (monitoring tunnel address only)
Wants=wg-quick@wg-mon.service
After=wg-quick@wg-mon.service
[Container]
ContainerName=node-exporter
Image=quay.io/prometheus/node-exporter:latest
AutoUpdate=registry
Network=host
PodmanArgs=--pid=host
Volume=/:/host:ro,rslave
Volume=/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket:ro
# Confined: its own SELinux type (post 4)
SecurityLabelType=node_exporter.process
Exec=--path.rootfs=/host --web.listen-address={{ tunnel_ip }}:9100 --collector.systemd --collector.textfile.directory=/host/var/lib/node_exporter/textfile
[Service]
Restart=always
[Install]
WantedBy=multi-user.target
&lt;/code>&lt;/pre>&lt;p>The textfile directory is how scripts report to Prometheus. A script writes a &lt;code>.prom&lt;/code> file, and node_exporter serves its content. My backup script and a daily update check use it.&lt;/p>
&lt;h3 id="prometheus">Prometheus&lt;/h3>
&lt;p>On the home server the targets are a short static list:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># prometheus.yml (part)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">scrape_configs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">job_name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">node-remote&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">static_configs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">targets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="m">10.99.0.11&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="m">9100&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">labels&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">mail1}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">targets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="m">10.99.0.12&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="m">9100&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">labels&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps2}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">targets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="m">10.99.0.13&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="m">9100&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">labels&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps1}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The tunnel addresses come from the inventory, so you could render this list from &lt;code>hostvars&lt;/code> too. With four hosts I kept it static and readable.&lt;/p>
&lt;p>Alertmanager sends alerts to a self-hosted ntfy. That covers &amp;ldquo;a host is down&amp;rdquo;. It does not cover &amp;ldquo;the home server, where Prometheus and Alertmanager run, is down&amp;rdquo;. For that, the next section uses an external dead man&amp;rsquo;s switch.&lt;/p>
&lt;h2 id="backups-one-script-for-every-host">Backups: one script for every host&lt;/h2>
&lt;p>Each host backs up with &lt;a href="https://restic.net/">restic&lt;/a> to its own repository on an SFTP backup box. The hosts differ in what they must dump first (databases, Mailcow) and in what they exclude. Everything else is the same. So there is one script template, with the host-specific parts as variables.&lt;/p>
&lt;h3 id="the-settings">The settings&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/vps1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_repository&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sftp:backup-box:vps1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_password_pass&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps1/restic&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_time&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;03:15&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_shell_vars&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">STORAGE=$APP_HOME/.local/share/containers/storage&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_args&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- --&lt;span class="l">exclude &amp;#34;$STORAGE/overlay&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- --&lt;span class="l">exclude &amp;#34;$STORAGE/overlay-images&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">/etc&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">$APP_HOME&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">$DUMP_DIR&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The container image layers are excluded. They can be pulled again and are the largest part of the storage. The volumes stay in.&lt;/p>
&lt;h3 id="the-script">The script&lt;/h3>
&lt;p>This is the template, shortened a little:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># /usr/local/sbin/backup-{{ backup_name }}.sh, from the backup role&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -uo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">RESTIC_REPOSITORY&lt;/span>&lt;span class="o">={{&lt;/span> backup_repository &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">RESTIC_PASSWORD_FILE&lt;/span>&lt;span class="o">=&lt;/span>/root/.config/restic/&lt;span class="o">{{&lt;/span> backup_name &lt;span class="o">}}&lt;/span>.pass
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># systemd services have no $HOME, so restic needs an explicit cache dir&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">RESTIC_CACHE_DIR&lt;/span>&lt;span class="o">=&lt;/span>/var/cache/restic
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">HC_PING_URL&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>cat /root/.config/restic/&lt;span class="o">{{&lt;/span> backup_name &lt;span class="o">}}&lt;/span>.hc 2&amp;gt;/dev/null &lt;span class="o">||&lt;/span> &lt;span class="nb">true&lt;/span>&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">APP_USER&lt;/span>&lt;span class="o">={{&lt;/span> app_user &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">APP_HOME&lt;/span>&lt;span class="o">={{&lt;/span> app_home &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">{{&lt;/span> backup_shell_vars &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">DUMP_DIR&lt;/span>&lt;span class="o">=&lt;/span>/var/backups/&lt;span class="o">{{&lt;/span> backup_name &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">failed&lt;/span>&lt;span class="o">=()&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">START&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>date +%s&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">hc_ping&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="o">[&lt;/span> -n &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$HC_PING_URL&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">]&lt;/span> &lt;span class="o">||&lt;/span> &lt;span class="k">return&lt;/span> &lt;span class="m">0&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> curl -fsS -m &lt;span class="m">10&lt;/span> --retry &lt;span class="m">3&lt;/span> -o /dev/null --data-binary @&amp;lt;&lt;span class="o">(&lt;/span>tail -n &lt;span class="m">100&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$RUN_LOG&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="o">)&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$HC_PING_URL$1&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">||&lt;/span> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;healthchecks ping failed&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Run a step, note a failure, go on with the next step&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">step&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> &lt;span class="nv">name&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nv">$1&lt;/span> rc
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">shift&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;=== &lt;/span>&lt;span class="k">$(&lt;/span>date &lt;span class="s1">&amp;#39;+%F %T&amp;#39;&lt;/span>&lt;span class="k">)&lt;/span>&lt;span class="s2"> &lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$@&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">rc&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nv">$?&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="o">[&lt;/span> &lt;span class="nv">$rc&lt;/span> -ne &lt;span class="m">0&lt;/span> &lt;span class="o">]&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;!!! &lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2"> failed (exit &lt;/span>&lt;span class="nv">$rc&lt;/span>&lt;span class="s2">)&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">failed&lt;/span>&lt;span class="o">+=(&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="o">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">return&lt;/span> &lt;span class="nv">$rc&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">app_podman&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> runuser -u &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$APP_USER&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> -- env &lt;span class="nv">XDG_RUNTIME_DIR&lt;/span>&lt;span class="o">=&lt;/span>/run/user/&lt;span class="k">$(&lt;/span>id -u &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$APP_USER&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span> podman &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$@&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Podman secrets are in no volume and not in Git: keep them in the encrypted backup&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">secrets_dump&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> s
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">for&lt;/span> s in &lt;span class="k">$(&lt;/span>app_podman secret ls --format &lt;span class="s1">&amp;#39;{{.Name}}&amp;#39;&lt;/span>&lt;span class="k">)&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> app_podman secret inspect --showsecret --format &lt;span class="s1">&amp;#39;{{.SecretData}}&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$s&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt; &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$DUMP_DIR&lt;/span>&lt;span class="s2">/secrets/&lt;/span>&lt;span class="nv">$s&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">||&lt;/span> &lt;span class="k">return&lt;/span> &lt;span class="m">1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">exec&lt;/span> 9&amp;gt;/run/backup-&lt;span class="o">{{&lt;/span> backup_name &lt;span class="o">}}&lt;/span>.lock
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> ! flock -n 9&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;previous backup run still in progress&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> hc_ping /fail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">exit&lt;/span> &lt;span class="m">1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">hc_ping /start
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">umask&lt;/span> &lt;span class="m">077&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">mkdir -p &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$DUMP_DIR&lt;/span>&lt;span class="s2">/secrets&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">{{&lt;/span> backup_pre_steps &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">step &lt;span class="s2">&amp;#34;podman secrets&amp;#34;&lt;/span> secrets_dump
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">step &lt;span class="s2">&amp;#34;restic unlock&amp;#34;&lt;/span> restic unlock
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> step &lt;span class="s2">&amp;#34;restic backup&amp;#34;&lt;/span> restic backup --verbose&lt;span class="o">=&lt;/span>&lt;span class="m">0&lt;/span> --exclude-caches &lt;span class="o">{{&lt;/span> backup_args &lt;span class="p">|&lt;/span> join&lt;span class="o">(&lt;/span>&lt;span class="s1">&amp;#39; &amp;#39;&lt;/span>&lt;span class="o">)&lt;/span> &lt;span class="o">}}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> step &lt;span class="s2">&amp;#34;restic forget/prune&amp;#34;&lt;/span> restic forget --prune --keep-daily &lt;span class="m">7&lt;/span> --keep-weekly &lt;span class="m">4&lt;/span> --keep-monthly &lt;span class="m">6&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">step &lt;span class="s2">&amp;#34;restic check&amp;#34;&lt;/span> restic check
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">write_metrics &lt;span class="k">$((&lt;/span> &lt;span class="si">${#&lt;/span>&lt;span class="nv">failed&lt;/span>&lt;span class="p">[@]&lt;/span>&lt;span class="si">}&lt;/span> &lt;span class="o">==&lt;/span> &lt;span class="m">0&lt;/span> &lt;span class="k">))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> &lt;span class="o">[&lt;/span> &lt;span class="si">${#&lt;/span>&lt;span class="nv">failed&lt;/span>&lt;span class="p">[@]&lt;/span>&lt;span class="si">}&lt;/span> -gt &lt;span class="m">0&lt;/span> &lt;span class="o">]&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> hc_ping /fail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">exit&lt;/span> &lt;span class="m">1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">hc_ping &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The Go template syntax in &lt;code>secrets_dump&lt;/code> (&lt;code>{{.Name}}&lt;/code>) clashes with Jinja. The real template therefore starts with a header that changes Jinja&amp;rsquo;s comment markers and wraps those lines in &lt;code>{% raw %}&lt;/code>. I left that out here to keep the script readable.&lt;/p>
&lt;p>Some decisions in it:&lt;/p>
&lt;ul>
&lt;li>The script does not stop at the first error. A failed database dump should not prevent the file backup. Each failed step is collected, and the end result is &amp;ldquo;failed&amp;rdquo; if any step failed.&lt;/li>
&lt;li>&lt;code>flock&lt;/code> prevents two runs at the same time, for example when a run takes longer than the interval.&lt;/li>
&lt;li>&lt;code>restic unlock&lt;/code> removes stale locks from a run that was killed. &lt;code>restic check&lt;/code> at the end verifies the repository structure. It does not read all data, which would be slow over SFTP. &lt;code>restic check --read-data-subset=5%&lt;/code> reads a sample, if you want more.&lt;/li>
&lt;/ul>
&lt;h3 id="host-specific-steps">Host-specific steps&lt;/h3>
&lt;p>A host adds dump functions and steps as shell code in its variables:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/mail1.yml (part)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_functions&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> forgejo_dump() {
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> app_podman exec forgejo-postgres pg_dumpall -U forgejo | gzip &amp;gt; &amp;#34;$DUMP_DIR/db/forgejo.sql.gz&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_pre_steps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> step &amp;#34;forgejo db&amp;#34; forgejo_dump&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="healthchecks-dot-io-the-dead-man-s-switch">healthchecks.io: the dead man&amp;rsquo;s switch&lt;/h3>
&lt;p>The script pings &lt;a href="https://healthchecks.io">healthchecks.io&lt;/a> at the start, at the end, and with &lt;code>/fail&lt;/code> on errors. The service alerts me when a ping is late. So I also hear about a backup that never started, because the host is down or the timer broke. Monitoring on the same host could never report that. The last 100 lines of the run&amp;rsquo;s log go with the ping, so the alert e-mail shows what failed.&lt;/p>
&lt;h3 id="metrics-for-prometheus">Metrics for Prometheus&lt;/h3>
&lt;p>At the end the script writes &lt;code>/var/lib/node_exporter/textfile/backup.prom&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">backup_last_run_timestamp_seconds 1761246000
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">backup_last_success_timestamp_seconds 1761246000
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">backup_last_run_success 1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">backup_last_duration_seconds 312
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>It writes to a temporary file first and then renames it. node_exporter never reads a half-written file. A Grafana overview shows the age of the last good backup per host. A failed run also makes the systemd unit fail, and an alert rule for failed units catches that.&lt;/p>
&lt;h3 id="the-timer">The timer&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># /etc/systemd/system/backup-vps1.timer&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Unit]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Description&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">Daily backup of vps1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Timer]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">OnCalendar&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">*-*-* 03:15&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">RandomizedDelaySec&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">15min&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Persistent&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">timers.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>Persistent=true&lt;/code> runs a missed backup after the host was off at 03:15.&lt;/p>
&lt;h2 id="restore-at-least-once">Restore, at least once&lt;/h2>
&lt;p>A backup that was never restored is a hope. I restored each host completely during the rehearsals in the next post. The restore from the old Arch host reported errors, because restic tried to set SELinux labels that the backup never had (see post 4). The files were fine.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">R&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;restic -r sftp:backup-box:vps1 --password-file /root/.config/restic/vps1.pass --no-lock&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">$R&lt;/span> snapshots
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">$R&lt;/span> restore latest --target /var/tmp/restore --include /home/svc/traefik/acme --exclude-xattr security.selinux
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>--no-lock&lt;/code> lets a test VM read the live repository without writing a lock into it.&lt;/p>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo systemctl start backup-vps1.service
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo tail -20 /var/log/backup-vps1.log
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -s http://10.99.0.13:9100/metrics &lt;span class="p">|&lt;/span> grep ^backup_ &lt;span class="c1"># from the hub&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Next: &lt;a href="../../blog/infra-08-rehearsal/">Rehearse before you reinstall&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://www.wireguard.com/quickstart/">WireGuard quick start&lt;/a>: keys, peers and &lt;code>wg-quick&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://firewalld.org/documentation/">firewalld documentation&lt;/a>: zones and policies, for the hub&amp;rsquo;s &lt;code>block&lt;/code> zone.&lt;/li>
&lt;li>&lt;a href="https://prometheus.io/docs/introduction/overview/">Prometheus documentation&lt;/a>: scrape configs, rules and alerts.&lt;/li>
&lt;li>&lt;a href="https://github.com/prometheus/node_exporter">node_exporter&lt;/a>: the collectors and the textfile directory.&lt;/li>
&lt;li>&lt;a href="https://github.com/prometheus/blackbox_exporter">blackbox_exporter&lt;/a>: HTTP, TCP and ICMP probes.&lt;/li>
&lt;li>&lt;a href="https://restic.readthedocs.io/en/stable/">restic documentation&lt;/a>: repositories, retention with &lt;code>forget --prune&lt;/code>, checks and restore.&lt;/li>
&lt;li>&lt;a href="https://healthchecks.io/docs/">healthchecks.io documentation&lt;/a>: pings, start and fail signals, and grace times.&lt;/li>
&lt;/ul></description></item><item><title>One generic Ansible role for every app</title><link>https://www.linux-lunatix.org/blog/infra-06-quadlet-app/</link><pubDate>Sun, 18 Oct 2026 16:30:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-06-quadlet-app/</guid><description>&lt;p>&lt;em>Part 6 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>My hosts run close to 30 apps, some of them with several containers. I did not want 30 Ansible roles. This post shows the one role that deploys all of them, and how I used it to take over running hosts without restarting anything.&lt;/p>
&lt;h2 id="the-idea">The idea&lt;/h2>
&lt;p>Each app is a directory:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">apps/forgejo/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── app.yml # optional: secrets, directories, units
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── quadlets/ # -&amp;gt; ~svc/.config/containers/systemd/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo.container
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo-postgres.container
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo.network
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo-data.volume
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── forgejo-db.volume
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── files/ # -&amp;gt; ~svc/ (same relative paths)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── .config/forgejo/...
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">└── tasks.yml # optional: extra tasks (downloads, checkouts)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A host lists its apps:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/vps1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">app_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">svc&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">apps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">traefik, ntfy, forgejo-runner]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The role &lt;code>quadlet_app&lt;/code> loops over the list. For each app it does the same steps:&lt;/p>
&lt;ol>
&lt;li>Read &lt;code>app.yml&lt;/code>.&lt;/li>
&lt;li>Create the Podman secrets it names, from pass.&lt;/li>
&lt;li>Copy &lt;code>files/&lt;/code> and &lt;code>quadlets/&lt;/code> into the app user&amp;rsquo;s home. Files that end in &lt;code>.j2&lt;/code> are templates.&lt;/li>
&lt;li>If anything changed: run &lt;code>daemon-reload&lt;/code> and restart the app&amp;rsquo;s units.&lt;/li>
&lt;li>Make sure the units run.&lt;/li>
&lt;/ol>
&lt;p>An app that did not change is not touched. That last point matters most. A playbook run on a host with 20 apps restarts exactly the apps I changed.&lt;/p>
&lt;h2 id="why-plain-files-and-not-a-module">Why plain files and not a module&lt;/h2>
&lt;p>&lt;code>containers.podman.podman_container&lt;/code> can create containers from Ansible, and newer versions can write quadlets too. I chose plain quadlet files in the repository instead:&lt;/p>
&lt;ul>
&lt;li>They are the same files that are on the host. &lt;code>diff&lt;/code> shows exactly what changes.&lt;/li>
&lt;li>I can copy one to a test machine and run it without Ansible.&lt;/li>
&lt;li>The Podman documentation for quadlets applies 1:1. There is no translation layer whose options I must look up.&lt;/li>
&lt;/ul>
&lt;h2 id="app-dot-yml">app.yml&lt;/h2>
&lt;p>Most apps need no &lt;code>app.yml&lt;/code> at all. When one is there, it can say:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># apps/forgejo/app.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">secrets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">name: forgejo-db-password, pass&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">infra/&amp;lt;host&amp;gt;/forgejo-db-password}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># apps/traefik/app.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">secrets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">name: dns-api-token, pass: infra/&amp;lt;host&amp;gt;/dns-api-token, if&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">traefik_dns_challenge}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">dirs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">path: .config/traefik, mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0700&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">traefik}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">path: traefik/acme, mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0700&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Key&lt;/th>
&lt;th>Meaning&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>secrets&lt;/code>&lt;/td>
&lt;td>Podman secrets: name, pass entry (&lt;code>&amp;lt;host&amp;gt;&lt;/code> is replaced), optional &lt;code>if&lt;/code> (a host variable that must be true)&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>secret_files&lt;/code>&lt;/td>
&lt;td>files from &lt;code>files/&lt;/code> with mode 0600 that never show up in a diff&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>dirs&lt;/code>&lt;/td>
&lt;td>directories to create, with an optional mode&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>units&lt;/code>&lt;/td>
&lt;td>the units to restart; default: one &lt;code>.service&lt;/code> per &lt;code>.container&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>enable&lt;/code>&lt;/td>
&lt;td>plain user units or timers to enable and start&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>restart&lt;/code>&lt;/td>
&lt;td>&lt;code>restart&lt;/code>, or &lt;code>stop-start&lt;/code> for containers that share a network namespace&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;h2 id="the-role-step-by-step">The role, step by step&lt;/h2>
&lt;p>The role&amp;rsquo;s &lt;code>main.yml&lt;/code> creates the quadlet directory once, then includes &lt;code>app.yml&lt;/code> for each app:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># roles/quadlet_app/tasks/main.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Quadlet directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_home }}/{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">owner&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">group&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">.config, .config/containers, .config/containers/systemd]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Apps&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.include_tasks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ apps }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_var&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_name&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>app_home&lt;/code>, &lt;code>app_uid&lt;/code> and &lt;code>app_gid&lt;/code> come from the &lt;code>podman&lt;/code> role, which creates the user, enables linger and starts the user&amp;rsquo;s &lt;code>podman.socket&lt;/code> and &lt;code>podman-auto-update.timer&lt;/code>.&lt;/p>
&lt;h3 id="read-the-definition-and-the-source-tree">Read the definition and the source tree&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: definition&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.set_fact&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_defaults | combine(app_file | from_yaml or {}) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ apps_dir }}/{{ app_name }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vars&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;ansible.builtin.file&amp;#39;, apps_dir ~ &amp;#39;/&amp;#39; ~ app_name ~ &amp;#39;/app.yml&amp;#39;, errors=&amp;#39;ignore&amp;#39;) | default(&amp;#39;&amp;#39;, true) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_defaults&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">secrets: [], secret_files: [], dirs: [], units: null, enable: [], restart&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">restart}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: source tree&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.set_fact&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_tree&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">files&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.filetree&amp;#39;, app_src ~ &amp;#39;/files&amp;#39;, wantlist=true) | selectattr(&amp;#39;state&amp;#39;, &amp;#39;eq&amp;#39;, &amp;#39;file&amp;#39;) | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dirs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.filetree&amp;#39;, app_src ~ &amp;#39;/files&amp;#39;, wantlist=true) | selectattr(&amp;#39;state&amp;#39;, &amp;#39;eq&amp;#39;, &amp;#39;directory&amp;#39;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> | map(attribute=&amp;#39;path&amp;#39;) | map(&amp;#39;community.general.dict_kv&amp;#39;, &amp;#39;path&amp;#39;) | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">quadlets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.filetree&amp;#39;, app_src ~ &amp;#39;/quadlets&amp;#39;, wantlist=true) | selectattr(&amp;#39;state&amp;#39;, &amp;#39;eq&amp;#39;, &amp;#39;file&amp;#39;) | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>filetree&lt;/code> lookup walks a directory on the control node and returns every file with its relative path and mode. That is all the role needs to know about an app.&lt;/p>
&lt;h3 id="secrets">Secrets&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: podman secrets&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">containers.podman.podman_secret&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.name }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">data&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.passwordstore&amp;#39;, item.pass | replace(&amp;#39;&amp;lt;host&amp;gt;&amp;#39;, inventory_hostname)) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">present&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app.secrets }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.name }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">item.if is not defined or lookup(&amp;#39;ansible.builtin.vars&amp;#39;, item.if) | bool&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">become_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">XDG_RUNTIME_DIR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/run/user/{{ app_uid }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">no_log&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_secrets_result&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>become_user&lt;/code> plus &lt;code>XDG_RUNTIME_DIR&lt;/code> is how every task in this role talks to the rootless user&amp;rsquo;s Podman and systemd. &lt;code>become_user&lt;/code> to an unprivileged user needs the &lt;code>acl&lt;/code> package on the host, so that Ansible can hand its temporary files to that user. The base role installs it.&lt;/p>
&lt;h3 id="files-and-quadlets">Files and quadlets&lt;/h3>
&lt;p>Plain files are copied, and &lt;code>.j2&lt;/code> files are rendered. Executables stay executable. Secret files get mode 0600 and no diff:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: files&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.src }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_home }}/{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">owner&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">group&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ &amp;#39;0600&amp;#39; if item.path in app.secret_files else (&amp;#39;0755&amp;#39; if item.mode[1] in &amp;#39;157&amp;#39; else &amp;#39;0644&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_tree.files | rejectattr(&amp;#39;path&amp;#39;, &amp;#39;search&amp;#39;, &amp;#39;\\.j2$&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">no_log&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path in app.secret_files }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_files_result&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: templated files&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.template&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.src }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_home }}/{{ item.path | regex_replace(&amp;#39;\\.j2$&amp;#39;, &amp;#39;&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">owner&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">group&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_tree.files | selectattr(&amp;#39;path&amp;#39;, &amp;#39;search&amp;#39;, &amp;#39;\\.j2$&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_templates_result&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The quadlets work the same way, with &lt;code>.config/containers/systemd/&lt;/code> as the destination. Why are only &lt;code>.j2&lt;/code> files templates? Some apps ship files that contain their own &lt;code>{{ }}&lt;/code> syntax: Grafana dashboards, Prometheus alert rules. Rendering those with Jinja would break them. So a file is a template only when its name says so.&lt;/p>
&lt;p>A quadlet template is useful for values that differ per host:&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2"># apps/traefik/quadlets/traefik.container.j2 (part)
UserNS=keep-id:uid={{ app_uid }},gid={{ app_gid }}
User={{ app_uid }}:{{ app_gid }}
{% for v in traefik_extra_volumes | default([]) %}
Volume={{ v }}
{% endfor %}
{% if traefik_dns_challenge %}
Secret=dns-api-token,type=env,target=CF_DNS_API_TOKEN
{% endif %}
&lt;/code>&lt;/pre>&lt;h3 id="restart-only-what-changed">Restart only what changed&lt;/h3>
&lt;p>Every task above registers its result. The role collects all of them and asks one question: did anything change for this app?&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: units&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.set_fact&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_units&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app.units if app.units is not none else
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> (app_tree.quadlets | map(attribute=&amp;#39;path&amp;#39;) | map(&amp;#39;regex_replace&amp;#39;, &amp;#39;\\.j2$&amp;#39;, &amp;#39;&amp;#39;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> | select(&amp;#39;match&amp;#39;, &amp;#39;.*\\.container$&amp;#39;) | map(&amp;#39;regex_replace&amp;#39;, &amp;#39;\\.container$&amp;#39;, &amp;#39;.service&amp;#39;) | list) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_changed&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ (app_secrets_result.results + app_files_result.results + app_templates_result.results
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> + app_quadlets_result.results + app_quadlet_templates_result.results
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> + (app_extra_results | default([]))) | selectattr(&amp;#39;changed&amp;#39;, &amp;#39;defined&amp;#39;) | selectattr(&amp;#39;changed&amp;#39;) | list | length &amp;gt; 0 }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: reload and restart&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_changed&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">become_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">XDG_RUNTIME_DIR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/run/user/{{ app_uid }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">block&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: daemon-reload&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">daemon_reload&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: restart&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">restarted&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.restart == &amp;#39;restart&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># Containers that share another&amp;#39;s network namespace must be gone before it can be&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># replaced: stop in reverse order, then start in order&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: stop (reverse order)&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">stopped&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units | reverse | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.restart == &amp;#39;stop-start&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: start&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">started&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.restart == &amp;#39;stop-start&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: running&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">started&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">become_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">XDG_RUNTIME_DIR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/run/user/{{ app_uid }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Why not handlers? Handlers run at the end of the play, and they run once per handler name, however many apps notified it. I want the restart right after the app&amp;rsquo;s own files changed, and only for that app&amp;rsquo;s units. Handlers cannot carry &amp;ldquo;which app&amp;rdquo; without tricks. A plain &lt;code>when: app_changed&lt;/code> is easier to read.&lt;/p>
&lt;p>Restarting is also how a container sees a changed config file. Ansible writes a new file (a new inode), and a running container keeps the old one in its mount.&lt;/p>
&lt;h3 id="extra-tasks">Extra tasks&lt;/h3>
&lt;p>Some apps need more than files: a Git checkout to build from, a binary to download. An optional &lt;code>tasks.yml&lt;/code> in the app directory is included, and it can add its results to &lt;code>app_extra_results&lt;/code> to trigger the restart:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: extra tasks&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.include_tasks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_src }}/tasks.yml&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">(app_src ~ &amp;#39;/tasks.yml&amp;#39;) is file&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="order-matters">Order matters&lt;/h2>
&lt;p>Apps run in the order of the &lt;code>apps&lt;/code> list. If one app provides a file that another app mounts, the provider must come first. On a fresh host, Traefik fails to start if a password file it mounts does not exist yet:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># the landing page app provides the htpasswd file that Traefik mounts&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">apps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">landing, traefik, ntfy, forgejo-runner]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">traefik_extra_volumes&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;%h/.config/traefik/landing_htpasswd:/landing_htpasswd:ro,Z&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="adopting-a-running-host">Adopting a running host&lt;/h2>
&lt;p>This is how I moved the existing quadlets of a running host into the role without a single restart:&lt;/p>
&lt;ol>
&lt;li>Copy the host&amp;rsquo;s quadlets and config files into &lt;code>apps/&amp;lt;name&amp;gt;/&lt;/code>, one app at a time.&lt;/li>
&lt;li>Run check mode with a diff against the live host:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l home1 -t apps --check --diff
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>Every difference is one of two things. Either the repository is wrong (fix the file), or the host has a change that was never written down (decide which version is right).&lt;/li>
&lt;li>Repeat until the run shows &lt;code>changed=0&lt;/code> in check mode.&lt;/li>
&lt;li>Run it for real. Nothing changes, so nothing restarts.&lt;/li>
&lt;/ol>
&lt;p>The loop also shows what nobody wrote down: an option changed by hand on the host, a file that exists only there. Each of those is now either in the repository or gone.&lt;/p>
&lt;h2 id="the-test-changed-0">The test: changed=0&lt;/h2>
&lt;p>After every change I run the playbook twice. The second run must report &lt;code>changed=0&lt;/code>. A task that reports a change every time hides the real changes, and it may restart an app on every run. Typical causes:&lt;/p>
&lt;ul>
&lt;li>&lt;code>command&lt;/code> or &lt;code>shell&lt;/code> tasks without &lt;code>changed_when&lt;/code>.&lt;/li>
&lt;li>&lt;code>file: state=touch&lt;/code> without &lt;code>modification_time: preserve&lt;/code>.&lt;/li>
&lt;li>A template that renders a timestamp or a random value.&lt;/li>
&lt;li>A module that compares in a different format than it writes (for example YAML key order).&lt;/li>
&lt;/ul>
&lt;p>Run it with fresh SSH connections, too (&lt;code>-o ControlPath=none&lt;/code>). The reason is in &lt;a href="../../blog/infra-05-ansible-base/">the previous post&lt;/a>.&lt;/p>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t apps
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t apps &lt;span class="c1"># changed=0&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh vps1 sudo -u svc &lt;span class="nv">XDG_RUNTIME_DIR&lt;/span>&lt;span class="o">=&lt;/span>/run/user/1001 systemctl --user list-units &lt;span class="s1">&amp;#39;traefik*&amp;#39;&lt;/span> &lt;span class="s1">&amp;#39;ntfy*&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Next: &lt;a href="../../blog/infra-07-monitoring-backup/">Monitoring and backups for every host&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_reuse_roles.html">Ansible roles&lt;/a>: the role layout, defaults and dependencies.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_handlers.html">Handlers&lt;/a>: notify, listen and &lt;code>flush_handlers&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_checkmode.html">Check mode and diff mode&lt;/a>: what &lt;code>--check --diff&lt;/code> can and cannot predict.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/collections/containers/podman/index.html">containers.podman collection&lt;/a>: the modules for secrets, images and containers. The alternative to plain files.&lt;/li>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html">podman-systemd.unit(5)&lt;/a>: the quadlet reference again, because the role only copies these files.&lt;/li>
&lt;li>&lt;a href="https://redhat-cop.github.io/automation-good-practices/">Good practices for Ansible (Red Hat CoP)&lt;/a>: the chapter on roles fits this post.&lt;/li>
&lt;/ul></description></item><item><title>Ansible groundwork: inventory, bootstrap and a base role</title><link>https://www.linux-lunatix.org/blog/infra-05-ansible-base/</link><pubDate>Wed, 14 Oct 2026 18:55:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-05-ansible-base/</guid><description>&lt;p>&lt;em>Part 5 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>This post sets up the parts that every host shares. At the end you have a control node, an inventory, a playbook for the first contact with a host, and a &lt;code>base&lt;/code> role. That role hardens sshd, runs firewalld and fail2ban, installs updates every night and caps the journal.&lt;/p>
&lt;h2 id="the-control-node">The control node&lt;/h2>
&lt;p>I run Ansible from my laptop, and from a second machine as a spare. Both use the same pinned versions in a virtualenv. The distribution&amp;rsquo;s Ansible package changes with each upgrade, and I want the same behaviour on both machines:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">python3 -m venv ~/.local/share/ops-venv
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">~/.local/share/ops-venv/bin/pip install &lt;span class="s2">&amp;#34;ansible-core==2.22.0&amp;#34;&lt;/span> ansible-lint yamllint
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">PATH&lt;/span>&lt;span class="o">=&lt;/span>~/.local/share/ops-venv/bin:&lt;span class="nv">$PATH&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The collections are pinned too, and installed into the repository:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># requirements.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">collections&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ansible.posix&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">version&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;2.2.2&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">community.general&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">version&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;14.0.0&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">containers.podman&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">version&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;1.21.2&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-galaxy collection install -r requirements.yml -p collections
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Each control node has its own key for Ansible, &lt;code>~/.ssh/ansible_ed25519&lt;/code>, separate from my login key. I can then revoke one machine without touching the other.&lt;/p>
&lt;h2 id="ansible-dot-cfg">ansible.cfg&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="k">[defaults]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">inventory&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">inventory/hosts.yml&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">roles_path&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">roles&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">collections_path&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">collections&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">remote_user&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">ansible&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">private_key_file&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">~/.ssh/ansible_ed25519&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">interpreter_python&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">auto_silent&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">callback_result_format&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">yaml&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">forks&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">4&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Handlers run even when a later task fails&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">force_handlers&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">True&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[privilege_escalation]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">become&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">True&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[diff]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">always&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">True&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[ssh_connection]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">pipelining&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">True&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Two settings are worth an explanation.&lt;/p>
&lt;p>&lt;code>force_handlers&lt;/code> runs the notified handlers even if a later task fails. Without it, this can happen: a task changes the sshd config and notifies &amp;ldquo;Reload sshd&amp;rdquo;, and then a later task fails. The handler never runs. On the next run the config file is unchanged, so nothing notifies the handler again. sshd runs with the old config until somebody notices.&lt;/p>
&lt;p>&lt;code>diff always&lt;/code> shows every change as a diff. Together with &lt;code>--check&lt;/code> this makes a run a review: I see what would change before it changes.&lt;/p>
&lt;h2 id="the-inventory">The inventory&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># inventory/hosts.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">all&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mail1&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">mail1.example.org&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bootstrap_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">admin&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">tunnel_ip&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">10.99.0.11&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vps1&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps1.example.org&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bootstrap_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fedora&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">tunnel_ip&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">10.99.0.13&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">svc&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vps2&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps2.example.org&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bootstrap_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fedora&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">tunnel_ip&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">10.99.0.12&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">home1&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">home1.example.net&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bootstrap_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">admin&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">tunnel_ip&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">10.99.0.1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">svc&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">children&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">mail1: , vps1: , vps2&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">home&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">home1&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mail&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">mail1&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">websites&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">vps2&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># The hosts that site.yml manages. A host joins after its reinstall with Fedora.&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fedora&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">vps1: , home1: , vps2&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>fedora&lt;/code> group is how I adopted the hosts one by one. A host that still ran Ubuntu or Arch was in the inventory, but &lt;code>site.yml&lt;/code> did not touch it. It joined the group after its reinstall.&lt;/p>
&lt;p>Everything else goes into &lt;code>group_vars/&lt;/code> and &lt;code>host_vars/&lt;/code>. A host file says what is special about that host:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># inventory/host_vars/vps1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">hostname&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps1.example.org&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">timezone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Europe/Berlin&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_allow_users&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">fedora, svc]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">firewall_public_services&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">http, https, ssh]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">dnf_automatic_reboot&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">when-needed&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">unprivileged_port_start&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">80&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">apps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">traefik, ntfy, forgejo-runner]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="secrets-from-pass">Secrets from pass&lt;/h2>
&lt;p>I keep all secrets in &lt;a href="https://www.passwordstore.org/">pass&lt;/a> on the control node. The lookup plugin reads them during the run:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Tunnel private key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/wireguard/wg-mon.key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.passwordstore&amp;#39;, &amp;#39;infra/&amp;#39; ~ inventory_hostname ~ &amp;#39;/wg-mon-key&amp;#39;) }}\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0600&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">no_log&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">diff&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>no_log&lt;/code> and &lt;code>diff: false&lt;/code> keep the secret out of the output. Without &lt;code>diff: false&lt;/code>, &lt;code>diff always&lt;/code> would print the old and the new key.&lt;/p>
&lt;p>Compared with Ansible Vault, the repository contains nothing encrypted at all, and there is no vault password to share between the control nodes. A secret is changed in one place, in pass, and the next run rolls it out.&lt;/p>
&lt;h2 id="first-contact-bootstrap-dot-yml">First contact: bootstrap.yml&lt;/h2>
&lt;p>A fresh host has one admin account that the provider set up (&lt;code>fedora&lt;/code> on a Fedora Cloud image). &lt;code>bootstrap.yml&lt;/code> logs in as that account once and creates the &lt;code>ansible&lt;/code> user. Every other playbook then uses that user.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># playbooks/bootstrap.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Bootstrap the ansible user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">all&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">remote_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ bootstrap_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vars&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_ssh_private_key_file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">~/.ssh/id_ed25519&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">tasks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Ansible user exists&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">comment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">shell&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/bin/bash&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">system&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">create_home&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Only the control-node keys may log in&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.authorized_key&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">key&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ansible_authorized_keys | join(&amp;#39;\n&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">exclusive&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># The dnf module needs it, and Fedora Server does not ship it&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">dnf bindings for Ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">dnf install -y python3-libdnf5&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">args&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">creates&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/usr/lib64/python3*/site-packages/libdnf5&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ansible_facts.pkg_mgr == &amp;#39;dnf5&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Passwordless sudo for ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/sudoers.d/ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;ansible ALL=(ALL) NOPASSWD: ALL\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0440&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">validate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">visudo -cf %s&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Read effective sshd settings&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd -T&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd_effective&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># AllowUsers lines add up; only needed where such a list is in force&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Allow ansible over SSH where AllowUsers is set&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/sshd_config.d/01-ansible.conf&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;AllowUsers ansible\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0600&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">validate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd -t -f %s&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd_effective.stdout_lines | select(&amp;#39;match&amp;#39;, &amp;#39;^allowusers &amp;#39;) | list | length &amp;gt; 0&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">handlers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">reloaded&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/bootstrap.yml -l vps1 &lt;span class="c1"># the admin has passwordless sudo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/bootstrap.yml -l vps1 -K &lt;span class="c1"># sudo asks for a password&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible vps1 -m ping
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>validate:&lt;/code> is your friend for every file that can lock you out. Ansible writes the new file to a temporary path, runs the command on it, and installs it only if the command succeeds. A sudoers file with a typo can no longer break sudo.&lt;/p>
&lt;h2 id="site-dot-yml">site.yml&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># playbooks/site.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Fedora hosts&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fedora&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">roles&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: base, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">base}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: podman, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">podman, apps, backup]}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: quadlet_app, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">apps}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: monitoring_agent, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: backup, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">backup}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Mail server&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">mail:&amp;amp;fedora&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">roles&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: mailcow, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">docker, mailcow]}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>mail:&amp;amp;fedora&lt;/code> is the intersection of the two groups. The mail host gets its Mailcow role only after it runs Fedora.&lt;/p>
&lt;p>I always run it like this:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 --check &lt;span class="c1"># what would change&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t base &lt;span class="c1"># only one part&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="the-base-role">The base role&lt;/h2>
&lt;p>The base role does the same things on every host. It starts with two checks: the host runs Fedora, and SELinux is enforcing (see &lt;a href="../../blog/infra-04-selinux/">the previous post&lt;/a>). Then it does the actual work.&lt;/p>
&lt;h3 id="sshd">sshd&lt;/h3>
&lt;p>One drop-in file. Fedora&amp;rsquo;s &lt;code>sshd_config&lt;/code> reads &lt;code>/etc/ssh/sshd_config.d/*.conf&lt;/code> first, and for most options the first value wins. So a file called &lt;code>00-hardening.conf&lt;/code> overrides the distribution&amp;rsquo;s defaults in &lt;code>50-redhat.conf&lt;/code>.&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2"># /etc/ssh/sshd_config.d/00-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
GSSAPIAuthentication no
{% if ssh_allow_users %}
AllowUsers {{ (ssh_allow_users + [&amp;#39;ansible&amp;#39;]) | unique | join(&amp;#39; &amp;#39;) }}
{% endif %}
# Ed25519 only; post-quantum key exchange first; AEAD ciphers and ETM MACs only
HostKey /etc/ssh/ssh_host_ed25519_key
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
PubkeyAcceptedAlgorithms ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com
KexAlgorithms mlkem768x25519-sha256,sntrup761x25519-sha512,curve25519-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com
X11Forwarding no
AllowAgentForwarding no
AllowStreamLocalForwarding no
PermitTunnel no
AllowTcpForwarding no
MaxAuthTries 3
LoginGraceTime 20
LogLevel VERBOSE
&lt;/code>&lt;/pre>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SSH hardening&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.template&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd-hardening.conf.j2&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/sshd_config.d/00-hardening.conf&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">validate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd -t -f %s&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Do not put &lt;code>Match&lt;/code> blocks into such a drop-in. A &lt;code>Match&lt;/code> block lasts until the end of the whole configuration, so it would also catch the rest of the main &lt;code>sshd_config&lt;/code>.&lt;/p>
&lt;p>The &lt;code>AllowUsers&lt;/code> line always adds &lt;code>ansible&lt;/code>. The first version did not. On a fresh test host, the first &lt;code>base&lt;/code> run then locked the &lt;code>ansible&lt;/code> user out of the host. I did not notice at first, because Ansible kept using its open SSH connection. More about that below.&lt;/p>
&lt;p>Check what sshd really uses with &lt;code>sudo sshd -T | grep -i allowusers&lt;/code>.&lt;/p>
&lt;h3 id="fail2ban">fail2ban&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># /etc/fail2ban/jail.d/sshd.local&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[DEFAULT]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ignoreip&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">127.0.0.1/8 ::1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[sshd]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">enabled&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">backend&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">systemd&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># aggressive: also scanners that disconnect before auth or fail the key exchange&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">mode&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">aggressive&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">maxretry&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">3&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">findtime&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">10m&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">bantime&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">1h&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">bantime.increment&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">bantime.maxtime&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">1w&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[recidive]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">enabled&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">bantime&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">1w&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">findtime&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">1d&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">maxretry&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">5&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>recidive&lt;/code> jail reads fail2ban&amp;rsquo;s own log file. On a fresh host that file does not exist yet, and the jail, and with it fail2ban, fails to start. So the role creates the file first:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fail2ban log file for the recidive jail&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/var/log/fail2ban.log&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">touch&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0600&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">setype&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fail2ban_log_t&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">modification_time&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">preserve&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">access_time&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">preserve&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>modification_time: preserve&lt;/code> keeps the task idempotent. A plain &lt;code>state: touch&lt;/code> reports a change on every run.&lt;/p>
&lt;h3 id="firewalld">firewalld&lt;/h3>
&lt;p>Only what the host lists in &lt;code>firewall_public_services&lt;/code> is open in the default zone:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Open services in the default zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ firewall_public_services }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>permanent&lt;/code> and &lt;code>immediate&lt;/code> together write the permanent config and apply it at once. No reload is needed, and a reboot changes nothing.&lt;/p>
&lt;h3 id="updates-every-night">Updates every night&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">dnf5-automatic settings&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">community.general.ini_file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/dnf/automatic.conf&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">section&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.section }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">option&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.option }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.value }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: commands, option: apply_updates, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;yes&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: commands, option: download_updates, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;yes&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: commands, option: upgrade_type, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">default}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: commands, option: reboot, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ dnf_automatic_reboot }}&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: commands, option: reboot_command, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;\&amp;#34;shutdown -r +5 &amp;#39;Rebooting after applying automatic updates&amp;#39;\&amp;#34;&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: emitters, option: emit_via, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">stdio}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">dnf5-automatic timer&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">dnf5-automatic.timer&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">enabled&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">started&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>reboot&lt;/code> is &lt;code>when-needed&lt;/code> on the VPS (a reboot after a new kernel or glibc) and &lt;code>never&lt;/code> on the home server, which I reboot myself.&lt;/p>
&lt;h3 id="a-cap-for-the-journal">A cap for the journal&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Journal size cap&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/systemd/journald.conf.d/00-size.conf&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> [Journal]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> SystemMaxUse=500M&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Restart journald&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Fedora&amp;rsquo;s default is 10% of the filesystem, up to 4 GB. On a small VPS with chatty containers that is a lot. The old Ubuntu mail host had 2.3 GB of journal when its root filesystem reached 91%.&lt;/p>
&lt;h2 id="things-to-watch">Things to watch&lt;/h2>
&lt;ul>
&lt;li>Ansible reuses its SSH connections (&lt;code>ControlPersist&lt;/code>). After a change to sshd, a run that passes may still use the old connection. Close the connections before you test such a change:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="k">for&lt;/span> s in ~/.ansible/cp/*&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span> ssh -O &lt;span class="nb">exit&lt;/span> -o &lt;span class="nv">ControlPath&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$s&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> x 2&amp;gt;/dev/null&lt;span class="p">;&lt;/span> &lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>Update a fresh host and reboot it before the first full run. Otherwise dnf5-automatic installs a new kernel during the rollout and schedules a reboot in five minutes. From then on, &lt;code>pam_nologin&lt;/code> lets only root log in, and the run fails halfway.&lt;/li>
&lt;li>&lt;code>--check&lt;/code> mode has limits. A task that depends on a package installed earlier in the same run fails in check mode, because the package is not there yet. Use &lt;code>when: not ansible_check_mode&lt;/code> for such tasks, or accept the error on a host&amp;rsquo;s very first check run.&lt;/li>
&lt;/ul>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t base
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t base &lt;span class="c1"># the second run: changed=0&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh -o &lt;span class="nv">ControlMaster&lt;/span>&lt;span class="o">=&lt;/span>no -o &lt;span class="nv">ControlPath&lt;/span>&lt;span class="o">=&lt;/span>none ansible@vps1.example.org &lt;span class="nb">true&lt;/span> &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> &lt;span class="nb">echo&lt;/span> ok
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Next: &lt;a href="../../blog/infra-06-quadlet-app/">One generic Ansible role for every app&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/tips_tricks/ansible_tips_tricks.html">Ansible tips and tricks&lt;/a>: the official list of good practices.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/tips_tricks/sample_setup.html">Sample Ansible setup&lt;/a>: a directory layout for inventories, group_vars and roles.&lt;/li>
&lt;li>&lt;a href="https://redhat-cop.github.io/automation-good-practices/">Good practices for Ansible (Red Hat CoP)&lt;/a>: role design, naming and variables. Opinionated, and worth reading.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html">How to build your inventory&lt;/a>: groups, host_vars and group_vars.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/collections/community/general/passwordstore_lookup.html">community.general.passwordstore lookup&lt;/a>: secrets from pass, with all options.&lt;/li>
&lt;li>&lt;a href="https://ansible.readthedocs.io/projects/lint/">ansible-lint&lt;/a>: catches many mistakes before a run.&lt;/li>
&lt;li>&lt;a href="https://firewalld.org/documentation/">firewalld documentation&lt;/a>: zones, services and rich rules.&lt;/li>
&lt;li>&lt;a href="https://github.com/fail2ban/fail2ban">fail2ban&lt;/a>: jails and filters.&lt;/li>
&lt;li>&lt;a href="https://dnf5.readthedocs.io/en/latest/dnf5_plugins/automatic.8.html">dnf5 automatic&lt;/a>: the settings in &lt;code>automatic.conf&lt;/code>.&lt;/li>
&lt;/ul></description></item><item><title>Four servers, three distributions: why I consolidated</title><link>https://www.linux-lunatix.org/blog/infra-01-consolidation/</link><pubDate>Sat, 26 Sep 2026 18:40:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-01-consolidation/</guid><description>&lt;h2 id="where-i-started">Where I started&lt;/h2>
&lt;p>I run a few small servers for myself, my family and a couple of friends. Over the years they grew the way such things grow: one at a time, each with the tools I liked at that moment. When I finally wrote down what I had, it looked like this:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Host&lt;/th>
&lt;th>Distribution&lt;/th>
&lt;th>Runs&lt;/th>
&lt;th>Managed with&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>mail1.example.org&lt;/code>&lt;/td>
&lt;td>Ubuntu 24.04&lt;/td>
&lt;td>Mailcow, Forgejo, Matrix, a paste bin, about 35 containers&lt;/td>
&lt;td>Docker Compose&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>vps2.example.org&lt;/code>&lt;/td>
&lt;td>Arch Linux&lt;/td>
&lt;td>three static websites&lt;/td>
&lt;td>rootless Podman, by hand&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>vps1.example.org&lt;/code>&lt;/td>
&lt;td>Fedora Cloud&lt;/td>
&lt;td>a CI runner, a push notification server&lt;/td>
&lt;td>rootless Podman, quadlets&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>home1.example.net&lt;/code>&lt;/td>
&lt;td>Fedora Server&lt;/td>
&lt;td>monitoring, media, DNS, about 20 containers&lt;/td>
&lt;td>rootless Podman, quadlets&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>(All names in this series are examples. My real hosts have other names.)&lt;/p>
&lt;p>Four servers, three distributions and four ways to manage them. The configuration was in four Git repositories. A script copied shared files between them, when I remembered to run it. How a host was set up was in READMEs, in shell history and in my head.&lt;/p>
&lt;p>Nothing was broken. But every change was slow, because I first had to remember how &lt;em>this&lt;/em> host did things. And I knew that a reinstall of the mail server would cost me a weekend of reading my own old notes.&lt;/p>
&lt;h2 id="what-i-wanted">What I wanted&lt;/h2>
&lt;p>I wrote down four goals before I touched anything:&lt;/p>
&lt;ol>
&lt;li>One distribution everywhere: Fedora.&lt;/li>
&lt;li>The OS setup and every app rolled out with Ansible, from one repository.&lt;/li>
&lt;li>Every app as a rootless Podman quadlet: a systemd unit, no Compose file, no root daemon.&lt;/li>
&lt;li>A reinstall of any host is a routine job: install, run the playbook, restore the data.&lt;/li>
&lt;/ol>
&lt;h2 id="the-decisions-and-why">The decisions, and why&lt;/h2>
&lt;h3 id="fedora-everywhere">Fedora everywhere&lt;/h3>
&lt;p>I already had two Fedora hosts, and I liked how they behaved. Fedora ships a current Podman with quadlet support, systemd, firewalld and SELinux, all of them integrated and enabled by default. A new release comes every six months, and &lt;code>dnf system-upgrade&lt;/code> takes a few minutes. Ubuntu and Arch would work too. But with one distribution there is one package manager, one firewall and one set of paths, and my Ansible roles can be much simpler.&lt;/p>
&lt;h3 id="rootless-podman-with-quadlets">Rootless Podman with quadlets&lt;/h3>
&lt;p>A quadlet is a small ini file, for example &lt;code>~/.config/containers/systemd/blog.container&lt;/code>. systemd turns it into a normal service at boot. Each container is then a unit that you start, stop and read logs from like any other service. Rootless means that the containers run as a normal user. A container that breaks out lands in an unprivileged account, and with SELinux it lands in a confined domain too.&lt;/p>
&lt;p>The next post shows the move from Compose to quadlets in detail.&lt;/p>
&lt;h3 id="docker-only-where-nothing-else-works">Docker only where nothing else works&lt;/h3>
&lt;p>Mailcow is the exception. It supports only Docker with Compose, and its update script expects exactly that. A mail server is not where I want to experiment, so Mailcow stays on Docker. Everything else on that host moves to rootless Podman.&lt;/p>
&lt;h3 id="one-repository">One repository&lt;/h3>
&lt;p>All hosts, all apps, all docs in one Git repository. A change that touches three hosts is one commit. The repository is the truth: if a setting is not in it, it does not exist.&lt;/p>
&lt;h3 id="secrets-in-pass-not-in-git">Secrets in pass, not in Git&lt;/h3>
&lt;p>I already use &lt;a href="https://www.passwordstore.org/">pass&lt;/a> (GPG-encrypted files in a Git repository) for my passwords. Ansible can read from it with the &lt;code>community.general.passwordstore&lt;/code> lookup. So there is no Ansible Vault and no encrypted blob in the infrastructure repository. The repository can be public in principle, and nothing secret leaks with it.&lt;/p>
&lt;h3 id="selinux-enforcing-no-exceptions">SELinux enforcing, no exceptions&lt;/h3>
&lt;p>Many container guides start with &lt;code>setenforce 0&lt;/code>. I wanted the opposite. Every host is enforcing, and the playbook stops if one is not. Every local policy change is code in the repository. Post 4 is about this.&lt;/p>
&lt;h3 id="ansible-owns-configuration-not-versions">Ansible owns configuration, not versions&lt;/h3>
&lt;p>Ansible writes the quadlets, config files and secrets. It does not pin image versions. Images are updated by &lt;code>podman auto-update&lt;/code> (daily). Images I build myself are updated by their CI pipelines. With this split, a playbook run is predictable: it changes what I changed in Git and nothing else.&lt;/p>
&lt;h2 id="the-strategy-adopt-first-then-rebuild">The strategy: adopt first, then rebuild&lt;/h2>
&lt;p>The obvious way would be to write roles, reinstall a server and see what breaks. I did not want that for hosts that other people use.&lt;/p>
&lt;p>So I turned it around:&lt;/p>
&lt;ol>
&lt;li>Adopt the hosts that already run Fedora. I wrote the roles against the live hosts. I ran &lt;code>ansible-playbook --check --diff&lt;/code> again and again and changed the code until the diff was empty. At that point the code describes the host exactly as it is. The real run then changes nothing. This tests the roles on real systems and puts nothing at risk.&lt;/li>
&lt;li>Rehearse on a VM. For the hosts that must be reinstalled, I made a local Fedora VM with the same settings. I ran the full rollout on it and restored the real backups into it. Then I tested the services against the VM.&lt;/li>
&lt;li>Reinstall for real. Only after the rehearsal passed did I reinstall the real host. Same steps, same order.&lt;/li>
&lt;/ol>
&lt;p>This takes longer than a &amp;ldquo;big bang&amp;rdquo;. But at no point was a service down because of untested code.&lt;/p>
&lt;h2 id="the-phases">The phases&lt;/h2>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Phase&lt;/th>
&lt;th>What&lt;/th>
&lt;th>Host(s)&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>0&lt;/td>
&lt;td>Repository, collections, inventory, &lt;code>ansible&lt;/code> user&lt;/td>
&lt;td>all&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>1&lt;/td>
&lt;td>Write the roles, adopt the first Fedora host&lt;/td>
&lt;td>&lt;code>vps1&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>2&lt;/td>
&lt;td>Adopt the home server (monitoring, about 20 apps)&lt;/td>
&lt;td>&lt;code>home1&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>3&lt;/td>
&lt;td>Rehearsal VMs&lt;/td>
&lt;td>VMs on a spare box&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>4&lt;/td>
&lt;td>Reinstall the Arch host with Fedora&lt;/td>
&lt;td>&lt;code>vps2&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>5&lt;/td>
&lt;td>Reinstall the Ubuntu mail host with Fedora&lt;/td>
&lt;td>&lt;code>mail1&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>6&lt;/td>
&lt;td>Clean up: archive the old repositories&lt;/td>
&lt;td>&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;h2 id="the-repository-layout">The repository layout&lt;/h2>
&lt;p>This is the layout I ended up with. The next posts go through each part.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">ops/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── ansible.cfg
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── requirements.yml # pinned collections
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── inventory/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── hosts.yml # hosts and groups
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── group_vars/all.yml # settings for every host
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── group_vars/&amp;lt;group&amp;gt;.yml
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── host_vars/&amp;lt;host&amp;gt;.yml # app_user, apps: [...], firewall, backup, ...
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── playbooks/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── bootstrap.yml # first contact: creates the ansible user
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── site.yml # everything else, with tags
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── roles/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── base/ # packages, sshd, firewalld, fail2ban, updates, SELinux
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── podman/ # the app user, linger, socket, auto-update timer
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── quadlet_app/ # one generic role for every app
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── monitoring_agent/ # WireGuard tunnel, node_exporter
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── backup/ # restic, timer, healthchecks
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── ...
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── apps/&amp;lt;name&amp;gt;/ # one directory per app: quadlets, files, app.yml
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── selinux/ # local policy modules as source
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── tools/ # small scripts: rehearsal VM, image build, restore
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">└── docs/
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The most important idea is in &lt;code>apps/&lt;/code>. There is no hand-written role per service. There is one generic role, and one directory per app with its quadlet files and config. Adding an app means adding a directory and putting its name into the host&amp;rsquo;s &lt;code>apps:&lt;/code> list. Post 6 shows how that role works.&lt;/p>
&lt;h2 id="how-i-check-that-it-works">How I check that it works&lt;/h2>
&lt;p>For every phase I used the same tests:&lt;/p>
&lt;ul>
&lt;li>&lt;code>ansible-playbook site.yml --check --diff&lt;/code> shows no changes on an adopted host.&lt;/li>
&lt;li>A second real run reports &lt;code>changed=0&lt;/code>. A role that changes something on every run hides real changes, so I treat that as a bug.&lt;/li>
&lt;li>After each phase all monitoring targets are up and no alert fires.&lt;/li>
&lt;li>After a reinstall the backup restore works. That was also my first real restore test in years.&lt;/li>
&lt;/ul>
&lt;h2 id="the-series">The series&lt;/h2>
&lt;ol>
&lt;li>This post: why and how.&lt;/li>
&lt;li>&lt;a href="../../blog/infra-02-quadlets/">From Docker Compose to rootless Podman quadlets&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-03-traefik/">Traefik in front of rootless containers&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-04-selinux/">SELinux enforcing with containers&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-05-ansible-base/">Ansible groundwork: inventory, bootstrap and a base role&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-06-quadlet-app/">One generic Ansible role for every app&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-07-monitoring-backup/">Monitoring and backups for every host&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-08-rehearsal/">Rehearse before you reinstall&lt;/a>&lt;/li>
&lt;/ol>
&lt;p>A second part of the series is about SSH. I replaced all &lt;code>authorized_keys&lt;/code> files with short-lived SSH certificates from my own CA, run on OpenBao. It starts with &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/p>
&lt;p>Next: &lt;a href="../../blog/infra-02-quadlets/">From Docker Compose to rootless Podman quadlets&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.fedoraproject.org/en-US/fedora-server/">Fedora Server documentation&lt;/a>: installation, administration and the release cycle.&lt;/li>
&lt;li>&lt;a href="https://podman.io/">Podman&lt;/a>: the project home, with links to the docs and the blog.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/">Ansible documentation&lt;/a>: the entry point for everything in part 1.&lt;/li>
&lt;li>&lt;a href="https://www.passwordstore.org/">pass, the standard Unix password manager&lt;/a>: GPG files in a git repository, the secrets store of this series.&lt;/li>
&lt;li>&lt;a href="https://docs.fedoraproject.org/en-US/quick-docs/selinux-getting-started/">Getting started with SELinux (Fedora Quick Docs)&lt;/a>: modes, contexts and the first commands.&lt;/li>
&lt;li>&lt;a href="https://docs.mailcow.email/">Mailcow documentation&lt;/a>: the one Docker exception in my setup.&lt;/li>
&lt;/ul></description></item></channel></rss>