<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Infrastructure on Linux Lunatix</title><link>https://www.linux-lunatix.org/categories/infrastructure/</link><description>Recent content in Infrastructure on Linux Lunatix</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><copyright>2025 Stefan Marx :: All written work is licensed under CC BY-SA 4.0 unless otherwise stated. Images are taken by me unless otherwise stated and need written permission to be used elsewhere.</copyright><lastBuildDate>Sun, 15 Nov 2026 16:50:00 +0100</lastBuildDate><atom:link href="https://www.linux-lunatix.org/categories/infrastructure/index.xml" rel="self" type="application/rss+xml"/><item><title>CI deploys without a deploy key</title><link>https://www.linux-lunatix.org/blog/ssh-04-ci-oidc/</link><pubDate>Sun, 15 Nov 2026 16:50:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/ssh-04-ci-oidc/</guid><description>&lt;p>&lt;em>Part 4 of the SSH series that starts with &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/em>&lt;/p>
&lt;p>My websites are built by Forgejo Actions. After a push to &lt;code>main&lt;/code>, a job builds a container image, pushes it to the registry and logs in to the web host by SSH. There it runs &lt;code>podman auto-update&lt;/code>, which pulls the new image and restarts the site.&lt;/p>
&lt;p>That login used a deploy key: a private SSH key, stored as a secret in Forgejo, valid forever. This post replaces it with a certificate that a job gets for itself, valid for five minutes, and only for pushes to &lt;code>main&lt;/code>. Forgejo stores no secret for it.&lt;/p>
&lt;h2 id="how-a-job-proves-who-it-is">How a job proves who it is&lt;/h2>
&lt;p>Forgejo Actions can give a job an OpenID Connect ID token, like GitHub Actions does (I run Forgejo 15). It is a JWT, signed by Forgejo, with claims about the job. Shortened:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-json" data-lang="json">&lt;span class="line">&lt;span class="cl">&lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;iss&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;https://git.example.org/api/actions&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;aud&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;https://bao.example.net&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;sub&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;repo:alice/blog:ref:refs/heads/main&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;repository&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;alice/blog&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;exp&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="mi">1763222400&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Anybody can check the signature with Forgejo&amp;rsquo;s public keys, which it publishes at &lt;code>https://git.example.org/api/actions/.well-known/openid-configuration&lt;/code>. Bao&amp;rsquo;s &lt;code>jwt&lt;/code> auth method does exactly that, and then it checks the claims.&lt;/p>
&lt;p>&lt;code>sub&lt;/code> is the claim to bind on. For a push it is &lt;code>repo:&amp;lt;repo&amp;gt;:ref:&amp;lt;ref&amp;gt;&lt;/code>. For a pull request it is &lt;code>repo:&amp;lt;repo&amp;gt;:pull_request&lt;/code>. A pull request from a fork can run the workflow, but it can never get a deploy certificate.&lt;/p>
&lt;h2 id="step-1-bao-s-side">Step 1: Bao&amp;rsquo;s side&lt;/h2>
&lt;p>Three parts, all in the configuration script from &lt;a href="../../blog/ssh-02-openbao/">the OpenBao post&lt;/a>.&lt;/p>
&lt;p>The role for the certificate, with the command already in it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">user_role deploy-websites deploy-websites 5m &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span> &lt;span class="s1">&amp;#39;{&amp;#34;force-command&amp;#34;: &amp;#34;podman auto-update&amp;#34;}&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A policy that can sign that role, and revoke its own token:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">bao policy write deploy-websites - &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;HCL&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"># CI deploys of the sites (jwt login): one 5-minute cert per job
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;ssh-user/sign/deploy-websites&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;auth/token/revoke-self&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">HCL&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>jwt&lt;/code> login, with Forgejo as the issuer, and a role that binds the claims:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">auth jwt jwt
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao write auth/jwt/config &lt;span class="nv">oidc_discovery_url&lt;/span>&lt;span class="o">=&lt;/span>https://git.example.org/api/actions &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="nv">bound_issuer&lt;/span>&lt;span class="o">=&lt;/span>https://git.example.org/api/actions &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao write auth/jwt/role/websites - &amp;gt;/dev/null &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;JSON&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">{
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;role_type&amp;#34;: &amp;#34;jwt&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;bound_audiences&amp;#34;: &amp;#34;https://bao.example.net&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;user_claim&amp;#34;: &amp;#34;repository&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;bound_claims_type&amp;#34;: &amp;#34;string&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;bound_claims&amp;#34;: {&amp;#34;sub&amp;#34;: [
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;repo:alice/blog:ref:refs/heads/main&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;repo:alice/shop:ref:refs/heads/main&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;repo:alice/portfolio:ref:refs/heads/main&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> ]},
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_policies&amp;#34;: &amp;#34;deploy-websites&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_no_default_policy&amp;#34;: true,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_ttl&amp;#34;: &amp;#34;5m&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_max_ttl&amp;#34;: &amp;#34;5m&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">JSON&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The fields that matter:&lt;/p>
&lt;ul>
&lt;li>&lt;code>bound_audiences&lt;/code>: the job must ask for a token for Bao. A token that the job got for another service is refused.&lt;/li>
&lt;li>&lt;code>bound_claims&lt;/code> on &lt;code>sub&lt;/code>: only pushes to &lt;code>main&lt;/code> of these three repositories. Another branch, a tag or a pull request gets &amp;ldquo;claim is not in the bound claims&amp;rdquo;.&lt;/li>
&lt;li>&lt;code>user_claim: repository&lt;/code>: the Bao token&amp;rsquo;s display name becomes &lt;code>jwt-alice/blog&lt;/code>. The certificate&amp;rsquo;s key ID is then &lt;code>jwt-alice/blog-deploy-websites&lt;/code>, and that is what the web host logs for each deploy.&lt;/li>
&lt;li>&lt;code>token_no_default_policy&lt;/code>: Bao&amp;rsquo;s &lt;code>default&lt;/code> policy lets a token look itself up and use the cubbyhole, among other things. A CI token needs none of that. Without the default policy it also cannot revoke itself, so the &lt;code>deploy-websites&lt;/code> policy grants &lt;code>revoke-self&lt;/code> explicitly.&lt;/li>
&lt;/ul>
&lt;h2 id="step-2-the-web-host-s-side">Step 2: the web host&amp;rsquo;s side&lt;/h2>
&lt;p>The principals file for the account that runs the sites has the option, as shown in &lt;a href="../../blog/ssh-03-hosts/">the previous post&lt;/a>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># group_vars/websites.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_principals&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">svc&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">alice, &amp;#39;restrict,command=&amp;#34;podman auto-update&amp;#34; deploy-websites&amp;#39;]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>restrict&lt;/code> turns off everything: a pty, port forwarding, agent forwarding, X11. &lt;code>command=&lt;/code> forces the command. The CA sets &lt;code>force-command&lt;/code> too, but the host does not depend on that.&lt;/p>
&lt;h2 id="step-3-the-network">Step 3: the network&lt;/h2>
&lt;p>The CI runner runs on my mail host, on the internet. Bao runs at home and is not on the internet. The runner reaches the home server through the monitoring tunnel from &lt;a href="../../blog/infra-07-monitoring-backup/">the monitoring post&lt;/a>. That tunnel ends in the hub&amp;rsquo;s &lt;code>block&lt;/code> zone, where the VPS cannot open any connection.&lt;/p>
&lt;p>I did not want to open port 443 on the hub for the runner. Port 443 is Traefik, and Traefik routes to every app on the home server. So Traefik gets an extra entrypoint, and only Bao&amp;rsquo;s router uses it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># traefik.yml on home1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">entryPoints&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">https&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">address&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;:443&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># Only OpenBao&amp;#39;s router uses this entrypoint: the CI runner may reach Bao and nothing else&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bao&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">address&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;:8443&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># traefik.container: publish it&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">8443:8443/tcp&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># openbao.container: the router listens on both&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.openbao.entrypoints=https,bao&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The hub&amp;rsquo;s firewall then has one exception, for the runner&amp;rsquo;s tunnel address and this port:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/home1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">wg_hub_block_exceptions&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">source: 10.99.0.11, port&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">8443&lt;/span>&lt;span class="l">/tcp}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>From the runner host, the result:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">curl -s -o /dev/null -w &lt;span class="s1">&amp;#39;%{http_code}\n&amp;#39;&lt;/span> https://bao.example.net:8443/v1/sys/health &lt;span class="c1"># 200&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -s -o /dev/null -w &lt;span class="s1">&amp;#39;%{http_code}\n&amp;#39;&lt;/span> --resolve grafana.example.net:8443:10.99.0.1 &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> https://grafana.example.net:8443/ &lt;span class="c1"># 404&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -s -m &lt;span class="m">5&lt;/span> https://bao.example.net/ &lt;span class="o">||&lt;/span> &lt;span class="nb">echo&lt;/span> blocked &lt;span class="c1"># blocked&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Another app&amp;rsquo;s name on 8443 gives a 404, because no other router listens there. Port 443 stays blocked.&lt;/p>
&lt;p>The runner needs the name &lt;code>bao.example.net&lt;/code> to resolve to the tunnel address. It does, because the public record points there (see the OpenBao post).&lt;/p>
&lt;h2 id="step-4-the-workflow">Step 4: the workflow&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="nt">env&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">DEPLOY_HOST&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps2.example.org&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">DEPLOY_USER&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">svc&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">BAO_ADDR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">https://bao.example.net:8443&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># The host CA: the web host shows a certificate signed by it, no pinned host key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">HOST_CA&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;ssh-ed25519 AAAA...host-ca-public-key...&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">jobs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># test and build jobs as before ...&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">deploy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">runs-on&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ubuntu-latest&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">needs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">build&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">if&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">github.ref == &amp;#39;refs/heads/main&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># The job&amp;#39;s ID token (repository, ref) is its login at OpenBao&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">enable-openid-connect&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">steps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Trigger podman auto-update&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">run&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> dir=$(mktemp -d) &amp;amp;&amp;amp; trap &amp;#39;rm -rf &amp;#34;$dir&amp;#34;&amp;#39; EXIT
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> # 1. the job&amp;#39;s ID token, for Bao as the audience
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> jwt=$(curl -fsS -H &amp;#34;Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN&amp;#34; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> &amp;#34;$ACTIONS_ID_TOKEN_REQUEST_URL&amp;amp;audience=https://bao.example.net&amp;#34; | jq -er .value)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> echo &amp;#34;::add-mask::$jwt&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> # 2. log in to Bao with it
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> token=$(jq -n --arg jwt &amp;#34;$jwt&amp;#34; &amp;#39;{role: &amp;#34;websites&amp;#34;, jwt: $jwt}&amp;#39; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> | curl -fsS -X POST -d @- &amp;#34;$BAO_ADDR/v1/auth/jwt/login&amp;#34; | jq -er .auth.client_token)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> echo &amp;#34;::add-mask::$token&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> # 3. a throwaway key, signed for five minutes
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> ssh-keygen -q -t ed25519 -N &amp;#39;&amp;#39; -C &amp;#34;$GITHUB_REPOSITORY&amp;#34; -f &amp;#34;$dir/key&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> jq -n --rawfile k &amp;#34;$dir/key.pub&amp;#34; &amp;#39;{public_key: $k}&amp;#39; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> | curl -fsS -X POST -H &amp;#34;X-Vault-Token: $token&amp;#34; -d @- &amp;#34;$BAO_ADDR/v1/ssh-user/sign/deploy-websites&amp;#34; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> | jq -er .data.signed_key &amp;gt; &amp;#34;$dir/key-cert.pub&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> # 4. the token is not needed any more
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> curl -fsS -X POST -H &amp;#34;X-Vault-Token: $token&amp;#34; &amp;#34;$BAO_ADDR/v1/auth/token/revoke-self&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> ssh-keygen -Lf &amp;#34;$dir/key-cert.pub&amp;#34; | sed -n &amp;#39;/Key ID/p;/Valid/p&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> # 5. the host is checked through the CA, then: podman auto-update, nothing else
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> echo &amp;#34;@cert-authority $DEPLOY_HOST $HOST_CA&amp;#34; &amp;gt; &amp;#34;$dir/known_hosts&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> ssh -i &amp;#34;$dir/key&amp;#34; -o IdentitiesOnly=yes -o UserKnownHostsFile=&amp;#34;$dir/known_hosts&amp;#34; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> -o StrictHostKeyChecking=yes &amp;#34;$DEPLOY_USER@$DEPLOY_HOST&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Some notes:&lt;/p>
&lt;ul>
&lt;li>&lt;code>enable-openid-connect: true&lt;/code> is set on the job. Only this job gets the two environment variables &lt;code>ACTIONS_ID_TOKEN_REQUEST_URL&lt;/code> and &lt;code>ACTIONS_ID_TOKEN_REQUEST_TOKEN&lt;/code>. The test and build jobs do not.&lt;/li>
&lt;li>&lt;code>::add-mask::&lt;/code> hides the JWT and the Bao token in the job log, in case a later command prints them.&lt;/li>
&lt;li>The private key exists only in the job&amp;rsquo;s temporary directory. The &lt;code>trap&lt;/code> removes it.&lt;/li>
&lt;li>The job uses &lt;code>curl&lt;/code> and &lt;code>jq&lt;/code> and needs no &lt;code>bao&lt;/code> CLI. The usual runner images have both.&lt;/li>
&lt;li>The pinned host key in the workflow is gone. After a reinstall of the web host, Ansible signs the new host key, and the workflow does not change.&lt;/li>
&lt;/ul>
&lt;p>The job log shows the certificate:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Key ID: &amp;#34;jwt-alice/blog-deploy-websites&amp;#34;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Valid: from 2026-11-15T16:51:02 to 2026-11-15T16:56:32
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>And the web host logs the deploy:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">sshd[51234]: Accepted publickey for svc from 10.99.0.11 port 40122 ssh2: ED25519-CERT SHA256:... ID jwt-alice/blog-deploy-websites (serial 7342...) CA ED25519 SHA256:...
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="test-what-must-fail">Test what must fail&lt;/h2>
&lt;ul>
&lt;li>Push the same workflow to another branch. Bao answers the login with an error about the bound claims, and the job fails at step 2.&lt;/li>
&lt;li>Open a pull request. Here the deploy job does not run at all (&lt;code>if: github.ref =&lt;/code> &amp;lsquo;refs/heads/main&amp;rsquo;&lt;code>). If somebody removes that line, Bao still refuses the =pull_request&lt;/code> subject.&lt;/li>
&lt;li>Try a shell with a deploy certificate. The host runs &lt;code>podman auto-update&lt;/code> and closes the connection.&lt;/li>
&lt;li>Try the certificate after six minutes. It has expired, and the host refuses it.&lt;/li>
&lt;li>Try the old deploy key. It must be refused: remove it from the host&amp;rsquo;s &lt;code>authorized_keys&lt;/code> and from Forgejo&amp;rsquo;s secrets.&lt;/li>
&lt;/ul>
&lt;p>For the branch test I pushed the workflow to a temporary branch &lt;code>oidc-test&lt;/code>, watched the job fail with the bound-claims error, and deleted the branch.&lt;/p>
&lt;h2 id="add-a-new-site">Add a new site&lt;/h2>
&lt;ol>
&lt;li>Add its &lt;code>sub&lt;/code> to the &lt;code>websites&lt;/code> role in the configuration script and run the script.&lt;/li>
&lt;li>Copy the deploy job into the new repository.&lt;/li>
&lt;/ol>
&lt;p>There is no secret to create and no key to distribute.&lt;/p>
&lt;p>Next, and last: &lt;a href="../../blog/ssh-05-plain-keys/">Removing the plain SSH keys&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://forgejo.org/docs/latest/user/actions/security-openid-connect/">Forgejo Actions: OpenID Connect&lt;/a>: &lt;code>enable-openid-connect&lt;/code> and the token request.&lt;/li>
&lt;li>&lt;a href="https://forgejo.org/docs/latest/user/actions/reference/">Forgejo Actions reference&lt;/a>: workflow syntax, contexts and variables.&lt;/li>
&lt;li>&lt;a href="https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/about-security-hardening-with-openid-connect">About security hardening with OpenID Connect (GitHub)&lt;/a>: the same idea on GitHub, with a good explanation of the claims.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/auth/jwt/">OpenBao JWT/OIDC auth method&lt;/a>: &lt;code>bound_claims&lt;/code>, &lt;code>bound_audiences&lt;/code> and &lt;code>user_claim&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/api-docs/secret/ssh/">OpenBao SSH secrets engine API&lt;/a>: &lt;code>default_critical_options&lt;/code> and the TTLs of a role.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd#AUTHORIZED_KEYS_FILE_FORMAT">sshd(8), authorized_keys format&lt;/a>: &lt;code>restrict&lt;/code> and &lt;code>command=&lt;/code>, which also work in a principals file.&lt;/li>
&lt;/ul></description></item><item><title>Why SSH certificates</title><link>https://www.linux-lunatix.org/blog/ssh-01-why-certificates/</link><pubDate>Sun, 01 Nov 2026 17:10:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/ssh-01-why-certificates/</guid><description>&lt;p>&lt;em>Part 1 of the SSH series. It builds on the hosts from &lt;a href="../../blog/infra-01-consolidation/">the consolidation series&lt;/a>, but you can read it on its own.&lt;/em>&lt;/p>
&lt;p>This post explains OpenSSH certificates. It has a demo that you can run on one machine in ten minutes, and it explains why I put my CA on OpenBao. The next four posts build the real thing.&lt;/p>
&lt;h2 id="the-problems-with-plain-keys">The problems with plain keys&lt;/h2>
&lt;p>SSH with plain keys has two trust lists, and both are maintained by hand.&lt;/p>
&lt;p>&lt;code>authorized_keys&lt;/code> on the server says which keys may log in. With four hosts, a few accounts each, two laptops, a tablet and a CI job, that is a lot of lines. A new device means editing every host. A lost device means finding its key everywhere and removing it. A CI deploy key lives forever, unless somebody rotates it.&lt;/p>
&lt;p>&lt;code>known_hosts&lt;/code> on the client says which host keys to trust. The first connection asks you to confirm a fingerprint. Almost nobody checks it, which is why this is called &amp;ldquo;trust on first use&amp;rdquo;. After a reinstall, ssh shows a big warning, and you either check the new key over a console or you click it away. In &lt;a href="../../blog/infra-08-rehearsal/">the rehearsal post&lt;/a> I showed how much work a careful check is.&lt;/p>
&lt;p>Both lists work. But they grow, nobody removes old entries, and every change touches many places.&lt;/p>
&lt;h2 id="what-a-certificate-is">What a certificate is&lt;/h2>
&lt;p>An OpenSSH certificate is a public key plus some data, signed by a CA key:&lt;/p>
&lt;ul>
&lt;li>the public key itself,&lt;/li>
&lt;li>the type: user or host,&lt;/li>
&lt;li>the principals: the user names or host names it is valid for,&lt;/li>
&lt;li>the validity: from when until when,&lt;/li>
&lt;li>the critical options, for example &lt;code>force-command&lt;/code> or &lt;code>source-address&lt;/code>,&lt;/li>
&lt;li>the extensions, for example &lt;code>permit-pty&lt;/code> or &lt;code>permit-port-forwarding&lt;/code>,&lt;/li>
&lt;li>a key ID, a free text that appears in the server&amp;rsquo;s log,&lt;/li>
&lt;li>the signature of the CA.&lt;/li>
&lt;/ul>
&lt;p>A server that trusts the CA accepts any user certificate signed by it, if the certificate is still valid and lists a principal that may use the account. A client that trusts a host CA accepts any host that presents a certificate for its name.&lt;/p>
&lt;p>These are not X.509 certificates. There is no chain and no CRL distribution. OpenSSH has its own simple format, and it has supported it since 2010 (OpenSSH 5.4).&lt;/p>
&lt;h2 id="what-this-changes">What this changes&lt;/h2>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Before&lt;/th>
&lt;th>With certificates&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>every host lists every user key&lt;/td>
&lt;td>every host lists one CA key and the allowed principals&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>a new device: edit every host&lt;/td>
&lt;td>a new device: get a certificate&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>a lost device: remove its key everywhere&lt;/td>
&lt;td>a lost device: its certificates expire in hours&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>a CI key that never expires&lt;/td>
&lt;td>a certificate for five minutes, per job&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>known_hosts&lt;/code> per host, TOFU&lt;/td>
&lt;td>one &lt;code>@cert-authority&lt;/code> line on the client&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>reinstall: check the key over a console&lt;/td>
&lt;td>reinstall: sign the new host key, done&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>The short validity is the most important part. Revocation is hard in any PKI. A certificate that expires after 16 hours needs no revocation in most cases.&lt;/p>
&lt;h2 id="try-it-a-ca-in-ten-minutes">Try it: a CA in ten minutes&lt;/h2>
&lt;p>You can try all of this on one machine, with &lt;code>ssh-keygen&lt;/code> and a local sshd. Do it in a VM or a container if you do not want to touch your machine&amp;rsquo;s sshd.&lt;/p>
&lt;h3 id="a-user-ca">A user CA&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">mkdir -p ~/ca-demo &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> &lt;span class="nb">cd&lt;/span> ~/ca-demo
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -t ed25519 -f user_ca -C &lt;span class="s2">&amp;#34;demo user CA&amp;#34;&lt;/span> -N &lt;span class="s1">&amp;#39;&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>user_ca&lt;/code> is the CA&amp;rsquo;s private key and &lt;code>user_ca.pub&lt;/code> its public key. In real life the private key is the most sensitive file of the whole setup. Here it is a demo.&lt;/p>
&lt;h3 id="sign-a-user-key">Sign a user key&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -t ed25519 -f alice -N &lt;span class="s1">&amp;#39;&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -s user_ca -I &lt;span class="s2">&amp;#34;alice-laptop&amp;#34;&lt;/span> -n alice -V +1h alice.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -Lf alice-cert.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">alice-cert.pub:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Type: ssh-ed25519-cert-v01@openssh.com user certificate
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Public key: ED25519-CERT SHA256:Xq2...
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Signing CA: ED25519 SHA256:k9T... (using ssh-ed25519)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Key ID: &amp;#34;alice-laptop&amp;#34;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Serial: 0
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Valid: from 2026-11-01T17:10:00 to 2026-11-01T18:10:00
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Principals:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> alice
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Critical Options: (none)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Extensions:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> permit-X11-forwarding
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> permit-agent-forwarding
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> permit-port-forwarding
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> permit-pty
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> permit-user-rc
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>-I&lt;/code> is the key ID, &lt;code>-n&lt;/code> the principals and &lt;code>-V&lt;/code> the validity. By default &lt;code>ssh-keygen&lt;/code> adds all the extensions. A real CA should add only what you need.&lt;/p>
&lt;p>ssh finds &lt;code>alice-cert.pub&lt;/code> by itself when you use the key &lt;code>alice&lt;/code>: it looks for &lt;code>&amp;lt;key&amp;gt;-cert.pub&lt;/code> next to the key.&lt;/p>
&lt;h3 id="trust-the-ca-on-the-server">Trust the CA on the server&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo install -m &lt;span class="m">0644&lt;/span> user_ca.pub /etc/ssh/user_ca.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo install -d /etc/ssh/principals
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># which principals may log in as the account &amp;#34;demo&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> alice &lt;span class="p">|&lt;/span> sudo tee /etc/ssh/principals/demo
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo tee /etc/ssh/sshd_config.d/10-ca.conf &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;EOF&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">TrustedUserCAKeys /etc/ssh/user_ca.pub
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">AuthorizedPrincipalsFile /etc/ssh/principals/%u
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo useradd -m demo
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo sshd -t &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> sudo systemctl reload sshd
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh -i ~/ca-demo/alice -o &lt;span class="nv">IdentitiesOnly&lt;/span>&lt;span class="o">=&lt;/span>yes demo@localhost
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>You are in, and &lt;code>demo&lt;/code> has no &lt;code>authorized_keys&lt;/code> file. The server log shows who it was:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">sshd[4242]: Accepted publickey for demo from ::1 port 50122 ssh2: ED25519-CERT SHA256:Xq2... ID alice-laptop (serial 0) CA ED25519 SHA256:k9T...
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>AuthorizedPrincipalsFile&lt;/code> is the important line. Without it, sshd accepts a certificate for an account only if the account name itself is one of its principals. With it, you decide per account which principals may use it. One principal can then log in as several accounts, and a certificate for &lt;code>alice&lt;/code> cannot log in as &lt;code>root&lt;/code> or &lt;code>ansible&lt;/code>.&lt;/p>
&lt;h3 id="a-host-ca">A host CA&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -t ed25519 -f host_ca -C &lt;span class="s2">&amp;#34;demo host CA&amp;#34;&lt;/span> -N &lt;span class="s1">&amp;#39;&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo ssh-keygen -s host_ca -I &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">$(&lt;/span>hostname&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> -h -n localhost,&lt;span class="k">$(&lt;/span>hostname -f&lt;span class="k">)&lt;/span> -V +52w &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> /etc/ssh/ssh_host_ed25519_key.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s1">&amp;#39;HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub&amp;#39;&lt;/span> &lt;span class="p">|&lt;/span> sudo tee /etc/ssh/sshd_config.d/11-hostcert.conf
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo systemctl reload sshd
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>-h&lt;/code> makes it a host certificate. The principals are the names that clients use for the host.&lt;/p>
&lt;p>On the client, one line in &lt;code>known_hosts&lt;/code> replaces all the host entries:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;@cert-authority localhost,*.example.org &lt;/span>&lt;span class="k">$(&lt;/span>cat host_ca.pub&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt; ~/ca-demo/known_hosts
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh -o &lt;span class="nv">UserKnownHostsFile&lt;/span>&lt;span class="o">=&lt;/span>~/ca-demo/known_hosts -o &lt;span class="nv">StrictHostKeyChecking&lt;/span>&lt;span class="o">=&lt;/span>yes &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -i ~/ca-demo/alice demo@localhost
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>No question about a fingerprint. &lt;code>StrictHostKeyChecking=yes&lt;/code> proves it: ssh would refuse a host it does not know.&lt;/p>
&lt;h3 id="clean-up">Clean up&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo rm /etc/ssh/sshd_config.d/10-ca.conf /etc/ssh/sshd_config.d/11-hostcert.conf &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> /etc/ssh/ssh_host_ed25519_key-cert.pub /etc/ssh/user_ca.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo rm -r /etc/ssh/principals
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo systemctl reload sshd &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> sudo userdel -r demo
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="options-that-the-server-enforces">Options that the server enforces&lt;/h2>
&lt;p>Critical options restrict what a certificate can do:&lt;/p>
&lt;ul>
&lt;li>&lt;code>force-command=&amp;quot;podman auto-update&amp;quot;&lt;/code>: the certificate runs only this command, whatever the client asks for.&lt;/li>
&lt;li>&lt;code>source-address=10.0.0.0/8&lt;/code>: the certificate works only from these addresses.&lt;/li>
&lt;/ul>
&lt;p>There is a catch here. Whoever may ask the CA for a certificate with an option may also ask for another value. If the CA&amp;rsquo;s policy lets a CI job choose its own &lt;code>force-command&lt;/code>, the job can choose &lt;code>bash&lt;/code>. So my hosts enforce the important options themselves, in the principals file. Each line there can carry the same options as an &lt;code>authorized_keys&lt;/code> line:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl"># /etc/ssh/principals/svc
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">alice
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">restrict,command=&amp;#34;podman auto-update&amp;#34; deploy-websites
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A certificate for &lt;code>alice&lt;/code> logs in normally. A certificate for &lt;code>deploy-websites&lt;/code> gets &lt;code>podman auto-update&lt;/code> and nothing else, even if the CA signed it without the option. &lt;a href="../../blog/ssh-03-hosts/">Post 3 of this series&lt;/a> shows the Ansible role that writes these files.&lt;/p>
&lt;h2 id="more-details-worth-knowing">More details worth knowing&lt;/h2>
&lt;p>The demo covers the basics. These details matter as soon as you run a CA for real.&lt;/p>
&lt;h3 id="sign-only-the-extensions-you-need">Sign only the extensions you need&lt;/h3>
&lt;p>&lt;code>ssh-keygen&lt;/code> adds all five extensions by default. &lt;code>-O clear&lt;/code> removes them, and each further &lt;code>-O&lt;/code> adds one back:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -s user_ca -I &lt;span class="s2">&amp;#34;alice-laptop&amp;#34;&lt;/span> -n alice -V +1h &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -O clear -O permit-pty alice.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This certificate can open a shell, but it cannot forward ports, agents or X11. A certificate for a backup job needs no &lt;code>permit-pty&lt;/code> either. The critical options work the same way:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -s user_ca -I &lt;span class="s2">&amp;#34;backup-job&amp;#34;&lt;/span> -n backup -V +10m &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -O clear -O force-command&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;/usr/local/bin/backup.sh&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -O source-address&lt;span class="o">=&lt;/span>192.168.50.0/24 backup.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="validity-and-clocks">Validity and clocks&lt;/h3>
&lt;p>The server checks the validity against its own clock. If the client&amp;rsquo;s clock or the CA&amp;rsquo;s clock runs a little ahead, a fresh certificate can be &amp;ldquo;not yet valid&amp;rdquo; on the server. Let it start a few minutes in the past:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -s user_ca -I &lt;span class="s2">&amp;#34;alice-laptop&amp;#34;&lt;/span> -n alice -V -5m:+16h alice.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>-V&lt;/code> also takes absolute times, for example &lt;code>-V 20261101:20261201&lt;/code>. Keep the clocks of all hosts in sync with NTP. Certificates depend on it.&lt;/p>
&lt;h3 id="always-set-principals">Always set principals&lt;/h3>
&lt;p>&lt;code>-n&lt;/code> is not optional in practice. In the certificate format, an empty principal list means &amp;ldquo;any principal&amp;rdquo;. With &lt;code>TrustedUserCAKeys&lt;/code>, sshd refuses a user certificate without principals, but other programs may handle it differently. A CA that signs whatever it gets is a risk. Give every certificate its principals, and give your CA a list of principals it may sign.&lt;/p>
&lt;h3 id="serial-numbers-and-key-ids">Serial numbers and key IDs&lt;/h3>
&lt;p>&lt;code>-z&lt;/code> sets a serial number:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -s user_ca -I &lt;span class="s2">&amp;#34;alice-laptop&amp;#34;&lt;/span> -z &lt;span class="m">1001&lt;/span> -n alice -V +16h alice.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The key ID and the serial appear in the server log for each login. They are also the handles for revocation. Choose key IDs that tell you who and what, for example &lt;code>alice-laptop&lt;/code> or &lt;code>ci-blog-main&lt;/code>. A CA such as OpenBao sets a random serial by itself.&lt;/p>
&lt;h3 id="revocation-when-you-need-it">Revocation, when you need it&lt;/h3>
&lt;p>Short certificates make revocation rare, but not impossible. The certificate of a tablet lives for weeks, and a host certificate for a year. sshd reads a revocation list from &lt;code>RevokedKeys&lt;/code>. That can be a plain list of public keys, or a key revocation list (KRL) in OpenSSH&amp;rsquo;s own compact format. A KRL can revoke by serial, by key ID or by key:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">cat &amp;gt; revoke.txt &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;EOF2&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">serial: 1001
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">id: alice-tablet
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF2&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -k -f revoked.krl -s user_ca.pub revoke.txt
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># test a certificate against the KRL&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -Q -f revoked.krl alice-cert.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>-u&lt;/code> adds to an existing KRL instead of replacing it. On the server:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">RevokedKeys /etc/ssh/revoked.krl
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>On the client, a line with &lt;code>@revoked&lt;/code> in &lt;code>known_hosts&lt;/code> refuses a host key, even with a valid host certificate:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">@revoked * ssh-ed25519 AAAA...the-old-host-key...
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="rotate-the-ca-key">Rotate the CA key&lt;/h3>
&lt;p>One day you will want a new CA key, because the old one leaked or because you want a new algorithm. &lt;code>TrustedUserCAKeys&lt;/code> can list several keys, one per line. That makes the change smooth:&lt;/p>
&lt;ol>
&lt;li>Create the new CA key and add its public key to &lt;code>TrustedUserCAKeys&lt;/code> on every host. Both keys are now trusted.&lt;/li>
&lt;li>Sign new certificates with the new key only.&lt;/li>
&lt;li>Wait until the last certificate from the old key has expired.&lt;/li>
&lt;li>Remove the old public key from every host.&lt;/li>
&lt;/ol>
&lt;p>On the client, two &lt;code>@cert-authority&lt;/code> lines do the same for host certificates. With short certificates, step 3 takes one day.&lt;/p>
&lt;h3 id="a-ca-per-account-without-sshd-config">A CA per account, without sshd_config&lt;/h3>
&lt;p>If you cannot change &lt;code>sshd_config&lt;/code>, the CA can go into a user&amp;rsquo;s &lt;code>authorized_keys&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">cert-authority,principals=&amp;#34;alice&amp;#34; ssh-ed25519 AAAA...user-ca-public-key...
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This trusts the CA only for this one account. It is useful on a shared host where you are not root. &lt;code>principals=&lt;/code> has the same role as the principals file.&lt;/p>
&lt;h3 id="host-certificates-and-names">Host certificates and names&lt;/h3>
&lt;p>The client checks that the host certificate lists the name it used to connect. &lt;code>ssh vps1&lt;/code> with a certificate for &lt;code>vps1.example.org&lt;/code> fails, unless your SSH config maps the short name to the full name with &lt;code>HostName&lt;/code> or &lt;code>HostKeyAlias&lt;/code>. Put every name that clients use into the host certificate&amp;rsquo;s principals. &lt;a href="../../blog/ssh-03-hosts/">Post 3&lt;/a> shows how I handle short names.&lt;/p>
&lt;h3 id="see-what-happens">See what happens&lt;/h3>
&lt;p>&lt;code>ssh -v&lt;/code> shows which keys and certificates the client offers and what the server answers:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">debug1: Offering public key: /home/alice/.ssh/id_ed25519 ED25519-CERT SHA256:... explicit
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">debug1: Server accepts key: /home/alice/.ssh/id_ed25519 ED25519-CERT SHA256:... explicit
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">debug1: Host &amp;#39;vps1.example.org&amp;#39; is known and matches the ED25519-CERT host certificate.
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>On the server, &lt;code>journalctl -u sshd&lt;/code> shows the key ID, the serial and the CA of each accepted certificate. It also shows the reason for a refusal, for example an expired certificate or a principal that is not allowed.&lt;/p>
&lt;h2 id="where-should-the-ca-key-live">Where should the CA key live?&lt;/h2>
&lt;p>The demo kept the CA key in a file. That works for a homelab, but it has problems:&lt;/p>
&lt;ul>
&lt;li>Whoever has the file can sign anything: any principal, any validity. A stolen laptop is a stolen CA.&lt;/li>
&lt;li>Nothing logs what was signed.&lt;/li>
&lt;li>A CI job cannot get a certificate without a copy of the key or a person in the loop.&lt;/li>
&lt;/ul>
&lt;p>What I wanted from a CA:&lt;/p>
&lt;ul>
&lt;li>The private keys never leave it.&lt;/li>
&lt;li>Signing needs a login, and the login decides what may be signed: which principals, how long, which options.&lt;/li>
&lt;li>My daily login uses my existing single sign-on, my self-hosted Forgejo.&lt;/li>
&lt;li>A CI job proves who it is with the identity token that Forgejo Actions gives it, without a stored secret.&lt;/li>
&lt;li>Every signing is logged.&lt;/li>
&lt;/ul>
&lt;h2 id="why-openbao">Why OpenBao&lt;/h2>
&lt;p>I looked at four options.&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Option&lt;/th>
&lt;th>For it&lt;/th>
&lt;th>Against it&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>ssh-keygen&lt;/code> and a file&lt;/td>
&lt;td>simple, nothing to run&lt;/td>
&lt;td>everything in the list above&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>step-ca (Smallstep)&lt;/td>
&lt;td>built for this, OIDC, good SSH support&lt;/td>
&lt;td>one more tool only for certificates&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>HashiCorp Vault&lt;/td>
&lt;td>the standard, SSH secrets engine&lt;/td>
&lt;td>Business Source License since 2023&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>OpenBao&lt;/td>
&lt;td>Vault&amp;rsquo;s open fork (MPL 2.0, Linux Foundation), same API and SSH engine&lt;/td>
&lt;td>younger project&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>I chose OpenBao. Its SSH secrets engine does exactly what I need. It has OIDC and JWT login, policies, an audit log, and integrated storage with snapshots. It can also hold other secrets later, and step-ca cannot do that. Most Vault documentation and tools work with it unchanged: the CLI is &lt;code>bao&lt;/code> instead of &lt;code>vault&lt;/code>, and the API is the same. Fedora packages the &lt;code>openbao&lt;/code> CLI, and the server runs from the official container image.&lt;/p>
&lt;h2 id="the-design">The design&lt;/h2>
&lt;p>Two CAs, one for users and one for hosts, so that a host key can never log in as a user. Each kind of certificate gets a role in Bao with its own rules:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Role&lt;/th>
&lt;th>Principal&lt;/th>
&lt;th>Valid&lt;/th>
&lt;th>For&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>admin&lt;/code>&lt;/td>
&lt;td>&lt;code>alice&lt;/code>&lt;/td>
&lt;td>16 h&lt;/td>
&lt;td>my laptop, every day&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>device&lt;/code>&lt;/td>
&lt;td>&lt;code>alice&lt;/code>&lt;/td>
&lt;td>30 days&lt;/td>
&lt;td>a tablet that cannot talk to Bao&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>ansible&lt;/code>&lt;/td>
&lt;td>&lt;code>ansible&lt;/code>&lt;/td>
&lt;td>16 h&lt;/td>
&lt;td>the Ansible key on the control node&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>deploy-websites&lt;/code>&lt;/td>
&lt;td>&lt;code>deploy-websites&lt;/code>&lt;/td>
&lt;td>5 min&lt;/td>
&lt;td>CI jobs; only &lt;code>podman auto-update&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>hosts&lt;/code> (host CA)&lt;/td>
&lt;td>the host&amp;rsquo;s FQDN&lt;/td>
&lt;td>1 year&lt;/td>
&lt;td>host certificates, signed during the Ansible run&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>16 hours covers a working day. In the morning I run one command, log in through the browser, and have certificates for the day.&lt;/p>
&lt;p>The plain keys go away, except for one break-glass key per host. That key is for the day when Bao is down and all certificates have expired.&lt;/p>
&lt;h2 id="the-series">The series&lt;/h2>
&lt;ol>
&lt;li>This post: why and how it works.&lt;/li>
&lt;li>&lt;a href="../../blog/ssh-02-openbao/">OpenBao rootless on Podman&lt;/a>: the server, its configuration as a script, unseal, backup.&lt;/li>
&lt;li>&lt;a href="../../blog/ssh-03-hosts/">Hosts that trust the SSH CA&lt;/a>: the Ansible side, host certificates, the client.&lt;/li>
&lt;li>&lt;a href="../../blog/ssh-04-ci-oidc/">CI deploys without a deploy key&lt;/a>: Forgejo Actions OIDC and five-minute certificates.&lt;/li>
&lt;li>&lt;a href="../../blog/ssh-05-plain-keys/">Removing the plain SSH keys&lt;/a>: break-glass, devices without Bao, and how not to lock yourself out.&lt;/li>
&lt;/ol>
&lt;p>Next: &lt;a href="../../blog/ssh-02-openbao/">OpenBao rootless on Podman&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://man.openbsd.org/ssh-keygen#CERTIFICATES">ssh-keygen(1), section CERTIFICATES&lt;/a>: signing, &lt;code>-O&lt;/code> options, validity formats and the KRL commands. Start here.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd#AUTHORIZED_KEYS_FILE_FORMAT">sshd(8), authorized_keys format&lt;/a>: &lt;code>cert-authority&lt;/code>, &lt;code>principals=&lt;/code> and the options that a principals file also accepts.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd#SSH_KNOWN_HOSTS_FILE_FORMAT">sshd(8), known_hosts format&lt;/a>: &lt;code>@cert-authority&lt;/code> and &lt;code>@revoked&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd_config">sshd_config(5)&lt;/a>: &lt;code>TrustedUserCAKeys&lt;/code>, &lt;code>AuthorizedPrincipalsFile&lt;/code>, &lt;code>HostCertificate&lt;/code> and &lt;code>RevokedKeys&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://datatracker.ietf.org/doc/draft-miller-ssh-cert/">SSH certificate format (IETF draft by Damien Miller)&lt;/a>: the exact wire format of user and host certificates, and every option and extension.&lt;/li>
&lt;li>&lt;a href="https://github.com/openssh/openssh-portable/blob/master/PROTOCOL.krl">PROTOCOL.krl&lt;/a>: the format of key revocation lists.&lt;/li>
&lt;li>&lt;a href="https://www.openssh.com/txt/release-5.4">OpenSSH 5.4 release notes&lt;/a>: the release that introduced certificates in 2010.&lt;/li>
&lt;li>&lt;a href="https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Certificate-based_Authentication">OpenSSH Cookbook: certificate-based authentication&lt;/a>: a long tutorial with many examples.&lt;/li>
&lt;li>&lt;a href="https://smallstep.com/blog/use-ssh-certificates/">If you&amp;rsquo;re not using SSH certificates you&amp;rsquo;re doing SSH wrong (Smallstep)&lt;/a>: a good argument for certificates, from the makers of step-ca.&lt;/li>
&lt;li>&lt;a href="https://engineering.fb.com/2016/09/12/security/scalable-and-secure-access-with-ssh/">Scalable and secure access with SSH (Facebook engineering)&lt;/a>: how a large company uses SSH certificates.&lt;/li>
&lt;li>&lt;a href="https://github.com/Netflix/bless">Netflix BLESS&lt;/a>: another CA design: short certificates from a serverless function.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/secrets/ssh/signed-ssh-certificates/">OpenBao signed SSH certificates&lt;/a>: the CA I use in the next posts.&lt;/li>
&lt;li>&lt;a href="https://smallstep.com/docs/step-ca/">step-ca&lt;/a>: the alternative I did not choose.&lt;/li>
&lt;/ul></description></item><item><title>Rehearse before you reinstall</title><link>https://www.linux-lunatix.org/blog/infra-08-rehearsal/</link><pubDate>Tue, 27 Oct 2026 19:45:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/infra-08-rehearsal/</guid><description>&lt;p>&lt;em>Part 8 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>Two of my hosts had to be reinstalled: the Arch VPS with the websites, and the Ubuntu host with mail and Git. Before I touched either of them, I ran the whole reinstall on a local VM: the install, the playbook, the restore of the real backups and the tests. This post shows how. The same approach works for any host you are afraid to touch.&lt;/p>
&lt;h2 id="the-idea">The idea&lt;/h2>
&lt;p>A rehearsal VM is the real host with three differences:&lt;/p>
&lt;ol>
&lt;li>It runs on a machine at home, on libvirt&amp;rsquo;s NAT network.&lt;/li>
&lt;li>It cannot talk to the world in ways that would hurt: no outgoing mail, no Matrix federation, no certificate requests, no CI jobs.&lt;/li>
&lt;li>It never runs the real host&amp;rsquo;s backup timer or monitoring tunnel.&lt;/li>
&lt;/ol>
&lt;p>Everything else is real: the real inventory settings, the real secrets from pass and the real backup repository (read-only). If the rehearsal passes, the cut-over is a repeat of known steps.&lt;/p>
&lt;h2 id="step-1-a-vm-from-the-fedora-cloud-image">Step 1: a VM from the Fedora Cloud image&lt;/h2>
&lt;p>A provider&amp;rsquo;s Fedora image is the Fedora Cloud image with cloud-init. So I start from the same image. Download it and check its signature first:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">cd&lt;/span> /var/lib/libvirt/images/base
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -LO https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-44-1.7-x86_64-CHECKSUM
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -O https://fedoraproject.org/fedora.gpg
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">gpgv --keyring ./fedora.gpg Fedora-Cloud-44-1.7-x86_64-CHECKSUM
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sha256sum -c --ignore-missing Fedora-Cloud-44-1.7-x86_64-CHECKSUM
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then a small script creates a VM from it. Each VM gets a thin qcow2 overlay on the base image. cloud-init creates the user &lt;code>fedora&lt;/code> with my key and passwordless sudo, the same starting point as a fresh VPS:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># rehearsal-vm.sh create &amp;lt;name&amp;gt; &amp;lt;memory MiB&amp;gt; &amp;lt;vcpus&amp;gt; &amp;lt;disk GB&amp;gt; &amp;lt;ssh pubkey file&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># rehearsal-vm.sh destroy &amp;lt;name&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># rehearsal-vm.sh ip &amp;lt;name&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">BASE_DIR&lt;/span>&lt;span class="o">=&lt;/span>/var/lib/libvirt/images/base
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">IMAGE&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">IMAGE&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="nv">$BASE_DIR&lt;/span>&lt;span class="p">/Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">VM_DIR&lt;/span>&lt;span class="o">=&lt;/span>/var/lib/libvirt/images
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">LIBVIRT_DEFAULT_URI&lt;/span>&lt;span class="o">=&lt;/span>qemu:///system
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">cmd&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">1&lt;/span>&lt;span class="p">:?command&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nv">name&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">2&lt;/span>&lt;span class="p">:?name&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">case&lt;/span> &lt;span class="nv">$cmd&lt;/span> in
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">create&lt;span class="o">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">mem&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">3&lt;/span>&lt;span class="p">:?memory MiB&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nv">cpus&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">4&lt;/span>&lt;span class="p">:?vcpus&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nv">disk&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">5&lt;/span>&lt;span class="p">:?disk GB&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nv">key&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>cat &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">6&lt;/span>&lt;span class="p">:?ssh pubkey file&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">tmp&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>mktemp -d&lt;span class="k">)&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nb">trap&lt;/span> &lt;span class="s1">&amp;#39;rm -rf &amp;#34;$tmp&amp;#34;&amp;#39;&lt;/span> EXIT
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> cat &amp;gt;&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$tmp&lt;/span>&lt;span class="s2">/user-data&amp;#34;&lt;/span> &lt;span class="s">&amp;lt;&amp;lt;EOF
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">#cloud-config
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">hostname: $name
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">users:
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> - name: fedora
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> groups: [wheel]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> sudo: &amp;#34;ALL=(ALL) NOPASSWD:ALL&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> ssh_authorized_keys: [&amp;#34;$key&amp;#34;]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> sudo qemu-img create -q -f qcow2 -F qcow2 -b &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$IMAGE&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$VM_DIR&lt;/span>&lt;span class="s2">/&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">.qcow2&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">disk&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">G&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> virt-install --name &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --memory &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$mem&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --vcpus &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$cpus&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --disk &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$VM_DIR&lt;/span>&lt;span class="s2">/&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">.qcow2,bus=virtio&amp;#34;&lt;/span> --import &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --os-variant fedora-unknown --network &lt;span class="nv">network&lt;/span>&lt;span class="o">=&lt;/span>default &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --cloud-init user-data&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$tmp&lt;/span>&lt;span class="s2">/user-data&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --graphics none --noautoconsole
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">destroy&lt;span class="o">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> virsh destroy &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> 2&amp;gt;/dev/null &lt;span class="o">||&lt;/span> &lt;span class="nb">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> virsh undefine &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --remove-all-storage
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ip&lt;span class="o">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> virsh domifaddr &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --source lease &lt;span class="p">|&lt;/span> awk &lt;span class="s1">&amp;#39;/ipv4/ {sub(&amp;#34;/.*&amp;#34;, &amp;#34;&amp;#34;, $4); print $4}&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">esac&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">rehearsal-vm.sh create mail1-rehearsal &lt;span class="m">8192&lt;/span> &lt;span class="m">4&lt;/span> &lt;span class="m">100&lt;/span> ~/.ssh/id_ed25519.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">rehearsal-vm.sh ip mail1-rehearsal &lt;span class="c1"># e.g. 192.168.122.76&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>virt-install --cloud-init&lt;/code> builds the seed ISO by itself. You need no separate &lt;code>cloud-localds&lt;/code> step.&lt;/p>
&lt;h2 id="step-2-an-inventory-for-the-vms">Step 2: an inventory for the VMs&lt;/h2>
&lt;p>The VMs are in a separate inventory file, which I pass only when I mean it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># inventory/rehearsal.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">all&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mail1-rehearsal&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">192.168.122.76&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_ssh_common_args&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">&amp;gt;-&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> -J me@vmhost.example.net
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> -o UserKnownHostsFile=~/.ssh/known_hosts.rehearsal -o StrictHostKeyChecking=accept-new&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bootstrap_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fedora&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hostname&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">mail1-rehearsal&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># use the real host&amp;#39;s secrets from pass&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">pass_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">mail1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">backup_timer&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">monitoring_tunnel&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># the CI runner would register under the real name and take real jobs&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">docker_compose_projects&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">children&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fedora&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">mail1-rehearsal&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mail&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">mail1-rehearsal&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The trick is where the real host&amp;rsquo;s settings live. They are not in &lt;code>host_vars/mail1.yml&lt;/code>, but in &lt;code>group_vars/mail.yml&lt;/code>, for a group with one member. The VM joins the same group and gets all of them. The few variables in &lt;code>rehearsal.yml&lt;/code> are host variables, and those beat group variables. So the VM can switch off the backup timer, the tunnel and the runner.&lt;/p>
&lt;p>The VMs are recreated often. Each new VM has new host keys, so the VMs get their own &lt;code>known_hosts&lt;/code> file. My real &lt;code>known_hosts&lt;/code> stays clean. &lt;code>accept-new&lt;/code> is acceptable here, and only here: the VM is on a NAT network behind my own machine.&lt;/p>
&lt;h2 id="step-3-guards">Step 3: guards&lt;/h2>
&lt;p>The rehearsal VM holds the real data and the real credentials. Left alone, it would:&lt;/p>
&lt;ul>
&lt;li>deliver mail that is waiting in the restored queue,&lt;/li>
&lt;li>federate with other Matrix servers under the real server name,&lt;/li>
&lt;li>ask Let&amp;rsquo;s Encrypt for certificates of the real domains,&lt;/li>
&lt;li>take real CI jobs as the real runner.&lt;/li>
&lt;/ul>
&lt;p>The inventory handles the runner. For the rest I put an nftables table on the VM that drops the dangerous outgoing traffic. Something like this:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">table inet rehearsal_guard {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> chain output {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> type filter hook output priority 0; policy accept;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> oifname &amp;#34;lo&amp;#34; accept
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> tcp dport { 25, 465, 587, 8448 } drop
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> # once the images are pulled: no new connections from the app user
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> meta skuid &amp;#34;svc&amp;#34; ct state new drop
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> }
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The app user&amp;rsquo;s rule goes in only after the first full run, because Podman must pull the images first. DNS needs one more step: unbound marks the root servers as down while the block is on. Restart it after you lift the block for a test.&lt;/p>
&lt;h2 id="step-4-the-rollout-in-this-order">Step 4: the rollout, in this order&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">I&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;-i inventory/hosts.yml -i inventory/rehearsal.yml&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 1. the ansible user&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook &lt;span class="nv">$I&lt;/span> playbooks/bootstrap.yml -l mail1-rehearsal
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 2. update and reboot at once (see post 5: dnf-automatic would do it mid-run)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible &lt;span class="nv">$I&lt;/span> mail1-rehearsal -b -m shell -a &lt;span class="s1">&amp;#39;dnf -y upgrade &amp;amp;&amp;amp; systemctl reboot&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 3. everything except the apps&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook &lt;span class="nv">$I&lt;/span> playbooks/site.yml -l mail1-rehearsal -t base,monitoring,backup
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 4. restore the state (below), before Traefik starts&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 5. everything&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook &lt;span class="nv">$I&lt;/span> playbooks/site.yml -l mail1-rehearsal
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># 6. once more, with fresh connections: changed=0&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook &lt;span class="nv">$I&lt;/span> playbooks/site.yml -l mail1-rehearsal &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -e &lt;span class="nv">ansible_ssh_common_args&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;-J me@vmhost.example.net -o ControlPath=none -o UserKnownHostsFile=~/.ssh/known_hosts.rehearsal&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Step 3 installs restic and the backup credentials, but no apps. The restore can then put the volumes in place before any container starts on empty data.&lt;/p>
&lt;h2 id="step-5-the-restore">Step 5: the restore&lt;/h2>
&lt;p>The restore script runs on the new host as root and reads the old host&amp;rsquo;s restic repository with &lt;code>--no-lock&lt;/code>. It has three parts, because the order matters:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">restore.sh files # volumes and Traefik&amp;#39;s acme.json: before the apps start
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">restore.sh dbs # database dumps: after the full run (the databases must run)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">restore.sh mailcow # Mailcow&amp;#39;s own backup and its local config: after the full run
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>files&lt;/code> part copies each old bind mount into a new Podman volume with &lt;code>podman unshare tar&lt;/code> (shown in &lt;a href="../../blog/infra-02-quadlets/">post 2&lt;/a>). The &lt;code>dbs&lt;/code> part drops the empty databases that the containers created at their first start, then loads the dumps. The &lt;code>mailcow&lt;/code> part restores Mailcow&amp;rsquo;s own config files over the checkout. Then it runs Mailcow&amp;rsquo;s &lt;code>backup_and_restore.sh restore&lt;/code>, which asks four questions. A &lt;code>printf&lt;/code> answers them:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">printf&lt;/span> &lt;span class="s1">&amp;#39;1\n0\nY\ny\n&amp;#39;&lt;/span> &lt;span class="p">|&lt;/span> &lt;span class="nv">MAILCOW_BACKUP_LOCATION&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$tmp&lt;/span>&lt;span class="s2">/backup&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> /opt/mailcow-dockerized/helper-scripts/backup_and_restore.sh restore
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="step-6-tests">Step 6: tests&lt;/h2>
&lt;p>Test the services against the VM without DNS changes:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">ip&lt;/span>&lt;span class="o">=&lt;/span>192.168.122.76
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">for&lt;/span> h in git.example.org chat.example.org paste.example.org mail1.example.org&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> curl -s -o /dev/null -w &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$h&lt;/span>&lt;span class="s2"> %{http_code} verify=%{ssl_verify_result}\n&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --resolve &lt;span class="nv">$h&lt;/span>:443:&lt;span class="nv">$ip&lt;/span> https://&lt;span class="nv">$h&lt;/span>/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>verify=0&lt;/code> means that the restored certificate is valid for the name. I also compared counts from the old host with the VM: mail domains, mailboxes, aliases, DKIM keys, the number of messages in one INBOX, and the repositories in Forgejo. They must be identical.&lt;/p>
&lt;p>On the VM itself:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo ausearch -m avc -ts boot &lt;span class="c1"># no SELinux denials&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ps -eZ &lt;span class="p">|&lt;/span> grep -c container_t &lt;span class="c1"># every container confined&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Mailcow on Docker with SELinux enforcing gave no denials at all, which surprised me.&lt;/p>
&lt;p>Things a VM cannot test: real mail delivery, certificates from the HTTP challenge (no inbound port 80), the CI runner. I wrote those down as checks for the real cut-over.&lt;/p>
&lt;h2 id="a-provider-without-a-fedora-image">A provider without a Fedora image&lt;/h2>
&lt;p>My VPS provider for the website host offers no Fedora image. It does accept an uploaded qcow2 image, but it has no cloud metadata service. A plain Fedora Cloud image would boot there and wait for cloud-init data that never comes, so there would be no user and no key.&lt;/p>
&lt;p>The fix is to bake the cloud-init data into the image, as a NoCloud seed. I built it from a rehearsal VM:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">rehearsal-vm.sh create image-build &lt;span class="m">2048&lt;/span> &lt;span class="m">2&lt;/span> &lt;span class="m">6&lt;/span> ~/.ssh/id_ed25519.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh fedora@&amp;lt;vm&amp;gt; &lt;span class="s1">&amp;#39;sudo dnf -y upgrade &amp;amp;&amp;amp; sudo systemctl reboot&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh fedora@&amp;lt;vm&amp;gt; &lt;span class="s1">&amp;#39;sudo bash -s&amp;#39;&lt;/span> -- vps2.example.org &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">$(&lt;/span>cat ~/.ssh/id_ed25519.pub&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;lt; seal-image.sh
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The script, slightly shortened:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># seal-image.sh &amp;lt;fqdn&amp;gt; &amp;lt;ssh public key&amp;gt;: run as root inside the build VM&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">fqdn&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">1&lt;/span>&lt;span class="p">:?fqdn&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="nv">key&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">2&lt;/span>&lt;span class="p">:?ssh public key&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">install -d -m &lt;span class="m">0755&lt;/span> /var/lib/cloud/seed/nocloud
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cat &amp;gt; /var/lib/cloud/seed/nocloud/meta-data &lt;span class="s">&amp;lt;&amp;lt;EOF
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">instance-id: ${fqdn%%.*}-$(date +%Y%m%d)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">local-hostname: ${fqdn%%.*}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cat &amp;gt; /var/lib/cloud/seed/nocloud/user-data &lt;span class="s">&amp;lt;&amp;lt;EOF
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">#cloud-config
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">ssh_pwauth: false
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">disable_root: true
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">users:
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> - name: fedora
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> groups: [wheel]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> sudo: &amp;#34;ALL=(ALL) NOPASSWD:ALL&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> lock_passwd: true
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> ssh_authorized_keys: [&amp;#34;$key&amp;#34;]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cat &amp;gt; /etc/cloud/cloud.cfg.d/90-datasource.cfg &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;EOF&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"># No metadata service: only the seed in /var/lib/cloud/seed/nocloud
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">datasource_list: [ NoCloud, None ]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">preserve_hostname: true
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">restorecon -R /var/lib/cloud /etc/cloud
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># The provider&amp;#39;s VMs have no serial port: drop console=ttyS0, keep GRUB on the VNC console&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">grubby --update-kernel&lt;span class="o">=&lt;/span>ALL --remove-args&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;console=ttyS0,115200n8&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sed -i -e &lt;span class="s1">&amp;#39;s/ console=ttyS0,115200n8//&amp;#39;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -e &lt;span class="s1">&amp;#39;s/^GRUB_TERMINAL_INPUT=.*/GRUB_TERMINAL_INPUT=&amp;#34;console&amp;#34;/&amp;#39;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -e &lt;span class="s1">&amp;#39;s/^GRUB_TERMINAL_OUTPUT=.*/GRUB_TERMINAL_OUTPUT=&amp;#34;console&amp;#34;/&amp;#39;&lt;/span> /etc/default/grub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sed -i &lt;span class="s1">&amp;#39;/^GRUB_SERIAL_COMMAND=/d&amp;#39;&lt;/span> /etc/default/grub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">grub2-mkconfig -o /boot/grub2/grub.cfg 2&amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># First-boot state: new machine-id, new host keys, no network profile bound to this MAC&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">rm -f /etc/NetworkManager/system-connections/cloud-init-*
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cloud-init clean --logs --machine-id
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">rm -f /etc/ssh/ssh_host_* /home/fedora/.bash_history /root/.bash_history
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$fqdn&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt; /etc/hostname
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">restorecon /etc/hostname
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">dnf clean all -q
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --rotate -q
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --vacuum-time&lt;span class="o">=&lt;/span>1s -q
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">fstrim -a
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> SEALED
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then power the VM off and compress its disk into the image:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo qemu-img convert -c -O qcow2 /var/lib/libvirt/images/image-build.qcow2 vps2-fedora44.qcow2
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sha256sum vps2-fedora44.qcow2 &amp;gt; vps2-fedora44.qcow2.sha256
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Some details matter here:&lt;/p>
&lt;ul>
&lt;li>&lt;code>preserve_hostname: true&lt;/code>. cloud-init sets the hostname with &lt;code>hostnamectl&lt;/code> so early that D-Bus is not up yet. That fails, and &lt;code>cloud-init status&lt;/code> then reports &amp;ldquo;degraded&amp;rdquo;. The hostname is already in &lt;code>/etc/hostname&lt;/code>, and the base role sets it again anyway.&lt;/li>
&lt;li>The serial console. The provider&amp;rsquo;s VMs have no serial port. With &lt;code>console=ttyS0&lt;/code> on the kernel command line, &lt;code>serial-getty&lt;/code> restarts every few seconds, and GRUB shows its menu only on the serial line. You would never see it in the VNC console.&lt;/li>
&lt;li>&lt;code>cloud-init clean --machine-id&lt;/code>. Without a new machine-id, every host made from the image would have the same one, and the same DHCP client ID too.&lt;/li>
&lt;li>The image contains only the user &lt;code>fedora&lt;/code> with my public key. No secret is in it.&lt;/li>
&lt;/ul>
&lt;p>Test the image the way the provider runs it. Mine boots with BIOS, a virtio-scsi disk and no cloud-init ISO:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">cp vps2-fedora44.qcow2 test.qcow2 &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> qemu-img resize test.qcow2 117G
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">virt-install --name image-test --memory &lt;span class="m">2048&lt;/span> --vcpus &lt;span class="m">2&lt;/span> --import &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --controller &lt;span class="nv">type&lt;/span>&lt;span class="o">=&lt;/span>scsi,model&lt;span class="o">=&lt;/span>virtio-scsi --disk test.qcow2,bus&lt;span class="o">=&lt;/span>scsi &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> --os-variant fedora-unknown --network &lt;span class="nv">network&lt;/span>&lt;span class="o">=&lt;/span>default --graphics none --noautoconsole
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh fedora@&amp;lt;&lt;span class="nb">test&lt;/span> vm&amp;gt; cloud-init status --long &lt;span class="c1"># status: done, DataSourceNoCloud&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>cloud-init grows the root filesystem to the full disk at the first boot.&lt;/p>
&lt;h2 id="the-cut-over">The cut-over&lt;/h2>
&lt;p>With a passed rehearsal, the cut-over is a list:&lt;/p>
&lt;ol>
&lt;li>On the old host: stop what writes (the web proxy, the mail daemons), run a last backup, check its log and note the snapshot ID.&lt;/li>
&lt;li>Reinstall the host in the provider&amp;rsquo;s panel, with the provider&amp;rsquo;s Fedora image or the uploaded one. The IP address stays.&lt;/li>
&lt;li>Check the host key over the console (below).&lt;/li>
&lt;li>Inventory: the host&amp;rsquo;s &lt;code>bootstrap_user&lt;/code>, and the host into the &lt;code>fedora&lt;/code> group.&lt;/li>
&lt;li>The same six rollout steps as on the VM, with &lt;code>-l mail1&lt;/code>, and with the snapshot ID for the restore.&lt;/li>
&lt;li>The checks that the VM could not do: real mail in both directions, Mailcow&amp;rsquo;s certificate, the runner online, a deploy of a site, the monitoring targets, the first backup run.&lt;/li>
&lt;/ol>
&lt;p>The provider&amp;rsquo;s reinstall offered Fedora 43 only. Upgrading to 44 right after the bootstrap took under two minutes on the VM:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo dnf -y upgrade --refresh
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo dnf -y system-upgrade download --releasever&lt;span class="o">=&lt;/span>&lt;span class="m">44&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo dnf -y offline reboot
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="checking-the-new-host-key">Checking the new host key&lt;/h2>
&lt;p>After a reinstall, ssh warns that the host key changed. That is expected, and it is exactly the moment an attacker in the middle would use. So I check the new key over a channel that the attacker does not control: the provider&amp;rsquo;s VNC console.&lt;/p>
&lt;p>cloud-init prints the host key fingerprints during the first boot, but they scroll by quickly. A trick that always works:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># keep the new key apart from the real known_hosts&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh -o &lt;span class="nv">UserKnownHostsFile&lt;/span>&lt;span class="o">=&lt;/span>/tmp/new_host -o &lt;span class="nv">StrictHostKeyChecking&lt;/span>&lt;span class="o">=&lt;/span>accept-new fedora@vps2.example.org &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="s2">&amp;#34;printf &amp;#39;%s\n&amp;#39; \&amp;#34;code 7341: \$(ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub)\&amp;#34; | sudo tee /dev/tty1&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Pick a new random code each time. If the code appears in the VNC console with the fingerprint, the connection reached the real host and not some machine in between. Then compare the fingerprint with &lt;code>ssh-keygen -lf /tmp/new_host&lt;/code>, and only then put the key into your real &lt;code>known_hosts&lt;/code>.&lt;/p>
&lt;p>The second part of this series removes this step: with host certificates, the host proves its identity by itself.&lt;/p>
&lt;h2 id="what-the-rehearsals-found">What the rehearsals found&lt;/h2>
&lt;ul>
&lt;li>Pi-hole binds its DNS port to an address on the WireGuard interface. The WireGuard role therefore has to run before the apps, not after them.&lt;/li>
&lt;li>Fedora&amp;rsquo;s &lt;code>dockerd&lt;/code> already runs with &lt;code>--selinux-enabled&lt;/code>. Adding it to &lt;code>daemon.json&lt;/code> as well stops the daemon. Also, &lt;code>moby-engine&lt;/code> creates no &lt;code>/etc/docker&lt;/code> directory.&lt;/li>
&lt;li>Element&amp;rsquo;s nginx cannot bind port 80 as a rootless user. It has a setting for another port.&lt;/li>
&lt;li>Mailcow&amp;rsquo;s &lt;code>update.sh&lt;/code> makes local Git commits in its checkout, so my role checks out the release tag and not the branch.&lt;/li>
&lt;li>The first &lt;code>base&lt;/code> run locked out the &lt;code>ansible&lt;/code> user on a fresh host (the &lt;code>AllowUsers&lt;/code> issue in post 5). A reused SSH connection had hidden it during the first rehearsal.&lt;/li>
&lt;/ul>
&lt;p>Each of these would have cost me time during a real cut-over, with mail down.&lt;/p>
&lt;p>This was the last post of part 1. Part 2 is about SSH: &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://fedoraproject.org/cloud/">Fedora Cloud&lt;/a>: the qcow2 images used for the rehearsal VMs.&lt;/li>
&lt;li>&lt;a href="https://cloudinit.readthedocs.io/en/latest/reference/datasources/nocloud.html">cloud-init NoCloud datasource&lt;/a>: the seed that the custom image carries.&lt;/li>
&lt;li>&lt;a href="https://libvirt.org/docs.html">libvirt documentation&lt;/a>: domains, networks and storage.&lt;/li>
&lt;li>&lt;a href="https://github.com/virt-manager/virt-manager/blob/main/man/virt-install.rst">virt-install&lt;/a>: all options, including &lt;code>--cloud-init&lt;/code> and &lt;code>--import&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://libguestfs.org/virt-customize.1.html">virt-customize&lt;/a>: changes a disk image without booting it. An alternative to a build VM.&lt;/li>
&lt;li>&lt;a href="https://docs.fedoraproject.org/en-US/quick-docs/upgrading-fedora-offline/">Upgrading Fedora using the DNF system upgrade&lt;/a>: the upgrade path from the end of this post.&lt;/li>
&lt;/ul></description></item><item><title>Monitoring and backups for every host</title><link>https://www.linux-lunatix.org/blog/infra-07-monitoring-backup/</link><pubDate>Fri, 23 Oct 2026 20:20:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-07-monitoring-backup/</guid><description>&lt;p>&lt;em>Part 7 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>Every host gets the same two things from Ansible: a monitoring agent and a backup. This post shows both. The monitoring part is mostly about the network: how Prometheus at home reaches the VPS without exposing anything to the internet. The backup part is one script template that works for all hosts.&lt;/p>
&lt;h2 id="the-monitoring-network">The monitoring network&lt;/h2>
&lt;p>Prometheus runs on my home server. The VPS are somewhere on the internet. I did not want node_exporter&amp;rsquo;s port open to the world, and I did not want the VPS to be able to reach my home network.&lt;/p>
&lt;p>The solution is a WireGuard tunnel from each VPS to the home server. I call it &lt;code>wg-mon&lt;/code>.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">vps1 (10.99.0.13) ──┐
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">vps2 (10.99.0.12) ──┼── WireGuard ──► home1 wg0 (10.99.0.1) ── Prometheus
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">mail1 (10.99.0.11) ─┘
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Two firewall rules make it safe in both directions:&lt;/p>
&lt;ol>
&lt;li>On each VPS, the tunnel interface is in a firewalld zone of its own, and only port 9100 (node_exporter) is open there.&lt;/li>
&lt;li>On the home server, the VPS tunnel addresses land in firewalld&amp;rsquo;s &lt;code>block&lt;/code> zone. Prometheus can open connections to them, and the replies come back. A VPS cannot open a single connection to the home server.&lt;/li>
&lt;/ol>
&lt;p>If somebody takes over a VPS, they get a tunnel that leads nowhere.&lt;/p>
&lt;p>My own devices (laptop, phone) use the same &lt;code>wg0&lt;/code> interface as &amp;ldquo;road warriors&amp;rdquo;. Their addresses go into the &lt;code>trusted&lt;/code> zone, so they reach the LAN.&lt;/p>
&lt;h3 id="the-vps-side">The VPS side&lt;/h3>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2"># /etc/wireguard/wg-mon.conf
[Interface]
Address = {{ tunnel_ip }}/32
PostUp = wg set %i private-key /etc/wireguard/wg-mon.key
[Peer]
PublicKey = {{ wg_mon_server_public_key }}
Endpoint = {{ wg_mon_server_endpoint }}
AllowedIPs = {{ wg_mon_server_ip }}/32
PersistentKeepalive = 25
&lt;/code>&lt;/pre>&lt;p>The private key is in a separate file, written from pass with &lt;code>no_log&lt;/code>. The config file can then show in a diff without leaking the key. &lt;code>PersistentKeepalive&lt;/code> keeps the NAT mapping at home open, so that Prometheus can always reach the VPS.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Tunnel up&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">wg-quick@wg-mon&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">enabled&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">started&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">firewalld zone for the tunnel&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">present&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring_zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># A new zone exists only after a reload&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Load the new zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">firewall-cmd --reload&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring_zone.changed and not ansible_check_mode&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Tunnel interface in the zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">interface&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">wg-mon&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">node_exporter port in the zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">port&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">9100&lt;/span>&lt;span class="l">/tcp&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="the-hub-side-generated-from-the-inventory">The hub side, generated from the inventory&lt;/h3>
&lt;p>Each host&amp;rsquo;s public key and tunnel address are in the inventory (&lt;code>wg_mon_public_key&lt;/code>, &lt;code>tunnel_ip&lt;/code>). The hub&amp;rsquo;s config loops over all hosts, so a new VPS needs no extra step:&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2"># /etc/wireguard/wg0.conf on home1
[Interface]
PrivateKey = {{ lookup(&amp;#39;community.general.passwordstore&amp;#39;, wg_hub_key_pass) }}
Address = {{ wg_hub_address }}
ListenPort = {{ wg_hub_port }}
{% for p in wg_hub_peers %}
# {{ p.name }}
[Peer]
PublicKey = {{ p.public_key }}
PresharedKey = {{ lookup(&amp;#39;community.general.passwordstore&amp;#39;, p.psk_pass) }}
AllowedIPs = {{ p.ip }}/32
{% endfor %}
{% for h in groups[&amp;#39;all&amp;#39;] if h != inventory_hostname and hostvars[h].wg_mon_public_key is defined %}
# {{ h }} (monitoring)
[Peer]
PublicKey = {{ hostvars[h].wg_mon_public_key }}
AllowedIPs = {{ hostvars[h].tunnel_ip }}/32
{% endfor %}
&lt;/code>&lt;/pre>&lt;p>This file contains the private and preshared keys, so its task has &lt;code>no_log: true&lt;/code> and &lt;code>diff: false&lt;/code>. The handler reloads the interface with &lt;code>wg syncconf&lt;/code> (&lt;code>systemctl reload wg-quick@wg0&lt;/code>). Peers change without dropping the open tunnels.&lt;/p>
&lt;p>The firewall on the hub:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Road-warrior peers are trusted&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">trusted&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">source&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.ip }}/32&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ wg_hub_peers }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.name }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">All other wg0 traffic is blocked&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">block&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">interface&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">wg0&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>firewalld picks a zone by source address first, and by interface second. A packet from a road warrior&amp;rsquo;s address matches &lt;code>trusted&lt;/code>. Everything else on &lt;code>wg0&lt;/code> falls through to &lt;code>block&lt;/code>.&lt;/p>
&lt;p>A sender on the LAN could fake a road warrior&amp;rsquo;s address. The base role&amp;rsquo;s sysctl settings turn on strict reverse path filtering (&lt;code>net.ipv4.conf.all.rp_filter = 1&lt;/code>), and the kernel drops such packets.&lt;/p>
&lt;p>Sometimes one VPS must reach exactly one service at home. My CI runner needs one port for a later part of this series. For that the role has a list of exceptions, as firewalld rich rules in the &lt;code>block&lt;/code> zone:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/home1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">wg_hub_block_exceptions&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">source: 10.99.0.11, port&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">8443&lt;/span>&lt;span class="l">/tcp}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Exceptions in the block zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">zone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">block&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">rich_rule&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">&amp;gt;-&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> rule family=&amp;#34;ipv4&amp;#34; source address=&amp;#34;{{ item.source }}&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> port port=&amp;#34;{{ item.port.split(&amp;#39;/&amp;#39;)[0] }}&amp;#34; protocol=&amp;#34;{{ item.port.split(&amp;#39;/&amp;#39;)[1] }}&amp;#34; accept&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ wg_hub_block_exceptions }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="node-exporter">node_exporter&lt;/h3>
&lt;p>node_exporter runs as a root quadlet (&lt;code>/etc/containers/systemd/&lt;/code>), because it needs to see the whole host. It listens only on the tunnel address:&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2"># /etc/containers/systemd/node-exporter.container
[Unit]
Description=Prometheus node_exporter (monitoring tunnel address only)
Wants=wg-quick@wg-mon.service
After=wg-quick@wg-mon.service
[Container]
ContainerName=node-exporter
Image=quay.io/prometheus/node-exporter:latest
AutoUpdate=registry
Network=host
PodmanArgs=--pid=host
Volume=/:/host:ro,rslave
Volume=/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket:ro
# Confined: its own SELinux type (post 4)
SecurityLabelType=node_exporter.process
Exec=--path.rootfs=/host --web.listen-address={{ tunnel_ip }}:9100 --collector.systemd --collector.textfile.directory=/host/var/lib/node_exporter/textfile
[Service]
Restart=always
[Install]
WantedBy=multi-user.target
&lt;/code>&lt;/pre>&lt;p>The textfile directory is how scripts report to Prometheus. A script writes a &lt;code>.prom&lt;/code> file, and node_exporter serves its content. My backup script and a daily update check use it.&lt;/p>
&lt;h3 id="prometheus">Prometheus&lt;/h3>
&lt;p>On the home server the targets are a short static list:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># prometheus.yml (part)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">scrape_configs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">job_name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">node-remote&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">static_configs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">targets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="m">10.99.0.11&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="m">9100&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">labels&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">mail1}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">targets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="m">10.99.0.12&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="m">9100&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">labels&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps2}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">targets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="m">10.99.0.13&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="m">9100&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">labels&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps1}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The tunnel addresses come from the inventory, so you could render this list from &lt;code>hostvars&lt;/code> too. With four hosts I kept it static and readable.&lt;/p>
&lt;p>Alertmanager sends alerts to a self-hosted ntfy. That covers &amp;ldquo;a host is down&amp;rdquo;. It does not cover &amp;ldquo;the home server, where Prometheus and Alertmanager run, is down&amp;rdquo;. For that, the next section uses an external dead man&amp;rsquo;s switch.&lt;/p>
&lt;h2 id="backups-one-script-for-every-host">Backups: one script for every host&lt;/h2>
&lt;p>Each host backs up with &lt;a href="https://restic.net/">restic&lt;/a> to its own repository on an SFTP backup box. The hosts differ in what they must dump first (databases, Mailcow) and in what they exclude. Everything else is the same. So there is one script template, with the host-specific parts as variables.&lt;/p>
&lt;h3 id="the-settings">The settings&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/vps1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_repository&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sftp:backup-box:vps1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_password_pass&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps1/restic&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_time&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;03:15&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_shell_vars&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">STORAGE=$APP_HOME/.local/share/containers/storage&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_args&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- --&lt;span class="l">exclude &amp;#34;$STORAGE/overlay&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- --&lt;span class="l">exclude &amp;#34;$STORAGE/overlay-images&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">/etc&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">$APP_HOME&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">$DUMP_DIR&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The container image layers are excluded. They can be pulled again and are the largest part of the storage. The volumes stay in.&lt;/p>
&lt;h3 id="the-script">The script&lt;/h3>
&lt;p>This is the template, shortened a little:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># /usr/local/sbin/backup-{{ backup_name }}.sh, from the backup role&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -uo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">RESTIC_REPOSITORY&lt;/span>&lt;span class="o">={{&lt;/span> backup_repository &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">RESTIC_PASSWORD_FILE&lt;/span>&lt;span class="o">=&lt;/span>/root/.config/restic/&lt;span class="o">{{&lt;/span> backup_name &lt;span class="o">}}&lt;/span>.pass
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># systemd services have no $HOME, so restic needs an explicit cache dir&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">RESTIC_CACHE_DIR&lt;/span>&lt;span class="o">=&lt;/span>/var/cache/restic
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">HC_PING_URL&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>cat /root/.config/restic/&lt;span class="o">{{&lt;/span> backup_name &lt;span class="o">}}&lt;/span>.hc 2&amp;gt;/dev/null &lt;span class="o">||&lt;/span> &lt;span class="nb">true&lt;/span>&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">APP_USER&lt;/span>&lt;span class="o">={{&lt;/span> app_user &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">APP_HOME&lt;/span>&lt;span class="o">={{&lt;/span> app_home &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">{{&lt;/span> backup_shell_vars &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">DUMP_DIR&lt;/span>&lt;span class="o">=&lt;/span>/var/backups/&lt;span class="o">{{&lt;/span> backup_name &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">failed&lt;/span>&lt;span class="o">=()&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">START&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>date +%s&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">hc_ping&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="o">[&lt;/span> -n &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$HC_PING_URL&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">]&lt;/span> &lt;span class="o">||&lt;/span> &lt;span class="k">return&lt;/span> &lt;span class="m">0&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> curl -fsS -m &lt;span class="m">10&lt;/span> --retry &lt;span class="m">3&lt;/span> -o /dev/null --data-binary @&amp;lt;&lt;span class="o">(&lt;/span>tail -n &lt;span class="m">100&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$RUN_LOG&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="o">)&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$HC_PING_URL$1&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">||&lt;/span> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;healthchecks ping failed&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Run a step, note a failure, go on with the next step&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">step&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> &lt;span class="nv">name&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nv">$1&lt;/span> rc
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">shift&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;=== &lt;/span>&lt;span class="k">$(&lt;/span>date &lt;span class="s1">&amp;#39;+%F %T&amp;#39;&lt;/span>&lt;span class="k">)&lt;/span>&lt;span class="s2"> &lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$@&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">rc&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nv">$?&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="o">[&lt;/span> &lt;span class="nv">$rc&lt;/span> -ne &lt;span class="m">0&lt;/span> &lt;span class="o">]&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;!!! &lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2"> failed (exit &lt;/span>&lt;span class="nv">$rc&lt;/span>&lt;span class="s2">)&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">failed&lt;/span>&lt;span class="o">+=(&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$name&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="o">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">return&lt;/span> &lt;span class="nv">$rc&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">app_podman&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> runuser -u &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$APP_USER&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> -- env &lt;span class="nv">XDG_RUNTIME_DIR&lt;/span>&lt;span class="o">=&lt;/span>/run/user/&lt;span class="k">$(&lt;/span>id -u &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$APP_USER&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span> podman &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$@&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Podman secrets are in no volume and not in Git: keep them in the encrypted backup&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">secrets_dump&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> s
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">for&lt;/span> s in &lt;span class="k">$(&lt;/span>app_podman secret ls --format &lt;span class="s1">&amp;#39;{{.Name}}&amp;#39;&lt;/span>&lt;span class="k">)&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> app_podman secret inspect --showsecret --format &lt;span class="s1">&amp;#39;{{.SecretData}}&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$s&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt; &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$DUMP_DIR&lt;/span>&lt;span class="s2">/secrets/&lt;/span>&lt;span class="nv">$s&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">||&lt;/span> &lt;span class="k">return&lt;/span> &lt;span class="m">1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">exec&lt;/span> 9&amp;gt;/run/backup-&lt;span class="o">{{&lt;/span> backup_name &lt;span class="o">}}&lt;/span>.lock
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> ! flock -n 9&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;previous backup run still in progress&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> hc_ping /fail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">exit&lt;/span> &lt;span class="m">1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">hc_ping /start
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">umask&lt;/span> &lt;span class="m">077&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">mkdir -p &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$DUMP_DIR&lt;/span>&lt;span class="s2">/secrets&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">{{&lt;/span> backup_pre_steps &lt;span class="o">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">step &lt;span class="s2">&amp;#34;podman secrets&amp;#34;&lt;/span> secrets_dump
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">step &lt;span class="s2">&amp;#34;restic unlock&amp;#34;&lt;/span> restic unlock
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> step &lt;span class="s2">&amp;#34;restic backup&amp;#34;&lt;/span> restic backup --verbose&lt;span class="o">=&lt;/span>&lt;span class="m">0&lt;/span> --exclude-caches &lt;span class="o">{{&lt;/span> backup_args &lt;span class="p">|&lt;/span> join&lt;span class="o">(&lt;/span>&lt;span class="s1">&amp;#39; &amp;#39;&lt;/span>&lt;span class="o">)&lt;/span> &lt;span class="o">}}&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> step &lt;span class="s2">&amp;#34;restic forget/prune&amp;#34;&lt;/span> restic forget --prune --keep-daily &lt;span class="m">7&lt;/span> --keep-weekly &lt;span class="m">4&lt;/span> --keep-monthly &lt;span class="m">6&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">step &lt;span class="s2">&amp;#34;restic check&amp;#34;&lt;/span> restic check
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">write_metrics &lt;span class="k">$((&lt;/span> &lt;span class="si">${#&lt;/span>&lt;span class="nv">failed&lt;/span>&lt;span class="p">[@]&lt;/span>&lt;span class="si">}&lt;/span> &lt;span class="o">==&lt;/span> &lt;span class="m">0&lt;/span> &lt;span class="k">))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> &lt;span class="o">[&lt;/span> &lt;span class="si">${#&lt;/span>&lt;span class="nv">failed&lt;/span>&lt;span class="p">[@]&lt;/span>&lt;span class="si">}&lt;/span> -gt &lt;span class="m">0&lt;/span> &lt;span class="o">]&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> hc_ping /fail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">exit&lt;/span> &lt;span class="m">1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">hc_ping &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The Go template syntax in &lt;code>secrets_dump&lt;/code> (&lt;code>{{.Name}}&lt;/code>) clashes with Jinja. The real template therefore starts with a header that changes Jinja&amp;rsquo;s comment markers and wraps those lines in &lt;code>{% raw %}&lt;/code>. I left that out here to keep the script readable.&lt;/p>
&lt;p>Some decisions in it:&lt;/p>
&lt;ul>
&lt;li>The script does not stop at the first error. A failed database dump should not prevent the file backup. Each failed step is collected, and the end result is &amp;ldquo;failed&amp;rdquo; if any step failed.&lt;/li>
&lt;li>&lt;code>flock&lt;/code> prevents two runs at the same time, for example when a run takes longer than the interval.&lt;/li>
&lt;li>&lt;code>restic unlock&lt;/code> removes stale locks from a run that was killed. &lt;code>restic check&lt;/code> at the end verifies the repository structure. It does not read all data, which would be slow over SFTP. &lt;code>restic check --read-data-subset=5%&lt;/code> reads a sample, if you want more.&lt;/li>
&lt;/ul>
&lt;h3 id="host-specific-steps">Host-specific steps&lt;/h3>
&lt;p>A host adds dump functions and steps as shell code in its variables:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/mail1.yml (part)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_functions&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> forgejo_dump() {
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> app_podman exec forgejo-postgres pg_dumpall -U forgejo | gzip &amp;gt; &amp;#34;$DUMP_DIR/db/forgejo.sql.gz&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> }&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">backup_pre_steps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> step &amp;#34;forgejo db&amp;#34; forgejo_dump&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="healthchecks-dot-io-the-dead-man-s-switch">healthchecks.io: the dead man&amp;rsquo;s switch&lt;/h3>
&lt;p>The script pings &lt;a href="https://healthchecks.io">healthchecks.io&lt;/a> at the start, at the end, and with &lt;code>/fail&lt;/code> on errors. The service alerts me when a ping is late. So I also hear about a backup that never started, because the host is down or the timer broke. Monitoring on the same host could never report that. The last 100 lines of the run&amp;rsquo;s log go with the ping, so the alert e-mail shows what failed.&lt;/p>
&lt;h3 id="metrics-for-prometheus">Metrics for Prometheus&lt;/h3>
&lt;p>At the end the script writes &lt;code>/var/lib/node_exporter/textfile/backup.prom&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">backup_last_run_timestamp_seconds 1761246000
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">backup_last_success_timestamp_seconds 1761246000
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">backup_last_run_success 1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">backup_last_duration_seconds 312
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>It writes to a temporary file first and then renames it. node_exporter never reads a half-written file. A Grafana overview shows the age of the last good backup per host. A failed run also makes the systemd unit fail, and an alert rule for failed units catches that.&lt;/p>
&lt;h3 id="the-timer">The timer&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># /etc/systemd/system/backup-vps1.timer&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Unit]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Description&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">Daily backup of vps1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Timer]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">OnCalendar&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">*-*-* 03:15&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">RandomizedDelaySec&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">15min&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Persistent&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">timers.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>Persistent=true&lt;/code> runs a missed backup after the host was off at 03:15.&lt;/p>
&lt;h2 id="restore-at-least-once">Restore, at least once&lt;/h2>
&lt;p>A backup that was never restored is a hope. I restored each host completely during the rehearsals in the next post. The restore from the old Arch host reported errors, because restic tried to set SELinux labels that the backup never had (see post 4). The files were fine.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">R&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;restic -r sftp:backup-box:vps1 --password-file /root/.config/restic/vps1.pass --no-lock&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">$R&lt;/span> snapshots
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">$R&lt;/span> restore latest --target /var/tmp/restore --include /home/svc/traefik/acme --exclude-xattr security.selinux
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>--no-lock&lt;/code> lets a test VM read the live repository without writing a lock into it.&lt;/p>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo systemctl start backup-vps1.service
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo tail -20 /var/log/backup-vps1.log
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -s http://10.99.0.13:9100/metrics &lt;span class="p">|&lt;/span> grep ^backup_ &lt;span class="c1"># from the hub&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Next: &lt;a href="../../blog/infra-08-rehearsal/">Rehearse before you reinstall&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://www.wireguard.com/quickstart/">WireGuard quick start&lt;/a>: keys, peers and &lt;code>wg-quick&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://firewalld.org/documentation/">firewalld documentation&lt;/a>: zones and policies, for the hub&amp;rsquo;s &lt;code>block&lt;/code> zone.&lt;/li>
&lt;li>&lt;a href="https://prometheus.io/docs/introduction/overview/">Prometheus documentation&lt;/a>: scrape configs, rules and alerts.&lt;/li>
&lt;li>&lt;a href="https://github.com/prometheus/node_exporter">node_exporter&lt;/a>: the collectors and the textfile directory.&lt;/li>
&lt;li>&lt;a href="https://github.com/prometheus/blackbox_exporter">blackbox_exporter&lt;/a>: HTTP, TCP and ICMP probes.&lt;/li>
&lt;li>&lt;a href="https://restic.readthedocs.io/en/stable/">restic documentation&lt;/a>: repositories, retention with &lt;code>forget --prune&lt;/code>, checks and restore.&lt;/li>
&lt;li>&lt;a href="https://healthchecks.io/docs/">healthchecks.io documentation&lt;/a>: pings, start and fail signals, and grace times.&lt;/li>
&lt;/ul></description></item><item><title>One generic Ansible role for every app</title><link>https://www.linux-lunatix.org/blog/infra-06-quadlet-app/</link><pubDate>Sun, 18 Oct 2026 16:30:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-06-quadlet-app/</guid><description>&lt;p>&lt;em>Part 6 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>My hosts run close to 30 apps, some of them with several containers. I did not want 30 Ansible roles. This post shows the one role that deploys all of them, and how I used it to take over running hosts without restarting anything.&lt;/p>
&lt;h2 id="the-idea">The idea&lt;/h2>
&lt;p>Each app is a directory:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">apps/forgejo/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── app.yml # optional: secrets, directories, units
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── quadlets/ # -&amp;gt; ~svc/.config/containers/systemd/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo.container
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo-postgres.container
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo.network
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo-data.volume
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── forgejo-db.volume
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── files/ # -&amp;gt; ~svc/ (same relative paths)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── .config/forgejo/...
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">└── tasks.yml # optional: extra tasks (downloads, checkouts)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A host lists its apps:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/vps1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">app_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">svc&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">apps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">traefik, ntfy, forgejo-runner]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The role &lt;code>quadlet_app&lt;/code> loops over the list. For each app it does the same steps:&lt;/p>
&lt;ol>
&lt;li>Read &lt;code>app.yml&lt;/code>.&lt;/li>
&lt;li>Create the Podman secrets it names, from pass.&lt;/li>
&lt;li>Copy &lt;code>files/&lt;/code> and &lt;code>quadlets/&lt;/code> into the app user&amp;rsquo;s home. Files that end in &lt;code>.j2&lt;/code> are templates.&lt;/li>
&lt;li>If anything changed: run &lt;code>daemon-reload&lt;/code> and restart the app&amp;rsquo;s units.&lt;/li>
&lt;li>Make sure the units run.&lt;/li>
&lt;/ol>
&lt;p>An app that did not change is not touched. That last point matters most. A playbook run on a host with 20 apps restarts exactly the apps I changed.&lt;/p>
&lt;h2 id="why-plain-files-and-not-a-module">Why plain files and not a module&lt;/h2>
&lt;p>&lt;code>containers.podman.podman_container&lt;/code> can create containers from Ansible, and newer versions can write quadlets too. I chose plain quadlet files in the repository instead:&lt;/p>
&lt;ul>
&lt;li>They are the same files that are on the host. &lt;code>diff&lt;/code> shows exactly what changes.&lt;/li>
&lt;li>I can copy one to a test machine and run it without Ansible.&lt;/li>
&lt;li>The Podman documentation for quadlets applies 1:1. There is no translation layer whose options I must look up.&lt;/li>
&lt;/ul>
&lt;h2 id="app-dot-yml">app.yml&lt;/h2>
&lt;p>Most apps need no &lt;code>app.yml&lt;/code> at all. When one is there, it can say:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># apps/forgejo/app.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">secrets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">name: forgejo-db-password, pass&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">infra/&amp;lt;host&amp;gt;/forgejo-db-password}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># apps/traefik/app.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">secrets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">name: dns-api-token, pass: infra/&amp;lt;host&amp;gt;/dns-api-token, if&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">traefik_dns_challenge}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">dirs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">path: .config/traefik, mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0700&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">traefik}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">path: traefik/acme, mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0700&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Key&lt;/th>
&lt;th>Meaning&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>secrets&lt;/code>&lt;/td>
&lt;td>Podman secrets: name, pass entry (&lt;code>&amp;lt;host&amp;gt;&lt;/code> is replaced), optional &lt;code>if&lt;/code> (a host variable that must be true)&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>secret_files&lt;/code>&lt;/td>
&lt;td>files from &lt;code>files/&lt;/code> with mode 0600 that never show up in a diff&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>dirs&lt;/code>&lt;/td>
&lt;td>directories to create, with an optional mode&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>units&lt;/code>&lt;/td>
&lt;td>the units to restart; default: one &lt;code>.service&lt;/code> per &lt;code>.container&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>enable&lt;/code>&lt;/td>
&lt;td>plain user units or timers to enable and start&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>restart&lt;/code>&lt;/td>
&lt;td>&lt;code>restart&lt;/code>, or &lt;code>stop-start&lt;/code> for containers that share a network namespace&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;h2 id="the-role-step-by-step">The role, step by step&lt;/h2>
&lt;p>The role&amp;rsquo;s &lt;code>main.yml&lt;/code> creates the quadlet directory once, then includes &lt;code>app.yml&lt;/code> for each app:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># roles/quadlet_app/tasks/main.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Quadlet directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_home }}/{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">owner&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">group&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">.config, .config/containers, .config/containers/systemd]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Apps&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.include_tasks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ apps }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_var&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_name&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>app_home&lt;/code>, &lt;code>app_uid&lt;/code> and &lt;code>app_gid&lt;/code> come from the &lt;code>podman&lt;/code> role, which creates the user, enables linger and starts the user&amp;rsquo;s &lt;code>podman.socket&lt;/code> and &lt;code>podman-auto-update.timer&lt;/code>.&lt;/p>
&lt;h3 id="read-the-definition-and-the-source-tree">Read the definition and the source tree&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: definition&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.set_fact&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_defaults | combine(app_file | from_yaml or {}) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ apps_dir }}/{{ app_name }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vars&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;ansible.builtin.file&amp;#39;, apps_dir ~ &amp;#39;/&amp;#39; ~ app_name ~ &amp;#39;/app.yml&amp;#39;, errors=&amp;#39;ignore&amp;#39;) | default(&amp;#39;&amp;#39;, true) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_defaults&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">secrets: [], secret_files: [], dirs: [], units: null, enable: [], restart&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">restart}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: source tree&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.set_fact&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_tree&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">files&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.filetree&amp;#39;, app_src ~ &amp;#39;/files&amp;#39;, wantlist=true) | selectattr(&amp;#39;state&amp;#39;, &amp;#39;eq&amp;#39;, &amp;#39;file&amp;#39;) | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dirs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.filetree&amp;#39;, app_src ~ &amp;#39;/files&amp;#39;, wantlist=true) | selectattr(&amp;#39;state&amp;#39;, &amp;#39;eq&amp;#39;, &amp;#39;directory&amp;#39;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> | map(attribute=&amp;#39;path&amp;#39;) | map(&amp;#39;community.general.dict_kv&amp;#39;, &amp;#39;path&amp;#39;) | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">quadlets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.filetree&amp;#39;, app_src ~ &amp;#39;/quadlets&amp;#39;, wantlist=true) | selectattr(&amp;#39;state&amp;#39;, &amp;#39;eq&amp;#39;, &amp;#39;file&amp;#39;) | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>filetree&lt;/code> lookup walks a directory on the control node and returns every file with its relative path and mode. That is all the role needs to know about an app.&lt;/p>
&lt;h3 id="secrets">Secrets&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: podman secrets&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">containers.podman.podman_secret&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.name }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">data&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.passwordstore&amp;#39;, item.pass | replace(&amp;#39;&amp;lt;host&amp;gt;&amp;#39;, inventory_hostname)) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">present&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app.secrets }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.name }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">item.if is not defined or lookup(&amp;#39;ansible.builtin.vars&amp;#39;, item.if) | bool&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">become_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">XDG_RUNTIME_DIR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/run/user/{{ app_uid }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">no_log&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_secrets_result&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>become_user&lt;/code> plus &lt;code>XDG_RUNTIME_DIR&lt;/code> is how every task in this role talks to the rootless user&amp;rsquo;s Podman and systemd. &lt;code>become_user&lt;/code> to an unprivileged user needs the &lt;code>acl&lt;/code> package on the host, so that Ansible can hand its temporary files to that user. The base role installs it.&lt;/p>
&lt;h3 id="files-and-quadlets">Files and quadlets&lt;/h3>
&lt;p>Plain files are copied, and &lt;code>.j2&lt;/code> files are rendered. Executables stay executable. Secret files get mode 0600 and no diff:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: files&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.src }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_home }}/{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">owner&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">group&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ &amp;#39;0600&amp;#39; if item.path in app.secret_files else (&amp;#39;0755&amp;#39; if item.mode[1] in &amp;#39;157&amp;#39; else &amp;#39;0644&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_tree.files | rejectattr(&amp;#39;path&amp;#39;, &amp;#39;search&amp;#39;, &amp;#39;\\.j2$&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">no_log&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path in app.secret_files }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_files_result&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: templated files&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.template&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.src }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_home }}/{{ item.path | regex_replace(&amp;#39;\\.j2$&amp;#39;, &amp;#39;&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">owner&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">group&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_tree.files | selectattr(&amp;#39;path&amp;#39;, &amp;#39;search&amp;#39;, &amp;#39;\\.j2$&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_templates_result&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The quadlets work the same way, with &lt;code>.config/containers/systemd/&lt;/code> as the destination. Why are only &lt;code>.j2&lt;/code> files templates? Some apps ship files that contain their own &lt;code>{{ }}&lt;/code> syntax: Grafana dashboards, Prometheus alert rules. Rendering those with Jinja would break them. So a file is a template only when its name says so.&lt;/p>
&lt;p>A quadlet template is useful for values that differ per host:&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2"># apps/traefik/quadlets/traefik.container.j2 (part)
UserNS=keep-id:uid={{ app_uid }},gid={{ app_gid }}
User={{ app_uid }}:{{ app_gid }}
{% for v in traefik_extra_volumes | default([]) %}
Volume={{ v }}
{% endfor %}
{% if traefik_dns_challenge %}
Secret=dns-api-token,type=env,target=CF_DNS_API_TOKEN
{% endif %}
&lt;/code>&lt;/pre>&lt;h3 id="restart-only-what-changed">Restart only what changed&lt;/h3>
&lt;p>Every task above registers its result. The role collects all of them and asks one question: did anything change for this app?&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: units&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.set_fact&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_units&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app.units if app.units is not none else
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> (app_tree.quadlets | map(attribute=&amp;#39;path&amp;#39;) | map(&amp;#39;regex_replace&amp;#39;, &amp;#39;\\.j2$&amp;#39;, &amp;#39;&amp;#39;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> | select(&amp;#39;match&amp;#39;, &amp;#39;.*\\.container$&amp;#39;) | map(&amp;#39;regex_replace&amp;#39;, &amp;#39;\\.container$&amp;#39;, &amp;#39;.service&amp;#39;) | list) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_changed&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ (app_secrets_result.results + app_files_result.results + app_templates_result.results
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> + app_quadlets_result.results + app_quadlet_templates_result.results
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> + (app_extra_results | default([]))) | selectattr(&amp;#39;changed&amp;#39;, &amp;#39;defined&amp;#39;) | selectattr(&amp;#39;changed&amp;#39;) | list | length &amp;gt; 0 }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: reload and restart&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_changed&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">become_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">XDG_RUNTIME_DIR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/run/user/{{ app_uid }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">block&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: daemon-reload&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">daemon_reload&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: restart&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">restarted&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.restart == &amp;#39;restart&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># Containers that share another&amp;#39;s network namespace must be gone before it can be&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># replaced: stop in reverse order, then start in order&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: stop (reverse order)&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">stopped&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units | reverse | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.restart == &amp;#39;stop-start&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: start&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">started&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.restart == &amp;#39;stop-start&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: running&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">started&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">become_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">XDG_RUNTIME_DIR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/run/user/{{ app_uid }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Why not handlers? Handlers run at the end of the play, and they run once per handler name, however many apps notified it. I want the restart right after the app&amp;rsquo;s own files changed, and only for that app&amp;rsquo;s units. Handlers cannot carry &amp;ldquo;which app&amp;rdquo; without tricks. A plain &lt;code>when: app_changed&lt;/code> is easier to read.&lt;/p>
&lt;p>Restarting is also how a container sees a changed config file. Ansible writes a new file (a new inode), and a running container keeps the old one in its mount.&lt;/p>
&lt;h3 id="extra-tasks">Extra tasks&lt;/h3>
&lt;p>Some apps need more than files: a Git checkout to build from, a binary to download. An optional &lt;code>tasks.yml&lt;/code> in the app directory is included, and it can add its results to &lt;code>app_extra_results&lt;/code> to trigger the restart:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: extra tasks&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.include_tasks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_src }}/tasks.yml&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">(app_src ~ &amp;#39;/tasks.yml&amp;#39;) is file&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="order-matters">Order matters&lt;/h2>
&lt;p>Apps run in the order of the &lt;code>apps&lt;/code> list. If one app provides a file that another app mounts, the provider must come first. On a fresh host, Traefik fails to start if a password file it mounts does not exist yet:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># the landing page app provides the htpasswd file that Traefik mounts&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">apps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">landing, traefik, ntfy, forgejo-runner]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">traefik_extra_volumes&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;%h/.config/traefik/landing_htpasswd:/landing_htpasswd:ro,Z&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="adopting-a-running-host">Adopting a running host&lt;/h2>
&lt;p>This is how I moved the existing quadlets of a running host into the role without a single restart:&lt;/p>
&lt;ol>
&lt;li>Copy the host&amp;rsquo;s quadlets and config files into &lt;code>apps/&amp;lt;name&amp;gt;/&lt;/code>, one app at a time.&lt;/li>
&lt;li>Run check mode with a diff against the live host:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l home1 -t apps --check --diff
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>Every difference is one of two things. Either the repository is wrong (fix the file), or the host has a change that was never written down (decide which version is right).&lt;/li>
&lt;li>Repeat until the run shows &lt;code>changed=0&lt;/code> in check mode.&lt;/li>
&lt;li>Run it for real. Nothing changes, so nothing restarts.&lt;/li>
&lt;/ol>
&lt;p>The loop also shows what nobody wrote down: an option changed by hand on the host, a file that exists only there. Each of those is now either in the repository or gone.&lt;/p>
&lt;h2 id="the-test-changed-0">The test: changed=0&lt;/h2>
&lt;p>After every change I run the playbook twice. The second run must report &lt;code>changed=0&lt;/code>. A task that reports a change every time hides the real changes, and it may restart an app on every run. Typical causes:&lt;/p>
&lt;ul>
&lt;li>&lt;code>command&lt;/code> or &lt;code>shell&lt;/code> tasks without &lt;code>changed_when&lt;/code>.&lt;/li>
&lt;li>&lt;code>file: state=touch&lt;/code> without &lt;code>modification_time: preserve&lt;/code>.&lt;/li>
&lt;li>A template that renders a timestamp or a random value.&lt;/li>
&lt;li>A module that compares in a different format than it writes (for example YAML key order).&lt;/li>
&lt;/ul>
&lt;p>Run it with fresh SSH connections, too (&lt;code>-o ControlPath=none&lt;/code>). The reason is in &lt;a href="../../blog/infra-05-ansible-base/">the previous post&lt;/a>.&lt;/p>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t apps
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t apps &lt;span class="c1"># changed=0&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh vps1 sudo -u svc &lt;span class="nv">XDG_RUNTIME_DIR&lt;/span>&lt;span class="o">=&lt;/span>/run/user/1001 systemctl --user list-units &lt;span class="s1">&amp;#39;traefik*&amp;#39;&lt;/span> &lt;span class="s1">&amp;#39;ntfy*&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Next: &lt;a href="../../blog/infra-07-monitoring-backup/">Monitoring and backups for every host&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_reuse_roles.html">Ansible roles&lt;/a>: the role layout, defaults and dependencies.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_handlers.html">Handlers&lt;/a>: notify, listen and &lt;code>flush_handlers&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_checkmode.html">Check mode and diff mode&lt;/a>: what &lt;code>--check --diff&lt;/code> can and cannot predict.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/collections/containers/podman/index.html">containers.podman collection&lt;/a>: the modules for secrets, images and containers. The alternative to plain files.&lt;/li>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html">podman-systemd.unit(5)&lt;/a>: the quadlet reference again, because the role only copies these files.&lt;/li>
&lt;li>&lt;a href="https://redhat-cop.github.io/automation-good-practices/">Good practices for Ansible (Red Hat CoP)&lt;/a>: the chapter on roles fits this post.&lt;/li>
&lt;/ul></description></item><item><title>Ansible groundwork: inventory, bootstrap and a base role</title><link>https://www.linux-lunatix.org/blog/infra-05-ansible-base/</link><pubDate>Wed, 14 Oct 2026 18:55:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-05-ansible-base/</guid><description>&lt;p>&lt;em>Part 5 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>This post sets up the parts that every host shares. At the end you have a control node, an inventory, a playbook for the first contact with a host, and a &lt;code>base&lt;/code> role. That role hardens sshd, runs firewalld and fail2ban, installs updates every night and caps the journal.&lt;/p>
&lt;h2 id="the-control-node">The control node&lt;/h2>
&lt;p>I run Ansible from my laptop, and from a second machine as a spare. Both use the same pinned versions in a virtualenv. The distribution&amp;rsquo;s Ansible package changes with each upgrade, and I want the same behaviour on both machines:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">python3 -m venv ~/.local/share/ops-venv
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">~/.local/share/ops-venv/bin/pip install &lt;span class="s2">&amp;#34;ansible-core==2.22.0&amp;#34;&lt;/span> ansible-lint yamllint
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">PATH&lt;/span>&lt;span class="o">=&lt;/span>~/.local/share/ops-venv/bin:&lt;span class="nv">$PATH&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The collections are pinned too, and installed into the repository:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># requirements.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">collections&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ansible.posix&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">version&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;2.2.2&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">community.general&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">version&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;14.0.0&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">containers.podman&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">version&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;1.21.2&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-galaxy collection install -r requirements.yml -p collections
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Each control node has its own key for Ansible, &lt;code>~/.ssh/ansible_ed25519&lt;/code>, separate from my login key. I can then revoke one machine without touching the other.&lt;/p>
&lt;h2 id="ansible-dot-cfg">ansible.cfg&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="k">[defaults]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">inventory&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">inventory/hosts.yml&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">roles_path&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">roles&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">collections_path&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">collections&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">remote_user&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">ansible&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">private_key_file&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">~/.ssh/ansible_ed25519&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">interpreter_python&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">auto_silent&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">callback_result_format&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">yaml&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">forks&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">4&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Handlers run even when a later task fails&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">force_handlers&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">True&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[privilege_escalation]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">become&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">True&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[diff]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">always&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">True&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[ssh_connection]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">pipelining&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">True&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Two settings are worth an explanation.&lt;/p>
&lt;p>&lt;code>force_handlers&lt;/code> runs the notified handlers even if a later task fails. Without it, this can happen: a task changes the sshd config and notifies &amp;ldquo;Reload sshd&amp;rdquo;, and then a later task fails. The handler never runs. On the next run the config file is unchanged, so nothing notifies the handler again. sshd runs with the old config until somebody notices.&lt;/p>
&lt;p>&lt;code>diff always&lt;/code> shows every change as a diff. Together with &lt;code>--check&lt;/code> this makes a run a review: I see what would change before it changes.&lt;/p>
&lt;h2 id="the-inventory">The inventory&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># inventory/hosts.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">all&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mail1&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">mail1.example.org&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bootstrap_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">admin&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">tunnel_ip&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">10.99.0.11&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vps1&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps1.example.org&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bootstrap_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fedora&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">tunnel_ip&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">10.99.0.13&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">svc&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vps2&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps2.example.org&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bootstrap_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fedora&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">tunnel_ip&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">10.99.0.12&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">home1&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_host&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">home1.example.net&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bootstrap_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">admin&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">tunnel_ip&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">10.99.0.1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">svc&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">children&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">mail1: , vps1: , vps2&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">home&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">home1&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mail&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">mail1&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">websites&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">vps2&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># The hosts that site.yml manages. A host joins after its reinstall with Fedora.&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fedora&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">vps1: , home1: , vps2&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>fedora&lt;/code> group is how I adopted the hosts one by one. A host that still ran Ubuntu or Arch was in the inventory, but &lt;code>site.yml&lt;/code> did not touch it. It joined the group after its reinstall.&lt;/p>
&lt;p>Everything else goes into &lt;code>group_vars/&lt;/code> and &lt;code>host_vars/&lt;/code>. A host file says what is special about that host:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># inventory/host_vars/vps1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">hostname&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps1.example.org&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">timezone&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Europe/Berlin&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_allow_users&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">fedora, svc]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">firewall_public_services&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">http, https, ssh]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">dnf_automatic_reboot&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">when-needed&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">unprivileged_port_start&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">80&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">apps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">traefik, ntfy, forgejo-runner]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="secrets-from-pass">Secrets from pass&lt;/h2>
&lt;p>I keep all secrets in &lt;a href="https://www.passwordstore.org/">pass&lt;/a> on the control node. The lookup plugin reads them during the run:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Tunnel private key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/wireguard/wg-mon.key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.passwordstore&amp;#39;, &amp;#39;infra/&amp;#39; ~ inventory_hostname ~ &amp;#39;/wg-mon-key&amp;#39;) }}\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0600&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">no_log&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">diff&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>no_log&lt;/code> and &lt;code>diff: false&lt;/code> keep the secret out of the output. Without &lt;code>diff: false&lt;/code>, &lt;code>diff always&lt;/code> would print the old and the new key.&lt;/p>
&lt;p>Compared with Ansible Vault, the repository contains nothing encrypted at all, and there is no vault password to share between the control nodes. A secret is changed in one place, in pass, and the next run rolls it out.&lt;/p>
&lt;h2 id="first-contact-bootstrap-dot-yml">First contact: bootstrap.yml&lt;/h2>
&lt;p>A fresh host has one admin account that the provider set up (&lt;code>fedora&lt;/code> on a Fedora Cloud image). &lt;code>bootstrap.yml&lt;/code> logs in as that account once and creates the &lt;code>ansible&lt;/code> user. Every other playbook then uses that user.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># playbooks/bootstrap.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Bootstrap the ansible user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">all&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">remote_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ bootstrap_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vars&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible_ssh_private_key_file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">~/.ssh/id_ed25519&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">tasks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Ansible user exists&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">comment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">shell&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/bin/bash&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">system&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">create_home&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Only the control-node keys may log in&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.authorized_key&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">key&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ansible_authorized_keys | join(&amp;#39;\n&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">exclusive&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># The dnf module needs it, and Fedora Server does not ship it&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">dnf bindings for Ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">dnf install -y python3-libdnf5&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">args&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">creates&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/usr/lib64/python3*/site-packages/libdnf5&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ansible_facts.pkg_mgr == &amp;#39;dnf5&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Passwordless sudo for ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/sudoers.d/ansible&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;ansible ALL=(ALL) NOPASSWD: ALL\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0440&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">validate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">visudo -cf %s&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Read effective sshd settings&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd -T&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd_effective&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># AllowUsers lines add up; only needed where such a list is in force&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Allow ansible over SSH where AllowUsers is set&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/sshd_config.d/01-ansible.conf&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;AllowUsers ansible\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0600&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">validate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd -t -f %s&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd_effective.stdout_lines | select(&amp;#39;match&amp;#39;, &amp;#39;^allowusers &amp;#39;) | list | length &amp;gt; 0&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">handlers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">reloaded&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/bootstrap.yml -l vps1 &lt;span class="c1"># the admin has passwordless sudo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/bootstrap.yml -l vps1 -K &lt;span class="c1"># sudo asks for a password&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible vps1 -m ping
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>validate:&lt;/code> is your friend for every file that can lock you out. Ansible writes the new file to a temporary path, runs the command on it, and installs it only if the command succeeds. A sudoers file with a typo can no longer break sudo.&lt;/p>
&lt;h2 id="site-dot-yml">site.yml&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># playbooks/site.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Fedora hosts&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fedora&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">roles&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: base, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">base}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: podman, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">podman, apps, backup]}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: quadlet_app, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">apps}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: monitoring_agent, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">monitoring}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: backup, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">backup}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Mail server&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">hosts&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">mail:&amp;amp;fedora&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">roles&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">role: mailcow, tags&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">docker, mailcow]}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>mail:&amp;amp;fedora&lt;/code> is the intersection of the two groups. The mail host gets its Mailcow role only after it runs Fedora.&lt;/p>
&lt;p>I always run it like this:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 --check &lt;span class="c1"># what would change&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t base &lt;span class="c1"># only one part&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="the-base-role">The base role&lt;/h2>
&lt;p>The base role does the same things on every host. It starts with two checks: the host runs Fedora, and SELinux is enforcing (see &lt;a href="../../blog/infra-04-selinux/">the previous post&lt;/a>). Then it does the actual work.&lt;/p>
&lt;h3 id="sshd">sshd&lt;/h3>
&lt;p>One drop-in file. Fedora&amp;rsquo;s &lt;code>sshd_config&lt;/code> reads &lt;code>/etc/ssh/sshd_config.d/*.conf&lt;/code> first, and for most options the first value wins. So a file called &lt;code>00-hardening.conf&lt;/code> overrides the distribution&amp;rsquo;s defaults in &lt;code>50-redhat.conf&lt;/code>.&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2"># /etc/ssh/sshd_config.d/00-hardening.conf
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AuthenticationMethods publickey
GSSAPIAuthentication no
{% if ssh_allow_users %}
AllowUsers {{ (ssh_allow_users + [&amp;#39;ansible&amp;#39;]) | unique | join(&amp;#39; &amp;#39;) }}
{% endif %}
# Ed25519 only; post-quantum key exchange first; AEAD ciphers and ETM MACs only
HostKey /etc/ssh/ssh_host_ed25519_key
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
PubkeyAcceptedAlgorithms ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com
KexAlgorithms mlkem768x25519-sha256,sntrup761x25519-sha512,curve25519-sha256
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com
X11Forwarding no
AllowAgentForwarding no
AllowStreamLocalForwarding no
PermitTunnel no
AllowTcpForwarding no
MaxAuthTries 3
LoginGraceTime 20
LogLevel VERBOSE
&lt;/code>&lt;/pre>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SSH hardening&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.template&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd-hardening.conf.j2&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/sshd_config.d/00-hardening.conf&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">validate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd -t -f %s&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Do not put &lt;code>Match&lt;/code> blocks into such a drop-in. A &lt;code>Match&lt;/code> block lasts until the end of the whole configuration, so it would also catch the rest of the main &lt;code>sshd_config&lt;/code>.&lt;/p>
&lt;p>The &lt;code>AllowUsers&lt;/code> line always adds &lt;code>ansible&lt;/code>. The first version did not. On a fresh test host, the first &lt;code>base&lt;/code> run then locked the &lt;code>ansible&lt;/code> user out of the host. I did not notice at first, because Ansible kept using its open SSH connection. More about that below.&lt;/p>
&lt;p>Check what sshd really uses with &lt;code>sudo sshd -T | grep -i allowusers&lt;/code>.&lt;/p>
&lt;h3 id="fail2ban">fail2ban&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># /etc/fail2ban/jail.d/sshd.local&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[DEFAULT]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ignoreip&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">127.0.0.1/8 ::1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[sshd]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">enabled&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">backend&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">systemd&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># aggressive: also scanners that disconnect before auth or fail the key exchange&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">mode&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">aggressive&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">maxretry&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">3&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">findtime&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">10m&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">bantime&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">1h&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">bantime.increment&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">bantime.maxtime&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">1w&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[recidive]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">enabled&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">bantime&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">1w&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">findtime&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">1d&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">maxretry&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s">5&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>recidive&lt;/code> jail reads fail2ban&amp;rsquo;s own log file. On a fresh host that file does not exist yet, and the jail, and with it fail2ban, fails to start. So the role creates the file first:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fail2ban log file for the recidive jail&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/var/log/fail2ban.log&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">touch&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0600&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">setype&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">fail2ban_log_t&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">modification_time&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">preserve&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">access_time&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">preserve&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>modification_time: preserve&lt;/code> keeps the task idempotent. A plain &lt;code>state: touch&lt;/code> reports a change on every run.&lt;/p>
&lt;h3 id="firewalld">firewalld&lt;/h3>
&lt;p>Only what the host lists in &lt;code>firewall_public_services&lt;/code> is open in the default zone:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Open services in the default zone&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.firewalld&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">permanent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">immediate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">enabled&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ firewall_public_services }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>permanent&lt;/code> and &lt;code>immediate&lt;/code> together write the permanent config and apply it at once. No reload is needed, and a reboot changes nothing.&lt;/p>
&lt;h3 id="updates-every-night">Updates every night&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">dnf5-automatic settings&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">community.general.ini_file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/dnf/automatic.conf&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">section&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.section }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">option&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.option }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.value }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: commands, option: apply_updates, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;yes&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: commands, option: download_updates, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;yes&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: commands, option: upgrade_type, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">default}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: commands, option: reboot, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ dnf_automatic_reboot }}&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: commands, option: reboot_command, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;\&amp;#34;shutdown -r +5 &amp;#39;Rebooting after applying automatic updates&amp;#39;\&amp;#34;&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">section: emitters, option: emit_via, value&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">stdio}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">dnf5-automatic timer&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">dnf5-automatic.timer&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">enabled&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">started&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>reboot&lt;/code> is &lt;code>when-needed&lt;/code> on the VPS (a reboot after a new kernel or glibc) and &lt;code>never&lt;/code> on the home server, which I reboot myself.&lt;/p>
&lt;h3 id="a-cap-for-the-journal">A cap for the journal&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Journal size cap&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/systemd/journald.conf.d/00-size.conf&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> [Journal]
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> SystemMaxUse=500M&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Restart journald&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Fedora&amp;rsquo;s default is 10% of the filesystem, up to 4 GB. On a small VPS with chatty containers that is a lot. The old Ubuntu mail host had 2.3 GB of journal when its root filesystem reached 91%.&lt;/p>
&lt;h2 id="things-to-watch">Things to watch&lt;/h2>
&lt;ul>
&lt;li>Ansible reuses its SSH connections (&lt;code>ControlPersist&lt;/code>). After a change to sshd, a run that passes may still use the old connection. Close the connections before you test such a change:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="k">for&lt;/span> s in ~/.ansible/cp/*&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span> ssh -O &lt;span class="nb">exit&lt;/span> -o &lt;span class="nv">ControlPath&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$s&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> x 2&amp;gt;/dev/null&lt;span class="p">;&lt;/span> &lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>Update a fresh host and reboot it before the first full run. Otherwise dnf5-automatic installs a new kernel during the rollout and schedules a reboot in five minutes. From then on, &lt;code>pam_nologin&lt;/code> lets only root log in, and the run fails halfway.&lt;/li>
&lt;li>&lt;code>--check&lt;/code> mode has limits. A task that depends on a package installed earlier in the same run fails in check mode, because the package is not there yet. Use &lt;code>when: not ansible_check_mode&lt;/code> for such tasks, or accept the error on a host&amp;rsquo;s very first check run.&lt;/li>
&lt;/ul>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t base
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t base &lt;span class="c1"># the second run: changed=0&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh -o &lt;span class="nv">ControlMaster&lt;/span>&lt;span class="o">=&lt;/span>no -o &lt;span class="nv">ControlPath&lt;/span>&lt;span class="o">=&lt;/span>none ansible@vps1.example.org &lt;span class="nb">true&lt;/span> &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> &lt;span class="nb">echo&lt;/span> ok
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Next: &lt;a href="../../blog/infra-06-quadlet-app/">One generic Ansible role for every app&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/tips_tricks/ansible_tips_tricks.html">Ansible tips and tricks&lt;/a>: the official list of good practices.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/tips_tricks/sample_setup.html">Sample Ansible setup&lt;/a>: a directory layout for inventories, group_vars and roles.&lt;/li>
&lt;li>&lt;a href="https://redhat-cop.github.io/automation-good-practices/">Good practices for Ansible (Red Hat CoP)&lt;/a>: role design, naming and variables. Opinionated, and worth reading.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/inventory_guide/intro_inventory.html">How to build your inventory&lt;/a>: groups, host_vars and group_vars.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/collections/community/general/passwordstore_lookup.html">community.general.passwordstore lookup&lt;/a>: secrets from pass, with all options.&lt;/li>
&lt;li>&lt;a href="https://ansible.readthedocs.io/projects/lint/">ansible-lint&lt;/a>: catches many mistakes before a run.&lt;/li>
&lt;li>&lt;a href="https://firewalld.org/documentation/">firewalld documentation&lt;/a>: zones, services and rich rules.&lt;/li>
&lt;li>&lt;a href="https://github.com/fail2ban/fail2ban">fail2ban&lt;/a>: jails and filters.&lt;/li>
&lt;li>&lt;a href="https://dnf5.readthedocs.io/en/latest/dnf5_plugins/automatic.8.html">dnf5 automatic&lt;/a>: the settings in &lt;code>automatic.conf&lt;/code>.&lt;/li>
&lt;/ul></description></item><item><title>SELinux enforcing with containers</title><link>https://www.linux-lunatix.org/blog/infra-04-selinux/</link><pubDate>Fri, 09 Oct 2026 21:10:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-04-selinux/</guid><description>&lt;p>&lt;em>Part 4 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>Every one of my hosts runs SELinux in enforcing mode. Every container runs in the confined type &lt;code>container_t&lt;/code>, with no exceptions. This post shows the few tools that make this practical: volume labels, file contexts, booleans and small policy modules. It also shows how to debug a denial.&lt;/p>
&lt;h2 id="why-bother">Why bother&lt;/h2>
&lt;p>A container that breaks out of its namespaces lands in a process on the host. With rootless Podman that process runs as an unprivileged user. With SELinux it is also confined to &lt;code>container_t&lt;/code>. That type may read and write only files labelled &lt;code>container_file_t&lt;/code>, and each container gets its own MCS category pair (for example &lt;code>s0:c123,c456&lt;/code>). So one container cannot read another container&amp;rsquo;s files, even when both belong to the same user.&lt;/p>
&lt;p>The cost is small once you know the patterns. Most of the time, the right volume flag is all you need.&lt;/p>
&lt;h2 id="volume-labels-z-and-z">Volume labels: &lt;code>:Z&lt;/code> and &lt;code>:z&lt;/code>&lt;/h2>
&lt;p>When you mount a host directory into a container, its files usually have a type such as &lt;code>user_home_t&lt;/code>. A confined container may not read that. Podman can relabel the files at start:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Flag&lt;/th>
&lt;th>Label&lt;/th>
&lt;th>Use for&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>:Z&lt;/code>&lt;/td>
&lt;td>&lt;code>container_file_t&lt;/code> with this container&amp;rsquo;s private categories&lt;/td>
&lt;td>data of one container&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>:z&lt;/code>&lt;/td>
&lt;td>&lt;code>container_file_t&lt;/code> without categories (shared)&lt;/td>
&lt;td>files that several containers mount&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Only this container uses it&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/ntfy/server.yml:/etc/ntfy/server.yml:ro,Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">ntfy-data:/var/lib/ntfy&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Three website containers mount the same file&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/nginx/realip.conf:/etc/nginx/conf.d/realip.conf:ro,z&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Named Podman volumes get the right label when Podman creates them. The &lt;code>:Z&lt;/code> on a volume does no harm and makes the intent clear.&lt;/p>
&lt;p>Two warnings:&lt;/p>
&lt;ul>
&lt;li>Never use &lt;code>:Z&lt;/code> on a system directory such as &lt;code>/etc&lt;/code> or &lt;code>/home&lt;/code>. Podman relabels everything below it, and the host then cannot use those files any more.&lt;/li>
&lt;li>If two containers mount the same file and both use &lt;code>:Z&lt;/code>, the second start relabels it for itself. The first container then gets &amp;ldquo;permission denied&amp;rdquo;. Use &lt;code>:z&lt;/code> for shared files.&lt;/li>
&lt;/ul>
&lt;h2 id="directories-outside-the-home-file-contexts">Directories outside the home: file contexts&lt;/h2>
&lt;p>Some files are not mounted from a home directory. For example, node_exporter reads text files with metrics from &lt;code>/var/lib/node_exporter/textfile&lt;/code>, and my backup script writes them. Instead of relabelling the directory at every start, make the label permanent with a file context rule:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo semanage fcontext -a -t container_file_t &lt;span class="s1">&amp;#39;/var/lib/node_exporter(/.*)?&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo restorecon -Rv /var/lib/node_exporter
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>restorecon&lt;/code> applies the rule now. The rule also survives a full relabel of the filesystem. With Ansible this becomes two tasks:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SELinux label for the textfile directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">community.general.sefcontext&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">target&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/var/lib/node_exporter(/.*)?&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">setype&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">container_file_t&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">present&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">textfile_fcontext&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Relabel the textfile directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">restorecon -R /var/lib/node_exporter&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">textfile_fcontext.changed and not ansible_check_mode&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="booleans">Booleans&lt;/h2>
&lt;p>The policy has switches for common cases. List them with &lt;code>getsebool -a | grep container&lt;/code>. One example: &lt;code>container_use_devices&lt;/code> lets containers use host devices. I keep the booleans per host in the inventory:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/&amp;lt;host&amp;gt;.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">selinux_booleans&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">container_use_devices&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SELinux booleans&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.seboolean&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.key }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.value }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">persistent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ selinux_booleans | dict2items }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Prefer a boolean over a policy module when one exists. Somebody else maintains it, and it is easy to see what it allows.&lt;/p>
&lt;h2 id="small-policy-modules-for-the-rest">Small policy modules for the rest&lt;/h2>
&lt;p>Two of my cases have no boolean.&lt;/p>
&lt;h3 id="traefik-and-the-podman-socket">Traefik and the Podman socket&lt;/h3>
&lt;p>Traefik reads container labels through the rootless Podman socket. The socket file can be relabelled with &lt;code>:z&lt;/code>. But connecting to the socket means talking to the process behind it, and that process runs as &lt;code>container_runtime_t&lt;/code>. &lt;code>container_t&lt;/code> may not do that. The denial in the audit log looks like this:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">type=AVC msg=audit(...): avc: denied { connectto } for pid=... comm=&amp;#34;traefik&amp;#34;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> path=&amp;#34;/run/user/1001/podman/podman.sock&amp;#34; scontext=system_u:system_r:container_t:s0:c12,c345
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> tcontext=unconfined_u:unconfined_r:container_runtime_t:s0 tclass=unix_stream_socket permissive=0
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The fix is one rule:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">module traefik_podman_socket 1.0;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">require {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> type container_t;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> type container_runtime_t;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> class unix_stream_socket connectto;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">}
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">allow container_t container_runtime_t:unix_stream_socket connectto;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Build and load it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">checkmodule -M -m -o traefik_podman_socket.mod traefik_podman_socket.te
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">semodule_package -o traefik_podman_socket.pp -m traefik_podman_socket.mod
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo semodule -i traefik_podman_socket.pp
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This rule applies to every &lt;code>container_t&lt;/code> process, not only to Traefik. A container still needs the socket mounted to use it, and only Traefik and my Podman exporter get that mount. Think about this trade-off before you load such a rule. Access to the Podman socket is close to access to the user&amp;rsquo;s account.&lt;/p>
&lt;p>A VPN container (gluetun) needs the same kind of module for &lt;code>/dev/net/tun&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">allow container_t tun_tap_device_t:chr_file { getattr ioctl open read write };
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="node-exporter-a-type-of-its-own">node_exporter: a type of its own&lt;/h3>
&lt;p>node_exporter reads the host&amp;rsquo;s mount table and asks systemd about unit states. Most guides run it with &lt;code>SecurityLabelDisable=true&lt;/code> (unconfined), and I did that too at first. I replaced it with a confined type of its own, written in CIL. The base is the container template from &lt;code>udica&lt;/code> (it ships with &lt;code>container-selinux&lt;/code>), plus the few extra rules that node_exporter needs:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-lisp" data-lang="lisp">&lt;span class="line">&lt;span class="cl">&lt;span class="p">(&lt;/span>&lt;span class="k">block&lt;/span> &lt;span class="nv">node_exporter&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">blockinherit&lt;/span> &lt;span class="nv">container&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">blockinherit&lt;/span> &lt;span class="nv">net_container&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="c1">; filesystem collector: init&amp;#39;s mount table, statfs of every mount point&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dir&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">search&lt;/span> &lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">file&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">open&lt;/span> &lt;span class="nf">read&lt;/span> &lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">lnk_file&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">read&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">filesystem_type&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">filesystem&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">mountpoint&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dir&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">search&lt;/span> &lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="c1">; selinux collector: current mode&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">security_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dir&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">search&lt;/span> &lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">security_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">file&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">open&lt;/span> &lt;span class="nf">read&lt;/span> &lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="c1">; systemd collector: read-only queries to systemd over the system bus&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">system_dbusd_var_run_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dir&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">search&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">system_dbusd_var_run_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">sock_file&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">write&lt;/span> &lt;span class="nv">getattr&lt;/span> &lt;span class="nf">open&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">system_dbusd_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">unix_stream_socket&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">connectto&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">system_dbusd_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dbus&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">send_msg&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">system_dbusd_t&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dbus&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">send_msg&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dbus&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">send_msg&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dbus&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">send_msg&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">system&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">status&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">systemd_unit_file_type&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">service&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">status&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">t&lt;/span>&lt;span class="o">=&lt;/span>/usr/share/udica/templates
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo semodule -i node_exporter.cil &lt;span class="nv">$t&lt;/span>/base_container.cil &lt;span class="nv">$t&lt;/span>/net_container.cil
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The quadlet then uses the new type:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">quay.io/prometheus/node-exporter:latest&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">host&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PodmanArgs&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">--pid=host&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">/:/host:ro,rslave&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket:ro&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">SecurityLabelType&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">node_exporter.process&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="debugging-a-denial">Debugging a denial&lt;/h2>
&lt;p>When a container fails with &amp;ldquo;permission denied&amp;rdquo; and the file permissions look fine, look at the audit log:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo ausearch -m avc -ts recent &lt;span class="c1"># the last 10 minutes&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo ausearch -m avc -ts boot &lt;span class="p">|&lt;/span> audit2why &lt;span class="c1"># with an explanation&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Some denials are hidden by &lt;code>dontaudit&lt;/code> rules. If &lt;code>ausearch&lt;/code> shows nothing but the problem stays, switch those rules off for a moment:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo semodule -DB &lt;span class="c1"># disable dontaudit, rebuild&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># reproduce the problem, run ausearch again&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo semodule -B &lt;span class="c1"># enable dontaudit again&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>audit2allow&lt;/code> turns denials into rules:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo ausearch -m avc -ts recent &lt;span class="p">|&lt;/span> audit2allow -m mymodule
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Read its output, but do not load it as it is. It allows exactly what was denied, for the whole source type. A container that tried to read &lt;code>/etc/shadow&lt;/code> gets a rule that allows &lt;code>container_t&lt;/code> to read &lt;code>shadow_t&lt;/code>, for every container on the host. Often the right fix is something else: a &lt;code>:Z&lt;/code> on the mount, a file context or a boolean. Write a module only when none of those fit. Keep it to the rules you understand.&lt;/p>
&lt;h2 id="make-it-code">Make it code&lt;/h2>
&lt;p>I keep the module sources in the repository (&lt;code>selinux/*.te&lt;/code>, &lt;code>selinux/*.cil&lt;/code>), never just the compiled &lt;code>.pp&lt;/code>. If you have an old module and lost its source, you can extract it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo semodule -c -E gluetun_policy &lt;span class="c1"># writes gluetun_policy.cil&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>My base Ansible role copies each source to &lt;code>/usr/local/share/selinux/&lt;/code>. It builds and loads a module when the source changed, or when the module is missing:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SELinux module sources&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ playbook_dir }}/../selinux/{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/usr/local/share/selinux/{{ item }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ selinux_modules }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodule_src&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Installed SELinux modules&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodule -l&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodules&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">check_mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Build and load SELinux modules&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.shell&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> set -e
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> cd /usr/local/share/selinux
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> name={{ item.item | regex_replace(&amp;#39;\.(te|cil)$&amp;#39;, &amp;#39;&amp;#39;) }}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> case {{ item.item }} in
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> *.te)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> checkmodule -M -m -o &amp;#34;$name.mod&amp;#34; {{ item.item }}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> semodule_package -o &amp;#34;$name.pp&amp;#34; -m &amp;#34;$name.mod&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> semodule -i &amp;#34;$name.pp&amp;#34; ;;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> *.cil)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> t=/usr/share/udica/templates
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> semodule -i {{ item.item }} $t/base_container.cil $t/net_container.cil ;;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> esac&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ semodule_src.results }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">item.changed or (item.item | regex_replace(&amp;#39;\.(te|cil)$&amp;#39;, &amp;#39;&amp;#39;)) not in semodules.stdout_lines&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="guard-rails">Guard rails&lt;/h2>
&lt;p>Two checks keep me honest.&lt;/p>
&lt;p>The base role stops the play if a host is not enforcing, and also if somebody made a domain permissive with &lt;code>semanage permissive -a&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SELinux is enforcing (running and configured)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.assert&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">that&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">ansible_facts.selinux.status == &amp;#39;enabled&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">ansible_facts.selinux.mode == &amp;#39;enforcing&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">ansible_facts.selinux.config_mode == &amp;#39;enforcing&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fail_msg&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;SELinux on {{ inventory_hostname }}: {{ ansible_facts.selinux }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># semanage permissive -a stores a local permissive domain as module permissive_&amp;lt;type&amp;gt;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">No&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">local permissive domains&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodule -l&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodule_list&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">check_mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">failed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodule_list.stdout_lines | select(&amp;#39;match&amp;#39;, &amp;#39;^permissive_&amp;#39;) | list | length &amp;gt; 0&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The role does not switch enforcing on by itself. A host that is not enforcing needs a look first (a relabel, old denials), so the play stops.&lt;/p>
&lt;p>The app role refuses quadlets that disable confinement:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: containers stay confined&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.assert&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">that&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">lookup(&amp;#39;ansible.builtin.file&amp;#39;, item.src) is not search(unconfined)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fail_msg&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }} disables SELinux confinement; use a policy module in selinux/ instead&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ quadlet_files }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vars&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">unconfined&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;(?mi)^\s*(SecurityLabelDisable\s*=\s*true|SecurityLabelType\s*=\s*(spc_t|unconfined_t)|PodmanArgs\s*=.*(--privileged|label[=:]disable))&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>With this check, &amp;ldquo;just disable SELinux for this one container&amp;rdquo; is no longer an option.&lt;/p>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">getenforce &lt;span class="c1"># Enforcing&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ps -eZ &lt;span class="p">|&lt;/span> grep -E &lt;span class="s1">&amp;#39;container_t|node_exporter&amp;#39;&lt;/span> &lt;span class="c1"># every container process confined&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo ausearch -m avc -ts boot &lt;span class="c1"># ideally: &amp;lt;no matches&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="things-to-watch">Things to watch&lt;/h2>
&lt;ul>
&lt;li>Restores. &lt;code>restic&lt;/code> restores the SELinux label as an extended attribute when it was in the backup. A backup from a host without SELinux has no label, and &lt;code>restic&lt;/code> reports errors when it tries to set one. Restore with &lt;code>--exclude-xattr security.selinux&lt;/code>, then run &lt;code>restorecon&lt;/code> or let &lt;code>:Z&lt;/code> relabel at start.&lt;/li>
&lt;li>Docker. My one Docker host (Mailcow) runs with SELinux too. Fedora&amp;rsquo;s Docker package already starts &lt;code>dockerd&lt;/code> with &lt;code>--selinux-enabled&lt;/code>, so do not add it to &lt;code>daemon.json&lt;/code> as well. The daemon refuses to start when an option comes from both places.&lt;/li>
&lt;/ul>
&lt;p>Next: &lt;a href="../../blog/infra-05-ansible-base/">Ansible groundwork: inventory, bootstrap and a base role&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.fedoraproject.org/en-US/quick-docs/selinux-getting-started/">Getting started with SELinux (Fedora Quick Docs)&lt;/a>: the basics, if this post went too fast.&lt;/li>
&lt;li>&lt;a href="https://github.com/containers/container-selinux">container-selinux&lt;/a>: the policy behind &lt;code>container_t&lt;/code> and &lt;code>container_file_t&lt;/code>, and its booleans.&lt;/li>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-run.1.html">podman-run(1)&lt;/a>: the &lt;code>:z&lt;/code> and &lt;code>:Z&lt;/code> volume options and the label settings of &lt;code>--security-opt&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://github.com/containers/udica">udica&lt;/a>: generates SELinux policies for containers. The source of the templates in this post.&lt;/li>
&lt;li>&lt;a href="https://github.com/SELinuxProject/selinux-notebook">The SELinux Notebook&lt;/a>: the deep reference: types, CIL, modules and how the kernel decides.&lt;/li>
&lt;/ul></description></item><item><title>Traefik in front of rootless containers</title><link>https://www.linux-lunatix.org/blog/infra-03-traefik/</link><pubDate>Mon, 05 Oct 2026 19:05:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-03-traefik/</guid><description>&lt;p>&lt;em>Part 3 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>At the end of this post Traefik runs as a rootless quadlet. It gets Let&amp;rsquo;s Encrypt certificates and finds the apps by their container labels, as it did with Docker. You will also know which network mode to choose, and why.&lt;/p>
&lt;h2 id="the-problem-with-rootless-networking">The problem with rootless networking&lt;/h2>
&lt;p>With Docker, Traefik sits on a bridge network with the apps, and the root daemon forwards ports 80 and 443 to it. Traefik sees the real client address.&lt;/p>
&lt;p>Rootless Podman cannot create real bridges in the host&amp;rsquo;s network namespace. It runs the container network in a user namespace. A helper process (&lt;code>rootlessport&lt;/code>, or &lt;code>pasta&lt;/code>) accepts connections on the host&amp;rsquo;s port and forwards them into the namespace. On the way the client address gets lost: Traefik then sees every request as coming from the helper. Access logs, rate limits, IP allow lists and fail2ban on the proxy then do not work.&lt;/p>
&lt;p>There are two ways to deal with this:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Mode&lt;/th>
&lt;th>Client IP&lt;/th>
&lt;th>Apps reached at&lt;/th>
&lt;th>Use it when&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>Traefik on the host network&lt;/td>
&lt;td>real&lt;/td>
&lt;td>&lt;code>127.0.0.1:&amp;lt;port&amp;gt;&lt;/code> published by each app&lt;/td>
&lt;td>the host is on the internet&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Traefik on a Podman bridge&lt;/td>
&lt;td>lost&lt;/td>
&lt;td>the container IP on the bridge&lt;/td>
&lt;td>only you and your LAN use it&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>I use both. My two internet-facing VPS run Traefik on the host network. My home server is reachable only from the LAN and through WireGuard, so the client addresses there are my own devices anyway. That host keeps the bridge network, which was there first and needs fewer changes.&lt;/p>
&lt;h2 id="option-a-traefik-on-the-host-network">Option A: Traefik on the host network&lt;/h2>
&lt;h3 id="the-quadlet">The quadlet&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ~svc/.config/containers/systemd/traefik.container&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Unit]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Description&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">Traefik reverse proxy&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">After&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">network-online.target&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">docker.io/library/traefik:v3&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Host network namespace: real client IPs, binds 80/443 directly&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">host&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Run as svc itself, so the files it writes belong to svc on the host&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">UserNS&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">keep-id:uid=1001,gid=1001&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">User&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">1001:1001&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/traefik/acme:/acme:Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/traefik/traefik.yml:/etc/traefik/traefik.yml:ro,Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/traefik/dynamic:/dynamic:ro,Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># The rootless Podman socket, read-only, as Traefik&amp;#39;s &amp;#34;docker&amp;#34; provider&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%t/podman/podman.sock:/var/run/docker.sock:ro,z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Secret&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">dns-api-token,type=env,target=CF_DNS_API_TOKEN&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Replace &lt;code>1001&lt;/code> with the UID and GID of your app user (&lt;code>id svc&lt;/code>). Some notes:&lt;/p>
&lt;ul>
&lt;li>&lt;code>Network=host&lt;/code> needs &lt;code>net.ipv4.ip_unprivileged_port_start=80&lt;/code> (see &lt;a href="../../blog/infra-02-quadlets/">the previous post&lt;/a>). Without it Traefik cannot bind 80 and 443.&lt;/li>
&lt;li>&lt;code>%h&lt;/code> is the user&amp;rsquo;s home directory and &lt;code>%t&lt;/code> is &lt;code>$XDG_RUNTIME_DIR&lt;/code>. These are systemd specifiers, and quadlets expand them.&lt;/li>
&lt;li>The Podman socket must be enabled for the user: &lt;code>systemctl --user enable --now podman.socket&lt;/code>.&lt;/li>
&lt;li>The socket mount needs an SELinux module, so that a confined container may connect to it. &lt;a href="../../blog/infra-04-selinux/">Post 4&lt;/a> shows it. Without the module Traefik logs &amp;ldquo;permission denied&amp;rdquo; on the socket.&lt;/li>
&lt;/ul>
&lt;h3 id="the-static-configuration">The static configuration&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># ~svc/.config/traefik/traefik.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">entryPoints&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">http&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">address&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;:80&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">http&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">redirections&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">entryPoint&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">to&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">https&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scheme&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">https&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">https&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">address&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;:443&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># On the host network, ClientHost in this log is the real client address.&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># Written to stdout, so it ends up in the journal.&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">accessLog&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">format&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">json&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fields&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">headers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">defaultMode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">drop&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">names&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">User-Agent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">keep&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">Referer&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">keep&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">providers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">docker&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">endpoint&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;unix:///var/run/docker.sock&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">exposedByDefault&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># Apps publish their port on 127.0.0.1. Route to that address, not to the&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># container IP, which the host network namespace cannot reach.&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">useBindPortIP&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">directory&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/dynamic&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">watch&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">certificatesResolvers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">default&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">acme&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">email&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;admin@example.org&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">storage&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/acme/acme.json&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dnsChallenge&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">provider&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">cloudflare&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">resolvers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;1.1.1.1:53&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;8.8.8.8:53&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>useBindPortIP: true&lt;/code> is the key line for this mode. Each app publishes its port on localhost (&lt;code>PublishPort=127.0.0.1:3000:3000&lt;/code>). Traefik reads the port mapping from the container&amp;rsquo;s labels and metadata through the socket and connects to &lt;code>127.0.0.1:3000&lt;/code>. Each app needs its own host port.&lt;/p>
&lt;h3 id="the-certificate-challenge">The certificate challenge&lt;/h3>
&lt;p>Let&amp;rsquo;s Encrypt has to check that you control the domain. Traefik supports three ways:&lt;/p>
&lt;ul>
&lt;li>The DNS challenge: Traefik writes a TXT record through your DNS provider&amp;rsquo;s API. It works for wildcard certificates and for hosts that are not reachable from the internet. It needs an API token, which I keep in a Podman secret (&lt;code>Secret=dns-api-token,...&lt;/code>). Use your provider&amp;rsquo;s name in &lt;code>provider:&lt;/code>. Traefik&amp;rsquo;s ACME library supports many providers, and each one has its own environment variable for the token.&lt;/li>
&lt;li>The TLS challenge (&lt;code>tlsChallenge: {}&lt;/code>): Let&amp;rsquo;s Encrypt connects to port 443. No token is needed, but the host must be reachable from the internet. I use it for domains whose DNS is at a provider without an API.&lt;/li>
&lt;li>The HTTP challenge: the same as the TLS challenge, but on port 80.&lt;/li>
&lt;/ul>
&lt;p>&lt;code>acme.json&lt;/code> holds the certificates and the ACME account key. Keep it in your backup. After a reinstall, restore it before Traefik starts. Traefik then serves the old certificates and asks Let&amp;rsquo;s Encrypt for nothing. Let&amp;rsquo;s Encrypt has rate limits, and a rehearsal or two reinstalls in a row can hit them.&lt;/p>
&lt;h3 id="shared-middlewares-in-a-file">Shared middlewares in a file&lt;/h3>
&lt;p>Things that many apps use, like security headers, go into the file provider:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># ~svc/.config/traefik/dynamic/dynamic_conf.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">http&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">middlewares&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">secureHeaders&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">headers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">browserXssFilter&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">contentTypeNosniff&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">frameDeny&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">stsSeconds&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">31536000&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">tls&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">options&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">default&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">minVersion&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">VersionTLS12&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>An app uses it with &lt;code>Label=traefik.http.routers.&amp;lt;name&amp;gt;.middlewares=secureHeaders@file&lt;/code>.&lt;/p>
&lt;h3 id="an-app-behind-it">An app behind it&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ~svc/.config/containers/systemd/blog.container&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">blog&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">git.example.org/alice/blog:latest&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">127.0.0.1:8101:80&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.enable=true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">&amp;#34;traefik.http.routers.blog.rule=Host(`www.example.org`) || Host(`example.org`)&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.blog.entrypoints=https&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.blog.tls.certresolver=default&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.blog.middlewares=secureHeaders@file&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.services.blog.loadbalancer.server.port=80&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Quote the whole &lt;code>Label=&lt;/code> value if it contains spaces or &lt;code>||&lt;/code>. Otherwise systemd splits it.&lt;/p>
&lt;h3 id="real-client-ips-in-the-app-too">Real client IPs in the app, too&lt;/h3>
&lt;p>Traefik sees the client, but the app behind it sees &lt;code>127.0.0.1&lt;/code>. For nginx, tell it to trust Traefik&amp;rsquo;s &lt;code>X-Forwarded-For&lt;/code> header:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-nginx" data-lang="nginx">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># realip.conf, mounted into the nginx container at /etc/nginx/conf.d/realip.conf
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Traefik replaces any X-Forwarded-For sent by clients. The app port is published
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># on 127.0.0.1 only, so the only peer that can connect is the local Traefik.
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">&lt;/span>&lt;span class="k">set_real_ip_from&lt;/span> &lt;span class="n">0.0.0.0&lt;/span>&lt;span class="s">/0&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">set_real_ip_from&lt;/span> &lt;span class="p">::&lt;/span>&lt;span class="s">/0&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">real_ip_header&lt;/span> &lt;span class="s">X-Forwarded-For&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The wide &lt;code>set_real_ip_from&lt;/code> looks wrong at first. It is safe here only because nothing but Traefik can reach the app&amp;rsquo;s port.&lt;/p>
&lt;h2 id="option-b-traefik-on-a-podman-bridge-network">Option B: Traefik on a Podman bridge network&lt;/h2>
&lt;p>On the home server Traefik and the apps share a Podman network called &lt;code>proxy&lt;/code>. Traefik publishes the ports, and the apps publish none.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># proxy.network&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Network]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">NetworkName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">proxy&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># traefik.container (bridge mode, shortened)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">docker.io/library/traefik:v3&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">80:80/tcp&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">443:443/tcp&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%t/podman/podman.sock:/run/podman/podman.sock:ro,z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/traefik/traefik.yml:/etc/traefik/traefik.yml:ro,z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik-letsencrypt.volume:/letsencrypt:z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Secret&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">dns-api-token,type=env,target=CF_DNS_API_TOKEN&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">proxy.network:ip=10.89.0.2&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>An app joins the network with &lt;code>Network=proxy.network:ip=10.89.0.20&lt;/code>. I give each container a fixed IP, so that other configs (Prometheus targets, for example) can use it. In this mode the provider section of &lt;code>traefik.yml&lt;/code> has no &lt;code>useBindPortIP&lt;/code>, because Traefik reaches the container IPs directly.&lt;/p>
&lt;p>Ports below 1024 are still needed for &lt;code>PublishPort=80:80&lt;/code>, because the rootless helper binds them as the user.&lt;/p>
&lt;h2 id="which-one-should-you-pick">Which one should you pick?&lt;/h2>
&lt;p>If anybody on the internet uses the service, choose the host network. You will want the client addresses the first time somebody hammers your login page. If only you and your devices use it, the bridge is simpler. The apps then need no host ports, and no app can be reached past Traefik on the host.&lt;/p>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">systemctl --user daemon-reload
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">systemctl --user start traefik blog
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --user -u traefik -f &lt;span class="c1"># look for &amp;#34;certificate obtained&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -sI https://www.example.org/ &lt;span class="p">|&lt;/span> head -3
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Host network mode: the access log must show your real IP as ClientHost&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --user -u traefik -n &lt;span class="m">5&lt;/span> -o cat &lt;span class="p">|&lt;/span> jq -r .ClientHost
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="things-to-watch">Things to watch&lt;/h2>
&lt;ul>
&lt;li>Test VMs and Let&amp;rsquo;s Encrypt. I restored &lt;code>acme.json&lt;/code> into every test VM before Traefik&amp;rsquo;s first start, and blocked the VM&amp;rsquo;s outgoing connections while I tested. A VM that asks for certificates of the real domains counts against the rate limits of the live host.&lt;/li>
&lt;li>Test against a VM without touching DNS: &lt;code>curl --resolve www.example.org:443:&amp;lt;vm-ip&amp;gt; https://www.example.org/&lt;/code>. With the restored &lt;code>acme.json&lt;/code> the certificate is even valid.&lt;/li>
&lt;li>Bridge mode and fail2ban. On the bridge, the client address that Traefik logs is the helper&amp;rsquo;s address. A ban on that would block everyone. So do not point fail2ban at the access log in bridge mode.&lt;/li>
&lt;/ul>
&lt;p>Next: &lt;a href="../../blog/infra-04-selinux/">SELinux enforcing with containers&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://doc.traefik.io/traefik/">Traefik documentation&lt;/a>: entrypoints, routers, services and middlewares.&lt;/li>
&lt;li>&lt;a href="https://doc.traefik.io/traefik/providers/docker/">Traefik Docker provider&lt;/a>: the provider that reads the labels. It works with the Podman socket.&lt;/li>
&lt;li>&lt;a href="https://doc.traefik.io/traefik/https/acme/">Traefik ACME&lt;/a>: the HTTP, TLS and DNS challenges and the &lt;code>acme.json&lt;/code> storage.&lt;/li>
&lt;li>&lt;a href="https://go-acme.github.io/lego/dns/">lego DNS providers&lt;/a>: the list of DNS APIs that Traefik supports, with the environment variables each one needs.&lt;/li>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-network.1.html">podman-network(1)&lt;/a>: bridge networks for rootless containers.&lt;/li>
&lt;/ul></description></item><item><title>From Docker Compose to rootless Podman quadlets</title><link>https://www.linux-lunatix.org/blog/infra-02-quadlets/</link><pubDate>Wed, 30 Sep 2026 20:15:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-02-quadlets/</guid><description>&lt;p>&lt;em>Part 2 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>At the end of this post you have a two-container app (Forgejo with PostgreSQL) running as rootless Podman quadlets. They start at boot, update themselves and log to the journal. No root daemon and no Compose file are involved.&lt;/p>
&lt;h2 id="why-quadlets">Why quadlets&lt;/h2>
&lt;p>Podman can run Compose files with &lt;code>podman-compose&lt;/code> or &lt;code>docker-compose&lt;/code> against the Podman socket. I tried both. They work, but they keep the Compose model: one tool starts a group of containers, and systemd knows nothing about them. The older &lt;code>podman generate systemd&lt;/code> is deprecated.&lt;/p>
&lt;p>Quadlets put the containers into systemd directly. You write a &lt;code>.container&lt;/code> file. At boot, or at &lt;code>systemctl --user daemon-reload&lt;/code>, a systemd generator turns it into a normal service unit. From then on:&lt;/p>
&lt;ul>
&lt;li>&lt;code>systemctl --user start|stop|status forgejo&lt;/code> works as for any service.&lt;/li>
&lt;li>The logs are in the journal: &lt;code>journalctl --user -u forgejo&lt;/code>.&lt;/li>
&lt;li>Dependencies are plain systemd: &lt;code>Requires=&lt;/code> and &lt;code>After=&lt;/code>.&lt;/li>
&lt;li>&lt;code>Restart=always&lt;/code> and timers come from systemd too.&lt;/li>
&lt;/ul>
&lt;p>You need Podman 4.4 or newer. Fedora 44 ships Podman 5.&lt;/p>
&lt;h2 id="step-1-a-user-for-the-apps">Step 1: a user for the apps&lt;/h2>
&lt;p>Each host has one unprivileged user that owns all its containers. I call it &lt;code>svc&lt;/code>.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo useradd --create-home --shell /bin/bash svc
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Let svc&amp;#39;s systemd instance run without a login session, and start it at boot&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo loginctl enable-linger svc
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>useradd&lt;/code> on Fedora also writes ranges to &lt;code>/etc/subuid&lt;/code> and &lt;code>/etc/subgid&lt;/code>. Rootless Podman needs them to map the user IDs inside a container. Check them:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">grep svc /etc/subuid /etc/subgid
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># /etc/subuid:svc:524288:65536&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># /etc/subgid:svc:524288:65536&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>To work as that user, you need a real login session, or at least the right &lt;code>XDG_RUNTIME_DIR&lt;/code>. &lt;code>sudo -iu svc&lt;/code> alone gives you a shell where &lt;code>systemctl --user&lt;/code> fails. These work:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo machinectl shell svc@ &lt;span class="c1"># a full session (needs systemd-container)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># or, for single commands:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo -u svc &lt;span class="nv">XDG_RUNTIME_DIR&lt;/span>&lt;span class="o">=&lt;/span>/run/user/&lt;span class="k">$(&lt;/span>id -u svc&lt;span class="k">)&lt;/span> systemctl --user status
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Rootless containers cannot bind ports below 1024 by default. If an app must listen on 80 or 443 directly (Traefik in the next post), lower the limit:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s1">&amp;#39;net.ipv4.ip_unprivileged_port_start = 80&amp;#39;&lt;/span> &lt;span class="p">|&lt;/span> sudo tee /etc/sysctl.d/90-unprivileged-ports.conf
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo sysctl --system
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="step-2-the-compose-file-we-start-from">Step 2: the Compose file we start from&lt;/h2>
&lt;p>This is a shortened version of what ran on the old Ubuntu host:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="nt">services&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">forgejo&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">image&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">codeberg.org/forgejo/forgejo:15&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">restart&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">unless-stopped&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">DB_TYPE=postgres&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">DB_HOST=postgres:5432&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">DB_PASSWD=${FORGEJO_DB_PASSWORD}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">networks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">proxy, dbnet]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">depends_on&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">postgres]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">volumes&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">./forgejo:/data&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">labels&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;traefik.enable=true&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;traefik.http.routers.forgejo.rule=Host(`git.example.org`)&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">postgres&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">image&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">postgres:17-alpine&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">restart&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">always&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">POSTGRES_USER&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">forgejo&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">POSTGRES_PASSWORD&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">${FORGEJO_DB_PASSWORD}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">POSTGRES_DB&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">forgejo&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">networks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">dbnet]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">volumes&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">./postgres:/var/lib/postgresql/data&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">networks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dbnet&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">proxy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">external&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Each part of it becomes one quadlet file:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Compose&lt;/th>
&lt;th>Quadlet file&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>networks: dbnet&lt;/code>&lt;/td>
&lt;td>&lt;code>forgejo.network&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>a volume&lt;/td>
&lt;td>&lt;code>forgejo-data.volume&lt;/code>, &lt;code>forgejo-db.volume&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>a service&lt;/td>
&lt;td>&lt;code>forgejo.container&lt;/code>, &lt;code>forgejo-postgres.container&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>.env&lt;/code> with a password&lt;/td>
&lt;td>a Podman secret&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>depends_on&lt;/code>&lt;/td>
&lt;td>&lt;code>Requires=&lt;/code> and &lt;code>After=&lt;/code>&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>All files go into &lt;code>~svc/.config/containers/systemd/&lt;/code>.&lt;/p>
&lt;h2 id="step-3-network-and-volumes">Step 3: network and volumes&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># forgejo.network&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Network]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">NetworkName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># forgejo-data.volume&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Volume]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">VolumeName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-data&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># forgejo-db.volume&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Volume]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">VolumeName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-db&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Without &lt;code>NetworkName=&lt;/code> and &lt;code>VolumeName=&lt;/code>, Podman calls them &lt;code>systemd-forgejo&lt;/code> and &lt;code>systemd-forgejo-data&lt;/code>. That works, but the explicit names are easier to read in &lt;code>podman volume ls&lt;/code> and in backup scripts.&lt;/p>
&lt;h2 id="step-4-a-secret-instead-of-the-dot-env-file">Step 4: a secret instead of the .env file&lt;/h2>
&lt;p>Compose read the database password from &lt;code>.env&lt;/code>. Podman has secrets. Create one as &lt;code>svc&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">printf&lt;/span> &lt;span class="s1">&amp;#39;%s&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">$(&lt;/span>pwgen -s &lt;span class="m">32&lt;/span> 1&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="p">|&lt;/span> podman secret create forgejo-db-password -
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman secret ls
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The secret is stored in the user&amp;rsquo;s Podman storage and is not part of any image or quadlet file. In post 6 Ansible creates it from &lt;code>pass&lt;/code>, so I never type it.&lt;/p>
&lt;h2 id="step-5-the-containers">Step 5: the containers&lt;/h2>
&lt;p>The database first:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># forgejo-postgres.container&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Unit]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Description&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">Forgejo database&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-postgres&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">docker.io/library/postgres:17-alpine&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo.network&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># The host name the app uses for its database&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">NetworkAlias&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">postgres&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-db.volume:/var/lib/postgresql/data:Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Environment&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">POSTGRES_USER=forgejo POSTGRES_DB=forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Secret&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-db-password,type=env,target=POSTGRES_PASSWORD&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">HealthCmd&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">pg_isready -U forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">HealthInterval&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">30s&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then the app:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># forgejo.container&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Unit]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Description&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">Forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Requires&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-postgres.service&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">After&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-postgres.service&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">codeberg.org/forgejo/forgejo:15&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo.network&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-data.volume:/data:Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Web UI only on localhost: the reverse proxy forwards to it&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">127.0.0.1:3000:3000&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Git over SSH, published directly&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">2222:22&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.enable=true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.forgejo.rule=Host(`git.example.org`)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.forgejo.entrypoints=https&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.forgejo.tls.certresolver=default&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.services.forgejo.loadbalancer.server.port=3000&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Some details:&lt;/p>
&lt;ul>
&lt;li>&lt;code>Network=forgejo.network&lt;/code> refers to the quadlet file, not to the network name. systemd then also adds a dependency on the network unit.&lt;/li>
&lt;li>&lt;code>Volume=forgejo-db.volume:...&lt;/code> is the same for volumes.&lt;/li>
&lt;li>&lt;code>:Z&lt;/code> gives the volume a private SELinux label. Post 4 explains &lt;code>:Z&lt;/code> and &lt;code>:z&lt;/code>.&lt;/li>
&lt;li>&lt;code>[Install]&lt;/code> with &lt;code>WantedBy=default.target&lt;/code> starts the service at boot. You cannot &lt;code>systemctl --user enable&lt;/code> a quadlet, because the unit is generated. The &lt;code>[Install]&lt;/code> section does that job.&lt;/li>
&lt;li>The &lt;code>Label=&lt;/code> lines are for Traefik. Compose had them as &lt;code>labels:&lt;/code>, and they stay the same. The next post sets up Traefik.&lt;/li>
&lt;/ul>
&lt;h2 id="step-6-start-it">Step 6: start it&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">systemctl --user daemon-reload
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">systemctl --user start forgejo &lt;span class="c1"># starts forgejo-postgres first&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">systemctl --user status forgejo forgejo-postgres
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman ps
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --user -u forgejo -f
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>If a unit does not appear after &lt;code>daemon-reload&lt;/code>, the generator did not like the file. Ask it directly:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">/usr/libexec/podman/quadlet -dryrun -user
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>It prints the generated units, or the error with the file and line.&lt;/p>
&lt;h2 id="step-7-updates-with-podman-auto-update">Step 7: updates with podman auto-update&lt;/h2>
&lt;p>&lt;code>AutoUpdate=registry&lt;/code> marks a container for &lt;code>podman auto-update&lt;/code>. The command checks the registry for a new image under the same tag. If there is one, it pulls it and restarts the unit. If the new container fails to start, it rolls back to the old image.&lt;/p>
&lt;p>Enable the timer that ships with Podman:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">systemctl --user &lt;span class="nb">enable&lt;/span> --now podman-auto-update.timer
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman auto-update --dry-run &lt;span class="c1"># what would be updated now&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The timer runs once a day. To change that, add a drop-in:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ~/.config/systemd/user/podman-auto-update.timer.d/interval.conf&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Timer]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">OnCalendar&lt;/span>&lt;span class="o">=&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">OnCalendar&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">*:0/15&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">RandomizedDelaySec&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">0&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The empty &lt;code>OnCalendar=&lt;/code> clears the default first. Without it, you get both schedules.&lt;/p>
&lt;p>The tag decides what an update can bring. &lt;code>postgres:17-alpine&lt;/code> gets patch releases of PostgreSQL 17, never 18. A major PostgreSQL upgrade needs a dump and a restore, and I want to do that by hand. &lt;code>forgejo:15&lt;/code> is the same. I use &lt;code>:latest&lt;/code> only for images where every new version is fine, for example my own website images.&lt;/p>
&lt;h2 id="step-8-move-the-data">Step 8: move the data&lt;/h2>
&lt;p>The old data was in bind mounts under &lt;code>/opt&lt;/code>, owned by root and by the container&amp;rsquo;s user IDs. Rootless Podman maps IDs: UID 0 in the container is &lt;code>svc&lt;/code> on the host, and UID 1000 in the container is some high number from &lt;code>/etc/subuid&lt;/code>. A plain &lt;code>cp&lt;/code> therefore gives the wrong owners.&lt;/p>
&lt;p>&lt;code>podman unshare&lt;/code> runs a command inside the user&amp;rsquo;s ID mapping. Owners then look the same as they did inside the old container:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># as root: copy the old bind mount into the new volume, owners as numbers&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman_svc&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> runuser -u svc -- env &lt;span class="nv">XDG_RUNTIME_DIR&lt;/span>&lt;span class="o">=&lt;/span>/run/user/&lt;span class="k">$(&lt;/span>id -u svc&lt;span class="k">)&lt;/span> podman &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$@&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman_svc volume create --ignore forgejo-data
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">mp&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>podman_svc volume inspect --format &lt;span class="s1">&amp;#39;{{.Mountpoint}}&amp;#39;&lt;/span> forgejo-data&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">tar -C /opt/forgejo/forgejo --numeric-owner -cf - . &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> runuser -u svc -- podman unshare tar -C &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$mp&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --numeric-owner -xpf -
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Do not copy a database directory this way while the database runs. For PostgreSQL I used a dump: &lt;code>pg_dumpall&lt;/code> on the old host, then &lt;code>psql&lt;/code> in the new container.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># on the old host&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">docker &lt;span class="nb">exec&lt;/span> forgejo-postgres-1 pg_dumpall -U forgejo &lt;span class="p">|&lt;/span> gzip &amp;gt; forgejo.sql.gz
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># on the new host, as svc: start with an empty database, then load the dump&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">systemctl --user stop forgejo
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman &lt;span class="nb">exec&lt;/span> forgejo-postgres sh -c &lt;span class="s1">&amp;#39;dropdb -U forgejo --force forgejo &amp;amp;&amp;amp; createdb -U forgejo forgejo&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">gunzip -c forgejo.sql.gz &lt;span class="p">|&lt;/span> podman &lt;span class="nb">exec&lt;/span> -i forgejo-postgres psql -q -U forgejo -d postgres
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">systemctl --user start forgejo
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="what-bit-me">What bit me&lt;/h2>
&lt;ul>
&lt;li>The working directory. &lt;code>runuser -u svc -- podman ...&lt;/code> from a directory that &lt;code>svc&lt;/code> cannot read fails with strange errors. Run it from the root directory (&lt;code>cd / &amp;amp;&amp;amp; runuser ...&lt;/code>).&lt;/li>
&lt;li>Containers that share a network namespace. A VPN container with other containers inside its namespace (&lt;code>Network=container:gluetun&lt;/code>) cannot be restarted on its own. Stop the dependants first, in reverse order, then start everything in order.&lt;/li>
&lt;li>Files mounted into containers. An editor or Ansible replaces a file with a new inode. The container still sees the old file until it restarts. Restart the unit after you change a mounted config file.&lt;/li>
&lt;li>&lt;code>sudo -iu svc&lt;/code>. Without &lt;code>XDG_RUNTIME_DIR&lt;/code>, &lt;code>systemctl --user&lt;/code> says it cannot connect to the bus. Use &lt;code>machinectl shell&lt;/code> or set the variable.&lt;/li>
&lt;/ul>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo reboot
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># after the reboot, without logging in as svc:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo -u svc &lt;span class="nv">XDG_RUNTIME_DIR&lt;/span>&lt;span class="o">=&lt;/span>/run/user/&lt;span class="k">$(&lt;/span>id -u svc&lt;span class="k">)&lt;/span> systemctl --user is-active forgejo forgejo-postgres
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Both say &lt;code>active&lt;/code>. The containers run as &lt;code>svc&lt;/code>, and nobody logged in.&lt;/p>
&lt;p>Next: &lt;a href="../../blog/infra-03-traefik/">Traefik in front of rootless containers&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html">podman-systemd.unit(5)&lt;/a>: the quadlet reference: every key of &lt;code>.container&lt;/code>, &lt;code>.volume&lt;/code>, &lt;code>.network&lt;/code> and the other unit types.&lt;/li>
&lt;li>&lt;a href="https://www.redhat.com/en/blog/quadlet-podman">Make systemd better for Podman with Quadlet (Red Hat blog)&lt;/a>: the reasons behind quadlets, from the people who wrote them.&lt;/li>
&lt;li>&lt;a href="https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md">Basic setup and use of Podman in a rootless environment&lt;/a>: subuids, storage and the limits of rootless mode.&lt;/li>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-auto-update.1.html">podman-auto-update(1)&lt;/a>: how &lt;code>AutoUpdate=registry&lt;/code> decides to pull and how the rollback works.&lt;/li>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-secret-create.1.html">podman-secret-create(1)&lt;/a>: secret drivers and how a container gets them.&lt;/li>
&lt;li>&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/loginctl.html">loginctl(1)&lt;/a>: &lt;code>enable-linger&lt;/code>, so that user services run without a login.&lt;/li>
&lt;li>&lt;a href="https://github.com/containers/podlet">podlet&lt;/a>: converts compose files and &lt;code>podman run&lt;/code> commands to quadlets. A good start, but read what it writes.&lt;/li>
&lt;/ul></description></item><item><title>Four servers, three distributions: why I consolidated</title><link>https://www.linux-lunatix.org/blog/infra-01-consolidation/</link><pubDate>Sat, 26 Sep 2026 18:40:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-01-consolidation/</guid><description>&lt;h2 id="where-i-started">Where I started&lt;/h2>
&lt;p>I run a few small servers for myself, my family and a couple of friends. Over the years they grew the way such things grow: one at a time, each with the tools I liked at that moment. When I finally wrote down what I had, it looked like this:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Host&lt;/th>
&lt;th>Distribution&lt;/th>
&lt;th>Runs&lt;/th>
&lt;th>Managed with&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>mail1.example.org&lt;/code>&lt;/td>
&lt;td>Ubuntu 24.04&lt;/td>
&lt;td>Mailcow, Forgejo, Matrix, a paste bin, about 35 containers&lt;/td>
&lt;td>Docker Compose&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>vps2.example.org&lt;/code>&lt;/td>
&lt;td>Arch Linux&lt;/td>
&lt;td>three static websites&lt;/td>
&lt;td>rootless Podman, by hand&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>vps1.example.org&lt;/code>&lt;/td>
&lt;td>Fedora Cloud&lt;/td>
&lt;td>a CI runner, a push notification server&lt;/td>
&lt;td>rootless Podman, quadlets&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>home1.example.net&lt;/code>&lt;/td>
&lt;td>Fedora Server&lt;/td>
&lt;td>monitoring, media, DNS, about 20 containers&lt;/td>
&lt;td>rootless Podman, quadlets&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>(All names in this series are examples. My real hosts have other names.)&lt;/p>
&lt;p>Four servers, three distributions and four ways to manage them. The configuration was in four Git repositories. A script copied shared files between them, when I remembered to run it. How a host was set up was in READMEs, in shell history and in my head.&lt;/p>
&lt;p>Nothing was broken. But every change was slow, because I first had to remember how &lt;em>this&lt;/em> host did things. And I knew that a reinstall of the mail server would cost me a weekend of reading my own old notes.&lt;/p>
&lt;h2 id="what-i-wanted">What I wanted&lt;/h2>
&lt;p>I wrote down four goals before I touched anything:&lt;/p>
&lt;ol>
&lt;li>One distribution everywhere: Fedora.&lt;/li>
&lt;li>The OS setup and every app rolled out with Ansible, from one repository.&lt;/li>
&lt;li>Every app as a rootless Podman quadlet: a systemd unit, no Compose file, no root daemon.&lt;/li>
&lt;li>A reinstall of any host is a routine job: install, run the playbook, restore the data.&lt;/li>
&lt;/ol>
&lt;h2 id="the-decisions-and-why">The decisions, and why&lt;/h2>
&lt;h3 id="fedora-everywhere">Fedora everywhere&lt;/h3>
&lt;p>I already had two Fedora hosts, and I liked how they behaved. Fedora ships a current Podman with quadlet support, systemd, firewalld and SELinux, all of them integrated and enabled by default. A new release comes every six months, and &lt;code>dnf system-upgrade&lt;/code> takes a few minutes. Ubuntu and Arch would work too. But with one distribution there is one package manager, one firewall and one set of paths, and my Ansible roles can be much simpler.&lt;/p>
&lt;h3 id="rootless-podman-with-quadlets">Rootless Podman with quadlets&lt;/h3>
&lt;p>A quadlet is a small ini file, for example &lt;code>~/.config/containers/systemd/blog.container&lt;/code>. systemd turns it into a normal service at boot. Each container is then a unit that you start, stop and read logs from like any other service. Rootless means that the containers run as a normal user. A container that breaks out lands in an unprivileged account, and with SELinux it lands in a confined domain too.&lt;/p>
&lt;p>The next post shows the move from Compose to quadlets in detail.&lt;/p>
&lt;h3 id="docker-only-where-nothing-else-works">Docker only where nothing else works&lt;/h3>
&lt;p>Mailcow is the exception. It supports only Docker with Compose, and its update script expects exactly that. A mail server is not where I want to experiment, so Mailcow stays on Docker. Everything else on that host moves to rootless Podman.&lt;/p>
&lt;h3 id="one-repository">One repository&lt;/h3>
&lt;p>All hosts, all apps, all docs in one Git repository. A change that touches three hosts is one commit. The repository is the truth: if a setting is not in it, it does not exist.&lt;/p>
&lt;h3 id="secrets-in-pass-not-in-git">Secrets in pass, not in Git&lt;/h3>
&lt;p>I already use &lt;a href="https://www.passwordstore.org/">pass&lt;/a> (GPG-encrypted files in a Git repository) for my passwords. Ansible can read from it with the &lt;code>community.general.passwordstore&lt;/code> lookup. So there is no Ansible Vault and no encrypted blob in the infrastructure repository. The repository can be public in principle, and nothing secret leaks with it.&lt;/p>
&lt;h3 id="selinux-enforcing-no-exceptions">SELinux enforcing, no exceptions&lt;/h3>
&lt;p>Many container guides start with &lt;code>setenforce 0&lt;/code>. I wanted the opposite. Every host is enforcing, and the playbook stops if one is not. Every local policy change is code in the repository. Post 4 is about this.&lt;/p>
&lt;h3 id="ansible-owns-configuration-not-versions">Ansible owns configuration, not versions&lt;/h3>
&lt;p>Ansible writes the quadlets, config files and secrets. It does not pin image versions. Images are updated by &lt;code>podman auto-update&lt;/code> (daily). Images I build myself are updated by their CI pipelines. With this split, a playbook run is predictable: it changes what I changed in Git and nothing else.&lt;/p>
&lt;h2 id="the-strategy-adopt-first-then-rebuild">The strategy: adopt first, then rebuild&lt;/h2>
&lt;p>The obvious way would be to write roles, reinstall a server and see what breaks. I did not want that for hosts that other people use.&lt;/p>
&lt;p>So I turned it around:&lt;/p>
&lt;ol>
&lt;li>Adopt the hosts that already run Fedora. I wrote the roles against the live hosts. I ran &lt;code>ansible-playbook --check --diff&lt;/code> again and again and changed the code until the diff was empty. At that point the code describes the host exactly as it is. The real run then changes nothing. This tests the roles on real systems and puts nothing at risk.&lt;/li>
&lt;li>Rehearse on a VM. For the hosts that must be reinstalled, I made a local Fedora VM with the same settings. I ran the full rollout on it and restored the real backups into it. Then I tested the services against the VM.&lt;/li>
&lt;li>Reinstall for real. Only after the rehearsal passed did I reinstall the real host. Same steps, same order.&lt;/li>
&lt;/ol>
&lt;p>This takes longer than a &amp;ldquo;big bang&amp;rdquo;. But at no point was a service down because of untested code.&lt;/p>
&lt;h2 id="the-phases">The phases&lt;/h2>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Phase&lt;/th>
&lt;th>What&lt;/th>
&lt;th>Host(s)&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>0&lt;/td>
&lt;td>Repository, collections, inventory, &lt;code>ansible&lt;/code> user&lt;/td>
&lt;td>all&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>1&lt;/td>
&lt;td>Write the roles, adopt the first Fedora host&lt;/td>
&lt;td>&lt;code>vps1&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>2&lt;/td>
&lt;td>Adopt the home server (monitoring, about 20 apps)&lt;/td>
&lt;td>&lt;code>home1&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>3&lt;/td>
&lt;td>Rehearsal VMs&lt;/td>
&lt;td>VMs on a spare box&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>4&lt;/td>
&lt;td>Reinstall the Arch host with Fedora&lt;/td>
&lt;td>&lt;code>vps2&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>5&lt;/td>
&lt;td>Reinstall the Ubuntu mail host with Fedora&lt;/td>
&lt;td>&lt;code>mail1&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>6&lt;/td>
&lt;td>Clean up: archive the old repositories&lt;/td>
&lt;td>&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;h2 id="the-repository-layout">The repository layout&lt;/h2>
&lt;p>This is the layout I ended up with. The next posts go through each part.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">ops/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── ansible.cfg
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── requirements.yml # pinned collections
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── inventory/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── hosts.yml # hosts and groups
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── group_vars/all.yml # settings for every host
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── group_vars/&amp;lt;group&amp;gt;.yml
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── host_vars/&amp;lt;host&amp;gt;.yml # app_user, apps: [...], firewall, backup, ...
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── playbooks/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── bootstrap.yml # first contact: creates the ansible user
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── site.yml # everything else, with tags
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── roles/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── base/ # packages, sshd, firewalld, fail2ban, updates, SELinux
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── podman/ # the app user, linger, socket, auto-update timer
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── quadlet_app/ # one generic role for every app
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── monitoring_agent/ # WireGuard tunnel, node_exporter
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── backup/ # restic, timer, healthchecks
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── ...
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── apps/&amp;lt;name&amp;gt;/ # one directory per app: quadlets, files, app.yml
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── selinux/ # local policy modules as source
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── tools/ # small scripts: rehearsal VM, image build, restore
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">└── docs/
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The most important idea is in &lt;code>apps/&lt;/code>. There is no hand-written role per service. There is one generic role, and one directory per app with its quadlet files and config. Adding an app means adding a directory and putting its name into the host&amp;rsquo;s &lt;code>apps:&lt;/code> list. Post 6 shows how that role works.&lt;/p>
&lt;h2 id="how-i-check-that-it-works">How I check that it works&lt;/h2>
&lt;p>For every phase I used the same tests:&lt;/p>
&lt;ul>
&lt;li>&lt;code>ansible-playbook site.yml --check --diff&lt;/code> shows no changes on an adopted host.&lt;/li>
&lt;li>A second real run reports &lt;code>changed=0&lt;/code>. A role that changes something on every run hides real changes, so I treat that as a bug.&lt;/li>
&lt;li>After each phase all monitoring targets are up and no alert fires.&lt;/li>
&lt;li>After a reinstall the backup restore works. That was also my first real restore test in years.&lt;/li>
&lt;/ul>
&lt;h2 id="the-series">The series&lt;/h2>
&lt;ol>
&lt;li>This post: why and how.&lt;/li>
&lt;li>&lt;a href="../../blog/infra-02-quadlets/">From Docker Compose to rootless Podman quadlets&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-03-traefik/">Traefik in front of rootless containers&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-04-selinux/">SELinux enforcing with containers&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-05-ansible-base/">Ansible groundwork: inventory, bootstrap and a base role&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-06-quadlet-app/">One generic Ansible role for every app&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-07-monitoring-backup/">Monitoring and backups for every host&lt;/a>&lt;/li>
&lt;li>&lt;a href="../../blog/infra-08-rehearsal/">Rehearse before you reinstall&lt;/a>&lt;/li>
&lt;/ol>
&lt;p>A second part of the series is about SSH. I replaced all &lt;code>authorized_keys&lt;/code> files with short-lived SSH certificates from my own CA, run on OpenBao. It starts with &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/p>
&lt;p>Next: &lt;a href="../../blog/infra-02-quadlets/">From Docker Compose to rootless Podman quadlets&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.fedoraproject.org/en-US/fedora-server/">Fedora Server documentation&lt;/a>: installation, administration and the release cycle.&lt;/li>
&lt;li>&lt;a href="https://podman.io/">Podman&lt;/a>: the project home, with links to the docs and the blog.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/">Ansible documentation&lt;/a>: the entry point for everything in part 1.&lt;/li>
&lt;li>&lt;a href="https://www.passwordstore.org/">pass, the standard Unix password manager&lt;/a>: GPG files in a git repository, the secrets store of this series.&lt;/li>
&lt;li>&lt;a href="https://docs.fedoraproject.org/en-US/quick-docs/selinux-getting-started/">Getting started with SELinux (Fedora Quick Docs)&lt;/a>: modes, contexts and the first commands.&lt;/li>
&lt;li>&lt;a href="https://docs.mailcow.email/">Mailcow documentation&lt;/a>: the one Docker exception in my setup.&lt;/li>
&lt;/ul></description></item></channel></rss>