<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Podman on Linux Lunatix</title><link>https://www.linux-lunatix.org/categories/podman/</link><description>Recent content in Podman on Linux Lunatix</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><copyright>2025 Stefan Marx :: All written work is licensed under CC BY-SA 4.0 unless otherwise stated. Images are taken by me unless otherwise stated and need written permission to be used elsewhere.</copyright><lastBuildDate>Thu, 05 Nov 2026 20:35:00 +0100</lastBuildDate><atom:link href="https://www.linux-lunatix.org/categories/podman/index.xml" rel="self" type="application/rss+xml"/><item><title>OpenBao rootless on Podman</title><link>https://www.linux-lunatix.org/blog/ssh-02-openbao/</link><pubDate>Thu, 05 Nov 2026 20:35:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/ssh-02-openbao/</guid><description>&lt;p>&lt;em>Part 2 of the SSH series that starts with &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/em>&lt;/p>
&lt;p>At the end of this post OpenBao runs on the home server as a rootless container. It has two SSH CAs (users and hosts), roles for each kind of certificate, and a login through Forgejo in the browser. Its configuration is one script in Git, and a nightly Raft snapshot goes into the backup.&lt;/p>
&lt;h2 id="where-it-runs-and-who-can-reach-it">Where it runs, and who can reach it&lt;/h2>
&lt;p>Bao runs on my home server, &lt;code>home1&lt;/code>. It is reachable from the LAN and through my WireGuard tunnel, never from the internet. The home router does not forward port 443 to it.&lt;/p>
&lt;p>Two DNS records make the same name work in both places:&lt;/p>
&lt;ul>
&lt;li>On the LAN, the local DNS (Pi-hole) answers &lt;code>bao.example.net&lt;/code> with the LAN address, &lt;code>192.168.50.5&lt;/code>.&lt;/li>
&lt;li>The public record of &lt;code>bao.example.net&lt;/code> points to the WireGuard address of the home server, &lt;code>10.99.0.1&lt;/code>. That address is private and only works inside the tunnel. Away from home, I start the tunnel first.&lt;/li>
&lt;/ul>
&lt;p>Traefik gets the certificate with the DNS challenge, so Let&amp;rsquo;s Encrypt never needs to connect to the host. TLS ends at Traefik.&lt;/p>
&lt;p>Hosts do not need Bao to check a certificate. They only need the CA&amp;rsquo;s public key, which they have as a file. Bao must be reachable only to &lt;em>get&lt;/em> a certificate: from my laptop, from the Ansible control node and from the CI runner. If Bao is down, logins with existing certificates keep working.&lt;/p>
&lt;h2 id="step-1-the-quadlet">Step 1: the quadlet&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ~svc/.config/containers/systemd/openbao.container&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Unit]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Description&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">OpenBao&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">openbao&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Minor version pinned, patch releases through auto-update&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">quay.io/openbao/openbao:2.7&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.enable=true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.openbao.rule=Host(`bao.example.net`)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.openbao.entrypoints=https&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.openbao.tls.certresolver=default&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.openbao.middlewares=secureHeaders@file&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.services.openbao.loadbalancer.server.port=8200&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Exec&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">bao server -config=/openbao/config/openbao.hcl&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># The image&amp;#39;s user openbao (100:1000) is svc on the host: no chown needed&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">UserNS&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">keep-id:uid=100,gid=1000&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Environment&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">SKIP_CHOWN=1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/openbao/openbao.hcl:/openbao/config/openbao.hcl:ro,z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">openbao-data.volume:/openbao/file:Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ReadOnly&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Tmpfs&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">/tmp&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">DropCapability&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">ALL&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">NoNewPrivileges&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">proxy.network:ip=10.89.0.14&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># openbao-data.volume&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Volume]&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Some notes:&lt;/p>
&lt;ul>
&lt;li>&lt;code>UserNS=keep-id:uid=100,gid=1000&lt;/code> maps the image&amp;rsquo;s &lt;code>openbao&lt;/code> user to &lt;code>svc&lt;/code> on the host. The files in the volume then belong to &lt;code>svc&lt;/code>, and the entrypoint does not need to &lt;code>chown&lt;/code> anything. &lt;code>SKIP_CHOWN=1&lt;/code> tells it not to try.&lt;/li>
&lt;li>&lt;code>ReadOnly=true&lt;/code>, &lt;code>DropCapability=ALL&lt;/code> and &lt;code>NoNewPrivileges=true&lt;/code>: Bao needs to write only into its volume and &lt;code>/tmp&lt;/code>.&lt;/li>
&lt;li>Pin the minor version. I want to read the release notes before a minor upgrade of the thing that holds my CAs. Patch releases come through &lt;code>podman auto-update&lt;/code>.&lt;/li>
&lt;/ul>
&lt;h2 id="step-2-the-server-config">Step 2: the server config&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-hcl" data-lang="hcl">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ~svc/.config/openbao/openbao.hcl
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">&lt;/span>&lt;span class="n">ui&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="kt">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">api_addr&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;https://bao.example.net&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">cluster_addr&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;http://127.0.0.1:8201&amp;#34;&lt;/span>&lt;span class="c1">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Integrated storage: mlock is not wanted (memory-mapped database)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">&lt;/span>&lt;span class="n">disable_mlock&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="kt">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">storage&lt;/span> &lt;span class="s2">&amp;#34;raft&amp;#34;&lt;/span> {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n"> path&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;/openbao/file&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n"> node_id&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;home1&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">}
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">listener&lt;/span> &lt;span class="s2">&amp;#34;tcp&amp;#34;&lt;/span> {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n"> address&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;0.0.0.0:8200&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n"> tls_disable&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="kt">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">}&lt;span class="c1">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Every request, with secret values HMACed, to stdout: journald through Podman
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">&lt;/span>&lt;span class="k">audit&lt;/span> &lt;span class="s2">&amp;#34;file&amp;#34; &amp;#34;stdout&amp;#34;&lt;/span> {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">options&lt;/span> {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n"> file_path&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;stdout&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> }
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Integrated storage (Raft) is a single directory and needs no external database. A single node is a valid Raft cluster.&lt;/p>
&lt;p>&lt;code>tls_disable&lt;/code> looks scary, but the listener is reachable only on the Podman network, from Traefik. TLS ends there.&lt;/p>
&lt;p>The audit device is declared in the config file, so it exists from the very first start. The log is in the journal with everything else.&lt;/p>
&lt;h2 id="step-3-initialise">Step 3: initialise&lt;/h2>
&lt;p>Start it, then initialise it once:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">systemctl --user daemon-reload &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> systemctl --user start openbao
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="o">=&lt;/span>https://bao.example.net
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao operator init -key-shares&lt;span class="o">=&lt;/span>&lt;span class="m">1&lt;/span> -key-threshold&lt;span class="o">=&lt;/span>&lt;span class="m">1&lt;/span> -format&lt;span class="o">=&lt;/span>json &amp;gt; /dev/shm/init.json
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>init&lt;/code> prints the unseal key and the initial root token. They go straight into pass, and the file is deleted:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">jq -r &lt;span class="s1">&amp;#39;.unseal_keys_b64[0]&amp;#39;&lt;/span> /dev/shm/init.json &lt;span class="p">|&lt;/span> pass insert -m infra/home1/openbao-unseal-key
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">jq -r &lt;span class="s1">&amp;#39;.root_token&amp;#39;&lt;/span> /dev/shm/init.json &lt;span class="p">|&lt;/span> pass insert -m infra/home1/openbao-root-token
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">shred -u /dev/shm/init.json
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>One key share is enough for one person. Shamir splitting into several shares makes sense when several people must agree to unseal.&lt;/p>
&lt;p>I use &lt;code>/dev/shm&lt;/code> so that the file is in memory and never on disk.&lt;/p>
&lt;h2 id="step-4-unseal-and-why-not-automatically">Step 4: unseal, and why not automatically&lt;/h2>
&lt;p>Bao encrypts everything in its storage. After every start it is sealed: it has the data but not the key. Somebody must give it the unseal key.&lt;/p>
&lt;p>I wanted auto-unseal with the home server&amp;rsquo;s TPM. The plan: encrypt a static seal key with &lt;code>systemd-creds --with-key=tpm2&lt;/code> and give it to the container with &lt;code>LoadCredentialEncrypted=&lt;/code>. Bao would then unseal itself after a reboot, and a stolen disk would be useless without the board.&lt;/p>
&lt;p>It did not work. The board has a TPM 1.2, and &lt;code>systemd-creds&lt;/code> supports only TPM 2.0. So I unseal by hand, from my laptop, with the key from pass:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># bao-unseal: run on the laptop, at home or through the tunnel&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">addr&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="nv">https&lt;/span>&lt;span class="p">://bao.example.net&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sealed&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> s
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">s&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>curl -fsS -m &lt;span class="m">15&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$addr&lt;/span>&lt;span class="s2">/v1/sys/seal-status&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span> &lt;span class="o">||&lt;/span> &lt;span class="o">{&lt;/span> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;OpenBao not reachable at &lt;/span>&lt;span class="nv">$addr&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt;&lt;span class="p">&amp;amp;&lt;/span>2&lt;span class="p">;&lt;/span> &lt;span class="nb">exit&lt;/span> 1&lt;span class="p">;&lt;/span> &lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> python3 -c &lt;span class="s1">&amp;#39;import json,sys; sys.exit(0 if json.loads(sys.argv[1])[&amp;#34;sealed&amp;#34;] else 1)&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$s&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> ! sealed&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;OpenBao is not sealed&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">exit&lt;/span> &lt;span class="m">0&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># the key goes to curl on stdin, never onto a command line&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">pass show infra/home1/openbao-unseal-key &lt;span class="p">|&lt;/span> head -1 &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.dumps({&amp;#34;key&amp;#34;: sys.stdin.read().strip()}))&amp;#39;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> curl -fsS -m &lt;span class="m">15&lt;/span> -X PUT -d @- -o /dev/null &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$addr&lt;/span>&lt;span class="s2">/v1/sys/unseal&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> sealed&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;still sealed&amp;#34;&lt;/span> &amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">2&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">exit&lt;/span> &lt;span class="m">1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;OpenBao unsealed&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The key never appears in &lt;code>ps&lt;/code> or in the shell history. A Prometheus blackbox probe watches &lt;code>/v1/sys/health&lt;/code>, which answers 503 while Bao is sealed. An alert tells me after five minutes.&lt;/p>
&lt;p>This is less work than it sounds. The home server reboots only when I reboot it. But remember that a &lt;code>podman auto-update&lt;/code> of the image, or any change to the quadlet, restarts the container, and the restart seals it again.&lt;/p>
&lt;h2 id="step-5-the-configuration-as-a-script">Step 5: the configuration as a script&lt;/h2>
&lt;p>Everything inside Bao (mounts, roles, policies, logins) is set by one script in the repository. It is idempotent: I change the script and run it again. There is no click in the web UI that I would forget.&lt;/p>
&lt;p>The script needs a token with full rights. The first run uses the root token from pass. Later runs use my &lt;code>userpass&lt;/code> login, which the script creates:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">BAO_TOKEN&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>pass show infra/home1/openbao-root-token&lt;span class="k">)&lt;/span> ./bao-config.sh &lt;span class="c1"># first run&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao login -no-print -method&lt;span class="o">=&lt;/span>userpass &lt;span class="nv">username&lt;/span>&lt;span class="o">=&lt;/span>alice &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> ./bao-config.sh &lt;span class="c1"># later&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Always use &lt;code>-no-print&lt;/code>. Without it, &lt;code>bao login&lt;/code> prints the token on the screen.&lt;/p>
&lt;p>After the first run, revoke the root token. A new one can be made from the unseal key with &lt;code>bao operator generate-root&lt;/code> when you need it.&lt;/p>
&lt;h3 id="the-ssh-cas">The SSH CAs&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="nv">https&lt;/span>&lt;span class="p">://bao.example.net&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">mounted&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> bao &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$1&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> list -format&lt;span class="o">=&lt;/span>json &lt;span class="p">|&lt;/span> python3 -c &lt;span class="s1">&amp;#39;import json,sys; sys.exit(0 if sys.argv[1] + &amp;#34;/&amp;#34; in json.load(sys.stdin) else 1)&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$2&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">secrets&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> mounted secrets &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$2&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">||&lt;/span> bao secrets &lt;span class="nb">enable&lt;/span> -path&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$2&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$1&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">auth&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> mounted auth &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$2&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">||&lt;/span> bao auth &lt;span class="nb">enable&lt;/span> -path&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$2&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$1&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Two SSH CAs, keys generated inside Bao (ed25519)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">for&lt;/span> m in ssh-user ssh-host&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> secrets ssh &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$m&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> bao &lt;span class="nb">read&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$m&lt;/span>&lt;span class="s2">/config/ca&amp;#34;&lt;/span> &amp;gt;/dev/null 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">1&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="o">||&lt;/span> bao write &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$m&lt;/span>&lt;span class="s2">/config/ca&amp;#34;&lt;/span> &lt;span class="nv">generate_signing_key&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nb">true&lt;/span> &lt;span class="nv">key_type&lt;/span>&lt;span class="o">=&lt;/span>ed25519 &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>generate_signing_key=true&lt;/code> creates the key inside Bao. It never leaves it. &lt;code>bao read ssh-user/config/ca&lt;/code> shows the public key, which is what the hosts need.&lt;/p>
&lt;p>The &lt;code>|| bao write&lt;/code> part makes it idempotent. A second run would otherwise replace the CA key, and every certificate would be invalid.&lt;/p>
&lt;h3 id="the-user-roles">The user roles&lt;/h3>
&lt;p>A small function writes a role. The JSON body is easier to read than a long list of &lt;code>key=value&lt;/code> arguments:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Key ID: who asked (token display name) and with which role; it shows in sshd&amp;#39;s log&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">user_role&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> &lt;span class="c1"># name principal ttl extensions [critical options as JSON]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> exts &lt;span class="nv">crit&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">5&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="s1">&amp;#39;{}&amp;#39;&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">exts&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.dumps({e: &amp;#34;&amp;#34; for e in sys.argv[1].split(&amp;#34;,&amp;#34;) if e}))&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$4&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> bao write &lt;span class="s2">&amp;#34;ssh-user/roles/&lt;/span>&lt;span class="nv">$1&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> - &amp;gt;/dev/null &lt;span class="s">&amp;lt;&amp;lt;JSON
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">{
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;key_type&amp;#34;: &amp;#34;ca&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allow_user_certificates&amp;#34;: true,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_users&amp;#34;: &amp;#34;$2&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;default_user&amp;#34;: &amp;#34;$2&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;ttl&amp;#34;: &amp;#34;$3&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;max_ttl&amp;#34;: &amp;#34;$3&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_extensions&amp;#34;: &amp;#34;$4&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;default_extensions&amp;#34;: $exts,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_critical_options&amp;#34;: &amp;#34;$(python3 -c &amp;#39;import json,sys; print(&amp;#34;,&amp;#34;.join(json.loads(sys.argv[1])))&amp;#39; &amp;#34;$crit&amp;#34;)&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;default_critical_options&amp;#34;: $crit,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_user_key_lengths&amp;#34;: {&amp;#34;ed25519&amp;#34;: [0], &amp;#34;sk-ed25519&amp;#34;: [0]},
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;key_id_format&amp;#34;: &amp;#34;{{token_display_name}}-{{role_name}}&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">JSON&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">user_role admin alice 16h permit-pty,permit-port-forwarding
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">user_role device alice 720h permit-pty,permit-port-forwarding
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">user_role ansible ansible 16h permit-pty
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">user_role deploy-websites deploy-websites 5m &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span> &lt;span class="s1">&amp;#39;{&amp;#34;force-command&amp;#34;: &amp;#34;podman auto-update&amp;#34;}&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>What each field does:&lt;/p>
&lt;ul>
&lt;li>&lt;code>allowed_users&lt;/code> and &lt;code>default_user&lt;/code>: the role signs exactly one principal. A caller cannot ask for &lt;code>root&lt;/code>.&lt;/li>
&lt;li>&lt;code>ttl&lt;/code> equal to &lt;code>max_ttl&lt;/code>: a caller cannot ask for a longer validity.&lt;/li>
&lt;li>&lt;code>allowed_extensions&lt;/code>: only these. No agent forwarding, no X11.&lt;/li>
&lt;li>&lt;code>allowed_user_key_lengths&lt;/code>: only Ed25519 keys, also on a FIDO2 token (&lt;code>sk-ed25519&lt;/code>).&lt;/li>
&lt;li>&lt;code>key_id_format&lt;/code>: the key ID in the certificate says who asked, for example &lt;code>oidc-alice-admin&lt;/code>. sshd logs it with every login. This is how I see in a host&amp;rsquo;s log which role signed the certificate that just logged in.&lt;/li>
&lt;/ul>
&lt;h3 id="the-host-role">The host role&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">bao write ssh-host/roles/hosts - &amp;gt;/dev/null &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;JSON&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">{
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;key_type&amp;#34;: &amp;#34;ca&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allow_host_certificates&amp;#34;: true,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_domains&amp;#34;: &amp;#34;example.org,example.net&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allow_subdomains&amp;#34;: true,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;ttl&amp;#34;: &amp;#34;8760h&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;max_ttl&amp;#34;: &amp;#34;8760h&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_user_key_lengths&amp;#34;: {&amp;#34;ed25519&amp;#34;: [0]},
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;key_id_format&amp;#34;: &amp;#34;{{token_display_name}}-host&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">JSON&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Host certificates are valid for one year. The Ansible run renews them when less than 60 days are left (next post).&lt;/p>
&lt;h3 id="policies">Policies&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">bao policy write admin - &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;HCL&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"># Daily use (OIDC login): sign user and host certificates
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;ssh-user/sign/admin&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;ssh-user/sign/device&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;ssh-user/sign/ansible&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;ssh-host/sign/hosts&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">HCL&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao policy write operator - &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;HCL&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"># Running Bao itself (userpass login): everything
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;*&amp;#34; { capabilities = [&amp;#34;create&amp;#34;, &amp;#34;read&amp;#34;, &amp;#34;update&amp;#34;, &amp;#34;patch&amp;#34;, &amp;#34;delete&amp;#34;, &amp;#34;list&amp;#34;, &amp;#34;sudo&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">HCL&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao policy write backup - &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;HCL&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;sys/storage/raft/snapshot&amp;#34; { capabilities = [&amp;#34;read&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">HCL&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The daily login can sign and do nothing else. It cannot read the configuration, change roles or create tokens.&lt;/p>
&lt;h2 id="step-6-logins">Step 6: logins&lt;/h2>
&lt;h3 id="oidc-through-forgejo-for-every-day">OIDC through Forgejo, for every day&lt;/h3>
&lt;p>Forgejo is an OpenID Connect provider. In Forgejo, go to Settings, Applications, and create an OAuth2 application with two redirect URIs:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">http://localhost:8250/oidc/callback
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">https://bao.example.net/ui/vault/auth/oidc/oidc/callback
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The first one is for the CLI. &lt;code>bao login -method=oidc&lt;/code> opens the browser and waits on port 8250 for the answer. The second one is for the web UI. Store the client ID and the secret in pass, then:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">auth oidc oidc
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.dumps({&amp;#34;oidc_discovery_url&amp;#34;: &amp;#34;https://git.example.org&amp;#34;, &amp;#34;oidc_client_id&amp;#34;: sys.argv[1], &amp;#34;oidc_client_secret&amp;#34;: sys.stdin.read().strip(), &amp;#34;default_role&amp;#34;: &amp;#34;alice&amp;#34;}))&amp;#39;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">$(&lt;/span>pass show infra/home1/openbao-oidc-client-id &lt;span class="p">|&lt;/span> head -1&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;lt; &amp;lt;&lt;span class="o">(&lt;/span>pass show infra/home1/openbao-oidc-client-secret &lt;span class="p">|&lt;/span> head -1&lt;span class="o">)&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> bao write auth/oidc/config - &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao write auth/oidc/role/alice - &amp;gt;/dev/null &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;JSON&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">{
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;user_claim&amp;#34;: &amp;#34;preferred_username&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;bound_claims&amp;#34;: {&amp;#34;preferred_username&amp;#34;: &amp;#34;alice&amp;#34;},
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;oidc_scopes&amp;#34;: &amp;#34;openid,profile&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_redirect_uris&amp;#34;: &amp;#34;http://localhost:8250/oidc/callback,https://bao.example.net/ui/vault/auth/oidc/oidc/callback&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_policies&amp;#34;: &amp;#34;admin&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_ttl&amp;#34;: &amp;#34;16h&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_max_ttl&amp;#34;: &amp;#34;16h&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">JSON&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>bound_claims&lt;/code> is important. Without it, every account on the Forgejo instance could log in to Bao and get the &lt;code>admin&lt;/code> policy.&lt;/p>
&lt;p>The secret goes to &lt;code>bao&lt;/code> on stdin, inside the JSON, and never onto a command line.&lt;/p>
&lt;h3 id="userpass-for-running-bao">userpass, for running Bao&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">auth userpass userpass
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> ! bao &lt;span class="nb">read&lt;/span> auth/userpass/users/alice &amp;gt;/dev/null 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>1&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> pass show infra/home1/openbao-alice-password &amp;gt;/dev/null 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">1&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="o">||&lt;/span> pass generate -n infra/home1/openbao-alice-password &lt;span class="m">32&lt;/span> &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> pass show infra/home1/openbao-alice-password &lt;span class="p">|&lt;/span> head -1 &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.dumps({&amp;#34;password&amp;#34;: sys.stdin.read().strip(), &amp;#34;token_policies&amp;#34;: &amp;#34;admin,operator&amp;#34;, &amp;#34;token_ttl&amp;#34;: &amp;#34;1h&amp;#34;, &amp;#34;token_max_ttl&amp;#34;: &amp;#34;4h&amp;#34;}))&amp;#39;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> bao write auth/userpass/users/alice - &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This login has the &lt;code>operator&lt;/code> policy for changes to Bao, with a short token. It is also my way in when Forgejo is down, for example during the reinstall of the host it runs on.&lt;/p>
&lt;h3 id="approle-for-the-backup">AppRole, for the backup&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">auth approle approle
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao write auth/approle/role/backup &lt;span class="nv">token_policies&lt;/span>&lt;span class="o">=&lt;/span>backup &lt;span class="nv">token_ttl&lt;/span>&lt;span class="o">=&lt;/span>5m &lt;span class="nv">token_max_ttl&lt;/span>&lt;span class="o">=&lt;/span>10m &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="nv">secret_id_ttl&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="m">0&lt;/span> &lt;span class="nv">token_num_uses&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="m">0&lt;/span> &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> ! pass show infra/home1/openbao-backup-approle &amp;gt;/dev/null 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>1&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">role_id&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>bao &lt;span class="nb">read&lt;/span> -field&lt;span class="o">=&lt;/span>role_id auth/approle/role/backup/role-id&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> bao write -f -format&lt;span class="o">=&lt;/span>json auth/approle/role/backup/secret-id &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.dumps({&amp;#34;role_id&amp;#34;: sys.argv[1], &amp;#34;secret_id&amp;#34;: json.load(sys.stdin)[&amp;#34;data&amp;#34;][&amp;#34;secret_id&amp;#34;]}))&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$role_id&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> pass insert -m infra/home1/openbao-backup-approle &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Ansible writes this JSON to &lt;code>/root/.config/openbao/backup-approle.json&lt;/code> on the home server (mode 0600). The backup script uses it to log in.&lt;/p>
&lt;h2 id="step-7-backup-and-restore">Step 7: backup and restore&lt;/h2>
&lt;p>Do not back up the live Raft directory. It is a database that is being written to. Take a snapshot through the API instead. That is one more step in the backup script from &lt;a href="../../blog/infra-07-monitoring-backup/">the monitoring and backup post&lt;/a>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># a backup_functions entry for home1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">openbao_snapshot&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> &lt;span class="nv">bao&lt;/span>&lt;span class="o">=(&lt;/span>curl -fsS -m &lt;span class="m">60&lt;/span> --resolve bao.example.net:443:127.0.0.1&lt;span class="o">)&lt;/span> token
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">token&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">bao&lt;/span>&lt;span class="p">[@]&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> -d @/root/.config/openbao/backup-approle.json &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> https://bao.example.net/v1/auth/approle/login &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.load(sys.stdin)[&amp;#34;auth&amp;#34;][&amp;#34;client_token&amp;#34;])&amp;#39;&lt;/span>&lt;span class="k">)&lt;/span> &lt;span class="o">||&lt;/span> &lt;span class="k">return&lt;/span> &lt;span class="m">1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">bao&lt;/span>&lt;span class="p">[@]&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> -H &lt;span class="s2">&amp;#34;X-Vault-Token: &lt;/span>&lt;span class="nv">$token&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> -o &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$DUMP_DIR&lt;/span>&lt;span class="s2">/openbao/raft.snap.tmp&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> https://bao.example.net/v1/sys/storage/raft/snapshot &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> mv &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$DUMP_DIR&lt;/span>&lt;span class="s2">/openbao/raft.snap.tmp&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$DUMP_DIR&lt;/span>&lt;span class="s2">/openbao/raft.snap&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>--resolve ...:127.0.0.1&lt;/code> sends the request to the local Traefik, with the right name for the certificate. The live volume is excluded from restic. While Bao is sealed this step fails, and the backup reports it.&lt;/p>
&lt;p>The snapshot is encrypted with Bao&amp;rsquo;s keys. To restore it on any host with Podman:&lt;/p>
&lt;ol>
&lt;li>Start an empty Bao, initialise it with one share, and unseal it with the new key.&lt;/li>
&lt;li>Log in with the new root token, then run &lt;code>bao operator raft snapshot restore -force raft.snap&lt;/code>.&lt;/li>
&lt;li>Bao is sealed again, now with the old key. Unseal it with the old key from pass.&lt;/li>
&lt;li>Everything is back: both CAs, the roles, the policies, the logins. Revoke the new root token.&lt;/li>
&lt;/ol>
&lt;p>Try this once, on a VM, before you need it.&lt;/p>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">bao login -no-print -method&lt;span class="o">=&lt;/span>oidc &lt;span class="c1"># the browser opens, log in to Forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao token lookup -format&lt;span class="o">=&lt;/span>json &lt;span class="p">|&lt;/span> jq &lt;span class="s1">&amp;#39;.data | {display_name, policies, ttl}&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao &lt;span class="nb">read&lt;/span> -field&lt;span class="o">=&lt;/span>public_key ssh-user/config/ca
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao write -field&lt;span class="o">=&lt;/span>signed_key ssh-user/sign/admin &lt;span class="nv">public_key&lt;/span>&lt;span class="o">=&lt;/span>@&lt;span class="nv">$HOME&lt;/span>/.ssh/id_ed25519.pub &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> ssh-keygen -Lf /dev/stdin
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --user -u openbao -n &lt;span class="m">3&lt;/span> -o cat &lt;span class="p">|&lt;/span> jq -c &lt;span class="s1">&amp;#39;{type, path: .request.path}&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The certificate shows principal &lt;code>alice&lt;/code>, 16 hours, key ID &lt;code>oidc-alice-admin&lt;/code>. The audit log shows the request.&lt;/p>
&lt;p>Next: &lt;a href="../../blog/ssh-03-hosts/">Hosts that trust the SSH CA&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://openbao.org/docs/">OpenBao documentation&lt;/a>: concepts, configuration, auth methods and secrets engines.&lt;/li>
&lt;li>&lt;a href="https://github.com/openbao/openbao">OpenBao on GitHub&lt;/a>: releases and the container images.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/configuration/storage/raft/">Integrated storage (Raft)&lt;/a>: the storage backend, also for a single node.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/concepts/seal/">Seal and unseal&lt;/a>: Shamir keys, auto unseal, and why the barrier exists.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/secrets/ssh/signed-ssh-certificates/">Signed SSH certificates&lt;/a>: the SSH secrets engine, for user and host keys.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/api-docs/secret/ssh/">SSH secrets engine API&lt;/a>: every field of a role: &lt;code>allowed_users&lt;/code>, &lt;code>ttl&lt;/code>, &lt;code>default_extensions&lt;/code> and more.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/auth/jwt/">JWT/OIDC auth method&lt;/a>: the browser login with Forgejo and, in post 4, the CI login.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/auth/jwt/oidc-providers/">OIDC provider configuration&lt;/a>: examples for many identity providers.&lt;/li>
&lt;li>&lt;a href="https://forgejo.org/docs/latest/user/oauth2-provider/">Forgejo as an OAuth2 provider&lt;/a>: the client registration on the Forgejo side.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/auth/userpass/">Userpass auth method&lt;/a>: the fallback login.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/auth/approle/">AppRole auth method&lt;/a>: the login of the backup job.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/concepts/policies/">Policies&lt;/a>: paths, capabilities and how Bao combines policies.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/audit/file/">File audit device&lt;/a>: the log of every request.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/commands/operator/raft/">bao operator raft&lt;/a>: &lt;code>snapshot save&lt;/code> and &lt;code>snapshot restore&lt;/code>.&lt;/li>
&lt;/ul></description></item><item><title>One generic Ansible role for every app</title><link>https://www.linux-lunatix.org/blog/infra-06-quadlet-app/</link><pubDate>Sun, 18 Oct 2026 16:30:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-06-quadlet-app/</guid><description>&lt;p>&lt;em>Part 6 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>My hosts run close to 30 apps, some of them with several containers. I did not want 30 Ansible roles. This post shows the one role that deploys all of them, and how I used it to take over running hosts without restarting anything.&lt;/p>
&lt;h2 id="the-idea">The idea&lt;/h2>
&lt;p>Each app is a directory:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">apps/forgejo/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── app.yml # optional: secrets, directories, units
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── quadlets/ # -&amp;gt; ~svc/.config/containers/systemd/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo.container
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo-postgres.container
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo.network
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ ├── forgejo-data.volume
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── forgejo-db.volume
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">├── files/ # -&amp;gt; ~svc/ (same relative paths)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">│ └── .config/forgejo/...
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">└── tasks.yml # optional: extra tasks (downloads, checkouts)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A host lists its apps:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/vps1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">app_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">svc&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">apps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">traefik, ntfy, forgejo-runner]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The role &lt;code>quadlet_app&lt;/code> loops over the list. For each app it does the same steps:&lt;/p>
&lt;ol>
&lt;li>Read &lt;code>app.yml&lt;/code>.&lt;/li>
&lt;li>Create the Podman secrets it names, from pass.&lt;/li>
&lt;li>Copy &lt;code>files/&lt;/code> and &lt;code>quadlets/&lt;/code> into the app user&amp;rsquo;s home. Files that end in &lt;code>.j2&lt;/code> are templates.&lt;/li>
&lt;li>If anything changed: run &lt;code>daemon-reload&lt;/code> and restart the app&amp;rsquo;s units.&lt;/li>
&lt;li>Make sure the units run.&lt;/li>
&lt;/ol>
&lt;p>An app that did not change is not touched. That last point matters most. A playbook run on a host with 20 apps restarts exactly the apps I changed.&lt;/p>
&lt;h2 id="why-plain-files-and-not-a-module">Why plain files and not a module&lt;/h2>
&lt;p>&lt;code>containers.podman.podman_container&lt;/code> can create containers from Ansible, and newer versions can write quadlets too. I chose plain quadlet files in the repository instead:&lt;/p>
&lt;ul>
&lt;li>They are the same files that are on the host. &lt;code>diff&lt;/code> shows exactly what changes.&lt;/li>
&lt;li>I can copy one to a test machine and run it without Ansible.&lt;/li>
&lt;li>The Podman documentation for quadlets applies 1:1. There is no translation layer whose options I must look up.&lt;/li>
&lt;/ul>
&lt;h2 id="app-dot-yml">app.yml&lt;/h2>
&lt;p>Most apps need no &lt;code>app.yml&lt;/code> at all. When one is there, it can say:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># apps/forgejo/app.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">secrets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">name: forgejo-db-password, pass&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">infra/&amp;lt;host&amp;gt;/forgejo-db-password}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># apps/traefik/app.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">secrets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">name: dns-api-token, pass: infra/&amp;lt;host&amp;gt;/dns-api-token, if&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">traefik_dns_challenge}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">dirs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">path: .config/traefik, mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0700&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">traefik}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">path: traefik/acme, mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0700&amp;#34;&lt;/span>}&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Key&lt;/th>
&lt;th>Meaning&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>secrets&lt;/code>&lt;/td>
&lt;td>Podman secrets: name, pass entry (&lt;code>&amp;lt;host&amp;gt;&lt;/code> is replaced), optional &lt;code>if&lt;/code> (a host variable that must be true)&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>secret_files&lt;/code>&lt;/td>
&lt;td>files from &lt;code>files/&lt;/code> with mode 0600 that never show up in a diff&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>dirs&lt;/code>&lt;/td>
&lt;td>directories to create, with an optional mode&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>units&lt;/code>&lt;/td>
&lt;td>the units to restart; default: one &lt;code>.service&lt;/code> per &lt;code>.container&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>enable&lt;/code>&lt;/td>
&lt;td>plain user units or timers to enable and start&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>restart&lt;/code>&lt;/td>
&lt;td>&lt;code>restart&lt;/code>, or &lt;code>stop-start&lt;/code> for containers that share a network namespace&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;h2 id="the-role-step-by-step">The role, step by step&lt;/h2>
&lt;p>The role&amp;rsquo;s &lt;code>main.yml&lt;/code> creates the quadlet directory once, then includes &lt;code>app.yml&lt;/code> for each app:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># roles/quadlet_app/tasks/main.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Quadlet directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_home }}/{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">owner&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">group&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">.config, .config/containers, .config/containers/systemd]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Apps&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.include_tasks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ apps }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_var&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_name&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>app_home&lt;/code>, &lt;code>app_uid&lt;/code> and &lt;code>app_gid&lt;/code> come from the &lt;code>podman&lt;/code> role, which creates the user, enables linger and starts the user&amp;rsquo;s &lt;code>podman.socket&lt;/code> and &lt;code>podman-auto-update.timer&lt;/code>.&lt;/p>
&lt;h3 id="read-the-definition-and-the-source-tree">Read the definition and the source tree&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: definition&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.set_fact&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_defaults | combine(app_file | from_yaml or {}) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ apps_dir }}/{{ app_name }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vars&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;ansible.builtin.file&amp;#39;, apps_dir ~ &amp;#39;/&amp;#39; ~ app_name ~ &amp;#39;/app.yml&amp;#39;, errors=&amp;#39;ignore&amp;#39;) | default(&amp;#39;&amp;#39;, true) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_defaults&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>{&lt;span class="nt">secrets: [], secret_files: [], dirs: [], units: null, enable: [], restart&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">restart}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: source tree&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.set_fact&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_tree&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">files&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.filetree&amp;#39;, app_src ~ &amp;#39;/files&amp;#39;, wantlist=true) | selectattr(&amp;#39;state&amp;#39;, &amp;#39;eq&amp;#39;, &amp;#39;file&amp;#39;) | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dirs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.filetree&amp;#39;, app_src ~ &amp;#39;/files&amp;#39;, wantlist=true) | selectattr(&amp;#39;state&amp;#39;, &amp;#39;eq&amp;#39;, &amp;#39;directory&amp;#39;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> | map(attribute=&amp;#39;path&amp;#39;) | map(&amp;#39;community.general.dict_kv&amp;#39;, &amp;#39;path&amp;#39;) | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">quadlets&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.filetree&amp;#39;, app_src ~ &amp;#39;/quadlets&amp;#39;, wantlist=true) | selectattr(&amp;#39;state&amp;#39;, &amp;#39;eq&amp;#39;, &amp;#39;file&amp;#39;) | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>filetree&lt;/code> lookup walks a directory on the control node and returns every file with its relative path and mode. That is all the role needs to know about an app.&lt;/p>
&lt;h3 id="secrets">Secrets&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: podman secrets&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">containers.podman.podman_secret&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.name }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">data&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ lookup(&amp;#39;community.general.passwordstore&amp;#39;, item.pass | replace(&amp;#39;&amp;lt;host&amp;gt;&amp;#39;, inventory_hostname)) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">present&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app.secrets }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.name }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">item.if is not defined or lookup(&amp;#39;ansible.builtin.vars&amp;#39;, item.if) | bool&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">become_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">XDG_RUNTIME_DIR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/run/user/{{ app_uid }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">no_log&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_secrets_result&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>become_user&lt;/code> plus &lt;code>XDG_RUNTIME_DIR&lt;/code> is how every task in this role talks to the rootless user&amp;rsquo;s Podman and systemd. &lt;code>become_user&lt;/code> to an unprivileged user needs the &lt;code>acl&lt;/code> package on the host, so that Ansible can hand its temporary files to that user. The base role installs it.&lt;/p>
&lt;h3 id="files-and-quadlets">Files and quadlets&lt;/h3>
&lt;p>Plain files are copied, and &lt;code>.j2&lt;/code> files are rendered. Executables stay executable. Secret files get mode 0600 and no diff:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: files&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.src }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_home }}/{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">owner&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">group&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ &amp;#39;0600&amp;#39; if item.path in app.secret_files else (&amp;#39;0755&amp;#39; if item.mode[1] in &amp;#39;157&amp;#39; else &amp;#39;0644&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_tree.files | rejectattr(&amp;#39;path&amp;#39;, &amp;#39;search&amp;#39;, &amp;#39;\\.j2$&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">no_log&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path in app.secret_files }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_files_result&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: templated files&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.template&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.src }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_home }}/{{ item.path | regex_replace(&amp;#39;\\.j2$&amp;#39;, &amp;#39;&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">owner&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">group&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_tree.files | selectattr(&amp;#39;path&amp;#39;, &amp;#39;search&amp;#39;, &amp;#39;\\.j2$&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_templates_result&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The quadlets work the same way, with &lt;code>.config/containers/systemd/&lt;/code> as the destination. Why are only &lt;code>.j2&lt;/code> files templates? Some apps ship files that contain their own &lt;code>{{ }}&lt;/code> syntax: Grafana dashboards, Prometheus alert rules. Rendering those with Jinja would break them. So a file is a template only when its name says so.&lt;/p>
&lt;p>A quadlet template is useful for values that differ per host:&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2"># apps/traefik/quadlets/traefik.container.j2 (part)
UserNS=keep-id:uid={{ app_uid }},gid={{ app_gid }}
User={{ app_uid }}:{{ app_gid }}
{% for v in traefik_extra_volumes | default([]) %}
Volume={{ v }}
{% endfor %}
{% if traefik_dns_challenge %}
Secret=dns-api-token,type=env,target=CF_DNS_API_TOKEN
{% endif %}
&lt;/code>&lt;/pre>&lt;h3 id="restart-only-what-changed">Restart only what changed&lt;/h3>
&lt;p>Every task above registers its result. The role collects all of them and asks one question: did anything change for this app?&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: units&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.set_fact&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_units&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app.units if app.units is not none else
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> (app_tree.quadlets | map(attribute=&amp;#39;path&amp;#39;) | map(&amp;#39;regex_replace&amp;#39;, &amp;#39;\\.j2$&amp;#39;, &amp;#39;&amp;#39;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> | select(&amp;#39;match&amp;#39;, &amp;#39;.*\\.container$&amp;#39;) | map(&amp;#39;regex_replace&amp;#39;, &amp;#39;\\.container$&amp;#39;, &amp;#39;.service&amp;#39;) | list) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">app_changed&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ (app_secrets_result.results + app_files_result.results + app_templates_result.results
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> + app_quadlets_result.results + app_quadlet_templates_result.results
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s2"> + (app_extra_results | default([]))) | selectattr(&amp;#39;changed&amp;#39;, &amp;#39;defined&amp;#39;) | selectattr(&amp;#39;changed&amp;#39;) | list | length &amp;gt; 0 }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: reload and restart&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app_changed&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">become_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">XDG_RUNTIME_DIR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/run/user/{{ app_uid }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">block&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: daemon-reload&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">daemon_reload&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: restart&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">restarted&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.restart == &amp;#39;restart&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># Containers that share another&amp;#39;s network namespace must be gone before it can be&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># replaced: stop in reverse order, then start in order&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: stop (reverse order)&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">stopped&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units | reverse | list }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.restart == &amp;#39;stop-start&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: start&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">started&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">app.restart == &amp;#39;stop-start&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: running&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.systemd_service&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scope&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">user&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">started&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_units }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">become_user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_user }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">XDG_RUNTIME_DIR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/run/user/{{ app_uid }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Why not handlers? Handlers run at the end of the play, and they run once per handler name, however many apps notified it. I want the restart right after the app&amp;rsquo;s own files changed, and only for that app&amp;rsquo;s units. Handlers cannot carry &amp;ldquo;which app&amp;rdquo; without tricks. A plain &lt;code>when: app_changed&lt;/code> is easier to read.&lt;/p>
&lt;p>Restarting is also how a container sees a changed config file. Ansible writes a new file (a new inode), and a running container keeps the old one in its mount.&lt;/p>
&lt;h3 id="extra-tasks">Extra tasks&lt;/h3>
&lt;p>Some apps need more than files: a Git checkout to build from, a binary to download. An optional &lt;code>tasks.yml&lt;/code> in the app directory is included, and it can add its results to &lt;code>app_extra_results&lt;/code> to trigger the restart:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: extra tasks&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.include_tasks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_src }}/tasks.yml&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">(app_src ~ &amp;#39;/tasks.yml&amp;#39;) is file&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="order-matters">Order matters&lt;/h2>
&lt;p>Apps run in the order of the &lt;code>apps&lt;/code> list. If one app provides a file that another app mounts, the provider must come first. On a fresh host, Traefik fails to start if a password file it mounts does not exist yet:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># the landing page app provides the htpasswd file that Traefik mounts&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">apps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">landing, traefik, ntfy, forgejo-runner]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">traefik_extra_volumes&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;%h/.config/traefik/landing_htpasswd:/landing_htpasswd:ro,Z&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="adopting-a-running-host">Adopting a running host&lt;/h2>
&lt;p>This is how I moved the existing quadlets of a running host into the role without a single restart:&lt;/p>
&lt;ol>
&lt;li>Copy the host&amp;rsquo;s quadlets and config files into &lt;code>apps/&amp;lt;name&amp;gt;/&lt;/code>, one app at a time.&lt;/li>
&lt;li>Run check mode with a diff against the live host:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l home1 -t apps --check --diff
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>Every difference is one of two things. Either the repository is wrong (fix the file), or the host has a change that was never written down (decide which version is right).&lt;/li>
&lt;li>Repeat until the run shows &lt;code>changed=0&lt;/code> in check mode.&lt;/li>
&lt;li>Run it for real. Nothing changes, so nothing restarts.&lt;/li>
&lt;/ol>
&lt;p>The loop also shows what nobody wrote down: an option changed by hand on the host, a file that exists only there. Each of those is now either in the repository or gone.&lt;/p>
&lt;h2 id="the-test-changed-0">The test: changed=0&lt;/h2>
&lt;p>After every change I run the playbook twice. The second run must report &lt;code>changed=0&lt;/code>. A task that reports a change every time hides the real changes, and it may restart an app on every run. Typical causes:&lt;/p>
&lt;ul>
&lt;li>&lt;code>command&lt;/code> or &lt;code>shell&lt;/code> tasks without &lt;code>changed_when&lt;/code>.&lt;/li>
&lt;li>&lt;code>file: state=touch&lt;/code> without &lt;code>modification_time: preserve&lt;/code>.&lt;/li>
&lt;li>A template that renders a timestamp or a random value.&lt;/li>
&lt;li>A module that compares in a different format than it writes (for example YAML key order).&lt;/li>
&lt;/ul>
&lt;p>Run it with fresh SSH connections, too (&lt;code>-o ControlPath=none&lt;/code>). The reason is in &lt;a href="../../blog/infra-05-ansible-base/">the previous post&lt;/a>.&lt;/p>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t apps
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t apps &lt;span class="c1"># changed=0&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh vps1 sudo -u svc &lt;span class="nv">XDG_RUNTIME_DIR&lt;/span>&lt;span class="o">=&lt;/span>/run/user/1001 systemctl --user list-units &lt;span class="s1">&amp;#39;traefik*&amp;#39;&lt;/span> &lt;span class="s1">&amp;#39;ntfy*&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Next: &lt;a href="../../blog/infra-07-monitoring-backup/">Monitoring and backups for every host&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_reuse_roles.html">Ansible roles&lt;/a>: the role layout, defaults and dependencies.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_handlers.html">Handlers&lt;/a>: notify, listen and &lt;code>flush_handlers&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_checkmode.html">Check mode and diff mode&lt;/a>: what &lt;code>--check --diff&lt;/code> can and cannot predict.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/collections/containers/podman/index.html">containers.podman collection&lt;/a>: the modules for secrets, images and containers. The alternative to plain files.&lt;/li>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html">podman-systemd.unit(5)&lt;/a>: the quadlet reference again, because the role only copies these files.&lt;/li>
&lt;li>&lt;a href="https://redhat-cop.github.io/automation-good-practices/">Good practices for Ansible (Red Hat CoP)&lt;/a>: the chapter on roles fits this post.&lt;/li>
&lt;/ul></description></item><item><title>SELinux enforcing with containers</title><link>https://www.linux-lunatix.org/blog/infra-04-selinux/</link><pubDate>Fri, 09 Oct 2026 21:10:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-04-selinux/</guid><description>&lt;p>&lt;em>Part 4 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>Every one of my hosts runs SELinux in enforcing mode. Every container runs in the confined type &lt;code>container_t&lt;/code>, with no exceptions. This post shows the few tools that make this practical: volume labels, file contexts, booleans and small policy modules. It also shows how to debug a denial.&lt;/p>
&lt;h2 id="why-bother">Why bother&lt;/h2>
&lt;p>A container that breaks out of its namespaces lands in a process on the host. With rootless Podman that process runs as an unprivileged user. With SELinux it is also confined to &lt;code>container_t&lt;/code>. That type may read and write only files labelled &lt;code>container_file_t&lt;/code>, and each container gets its own MCS category pair (for example &lt;code>s0:c123,c456&lt;/code>). So one container cannot read another container&amp;rsquo;s files, even when both belong to the same user.&lt;/p>
&lt;p>The cost is small once you know the patterns. Most of the time, the right volume flag is all you need.&lt;/p>
&lt;h2 id="volume-labels-z-and-z">Volume labels: &lt;code>:Z&lt;/code> and &lt;code>:z&lt;/code>&lt;/h2>
&lt;p>When you mount a host directory into a container, its files usually have a type such as &lt;code>user_home_t&lt;/code>. A confined container may not read that. Podman can relabel the files at start:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Flag&lt;/th>
&lt;th>Label&lt;/th>
&lt;th>Use for&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>:Z&lt;/code>&lt;/td>
&lt;td>&lt;code>container_file_t&lt;/code> with this container&amp;rsquo;s private categories&lt;/td>
&lt;td>data of one container&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>:z&lt;/code>&lt;/td>
&lt;td>&lt;code>container_file_t&lt;/code> without categories (shared)&lt;/td>
&lt;td>files that several containers mount&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Only this container uses it&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/ntfy/server.yml:/etc/ntfy/server.yml:ro,Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">ntfy-data:/var/lib/ntfy&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Three website containers mount the same file&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/nginx/realip.conf:/etc/nginx/conf.d/realip.conf:ro,z&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Named Podman volumes get the right label when Podman creates them. The &lt;code>:Z&lt;/code> on a volume does no harm and makes the intent clear.&lt;/p>
&lt;p>Two warnings:&lt;/p>
&lt;ul>
&lt;li>Never use &lt;code>:Z&lt;/code> on a system directory such as &lt;code>/etc&lt;/code> or &lt;code>/home&lt;/code>. Podman relabels everything below it, and the host then cannot use those files any more.&lt;/li>
&lt;li>If two containers mount the same file and both use &lt;code>:Z&lt;/code>, the second start relabels it for itself. The first container then gets &amp;ldquo;permission denied&amp;rdquo;. Use &lt;code>:z&lt;/code> for shared files.&lt;/li>
&lt;/ul>
&lt;h2 id="directories-outside-the-home-file-contexts">Directories outside the home: file contexts&lt;/h2>
&lt;p>Some files are not mounted from a home directory. For example, node_exporter reads text files with metrics from &lt;code>/var/lib/node_exporter/textfile&lt;/code>, and my backup script writes them. Instead of relabelling the directory at every start, make the label permanent with a file context rule:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo semanage fcontext -a -t container_file_t &lt;span class="s1">&amp;#39;/var/lib/node_exporter(/.*)?&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo restorecon -Rv /var/lib/node_exporter
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>restorecon&lt;/code> applies the rule now. The rule also survives a full relabel of the filesystem. With Ansible this becomes two tasks:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SELinux label for the textfile directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">community.general.sefcontext&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">target&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/var/lib/node_exporter(/.*)?&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">setype&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">container_file_t&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">present&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">textfile_fcontext&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Relabel the textfile directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">restorecon -R /var/lib/node_exporter&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">textfile_fcontext.changed and not ansible_check_mode&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="booleans">Booleans&lt;/h2>
&lt;p>The policy has switches for common cases. List them with &lt;code>getsebool -a | grep container&lt;/code>. One example: &lt;code>container_use_devices&lt;/code> lets containers use host devices. I keep the booleans per host in the inventory:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/&amp;lt;host&amp;gt;.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">selinux_booleans&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">container_use_devices&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SELinux booleans&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.seboolean&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.key }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.value }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">persistent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ selinux_booleans | dict2items }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Prefer a boolean over a policy module when one exists. Somebody else maintains it, and it is easy to see what it allows.&lt;/p>
&lt;h2 id="small-policy-modules-for-the-rest">Small policy modules for the rest&lt;/h2>
&lt;p>Two of my cases have no boolean.&lt;/p>
&lt;h3 id="traefik-and-the-podman-socket">Traefik and the Podman socket&lt;/h3>
&lt;p>Traefik reads container labels through the rootless Podman socket. The socket file can be relabelled with &lt;code>:z&lt;/code>. But connecting to the socket means talking to the process behind it, and that process runs as &lt;code>container_runtime_t&lt;/code>. &lt;code>container_t&lt;/code> may not do that. The denial in the audit log looks like this:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">type=AVC msg=audit(...): avc: denied { connectto } for pid=... comm=&amp;#34;traefik&amp;#34;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> path=&amp;#34;/run/user/1001/podman/podman.sock&amp;#34; scontext=system_u:system_r:container_t:s0:c12,c345
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> tcontext=unconfined_u:unconfined_r:container_runtime_t:s0 tclass=unix_stream_socket permissive=0
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The fix is one rule:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">module traefik_podman_socket 1.0;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">require {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> type container_t;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> type container_runtime_t;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> class unix_stream_socket connectto;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">}
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">allow container_t container_runtime_t:unix_stream_socket connectto;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Build and load it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">checkmodule -M -m -o traefik_podman_socket.mod traefik_podman_socket.te
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">semodule_package -o traefik_podman_socket.pp -m traefik_podman_socket.mod
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo semodule -i traefik_podman_socket.pp
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This rule applies to every &lt;code>container_t&lt;/code> process, not only to Traefik. A container still needs the socket mounted to use it, and only Traefik and my Podman exporter get that mount. Think about this trade-off before you load such a rule. Access to the Podman socket is close to access to the user&amp;rsquo;s account.&lt;/p>
&lt;p>A VPN container (gluetun) needs the same kind of module for &lt;code>/dev/net/tun&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">allow container_t tun_tap_device_t:chr_file { getattr ioctl open read write };
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="node-exporter-a-type-of-its-own">node_exporter: a type of its own&lt;/h3>
&lt;p>node_exporter reads the host&amp;rsquo;s mount table and asks systemd about unit states. Most guides run it with &lt;code>SecurityLabelDisable=true&lt;/code> (unconfined), and I did that too at first. I replaced it with a confined type of its own, written in CIL. The base is the container template from &lt;code>udica&lt;/code> (it ships with &lt;code>container-selinux&lt;/code>), plus the few extra rules that node_exporter needs:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-lisp" data-lang="lisp">&lt;span class="line">&lt;span class="cl">&lt;span class="p">(&lt;/span>&lt;span class="k">block&lt;/span> &lt;span class="nv">node_exporter&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">blockinherit&lt;/span> &lt;span class="nv">container&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">blockinherit&lt;/span> &lt;span class="nv">net_container&lt;/span>&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="c1">; filesystem collector: init&amp;#39;s mount table, statfs of every mount point&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dir&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">search&lt;/span> &lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">file&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">open&lt;/span> &lt;span class="nf">read&lt;/span> &lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">lnk_file&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">read&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">filesystem_type&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">filesystem&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">mountpoint&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dir&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">search&lt;/span> &lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="c1">; selinux collector: current mode&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">security_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dir&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">search&lt;/span> &lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">security_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">file&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">open&lt;/span> &lt;span class="nf">read&lt;/span> &lt;span class="nv">getattr&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="c1">; systemd collector: read-only queries to systemd over the system bus&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">system_dbusd_var_run_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dir&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">search&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">system_dbusd_var_run_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">sock_file&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nf">write&lt;/span> &lt;span class="nv">getattr&lt;/span> &lt;span class="nf">open&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">system_dbusd_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">unix_stream_socket&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">connectto&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">system_dbusd_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dbus&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">send_msg&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">system_dbusd_t&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dbus&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">send_msg&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dbus&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">send_msg&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">dbus&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">send_msg&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">init_t&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">system&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">status&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="p">(&lt;/span>&lt;span class="nv">allow&lt;/span> &lt;span class="nv">process&lt;/span> &lt;span class="nv">systemd_unit_file_type&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">service&lt;/span> &lt;span class="p">(&lt;/span>&lt;span class="nv">status&lt;/span>&lt;span class="p">)))&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">)&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">t&lt;/span>&lt;span class="o">=&lt;/span>/usr/share/udica/templates
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo semodule -i node_exporter.cil &lt;span class="nv">$t&lt;/span>/base_container.cil &lt;span class="nv">$t&lt;/span>/net_container.cil
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The quadlet then uses the new type:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">quay.io/prometheus/node-exporter:latest&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">host&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PodmanArgs&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">--pid=host&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">/:/host:ro,rslave&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket:ro&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">SecurityLabelType&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">node_exporter.process&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="debugging-a-denial">Debugging a denial&lt;/h2>
&lt;p>When a container fails with &amp;ldquo;permission denied&amp;rdquo; and the file permissions look fine, look at the audit log:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo ausearch -m avc -ts recent &lt;span class="c1"># the last 10 minutes&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo ausearch -m avc -ts boot &lt;span class="p">|&lt;/span> audit2why &lt;span class="c1"># with an explanation&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Some denials are hidden by &lt;code>dontaudit&lt;/code> rules. If &lt;code>ausearch&lt;/code> shows nothing but the problem stays, switch those rules off for a moment:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo semodule -DB &lt;span class="c1"># disable dontaudit, rebuild&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># reproduce the problem, run ausearch again&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo semodule -B &lt;span class="c1"># enable dontaudit again&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>audit2allow&lt;/code> turns denials into rules:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo ausearch -m avc -ts recent &lt;span class="p">|&lt;/span> audit2allow -m mymodule
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Read its output, but do not load it as it is. It allows exactly what was denied, for the whole source type. A container that tried to read &lt;code>/etc/shadow&lt;/code> gets a rule that allows &lt;code>container_t&lt;/code> to read &lt;code>shadow_t&lt;/code>, for every container on the host. Often the right fix is something else: a &lt;code>:Z&lt;/code> on the mount, a file context or a boolean. Write a module only when none of those fit. Keep it to the rules you understand.&lt;/p>
&lt;h2 id="make-it-code">Make it code&lt;/h2>
&lt;p>I keep the module sources in the repository (&lt;code>selinux/*.te&lt;/code>, &lt;code>selinux/*.cil&lt;/code>), never just the compiled &lt;code>.pp&lt;/code>. If you have an old module and lost its source, you can extract it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo semodule -c -E gluetun_policy &lt;span class="c1"># writes gluetun_policy.cil&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>My base Ansible role copies each source to &lt;code>/usr/local/share/selinux/&lt;/code>. It builds and loads a module when the source changed, or when the module is missing:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SELinux module sources&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ playbook_dir }}/../selinux/{{ item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/usr/local/share/selinux/{{ item }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ selinux_modules }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodule_src&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Installed SELinux modules&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodule -l&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodules&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">check_mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Build and load SELinux modules&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.shell&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> set -e
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> cd /usr/local/share/selinux
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> name={{ item.item | regex_replace(&amp;#39;\.(te|cil)$&amp;#39;, &amp;#39;&amp;#39;) }}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> case {{ item.item }} in
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> *.te)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> checkmodule -M -m -o &amp;#34;$name.mod&amp;#34; {{ item.item }}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> semodule_package -o &amp;#34;$name.pp&amp;#34; -m &amp;#34;$name.mod&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> semodule -i &amp;#34;$name.pp&amp;#34; ;;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> *.cil)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> t=/usr/share/udica/templates
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> semodule -i {{ item.item }} $t/base_container.cil $t/net_container.cil ;;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> esac&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ semodule_src.results }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.item }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">item.changed or (item.item | regex_replace(&amp;#39;\.(te|cil)$&amp;#39;, &amp;#39;&amp;#39;)) not in semodules.stdout_lines&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="guard-rails">Guard rails&lt;/h2>
&lt;p>Two checks keep me honest.&lt;/p>
&lt;p>The base role stops the play if a host is not enforcing, and also if somebody made a domain permissive with &lt;code>semanage permissive -a&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SELinux is enforcing (running and configured)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.assert&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">that&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">ansible_facts.selinux.status == &amp;#39;enabled&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">ansible_facts.selinux.mode == &amp;#39;enforcing&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">ansible_facts.selinux.config_mode == &amp;#39;enforcing&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fail_msg&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;SELinux on {{ inventory_hostname }}: {{ ansible_facts.selinux }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># semanage permissive -a stores a local permissive domain as module permissive_&amp;lt;type&amp;gt;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">No&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">local permissive domains&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodule -l&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodule_list&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">check_mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">failed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">semodule_list.stdout_lines | select(&amp;#39;match&amp;#39;, &amp;#39;^permissive_&amp;#39;) | list | length &amp;gt; 0&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The role does not switch enforcing on by itself. A host that is not enforcing needs a look first (a relabel, old denials), so the play stops.&lt;/p>
&lt;p>The app role refuses quadlets that disable confinement:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ app_name }}: containers stay confined&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.assert&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">that&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">lookup(&amp;#39;ansible.builtin.file&amp;#39;, item.src) is not search(unconfined)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fail_msg&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }} disables SELinux confinement; use a policy module in selinux/ instead&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ quadlet_files }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vars&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">unconfined&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s1">&amp;#39;(?mi)^\s*(SecurityLabelDisable\s*=\s*true|SecurityLabelType\s*=\s*(spc_t|unconfined_t)|PodmanArgs\s*=.*(--privileged|label[=:]disable))&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>With this check, &amp;ldquo;just disable SELinux for this one container&amp;rdquo; is no longer an option.&lt;/p>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">getenforce &lt;span class="c1"># Enforcing&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ps -eZ &lt;span class="p">|&lt;/span> grep -E &lt;span class="s1">&amp;#39;container_t|node_exporter&amp;#39;&lt;/span> &lt;span class="c1"># every container process confined&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo ausearch -m avc -ts boot &lt;span class="c1"># ideally: &amp;lt;no matches&amp;gt;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="things-to-watch">Things to watch&lt;/h2>
&lt;ul>
&lt;li>Restores. &lt;code>restic&lt;/code> restores the SELinux label as an extended attribute when it was in the backup. A backup from a host without SELinux has no label, and &lt;code>restic&lt;/code> reports errors when it tries to set one. Restore with &lt;code>--exclude-xattr security.selinux&lt;/code>, then run &lt;code>restorecon&lt;/code> or let &lt;code>:Z&lt;/code> relabel at start.&lt;/li>
&lt;li>Docker. My one Docker host (Mailcow) runs with SELinux too. Fedora&amp;rsquo;s Docker package already starts &lt;code>dockerd&lt;/code> with &lt;code>--selinux-enabled&lt;/code>, so do not add it to &lt;code>daemon.json&lt;/code> as well. The daemon refuses to start when an option comes from both places.&lt;/li>
&lt;/ul>
&lt;p>Next: &lt;a href="../../blog/infra-05-ansible-base/">Ansible groundwork: inventory, bootstrap and a base role&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.fedoraproject.org/en-US/quick-docs/selinux-getting-started/">Getting started with SELinux (Fedora Quick Docs)&lt;/a>: the basics, if this post went too fast.&lt;/li>
&lt;li>&lt;a href="https://github.com/containers/container-selinux">container-selinux&lt;/a>: the policy behind &lt;code>container_t&lt;/code> and &lt;code>container_file_t&lt;/code>, and its booleans.&lt;/li>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-run.1.html">podman-run(1)&lt;/a>: the &lt;code>:z&lt;/code> and &lt;code>:Z&lt;/code> volume options and the label settings of &lt;code>--security-opt&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://github.com/containers/udica">udica&lt;/a>: generates SELinux policies for containers. The source of the templates in this post.&lt;/li>
&lt;li>&lt;a href="https://github.com/SELinuxProject/selinux-notebook">The SELinux Notebook&lt;/a>: the deep reference: types, CIL, modules and how the kernel decides.&lt;/li>
&lt;/ul></description></item><item><title>Traefik in front of rootless containers</title><link>https://www.linux-lunatix.org/blog/infra-03-traefik/</link><pubDate>Mon, 05 Oct 2026 19:05:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-03-traefik/</guid><description>&lt;p>&lt;em>Part 3 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>At the end of this post Traefik runs as a rootless quadlet. It gets Let&amp;rsquo;s Encrypt certificates and finds the apps by their container labels, as it did with Docker. You will also know which network mode to choose, and why.&lt;/p>
&lt;h2 id="the-problem-with-rootless-networking">The problem with rootless networking&lt;/h2>
&lt;p>With Docker, Traefik sits on a bridge network with the apps, and the root daemon forwards ports 80 and 443 to it. Traefik sees the real client address.&lt;/p>
&lt;p>Rootless Podman cannot create real bridges in the host&amp;rsquo;s network namespace. It runs the container network in a user namespace. A helper process (&lt;code>rootlessport&lt;/code>, or &lt;code>pasta&lt;/code>) accepts connections on the host&amp;rsquo;s port and forwards them into the namespace. On the way the client address gets lost: Traefik then sees every request as coming from the helper. Access logs, rate limits, IP allow lists and fail2ban on the proxy then do not work.&lt;/p>
&lt;p>There are two ways to deal with this:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Mode&lt;/th>
&lt;th>Client IP&lt;/th>
&lt;th>Apps reached at&lt;/th>
&lt;th>Use it when&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>Traefik on the host network&lt;/td>
&lt;td>real&lt;/td>
&lt;td>&lt;code>127.0.0.1:&amp;lt;port&amp;gt;&lt;/code> published by each app&lt;/td>
&lt;td>the host is on the internet&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>Traefik on a Podman bridge&lt;/td>
&lt;td>lost&lt;/td>
&lt;td>the container IP on the bridge&lt;/td>
&lt;td>only you and your LAN use it&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>I use both. My two internet-facing VPS run Traefik on the host network. My home server is reachable only from the LAN and through WireGuard, so the client addresses there are my own devices anyway. That host keeps the bridge network, which was there first and needs fewer changes.&lt;/p>
&lt;h2 id="option-a-traefik-on-the-host-network">Option A: Traefik on the host network&lt;/h2>
&lt;h3 id="the-quadlet">The quadlet&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ~svc/.config/containers/systemd/traefik.container&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Unit]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Description&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">Traefik reverse proxy&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">After&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">network-online.target&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">docker.io/library/traefik:v3&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Host network namespace: real client IPs, binds 80/443 directly&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">host&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Run as svc itself, so the files it writes belong to svc on the host&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">UserNS&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">keep-id:uid=1001,gid=1001&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">User&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">1001:1001&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/traefik/acme:/acme:Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/traefik/traefik.yml:/etc/traefik/traefik.yml:ro,Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/traefik/dynamic:/dynamic:ro,Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># The rootless Podman socket, read-only, as Traefik&amp;#39;s &amp;#34;docker&amp;#34; provider&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%t/podman/podman.sock:/var/run/docker.sock:ro,z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Secret&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">dns-api-token,type=env,target=CF_DNS_API_TOKEN&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Replace &lt;code>1001&lt;/code> with the UID and GID of your app user (&lt;code>id svc&lt;/code>). Some notes:&lt;/p>
&lt;ul>
&lt;li>&lt;code>Network=host&lt;/code> needs &lt;code>net.ipv4.ip_unprivileged_port_start=80&lt;/code> (see &lt;a href="../../blog/infra-02-quadlets/">the previous post&lt;/a>). Without it Traefik cannot bind 80 and 443.&lt;/li>
&lt;li>&lt;code>%h&lt;/code> is the user&amp;rsquo;s home directory and &lt;code>%t&lt;/code> is &lt;code>$XDG_RUNTIME_DIR&lt;/code>. These are systemd specifiers, and quadlets expand them.&lt;/li>
&lt;li>The Podman socket must be enabled for the user: &lt;code>systemctl --user enable --now podman.socket&lt;/code>.&lt;/li>
&lt;li>The socket mount needs an SELinux module, so that a confined container may connect to it. &lt;a href="../../blog/infra-04-selinux/">Post 4&lt;/a> shows it. Without the module Traefik logs &amp;ldquo;permission denied&amp;rdquo; on the socket.&lt;/li>
&lt;/ul>
&lt;h3 id="the-static-configuration">The static configuration&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># ~svc/.config/traefik/traefik.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">entryPoints&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">http&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">address&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;:80&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">http&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">redirections&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">entryPoint&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">to&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">https&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">scheme&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">https&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">https&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">address&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;:443&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># On the host network, ClientHost in this log is the real client address.&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># Written to stdout, so it ends up in the journal.&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">accessLog&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">format&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">json&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fields&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">headers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">defaultMode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">drop&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">names&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">User-Agent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">keep&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">Referer&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">keep&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">providers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">docker&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">endpoint&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;unix:///var/run/docker.sock&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">exposedByDefault&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># Apps publish their port on 127.0.0.1. Route to that address, not to the&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># container IP, which the host network namespace cannot reach.&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">useBindPortIP&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">directory&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/dynamic&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">watch&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">certificatesResolvers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">default&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">acme&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">email&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;admin@example.org&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">storage&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/acme/acme.json&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dnsChallenge&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">provider&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">cloudflare&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">resolvers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;1.1.1.1:53&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;8.8.8.8:53&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>useBindPortIP: true&lt;/code> is the key line for this mode. Each app publishes its port on localhost (&lt;code>PublishPort=127.0.0.1:3000:3000&lt;/code>). Traefik reads the port mapping from the container&amp;rsquo;s labels and metadata through the socket and connects to &lt;code>127.0.0.1:3000&lt;/code>. Each app needs its own host port.&lt;/p>
&lt;h3 id="the-certificate-challenge">The certificate challenge&lt;/h3>
&lt;p>Let&amp;rsquo;s Encrypt has to check that you control the domain. Traefik supports three ways:&lt;/p>
&lt;ul>
&lt;li>The DNS challenge: Traefik writes a TXT record through your DNS provider&amp;rsquo;s API. It works for wildcard certificates and for hosts that are not reachable from the internet. It needs an API token, which I keep in a Podman secret (&lt;code>Secret=dns-api-token,...&lt;/code>). Use your provider&amp;rsquo;s name in &lt;code>provider:&lt;/code>. Traefik&amp;rsquo;s ACME library supports many providers, and each one has its own environment variable for the token.&lt;/li>
&lt;li>The TLS challenge (&lt;code>tlsChallenge: {}&lt;/code>): Let&amp;rsquo;s Encrypt connects to port 443. No token is needed, but the host must be reachable from the internet. I use it for domains whose DNS is at a provider without an API.&lt;/li>
&lt;li>The HTTP challenge: the same as the TLS challenge, but on port 80.&lt;/li>
&lt;/ul>
&lt;p>&lt;code>acme.json&lt;/code> holds the certificates and the ACME account key. Keep it in your backup. After a reinstall, restore it before Traefik starts. Traefik then serves the old certificates and asks Let&amp;rsquo;s Encrypt for nothing. Let&amp;rsquo;s Encrypt has rate limits, and a rehearsal or two reinstalls in a row can hit them.&lt;/p>
&lt;h3 id="shared-middlewares-in-a-file">Shared middlewares in a file&lt;/h3>
&lt;p>Things that many apps use, like security headers, go into the file provider:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># ~svc/.config/traefik/dynamic/dynamic_conf.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">http&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">middlewares&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">secureHeaders&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">headers&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">browserXssFilter&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">contentTypeNosniff&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">frameDeny&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">stsSeconds&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">31536000&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">tls&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">options&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">default&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">minVersion&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">VersionTLS12&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>An app uses it with &lt;code>Label=traefik.http.routers.&amp;lt;name&amp;gt;.middlewares=secureHeaders@file&lt;/code>.&lt;/p>
&lt;h3 id="an-app-behind-it">An app behind it&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ~svc/.config/containers/systemd/blog.container&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">blog&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">git.example.org/alice/blog:latest&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">127.0.0.1:8101:80&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.enable=true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">&amp;#34;traefik.http.routers.blog.rule=Host(`www.example.org`) || Host(`example.org`)&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.blog.entrypoints=https&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.blog.tls.certresolver=default&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.blog.middlewares=secureHeaders@file&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.services.blog.loadbalancer.server.port=80&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Quote the whole &lt;code>Label=&lt;/code> value if it contains spaces or &lt;code>||&lt;/code>. Otherwise systemd splits it.&lt;/p>
&lt;h3 id="real-client-ips-in-the-app-too">Real client IPs in the app, too&lt;/h3>
&lt;p>Traefik sees the client, but the app behind it sees &lt;code>127.0.0.1&lt;/code>. For nginx, tell it to trust Traefik&amp;rsquo;s &lt;code>X-Forwarded-For&lt;/code> header:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-nginx" data-lang="nginx">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># realip.conf, mounted into the nginx container at /etc/nginx/conf.d/realip.conf
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Traefik replaces any X-Forwarded-For sent by clients. The app port is published
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># on 127.0.0.1 only, so the only peer that can connect is the local Traefik.
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">&lt;/span>&lt;span class="k">set_real_ip_from&lt;/span> &lt;span class="n">0.0.0.0&lt;/span>&lt;span class="s">/0&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">set_real_ip_from&lt;/span> &lt;span class="p">::&lt;/span>&lt;span class="s">/0&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">real_ip_header&lt;/span> &lt;span class="s">X-Forwarded-For&lt;/span>&lt;span class="p">;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The wide &lt;code>set_real_ip_from&lt;/code> looks wrong at first. It is safe here only because nothing but Traefik can reach the app&amp;rsquo;s port.&lt;/p>
&lt;h2 id="option-b-traefik-on-a-podman-bridge-network">Option B: Traefik on a Podman bridge network&lt;/h2>
&lt;p>On the home server Traefik and the apps share a Podman network called &lt;code>proxy&lt;/code>. Traefik publishes the ports, and the apps publish none.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># proxy.network&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Network]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">NetworkName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">proxy&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># traefik.container (bridge mode, shortened)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">docker.io/library/traefik:v3&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">80:80/tcp&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">443:443/tcp&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%t/podman/podman.sock:/run/podman/podman.sock:ro,z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/traefik/traefik.yml:/etc/traefik/traefik.yml:ro,z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik-letsencrypt.volume:/letsencrypt:z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Secret&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">dns-api-token,type=env,target=CF_DNS_API_TOKEN&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">proxy.network:ip=10.89.0.2&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>An app joins the network with &lt;code>Network=proxy.network:ip=10.89.0.20&lt;/code>. I give each container a fixed IP, so that other configs (Prometheus targets, for example) can use it. In this mode the provider section of &lt;code>traefik.yml&lt;/code> has no &lt;code>useBindPortIP&lt;/code>, because Traefik reaches the container IPs directly.&lt;/p>
&lt;p>Ports below 1024 are still needed for &lt;code>PublishPort=80:80&lt;/code>, because the rootless helper binds them as the user.&lt;/p>
&lt;h2 id="which-one-should-you-pick">Which one should you pick?&lt;/h2>
&lt;p>If anybody on the internet uses the service, choose the host network. You will want the client addresses the first time somebody hammers your login page. If only you and your devices use it, the bridge is simpler. The apps then need no host ports, and no app can be reached past Traefik on the host.&lt;/p>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">systemctl --user daemon-reload
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">systemctl --user start traefik blog
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --user -u traefik -f &lt;span class="c1"># look for &amp;#34;certificate obtained&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -sI https://www.example.org/ &lt;span class="p">|&lt;/span> head -3
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Host network mode: the access log must show your real IP as ClientHost&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --user -u traefik -n &lt;span class="m">5&lt;/span> -o cat &lt;span class="p">|&lt;/span> jq -r .ClientHost
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="things-to-watch">Things to watch&lt;/h2>
&lt;ul>
&lt;li>Test VMs and Let&amp;rsquo;s Encrypt. I restored &lt;code>acme.json&lt;/code> into every test VM before Traefik&amp;rsquo;s first start, and blocked the VM&amp;rsquo;s outgoing connections while I tested. A VM that asks for certificates of the real domains counts against the rate limits of the live host.&lt;/li>
&lt;li>Test against a VM without touching DNS: &lt;code>curl --resolve www.example.org:443:&amp;lt;vm-ip&amp;gt; https://www.example.org/&lt;/code>. With the restored &lt;code>acme.json&lt;/code> the certificate is even valid.&lt;/li>
&lt;li>Bridge mode and fail2ban. On the bridge, the client address that Traefik logs is the helper&amp;rsquo;s address. A ban on that would block everyone. So do not point fail2ban at the access log in bridge mode.&lt;/li>
&lt;/ul>
&lt;p>Next: &lt;a href="../../blog/infra-04-selinux/">SELinux enforcing with containers&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://doc.traefik.io/traefik/">Traefik documentation&lt;/a>: entrypoints, routers, services and middlewares.&lt;/li>
&lt;li>&lt;a href="https://doc.traefik.io/traefik/providers/docker/">Traefik Docker provider&lt;/a>: the provider that reads the labels. It works with the Podman socket.&lt;/li>
&lt;li>&lt;a href="https://doc.traefik.io/traefik/https/acme/">Traefik ACME&lt;/a>: the HTTP, TLS and DNS challenges and the &lt;code>acme.json&lt;/code> storage.&lt;/li>
&lt;li>&lt;a href="https://go-acme.github.io/lego/dns/">lego DNS providers&lt;/a>: the list of DNS APIs that Traefik supports, with the environment variables each one needs.&lt;/li>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-network.1.html">podman-network(1)&lt;/a>: bridge networks for rootless containers.&lt;/li>
&lt;/ul></description></item><item><title>From Docker Compose to rootless Podman quadlets</title><link>https://www.linux-lunatix.org/blog/infra-02-quadlets/</link><pubDate>Wed, 30 Sep 2026 20:15:00 +0200</pubDate><guid>https://www.linux-lunatix.org/blog/infra-02-quadlets/</guid><description>&lt;p>&lt;em>Part 2 of the series that starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/em>&lt;/p>
&lt;p>At the end of this post you have a two-container app (Forgejo with PostgreSQL) running as rootless Podman quadlets. They start at boot, update themselves and log to the journal. No root daemon and no Compose file are involved.&lt;/p>
&lt;h2 id="why-quadlets">Why quadlets&lt;/h2>
&lt;p>Podman can run Compose files with &lt;code>podman-compose&lt;/code> or &lt;code>docker-compose&lt;/code> against the Podman socket. I tried both. They work, but they keep the Compose model: one tool starts a group of containers, and systemd knows nothing about them. The older &lt;code>podman generate systemd&lt;/code> is deprecated.&lt;/p>
&lt;p>Quadlets put the containers into systemd directly. You write a &lt;code>.container&lt;/code> file. At boot, or at &lt;code>systemctl --user daemon-reload&lt;/code>, a systemd generator turns it into a normal service unit. From then on:&lt;/p>
&lt;ul>
&lt;li>&lt;code>systemctl --user start|stop|status forgejo&lt;/code> works as for any service.&lt;/li>
&lt;li>The logs are in the journal: &lt;code>journalctl --user -u forgejo&lt;/code>.&lt;/li>
&lt;li>Dependencies are plain systemd: &lt;code>Requires=&lt;/code> and &lt;code>After=&lt;/code>.&lt;/li>
&lt;li>&lt;code>Restart=always&lt;/code> and timers come from systemd too.&lt;/li>
&lt;/ul>
&lt;p>You need Podman 4.4 or newer. Fedora 44 ships Podman 5.&lt;/p>
&lt;h2 id="step-1-a-user-for-the-apps">Step 1: a user for the apps&lt;/h2>
&lt;p>Each host has one unprivileged user that owns all its containers. I call it &lt;code>svc&lt;/code>.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo useradd --create-home --shell /bin/bash svc
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Let svc&amp;#39;s systemd instance run without a login session, and start it at boot&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo loginctl enable-linger svc
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>useradd&lt;/code> on Fedora also writes ranges to &lt;code>/etc/subuid&lt;/code> and &lt;code>/etc/subgid&lt;/code>. Rootless Podman needs them to map the user IDs inside a container. Check them:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">grep svc /etc/subuid /etc/subgid
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># /etc/subuid:svc:524288:65536&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># /etc/subgid:svc:524288:65536&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>To work as that user, you need a real login session, or at least the right &lt;code>XDG_RUNTIME_DIR&lt;/code>. &lt;code>sudo -iu svc&lt;/code> alone gives you a shell where &lt;code>systemctl --user&lt;/code> fails. These work:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo machinectl shell svc@ &lt;span class="c1"># a full session (needs systemd-container)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># or, for single commands:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo -u svc &lt;span class="nv">XDG_RUNTIME_DIR&lt;/span>&lt;span class="o">=&lt;/span>/run/user/&lt;span class="k">$(&lt;/span>id -u svc&lt;span class="k">)&lt;/span> systemctl --user status
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Rootless containers cannot bind ports below 1024 by default. If an app must listen on 80 or 443 directly (Traefik in the next post), lower the limit:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s1">&amp;#39;net.ipv4.ip_unprivileged_port_start = 80&amp;#39;&lt;/span> &lt;span class="p">|&lt;/span> sudo tee /etc/sysctl.d/90-unprivileged-ports.conf
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo sysctl --system
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="step-2-the-compose-file-we-start-from">Step 2: the Compose file we start from&lt;/h2>
&lt;p>This is a shortened version of what ran on the old Ubuntu host:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="nt">services&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">forgejo&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">image&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">codeberg.org/forgejo/forgejo:15&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">restart&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">unless-stopped&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">DB_TYPE=postgres&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">DB_HOST=postgres:5432&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">DB_PASSWD=${FORGEJO_DB_PASSWORD}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">networks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">proxy, dbnet]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">depends_on&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">postgres]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">volumes&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">./forgejo:/data&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">labels&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;traefik.enable=true&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;traefik.http.routers.forgejo.rule=Host(`git.example.org`)&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">postgres&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">image&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">postgres:17-alpine&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">restart&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">always&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">POSTGRES_USER&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">forgejo&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">POSTGRES_PASSWORD&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">${FORGEJO_DB_PASSWORD}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">POSTGRES_DB&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">forgejo&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">networks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">dbnet]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">volumes&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">./postgres:/var/lib/postgresql/data&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">networks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dbnet&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">proxy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">external&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Each part of it becomes one quadlet file:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Compose&lt;/th>
&lt;th>Quadlet file&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>networks: dbnet&lt;/code>&lt;/td>
&lt;td>&lt;code>forgejo.network&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>a volume&lt;/td>
&lt;td>&lt;code>forgejo-data.volume&lt;/code>, &lt;code>forgejo-db.volume&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>a service&lt;/td>
&lt;td>&lt;code>forgejo.container&lt;/code>, &lt;code>forgejo-postgres.container&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>.env&lt;/code> with a password&lt;/td>
&lt;td>a Podman secret&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>depends_on&lt;/code>&lt;/td>
&lt;td>&lt;code>Requires=&lt;/code> and &lt;code>After=&lt;/code>&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>All files go into &lt;code>~svc/.config/containers/systemd/&lt;/code>.&lt;/p>
&lt;h2 id="step-3-network-and-volumes">Step 3: network and volumes&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># forgejo.network&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Network]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">NetworkName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># forgejo-data.volume&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Volume]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">VolumeName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-data&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># forgejo-db.volume&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Volume]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">VolumeName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-db&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Without &lt;code>NetworkName=&lt;/code> and &lt;code>VolumeName=&lt;/code>, Podman calls them &lt;code>systemd-forgejo&lt;/code> and &lt;code>systemd-forgejo-data&lt;/code>. That works, but the explicit names are easier to read in &lt;code>podman volume ls&lt;/code> and in backup scripts.&lt;/p>
&lt;h2 id="step-4-a-secret-instead-of-the-dot-env-file">Step 4: a secret instead of the .env file&lt;/h2>
&lt;p>Compose read the database password from &lt;code>.env&lt;/code>. Podman has secrets. Create one as &lt;code>svc&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">printf&lt;/span> &lt;span class="s1">&amp;#39;%s&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">$(&lt;/span>pwgen -s &lt;span class="m">32&lt;/span> 1&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="p">|&lt;/span> podman secret create forgejo-db-password -
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman secret ls
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The secret is stored in the user&amp;rsquo;s Podman storage and is not part of any image or quadlet file. In post 6 Ansible creates it from &lt;code>pass&lt;/code>, so I never type it.&lt;/p>
&lt;h2 id="step-5-the-containers">Step 5: the containers&lt;/h2>
&lt;p>The database first:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># forgejo-postgres.container&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Unit]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Description&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">Forgejo database&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-postgres&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">docker.io/library/postgres:17-alpine&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo.network&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># The host name the app uses for its database&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">NetworkAlias&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">postgres&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-db.volume:/var/lib/postgresql/data:Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Environment&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">POSTGRES_USER=forgejo POSTGRES_DB=forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Secret&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-db-password,type=env,target=POSTGRES_PASSWORD&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">HealthCmd&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">pg_isready -U forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">HealthInterval&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">30s&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then the app:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># forgejo.container&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Unit]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Description&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">Forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Requires&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-postgres.service&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">After&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-postgres.service&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">codeberg.org/forgejo/forgejo:15&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo.network&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">forgejo-data.volume:/data:Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Web UI only on localhost: the reverse proxy forwards to it&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">127.0.0.1:3000:3000&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Git over SSH, published directly&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">2222:22&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.enable=true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.forgejo.rule=Host(`git.example.org`)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.forgejo.entrypoints=https&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.forgejo.tls.certresolver=default&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.services.forgejo.loadbalancer.server.port=3000&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Some details:&lt;/p>
&lt;ul>
&lt;li>&lt;code>Network=forgejo.network&lt;/code> refers to the quadlet file, not to the network name. systemd then also adds a dependency on the network unit.&lt;/li>
&lt;li>&lt;code>Volume=forgejo-db.volume:...&lt;/code> is the same for volumes.&lt;/li>
&lt;li>&lt;code>:Z&lt;/code> gives the volume a private SELinux label. Post 4 explains &lt;code>:Z&lt;/code> and &lt;code>:z&lt;/code>.&lt;/li>
&lt;li>&lt;code>[Install]&lt;/code> with &lt;code>WantedBy=default.target&lt;/code> starts the service at boot. You cannot &lt;code>systemctl --user enable&lt;/code> a quadlet, because the unit is generated. The &lt;code>[Install]&lt;/code> section does that job.&lt;/li>
&lt;li>The &lt;code>Label=&lt;/code> lines are for Traefik. Compose had them as &lt;code>labels:&lt;/code>, and they stay the same. The next post sets up Traefik.&lt;/li>
&lt;/ul>
&lt;h2 id="step-6-start-it">Step 6: start it&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">systemctl --user daemon-reload
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">systemctl --user start forgejo &lt;span class="c1"># starts forgejo-postgres first&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">systemctl --user status forgejo forgejo-postgres
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman ps
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --user -u forgejo -f
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>If a unit does not appear after &lt;code>daemon-reload&lt;/code>, the generator did not like the file. Ask it directly:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">/usr/libexec/podman/quadlet -dryrun -user
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>It prints the generated units, or the error with the file and line.&lt;/p>
&lt;h2 id="step-7-updates-with-podman-auto-update">Step 7: updates with podman auto-update&lt;/h2>
&lt;p>&lt;code>AutoUpdate=registry&lt;/code> marks a container for &lt;code>podman auto-update&lt;/code>. The command checks the registry for a new image under the same tag. If there is one, it pulls it and restarts the unit. If the new container fails to start, it rolls back to the old image.&lt;/p>
&lt;p>Enable the timer that ships with Podman:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">systemctl --user &lt;span class="nb">enable&lt;/span> --now podman-auto-update.timer
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman auto-update --dry-run &lt;span class="c1"># what would be updated now&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The timer runs once a day. To change that, add a drop-in:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ~/.config/systemd/user/podman-auto-update.timer.d/interval.conf&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Timer]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">OnCalendar&lt;/span>&lt;span class="o">=&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">OnCalendar&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">*:0/15&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">RandomizedDelaySec&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">0&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The empty &lt;code>OnCalendar=&lt;/code> clears the default first. Without it, you get both schedules.&lt;/p>
&lt;p>The tag decides what an update can bring. &lt;code>postgres:17-alpine&lt;/code> gets patch releases of PostgreSQL 17, never 18. A major PostgreSQL upgrade needs a dump and a restore, and I want to do that by hand. &lt;code>forgejo:15&lt;/code> is the same. I use &lt;code>:latest&lt;/code> only for images where every new version is fine, for example my own website images.&lt;/p>
&lt;h2 id="step-8-move-the-data">Step 8: move the data&lt;/h2>
&lt;p>The old data was in bind mounts under &lt;code>/opt&lt;/code>, owned by root and by the container&amp;rsquo;s user IDs. Rootless Podman maps IDs: UID 0 in the container is &lt;code>svc&lt;/code> on the host, and UID 1000 in the container is some high number from &lt;code>/etc/subuid&lt;/code>. A plain &lt;code>cp&lt;/code> therefore gives the wrong owners.&lt;/p>
&lt;p>&lt;code>podman unshare&lt;/code> runs a command inside the user&amp;rsquo;s ID mapping. Owners then look the same as they did inside the old container:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># as root: copy the old bind mount into the new volume, owners as numbers&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman_svc&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> runuser -u svc -- env &lt;span class="nv">XDG_RUNTIME_DIR&lt;/span>&lt;span class="o">=&lt;/span>/run/user/&lt;span class="k">$(&lt;/span>id -u svc&lt;span class="k">)&lt;/span> podman &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$@&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman_svc volume create --ignore forgejo-data
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">mp&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>podman_svc volume inspect --format &lt;span class="s1">&amp;#39;{{.Mountpoint}}&amp;#39;&lt;/span> forgejo-data&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">tar -C /opt/forgejo/forgejo --numeric-owner -cf - . &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> runuser -u svc -- podman unshare tar -C &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$mp&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --numeric-owner -xpf -
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Do not copy a database directory this way while the database runs. For PostgreSQL I used a dump: &lt;code>pg_dumpall&lt;/code> on the old host, then &lt;code>psql&lt;/code> in the new container.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># on the old host&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">docker &lt;span class="nb">exec&lt;/span> forgejo-postgres-1 pg_dumpall -U forgejo &lt;span class="p">|&lt;/span> gzip &amp;gt; forgejo.sql.gz
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># on the new host, as svc: start with an empty database, then load the dump&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">systemctl --user stop forgejo
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">podman &lt;span class="nb">exec&lt;/span> forgejo-postgres sh -c &lt;span class="s1">&amp;#39;dropdb -U forgejo --force forgejo &amp;amp;&amp;amp; createdb -U forgejo forgejo&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">gunzip -c forgejo.sql.gz &lt;span class="p">|&lt;/span> podman &lt;span class="nb">exec&lt;/span> -i forgejo-postgres psql -q -U forgejo -d postgres
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">systemctl --user start forgejo
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="what-bit-me">What bit me&lt;/h2>
&lt;ul>
&lt;li>The working directory. &lt;code>runuser -u svc -- podman ...&lt;/code> from a directory that &lt;code>svc&lt;/code> cannot read fails with strange errors. Run it from the root directory (&lt;code>cd / &amp;amp;&amp;amp; runuser ...&lt;/code>).&lt;/li>
&lt;li>Containers that share a network namespace. A VPN container with other containers inside its namespace (&lt;code>Network=container:gluetun&lt;/code>) cannot be restarted on its own. Stop the dependants first, in reverse order, then start everything in order.&lt;/li>
&lt;li>Files mounted into containers. An editor or Ansible replaces a file with a new inode. The container still sees the old file until it restarts. Restart the unit after you change a mounted config file.&lt;/li>
&lt;li>&lt;code>sudo -iu svc&lt;/code>. Without &lt;code>XDG_RUNTIME_DIR&lt;/code>, &lt;code>systemctl --user&lt;/code> says it cannot connect to the bus. Use &lt;code>machinectl shell&lt;/code> or set the variable.&lt;/li>
&lt;/ul>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo reboot
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># after the reboot, without logging in as svc:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo -u svc &lt;span class="nv">XDG_RUNTIME_DIR&lt;/span>&lt;span class="o">=&lt;/span>/run/user/&lt;span class="k">$(&lt;/span>id -u svc&lt;span class="k">)&lt;/span> systemctl --user is-active forgejo forgejo-postgres
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Both say &lt;code>active&lt;/code>. The containers run as &lt;code>svc&lt;/code>, and nobody logged in.&lt;/p>
&lt;p>Next: &lt;a href="../../blog/infra-03-traefik/">Traefik in front of rootless containers&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html">podman-systemd.unit(5)&lt;/a>: the quadlet reference: every key of &lt;code>.container&lt;/code>, &lt;code>.volume&lt;/code>, &lt;code>.network&lt;/code> and the other unit types.&lt;/li>
&lt;li>&lt;a href="https://www.redhat.com/en/blog/quadlet-podman">Make systemd better for Podman with Quadlet (Red Hat blog)&lt;/a>: the reasons behind quadlets, from the people who wrote them.&lt;/li>
&lt;li>&lt;a href="https://github.com/containers/podman/blob/main/docs/tutorials/rootless_tutorial.md">Basic setup and use of Podman in a rootless environment&lt;/a>: subuids, storage and the limits of rootless mode.&lt;/li>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-auto-update.1.html">podman-auto-update(1)&lt;/a>: how &lt;code>AutoUpdate=registry&lt;/code> decides to pull and how the rollback works.&lt;/li>
&lt;li>&lt;a href="https://docs.podman.io/en/latest/markdown/podman-secret-create.1.html">podman-secret-create(1)&lt;/a>: secret drivers and how a container gets them.&lt;/li>
&lt;li>&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/loginctl.html">loginctl(1)&lt;/a>: &lt;code>enable-linger&lt;/code>, so that user services run without a login.&lt;/li>
&lt;li>&lt;a href="https://github.com/containers/podlet">podlet&lt;/a>: converts compose files and &lt;code>podman run&lt;/code> commands to quadlets. A good start, but read what it writes.&lt;/li>
&lt;/ul></description></item></channel></rss>