<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>SSH on Linux Lunatix</title><link>https://www.linux-lunatix.org/categories/ssh/</link><description>Recent content in SSH on Linux Lunatix</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><copyright>2025 Stefan Marx :: All written work is licensed under CC BY-SA 4.0 unless otherwise stated. Images are taken by me unless otherwise stated and need written permission to be used elsewhere.</copyright><lastBuildDate>Sat, 21 Nov 2026 10:15:00 +0100</lastBuildDate><atom:link href="https://www.linux-lunatix.org/categories/ssh/index.xml" rel="self" type="application/rss+xml"/><item><title>Removing the plain SSH keys</title><link>https://www.linux-lunatix.org/blog/ssh-05-plain-keys/</link><pubDate>Sat, 21 Nov 2026 10:15:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/ssh-05-plain-keys/</guid><description>&lt;p>&lt;em>Part 5 of the SSH series that starts with &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/em>&lt;/p>
&lt;p>Certificates work on every host now. The old plain keys are still in the &lt;code>authorized_keys&lt;/code> files, and while they are there, the certificates change nothing about security. A stolen laptop key would still open every door. This post removes the plain keys. Only one break-glass key per account stays, for the day when Bao is gone.&lt;/p>
&lt;h2 id="the-target">The target&lt;/h2>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Account&lt;/th>
&lt;th>Certificate principals&lt;/th>
&lt;th>Plain keys&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>the admin account (&lt;code>fedora&lt;/code>)&lt;/td>
&lt;td>&lt;code>alice&lt;/code>&lt;/td>
&lt;td>break-glass&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>svc&lt;/code> (the apps)&lt;/td>
&lt;td>&lt;code>alice&lt;/code> (+ &lt;code>deploy-websites&lt;/code> on the web host)&lt;/td>
&lt;td>none&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>ansible&lt;/code>&lt;/td>
&lt;td>&lt;code>ansible&lt;/code>&lt;/td>
&lt;td>break-glass&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>&lt;code>svc&lt;/code> gets no plain key at all. If I need it without Bao, I log in as the admin account and use &lt;code>sudo&lt;/code>.&lt;/p>
&lt;h2 id="step-1-the-break-glass-key">Step 1: the break-glass key&lt;/h2>
&lt;p>The break-glass key is a normal Ed25519 key. Its private part lives only in pass, never in &lt;code>~/.ssh&lt;/code>. Create it in memory and move it straight into pass:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">d&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>mktemp -d -p /dev/shm&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -q -t ed25519 -N &lt;span class="s1">&amp;#39;&amp;#39;&lt;/span> -C &lt;span class="s1">&amp;#39;break-glass@infra&amp;#39;&lt;/span> -f &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/key&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">pass insert -m infra/ssh-break-glass &amp;lt; &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/key&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">cp &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/key.pub&amp;#34;&lt;/span> ~/ops/client/ssh/break-glass.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">shred -u &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/key&amp;#34;&lt;/span> &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> rm -r &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The public part goes into the inventory:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># group_vars/all.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_break_glass_key&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;ssh-ed25519 AAAA...break-glass-public-key... break-glass@infra&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>To use it, a small script loads the key from pass into ssh-agent for 15 minutes. It never writes it to disk:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># ssh-break-glass: only when Bao is gone and the certificates have expired&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">#&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-break-glass vps1 # any ssh arguments&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-break-glass -- ansible-playbook ... # a command: Ansible with the key&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">pub&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>dirname &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">$(&lt;/span>readlink -f &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$0&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span>/../ssh/break-glass.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">pass show infra/ssh-break-glass &lt;span class="p">|&lt;/span> ssh-add -q -t 15m -
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> &lt;span class="o">[&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">1&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;--&amp;#34;&lt;/span> &lt;span class="o">]&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">shift&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">exec&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$@&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> --private-key &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$pub&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">exec&lt;/span> ssh -i &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$pub&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$@&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>ssh-add -&lt;/code> reads the private key from stdin. &lt;code>-t 15m&lt;/code> removes it from the agent after 15 minutes.&lt;/p>
&lt;p>&lt;code>-i&lt;/code> with a &lt;code>.pub&lt;/code> file looks odd. ssh then selects the matching key in the agent and offers it first. That matters with &lt;code>IdentitiesOnly yes&lt;/code> and &lt;code>MaxAuthTries 3&lt;/code>. Ansible&amp;rsquo;s &lt;code>--private-key&lt;/code> works the same way.&lt;/p>
&lt;p>Test it now, while everything works, and not on the day you need it.&lt;/p>
&lt;h2 id="step-2-authorized-keys-from-the-inventory">Step 2: authorized_keys from the inventory&lt;/h2>
&lt;p>Each host lists its plain keys per account:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/vps1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_authorized_keys&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fedora&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_break_glass_key }}&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">svc&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_break_glass_key }}&amp;#34;&lt;/span>&lt;span class="p">]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The rules: a listed account gets exactly these keys, and nothing else stays in its file. An empty list removes the file. An account that is not listed is left alone, so the role cannot touch accounts it does not know.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># roles/base/tasks/ssh_keys.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Accounts&amp;#39; home directories&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.getent&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">database&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">passwd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">authorized_keys with exactly the listed keys&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.posix.authorized_key&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">user&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.key }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">key&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.value | join(&amp;#39;\n&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">exclusive&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_authorized_keys | dict2items | selectattr(&amp;#39;value&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.key }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">No&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">authorized_keys for cert-only accounts&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ansible_facts.getent_passwd[item.key][4] }}/.ssh/authorized_keys&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">absent&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_authorized_keys | dict2items | rejectattr(&amp;#39;value&amp;#39;) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.key }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>exclusive: true&lt;/code> is what removes the old keys. The home directory comes from &lt;code>getent&lt;/code>, because &lt;code>svc&lt;/code> and the admin account do not have to live under &lt;code>/home/&amp;lt;name&amp;gt;&lt;/code>.&lt;/p>
&lt;p>In the role&amp;rsquo;s &lt;code>main.yml&lt;/code> these tasks come after the &lt;code>flush_handlers&lt;/code> from &lt;a href="../../blog/ssh-03-hosts/">the previous post&lt;/a>. On a host&amp;rsquo;s first run, sshd already trusts the CA when the keys go away.&lt;/p>
&lt;p>&lt;code>bootstrap.yml&lt;/code> from the Ansible post still installs the control nodes&amp;rsquo; plain keys for &lt;code>ansible&lt;/code>. That is for the first contact with a new host, before it trusts the CA. The first &lt;code>site.yml&lt;/code> run then replaces them with the break-glass key.&lt;/p>
&lt;h2 id="step-3-roll-it-out-one-host-at-a-time">Step 3: roll it out, one host at a time&lt;/h2>
&lt;p>I did this first on a throwaway VM, then host by host.&lt;/p>
&lt;ol>
&lt;li>Get fresh certificates: &lt;code>ssh-cert&lt;/code>. Check that both are valid: &lt;code>ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub&lt;/code> and the same for &lt;code>ansible_ed25519-cert.pub&lt;/code>.&lt;/li>
&lt;li>Open a standby session as the admin account and leave it open:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh -M -S ~/.ssh/cm/standby-vps1 -fN fedora@vps1.example.org
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>Close Ansible&amp;rsquo;s reused connections, then run the base role:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="k">for&lt;/span> s in ~/.ansible/cp/*&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span> ssh -O &lt;span class="nb">exit&lt;/span> -o &lt;span class="nv">ControlPath&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$s&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> x 2&amp;gt;/dev/null&lt;span class="p">;&lt;/span> &lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t base --check
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -l vps1 -t base
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>Test with fresh connections. Each of these must work:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">F&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;-o ControlMaster=no -o ControlPath=none&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh &lt;span class="nv">$F&lt;/span> fedora@vps1.example.org &lt;span class="nb">true&lt;/span> &lt;span class="c1"># certificate&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh &lt;span class="nv">$F&lt;/span> svc@vps1.example.org &lt;span class="nb">true&lt;/span> &lt;span class="c1"># certificate&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ansible vps1 -m ping &lt;span class="c1"># the ansible certificate&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-break-glass &lt;span class="nv">$F&lt;/span> fedora@vps1.example.org &lt;span class="nb">true&lt;/span> &lt;span class="c1"># break-glass key&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>And this must fail:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">d&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>mktemp -d&lt;span class="k">)&lt;/span> &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> cp ~/.ssh/id_ed25519 &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh &lt;span class="nv">$F&lt;/span> -o &lt;span class="nv">IdentityAgent&lt;/span>&lt;span class="o">=&lt;/span>none -o &lt;span class="nv">IdentitiesOnly&lt;/span>&lt;span class="o">=&lt;/span>yes -i &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">/id_ed25519&amp;#34;&lt;/span> fedora@vps1.example.org &lt;span class="nb">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">rm -r &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$d&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>The copy has no &lt;code>-cert.pub&lt;/code> next to it, and &lt;code>IdentityAgent=none&lt;/code> keeps the agent out, so ssh offers only the plain key. Before this step it was in &lt;code>authorized_keys&lt;/code>. Now it is refused.&lt;/li>
&lt;li>The second run must report &lt;code>changed=0&lt;/code>. Then close the standby session.&lt;/li>
&lt;/ol>
&lt;h2 id="devices-that-cannot-talk-to-bao">Devices that cannot talk to Bao&lt;/h2>
&lt;p>My tablet runs &lt;a href="https://blink.sh">Blink&lt;/a>, an SSH client for iOS. Blink cannot log in to Bao, but it can use a certificate. So the tablet gets a certificate for 30 days, which I sign on the laptop.&lt;/p>
&lt;ol>
&lt;li>In Blink, create an Ed25519 key and copy its public part to the laptop.&lt;/li>
&lt;li>Sign it with the &lt;code>device&lt;/code> role:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-cert device tablet.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>In Blink: Keys, the key, Certificate, paste.&lt;/li>
&lt;li>Put a reminder into the calendar a few days before the 30 days end. An expired certificate is simply refused.&lt;/li>
&lt;/ol>
&lt;p>Blink cannot check host certificates. It asks about each host key once. Compare the fingerprint with the host&amp;rsquo;s real key before you accept it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh vps1.example.org ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A second Ansible control node that is not always on gets a 30-day certificate for its &lt;code>ansible&lt;/code> key the same way, with a role of its own.&lt;/p>
&lt;h2 id="a-lost-device">A lost device&lt;/h2>
&lt;p>For the laptop, the certificates expire within 16 hours. But its Bao token can sign new ones until the token expires. Revoke it at once from another machine:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">bao login -no-print -method&lt;span class="o">=&lt;/span>userpass &lt;span class="nv">username&lt;/span>&lt;span class="o">=&lt;/span>alice
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao list auth/token/accessors
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao token lookup -accessor &amp;lt;accessor&amp;gt; &lt;span class="c1"># find the oidc-alice tokens&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao token revoke -accessor &amp;lt;accessor&amp;gt;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>For a device with a 30-day certificate, put its public key into the revoked list and roll it out:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># group_vars/all.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_revoked_keys&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="s2">&amp;#34;ssh-ed25519 AAAA...the-tablet&amp;#39;s-public-key...&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ansible-playbook playbooks/site.yml -t base
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>sshd then refuses that key with any certificate. The device&amp;rsquo;s other keys (GitHub, an SFTP backup box) are plain keys and must be removed there, as before.&lt;/p>
&lt;h2 id="when-bao-is-gone">When Bao is gone&lt;/h2>
&lt;ul>
&lt;li>Certificates that are already issued keep working until they expire. Hosts check them offline.&lt;/li>
&lt;li>After that: &lt;code>ssh-break-glass &amp;lt;host&amp;gt;&lt;/code> for a login, and &lt;code>ssh-break-glass -- ansible-playbook ...&lt;/code> for Ansible.&lt;/li>
&lt;li>Without pass, too: the provider&amp;rsquo;s VNC console for the VPS, the local console at home.&lt;/li>
&lt;li>Then bring Bao back, or restore it on another host from its Raft snapshot (&lt;a href="../../blog/ssh-02-openbao/">the OpenBao post&lt;/a>).&lt;/li>
&lt;/ul>
&lt;h2 id="how-i-nearly-locked-myself-out">How I nearly locked myself out&lt;/h2>
&lt;p>Two things went wrong during this work. Each one taught me a rule.&lt;/p>
&lt;h3 id="fail2ban-banned-my-home-ip">fail2ban banned my home IP&lt;/h3>
&lt;p>During a negative test I connected to a host by its bare FQDN, without the &lt;code>User&lt;/code> that my SSH config sets for its short name. ssh therefore used my local user name. That account is not in the host&amp;rsquo;s &lt;code>AllowUsers&lt;/code>, and fail2ban counts such a login as a failure. Together with my negative tests that made three failures within ten minutes, and fail2ban banned my home IP for an hour.&lt;/p>
&lt;p>Interestingly, a refused key alone (&amp;ldquo;Failed publickey&amp;rdquo;) does not count in fail2ban&amp;rsquo;s sshd filter. A login as an account that &lt;code>AllowUsers&lt;/code> does not allow does count.&lt;/p>
&lt;p>I got in again through another of my hosts, which still had a working connection, and unbanned the IP there:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh -J other-host vps1.example.org &lt;span class="s1">&amp;#39;sudo fail2ban-client set sshd unbanip 203.0.113.50&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The rule: test negative cases sparingly, with explicit user names, and know how you get in from another IP.&lt;/p>
&lt;h3 id="ansible-s-tests-passed-without-a-certificate">Ansible&amp;rsquo;s tests passed without a certificate&lt;/h3>
&lt;p>After the first host, all Ansible tests passed. Then I noticed that my laptop had no Ansible certificate at all. The &lt;code>ssh-cert&lt;/code> script signed only the login key at that point. Ansible had passed because it reused its SSH connections from before the change (&lt;code>ControlPersist&lt;/code>). A fresh connection would have failed.&lt;/p>
&lt;p>Since then &lt;code>ssh-cert&lt;/code> signs both keys by default, and I close &lt;code>~/.ansible/cp/*&lt;/code> before every test.&lt;/p>
&lt;h2 id="the-rules-in-short">The rules, in short&lt;/h2>
&lt;ul>
&lt;li>Keep a standby session open while you change anything that controls SSH access.&lt;/li>
&lt;li>Test with fresh connections: &lt;code>-o ControlMaster=no -o ControlPath=none&lt;/code>, and close Ansible&amp;rsquo;s connections.&lt;/li>
&lt;li>Test the break-glass path before you need it.&lt;/li>
&lt;li>Keep the principals and the keys in the inventory, and make each list exclusive.&lt;/li>
&lt;li>Short certificates beat revocation. Revoke the token that can sign, not the certificate.&lt;/li>
&lt;/ul>
&lt;h2 id="where-this-leaves-me">Where this leaves me&lt;/h2>
&lt;p>Every host trusts one CA key and lists, per account, which principals may log in. My laptop gets certificates for the day with one command and needs no &lt;code>known_hosts&lt;/code> entries for my hosts. CI jobs get a certificate for five minutes and can run one command. A reinstalled host gets a new host certificate during its first Ansible run, and nobody has to check a fingerprint over a console. The only plain key left lives in pass.&lt;/p>
&lt;p>That was the last post of the series. Part 1, about consolidating the servers, starts with &lt;a href="../../blog/infra-01-consolidation/">Four servers, three distributions: why I consolidated&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/collections/ansible/posix/authorized_key_module.html">ansible.posix.authorized_key&lt;/a>: &lt;code>exclusive&lt;/code> and the other options.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/ssh-add">ssh-add(1)&lt;/a>: &lt;code>-t&lt;/code> for a lifetime and &lt;code>-&lt;/code> for a key from stdin.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/ssh_config#ControlMaster">ssh_config(5), ControlMaster&lt;/a>: connection reuse, and why it hides a changed login.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/collections/ansible/builtin/ssh_connection.html">Ansible ssh connection plugin&lt;/a>: &lt;code>control_path&lt;/code> and &lt;code>ssh_args&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/commands/token/revoke/">bao token revoke&lt;/a>: revoke by accessor, for a lost laptop.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd_config#RevokedKeys">sshd_config(5), RevokedKeys&lt;/a>: plain key lists and KRLs.&lt;/li>
&lt;li>&lt;a href="https://docs.blink.sh/">Blink Shell documentation&lt;/a>: keys and certificates on iOS.&lt;/li>
&lt;li>&lt;a href="https://github.com/fail2ban/fail2ban">fail2ban&lt;/a>: what the sshd filter counts.&lt;/li>
&lt;/ul></description></item><item><title>CI deploys without a deploy key</title><link>https://www.linux-lunatix.org/blog/ssh-04-ci-oidc/</link><pubDate>Sun, 15 Nov 2026 16:50:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/ssh-04-ci-oidc/</guid><description>&lt;p>&lt;em>Part 4 of the SSH series that starts with &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/em>&lt;/p>
&lt;p>My websites are built by Forgejo Actions. After a push to &lt;code>main&lt;/code>, a job builds a container image, pushes it to the registry and logs in to the web host by SSH. There it runs &lt;code>podman auto-update&lt;/code>, which pulls the new image and restarts the site.&lt;/p>
&lt;p>That login used a deploy key: a private SSH key, stored as a secret in Forgejo, valid forever. This post replaces it with a certificate that a job gets for itself, valid for five minutes, and only for pushes to &lt;code>main&lt;/code>. Forgejo stores no secret for it.&lt;/p>
&lt;h2 id="how-a-job-proves-who-it-is">How a job proves who it is&lt;/h2>
&lt;p>Forgejo Actions can give a job an OpenID Connect ID token, like GitHub Actions does (I run Forgejo 15). It is a JWT, signed by Forgejo, with claims about the job. Shortened:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-json" data-lang="json">&lt;span class="line">&lt;span class="cl">&lt;span class="p">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;iss&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;https://git.example.org/api/actions&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;aud&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;https://bao.example.net&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;sub&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;repo:alice/blog:ref:refs/heads/main&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;repository&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="s2">&amp;#34;alice/blog&amp;#34;&lt;/span>&lt;span class="p">,&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nt">&amp;#34;exp&amp;#34;&lt;/span>&lt;span class="p">:&lt;/span> &lt;span class="mi">1763222400&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="p">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Anybody can check the signature with Forgejo&amp;rsquo;s public keys, which it publishes at &lt;code>https://git.example.org/api/actions/.well-known/openid-configuration&lt;/code>. Bao&amp;rsquo;s &lt;code>jwt&lt;/code> auth method does exactly that, and then it checks the claims.&lt;/p>
&lt;p>&lt;code>sub&lt;/code> is the claim to bind on. For a push it is &lt;code>repo:&amp;lt;repo&amp;gt;:ref:&amp;lt;ref&amp;gt;&lt;/code>. For a pull request it is &lt;code>repo:&amp;lt;repo&amp;gt;:pull_request&lt;/code>. A pull request from a fork can run the workflow, but it can never get a deploy certificate.&lt;/p>
&lt;h2 id="step-1-bao-s-side">Step 1: Bao&amp;rsquo;s side&lt;/h2>
&lt;p>Three parts, all in the configuration script from &lt;a href="../../blog/ssh-02-openbao/">the OpenBao post&lt;/a>.&lt;/p>
&lt;p>The role for the certificate, with the command already in it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">user_role deploy-websites deploy-websites 5m &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span> &lt;span class="s1">&amp;#39;{&amp;#34;force-command&amp;#34;: &amp;#34;podman auto-update&amp;#34;}&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A policy that can sign that role, and revoke its own token:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">bao policy write deploy-websites - &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;HCL&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"># CI deploys of the sites (jwt login): one 5-minute cert per job
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;ssh-user/sign/deploy-websites&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;auth/token/revoke-self&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">HCL&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>jwt&lt;/code> login, with Forgejo as the issuer, and a role that binds the claims:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">auth jwt jwt
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao write auth/jwt/config &lt;span class="nv">oidc_discovery_url&lt;/span>&lt;span class="o">=&lt;/span>https://git.example.org/api/actions &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="nv">bound_issuer&lt;/span>&lt;span class="o">=&lt;/span>https://git.example.org/api/actions &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao write auth/jwt/role/websites - &amp;gt;/dev/null &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;JSON&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">{
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;role_type&amp;#34;: &amp;#34;jwt&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;bound_audiences&amp;#34;: &amp;#34;https://bao.example.net&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;user_claim&amp;#34;: &amp;#34;repository&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;bound_claims_type&amp;#34;: &amp;#34;string&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;bound_claims&amp;#34;: {&amp;#34;sub&amp;#34;: [
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;repo:alice/blog:ref:refs/heads/main&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;repo:alice/shop:ref:refs/heads/main&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;repo:alice/portfolio:ref:refs/heads/main&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> ]},
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_policies&amp;#34;: &amp;#34;deploy-websites&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_no_default_policy&amp;#34;: true,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_ttl&amp;#34;: &amp;#34;5m&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_max_ttl&amp;#34;: &amp;#34;5m&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">JSON&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The fields that matter:&lt;/p>
&lt;ul>
&lt;li>&lt;code>bound_audiences&lt;/code>: the job must ask for a token for Bao. A token that the job got for another service is refused.&lt;/li>
&lt;li>&lt;code>bound_claims&lt;/code> on &lt;code>sub&lt;/code>: only pushes to &lt;code>main&lt;/code> of these three repositories. Another branch, a tag or a pull request gets &amp;ldquo;claim is not in the bound claims&amp;rdquo;.&lt;/li>
&lt;li>&lt;code>user_claim: repository&lt;/code>: the Bao token&amp;rsquo;s display name becomes &lt;code>jwt-alice/blog&lt;/code>. The certificate&amp;rsquo;s key ID is then &lt;code>jwt-alice/blog-deploy-websites&lt;/code>, and that is what the web host logs for each deploy.&lt;/li>
&lt;li>&lt;code>token_no_default_policy&lt;/code>: Bao&amp;rsquo;s &lt;code>default&lt;/code> policy lets a token look itself up and use the cubbyhole, among other things. A CI token needs none of that. Without the default policy it also cannot revoke itself, so the &lt;code>deploy-websites&lt;/code> policy grants &lt;code>revoke-self&lt;/code> explicitly.&lt;/li>
&lt;/ul>
&lt;h2 id="step-2-the-web-host-s-side">Step 2: the web host&amp;rsquo;s side&lt;/h2>
&lt;p>The principals file for the account that runs the sites has the option, as shown in &lt;a href="../../blog/ssh-03-hosts/">the previous post&lt;/a>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># group_vars/websites.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_principals&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">svc&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">alice, &amp;#39;restrict,command=&amp;#34;podman auto-update&amp;#34; deploy-websites&amp;#39;]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>restrict&lt;/code> turns off everything: a pty, port forwarding, agent forwarding, X11. &lt;code>command=&lt;/code> forces the command. The CA sets &lt;code>force-command&lt;/code> too, but the host does not depend on that.&lt;/p>
&lt;h2 id="step-3-the-network">Step 3: the network&lt;/h2>
&lt;p>The CI runner runs on my mail host, on the internet. Bao runs at home and is not on the internet. The runner reaches the home server through the monitoring tunnel from &lt;a href="../../blog/infra-07-monitoring-backup/">the monitoring post&lt;/a>. That tunnel ends in the hub&amp;rsquo;s &lt;code>block&lt;/code> zone, where the VPS cannot open any connection.&lt;/p>
&lt;p>I did not want to open port 443 on the hub for the runner. Port 443 is Traefik, and Traefik routes to every app on the home server. So Traefik gets an extra entrypoint, and only Bao&amp;rsquo;s router uses it:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># traefik.yml on home1&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">entryPoints&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">https&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">address&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;:443&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># Only OpenBao&amp;#39;s router uses this entrypoint: the CI runner may reach Bao and nothing else&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">bao&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">address&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;:8443&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># traefik.container: publish it&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">PublishPort&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">8443:8443/tcp&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># openbao.container: the router listens on both&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.openbao.entrypoints=https,bao&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The hub&amp;rsquo;s firewall then has one exception, for the runner&amp;rsquo;s tunnel address and this port:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/home1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">wg_hub_block_exceptions&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- {&lt;span class="nt">source: 10.99.0.11, port&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="m">8443&lt;/span>&lt;span class="l">/tcp}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>From the runner host, the result:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">curl -s -o /dev/null -w &lt;span class="s1">&amp;#39;%{http_code}\n&amp;#39;&lt;/span> https://bao.example.net:8443/v1/sys/health &lt;span class="c1"># 200&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -s -o /dev/null -w &lt;span class="s1">&amp;#39;%{http_code}\n&amp;#39;&lt;/span> --resolve grafana.example.net:8443:10.99.0.1 &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> https://grafana.example.net:8443/ &lt;span class="c1"># 404&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">curl -s -m &lt;span class="m">5&lt;/span> https://bao.example.net/ &lt;span class="o">||&lt;/span> &lt;span class="nb">echo&lt;/span> blocked &lt;span class="c1"># blocked&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Another app&amp;rsquo;s name on 8443 gives a 404, because no other router listens there. Port 443 stays blocked.&lt;/p>
&lt;p>The runner needs the name &lt;code>bao.example.net&lt;/code> to resolve to the tunnel address. It does, because the public record points there (see the OpenBao post).&lt;/p>
&lt;h2 id="step-4-the-workflow">Step 4: the workflow&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="nt">env&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">DEPLOY_HOST&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">vps2.example.org&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">DEPLOY_USER&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">svc&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">BAO_ADDR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">https://bao.example.net:8443&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># The host CA: the web host shows a certificate signed by it, no pinned host key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">HOST_CA&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;ssh-ed25519 AAAA...host-ca-public-key...&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">jobs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># test and build jobs as before ...&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">deploy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">runs-on&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ubuntu-latest&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">needs&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">build&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">if&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">github.ref == &amp;#39;refs/heads/main&amp;#39;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># The job&amp;#39;s ID token (repository, ref) is its login at OpenBao&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">enable-openid-connect&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">steps&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Trigger podman auto-update&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">run&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> dir=$(mktemp -d) &amp;amp;&amp;amp; trap &amp;#39;rm -rf &amp;#34;$dir&amp;#34;&amp;#39; EXIT
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> # 1. the job&amp;#39;s ID token, for Bao as the audience
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> jwt=$(curl -fsS -H &amp;#34;Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN&amp;#34; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> &amp;#34;$ACTIONS_ID_TOKEN_REQUEST_URL&amp;amp;audience=https://bao.example.net&amp;#34; | jq -er .value)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> echo &amp;#34;::add-mask::$jwt&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> # 2. log in to Bao with it
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> token=$(jq -n --arg jwt &amp;#34;$jwt&amp;#34; &amp;#39;{role: &amp;#34;websites&amp;#34;, jwt: $jwt}&amp;#39; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> | curl -fsS -X POST -d @- &amp;#34;$BAO_ADDR/v1/auth/jwt/login&amp;#34; | jq -er .auth.client_token)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> echo &amp;#34;::add-mask::$token&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> # 3. a throwaway key, signed for five minutes
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> ssh-keygen -q -t ed25519 -N &amp;#39;&amp;#39; -C &amp;#34;$GITHUB_REPOSITORY&amp;#34; -f &amp;#34;$dir/key&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> jq -n --rawfile k &amp;#34;$dir/key.pub&amp;#34; &amp;#39;{public_key: $k}&amp;#39; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> | curl -fsS -X POST -H &amp;#34;X-Vault-Token: $token&amp;#34; -d @- &amp;#34;$BAO_ADDR/v1/ssh-user/sign/deploy-websites&amp;#34; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> | jq -er .data.signed_key &amp;gt; &amp;#34;$dir/key-cert.pub&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> # 4. the token is not needed any more
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> curl -fsS -X POST -H &amp;#34;X-Vault-Token: $token&amp;#34; &amp;#34;$BAO_ADDR/v1/auth/token/revoke-self&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> ssh-keygen -Lf &amp;#34;$dir/key-cert.pub&amp;#34; | sed -n &amp;#39;/Key ID/p;/Valid/p&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> # 5. the host is checked through the CA, then: podman auto-update, nothing else
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> echo &amp;#34;@cert-authority $DEPLOY_HOST $HOST_CA&amp;#34; &amp;gt; &amp;#34;$dir/known_hosts&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> ssh -i &amp;#34;$dir/key&amp;#34; -o IdentitiesOnly=yes -o UserKnownHostsFile=&amp;#34;$dir/known_hosts&amp;#34; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> -o StrictHostKeyChecking=yes &amp;#34;$DEPLOY_USER@$DEPLOY_HOST&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Some notes:&lt;/p>
&lt;ul>
&lt;li>&lt;code>enable-openid-connect: true&lt;/code> is set on the job. Only this job gets the two environment variables &lt;code>ACTIONS_ID_TOKEN_REQUEST_URL&lt;/code> and &lt;code>ACTIONS_ID_TOKEN_REQUEST_TOKEN&lt;/code>. The test and build jobs do not.&lt;/li>
&lt;li>&lt;code>::add-mask::&lt;/code> hides the JWT and the Bao token in the job log, in case a later command prints them.&lt;/li>
&lt;li>The private key exists only in the job&amp;rsquo;s temporary directory. The &lt;code>trap&lt;/code> removes it.&lt;/li>
&lt;li>The job uses &lt;code>curl&lt;/code> and &lt;code>jq&lt;/code> and needs no &lt;code>bao&lt;/code> CLI. The usual runner images have both.&lt;/li>
&lt;li>The pinned host key in the workflow is gone. After a reinstall of the web host, Ansible signs the new host key, and the workflow does not change.&lt;/li>
&lt;/ul>
&lt;p>The job log shows the certificate:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Key ID: &amp;#34;jwt-alice/blog-deploy-websites&amp;#34;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Valid: from 2026-11-15T16:51:02 to 2026-11-15T16:56:32
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>And the web host logs the deploy:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">sshd[51234]: Accepted publickey for svc from 10.99.0.11 port 40122 ssh2: ED25519-CERT SHA256:... ID jwt-alice/blog-deploy-websites (serial 7342...) CA ED25519 SHA256:...
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="test-what-must-fail">Test what must fail&lt;/h2>
&lt;ul>
&lt;li>Push the same workflow to another branch. Bao answers the login with an error about the bound claims, and the job fails at step 2.&lt;/li>
&lt;li>Open a pull request. Here the deploy job does not run at all (&lt;code>if: github.ref =&lt;/code> &amp;lsquo;refs/heads/main&amp;rsquo;&lt;code>). If somebody removes that line, Bao still refuses the =pull_request&lt;/code> subject.&lt;/li>
&lt;li>Try a shell with a deploy certificate. The host runs &lt;code>podman auto-update&lt;/code> and closes the connection.&lt;/li>
&lt;li>Try the certificate after six minutes. It has expired, and the host refuses it.&lt;/li>
&lt;li>Try the old deploy key. It must be refused: remove it from the host&amp;rsquo;s &lt;code>authorized_keys&lt;/code> and from Forgejo&amp;rsquo;s secrets.&lt;/li>
&lt;/ul>
&lt;p>For the branch test I pushed the workflow to a temporary branch &lt;code>oidc-test&lt;/code>, watched the job fail with the bound-claims error, and deleted the branch.&lt;/p>
&lt;h2 id="add-a-new-site">Add a new site&lt;/h2>
&lt;ol>
&lt;li>Add its &lt;code>sub&lt;/code> to the &lt;code>websites&lt;/code> role in the configuration script and run the script.&lt;/li>
&lt;li>Copy the deploy job into the new repository.&lt;/li>
&lt;/ol>
&lt;p>There is no secret to create and no key to distribute.&lt;/p>
&lt;p>Next, and last: &lt;a href="../../blog/ssh-05-plain-keys/">Removing the plain SSH keys&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://forgejo.org/docs/latest/user/actions/security-openid-connect/">Forgejo Actions: OpenID Connect&lt;/a>: &lt;code>enable-openid-connect&lt;/code> and the token request.&lt;/li>
&lt;li>&lt;a href="https://forgejo.org/docs/latest/user/actions/reference/">Forgejo Actions reference&lt;/a>: workflow syntax, contexts and variables.&lt;/li>
&lt;li>&lt;a href="https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/about-security-hardening-with-openid-connect">About security hardening with OpenID Connect (GitHub)&lt;/a>: the same idea on GitHub, with a good explanation of the claims.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/auth/jwt/">OpenBao JWT/OIDC auth method&lt;/a>: &lt;code>bound_claims&lt;/code>, &lt;code>bound_audiences&lt;/code> and &lt;code>user_claim&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/api-docs/secret/ssh/">OpenBao SSH secrets engine API&lt;/a>: &lt;code>default_critical_options&lt;/code> and the TTLs of a role.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd#AUTHORIZED_KEYS_FILE_FORMAT">sshd(8), authorized_keys format&lt;/a>: &lt;code>restrict&lt;/code> and &lt;code>command=&lt;/code>, which also work in a principals file.&lt;/li>
&lt;/ul></description></item><item><title>Hosts that trust the SSH CA</title><link>https://www.linux-lunatix.org/blog/ssh-03-hosts/</link><pubDate>Tue, 10 Nov 2026 19:20:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/ssh-03-hosts/</guid><description>&lt;p>&lt;em>Part 3 of the SSH series that starts with &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/em>&lt;/p>
&lt;p>Bao from &lt;a href="../../blog/ssh-02-openbao/">the previous post&lt;/a> can sign certificates. Now the hosts must trust them, and the laptop must use them. At the end of this post:&lt;/p>
&lt;ul>
&lt;li>every host accepts user certificates for the accounts and principals listed in the inventory,&lt;/li>
&lt;li>every host has a host certificate, renewed by Ansible,&lt;/li>
&lt;li>the laptop gets its certificates with one command and trusts the hosts without &lt;code>known_hosts&lt;/code> entries.&lt;/li>
&lt;/ul>
&lt;p>The plain keys stay in place during this step. They go away in the last post of the series, when everything else works.&lt;/p>
&lt;h2 id="the-inventory">The inventory&lt;/h2>
&lt;p>The CA public keys go into &lt;code>group_vars/all.yml&lt;/code>. They are public, and a rollout needs no Bao for them:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># group_vars/all.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># bao read -field=public_key ssh-user/config/ca&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_user_ca&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;ssh-ed25519 AAAA...user-ca-public-key... openbao ssh-user&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># bao read -field=public_key ssh-host/config/ca&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_host_ca&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;ssh-ed25519 AAAA...host-ca-public-key... openbao ssh-host&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">bao_addr&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">https://bao.example.net&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Each host says who may log in as which account:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># host_vars/vps1.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_principals&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fedora&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">alice]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">svc&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">alice]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">ansible]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># group_vars/websites.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="nt">ssh_principals&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">fedora&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">alice]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">svc&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">alice, &amp;#39;restrict,command=&amp;#34;podman auto-update&amp;#34; deploy-websites&amp;#39;]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">[&lt;/span>&lt;span class="l">ansible]&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A line may start with options, in the same syntax as in &lt;code>authorized_keys&lt;/code>. The host enforces them, whatever the certificate says. &lt;a href="../../blog/ssh-01-why-certificates/">The first post&lt;/a> explains why that matters.&lt;/p>
&lt;h2 id="the-base-role-trust-the-ca">The base role: trust the CA&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">&lt;span class="c"># roles/base/tasks/ssh_certs.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SSH CA directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/ca&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0755&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">User CA&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/ca/user_ca.pub&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_user_ca }}\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ssh_user_ca | length &amp;gt; 0&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Principals directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/principals&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">directory&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0755&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Principals per account&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/principals/{{ item.key }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.value | join(&amp;#39;\n&amp;#39;) }}\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_principals | dict2items }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.key }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Principals files in place&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.find&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">paths&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/principals&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">principals_files&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>&lt;span class="c"># Exclusive: an account no longer listed has no principals file, so no cert logs in as it&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Principals files of accounts no longer listed&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.file&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">path&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">absent&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ principals_files.files | rejectattr(&amp;#39;path&amp;#39;, &amp;#39;in&amp;#39;, ssh_principals.keys() | map(&amp;#39;regex_replace&amp;#39;, &amp;#39;^&amp;#39;, &amp;#39;/etc/ssh/principals/&amp;#39;) | list) }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">loop_control&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">label&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ item.path }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Revoked keys&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/revoked_keys&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ (ssh_revoked_keys | join(&amp;#39;\n&amp;#39;) ~ &amp;#39;\n&amp;#39;) if ssh_revoked_keys else &amp;#39;&amp;#39; }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The principals files need no reload. sshd reads them at each login.&lt;/p>
&lt;p>The &amp;ldquo;no longer listed&amp;rdquo; task makes the list exclusive. If I remove an account from &lt;code>ssh_principals&lt;/code>, its file goes away, and no certificate can log in as that account any more. A list that only adds is a list that only grows.&lt;/p>
&lt;p>&lt;code>RevokedKeys&lt;/code> accepts a plain file of public keys. A key in it is refused even with a valid certificate. That covers a lost device whose certificate is still valid for weeks (the last post of the series has the details). The file must exist, even empty. If &lt;code>RevokedKeys&lt;/code> points to a missing file, sshd refuses all public key logins.&lt;/p>
&lt;h3 id="sshd-s-config">sshd&amp;rsquo;s config&lt;/h3>
&lt;p>The hardening drop-in from &lt;a href="../../blog/infra-05-ansible-base/">the Ansible groundwork post&lt;/a> gets these lines:&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-jinja2" data-lang="jinja2">HostKey /etc/ssh/ssh_host_ed25519_key
{% if ssh_host_cert_installed | default(false) %}
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
{% endif %}
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
PubkeyAcceptedAlgorithms ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com
# The CA must sign with Ed25519, too
CASignatureAlgorithms ssh-ed25519
{% if ssh_user_ca %}
# Who may log in as an account is the account&amp;#39;s principals file (with options),
# not the certificate alone
TrustedUserCAKeys /etc/ssh/ca/user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/principals/%u
{% endif %}
RevokedKeys /etc/ssh/revoked_keys
&lt;/code>&lt;/pre>&lt;p>&lt;code>HostKeyAlgorithms&lt;/code> must list the certificate type, or sshd never offers the host certificate. The same applies to &lt;code>PubkeyAcceptedAlgorithms&lt;/code> for user certificates.&lt;/p>
&lt;p>The &lt;code>HostCertificate&lt;/code> line appears only once the certificate exists, so sshd never points to a missing file.&lt;/p>
&lt;h2 id="host-certificates-from-the-ansible-run">Host certificates from the Ansible run&lt;/h2>
&lt;p>The host&amp;rsquo;s key is on the host. The CA is in Bao. The control node has a Bao login. So the playbook reads the host&amp;rsquo;s public key, has it signed on the control node and installs the certificate on the host.&lt;/p>
&lt;p>First the role decides whether a new certificate is needed. A shell task describes the current one as JSON:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Host cert state&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.shell&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">|&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> set -o pipefail
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> c=/etc/ssh/ssh_host_ed25519_key-cert.pub
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> k=/etc/ssh/ssh_host_ed25519_key.pub
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> if [ ! -s $c ]; then echo &amp;#39;{&amp;#34;present&amp;#34;: false}&amp;#39;; exit 0; fi
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> info=$(ssh-keygen -Lf $c)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> to=$(sed -n &amp;#39;s/^ *Valid: from .* to \(.*\)$/\1/p&amp;#39; &amp;lt;&amp;lt;&amp;lt;&amp;#34;$info&amp;#34;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> left=$(( ($(date -d &amp;#34;$to&amp;#34; +%s 2&amp;gt;/dev/null || date +%s) - $(date +%s)) / 86400 ))
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> ca=$(sed -n &amp;#39;s/^ *Signing CA: [A-Z0-9-]* \(SHA256:[^ ]*\).*/\1/p&amp;#39; &amp;lt;&amp;lt;&amp;lt;&amp;#34;$info&amp;#34;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> cert_key=$(sed -n &amp;#39;s/^ *Public key: [A-Z0-9-]* \(SHA256:[^ ]*\).*/\1/p&amp;#39; &amp;lt;&amp;lt;&amp;lt;&amp;#34;$info&amp;#34;)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> principals=$(sed -n &amp;#39;/^ *Principals:/,/^ *Critical Options:/p&amp;#39; &amp;lt;&amp;lt;&amp;lt;&amp;#34;$info&amp;#34; | sed &amp;#39;1d;$d&amp;#39; | tr -d &amp;#39; &amp;#39; | sort | paste -sd,)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> printf &amp;#39;{&amp;#34;present&amp;#34;: true, &amp;#34;days_left&amp;#34;: %d, &amp;#34;ca&amp;#34;: &amp;#34;%s&amp;#34;, &amp;#34;want_ca&amp;#34;: &amp;#34;%s&amp;#34;, &amp;#34;cert_key&amp;#34;: &amp;#34;%s&amp;#34;, &amp;#34;host_key&amp;#34;: &amp;#34;%s&amp;#34;, &amp;#34;principals&amp;#34;: &amp;#34;%s&amp;#34;}\n&amp;#39; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> &amp;#34;$left&amp;#34; &amp;#34;$ca&amp;#34; &amp;#34;$(ssh-keygen -lf /dev/stdin &amp;lt;&amp;lt;&amp;lt;&amp;#34;$HOST_CA&amp;#34; | awk &amp;#39;{print $2}&amp;#39;)&amp;#34; \
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> &amp;#34;$cert_key&amp;#34; &amp;#34;$(ssh-keygen -lf $k | awk &amp;#39;{print $2}&amp;#39;)&amp;#34; &amp;#34;$principals&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">args&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">executable&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/bin/bash&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">HOST_CA&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ ssh_host_ca }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">host_cert_state&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">check_mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Host cert needed?&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.set_fact&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">host_cert_sign&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">&amp;gt;-&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> {{ not state.present
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> or state.days_left &amp;lt; 60
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> or state.ca != state.want_ca
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> or state.cert_key != state.host_key
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> or state.principals != (ssh_host_cert_principals | sort | join(&amp;#39;,&amp;#39;)) }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">vars&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">state&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ host_cert_state.stdout | from_json }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A new certificate is signed when there is none, when less than 60 days are left, or when it is from another CA (after a CA rotation). It is also signed when it is for another key (after a reinstall) or for other names. Otherwise the run leaves it alone and stays at &lt;code>changed=0&lt;/code>.&lt;/p>
&lt;p>Then the signing:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Host cert&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">host_cert_sign&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">block&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Host public key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.slurp&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/ssh_host_ed25519_key.pub&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">host_pubkey&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="c"># On the control node, with its Bao login (the admin policy may sign host certs)&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Sign the host key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.command&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">argv&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">bao&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">write&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- -&lt;span class="l">field=signed_key&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">ssh-host/sign/hosts&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">cert_type=host&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">valid_principals={{ ssh_host_cert_principals | join(&amp;#39;,&amp;#39;) }}&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="l">public_key=-&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">stdin&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ host_pubkey.content | b64decode }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">environment&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">BAO_ADDR&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ bao_addr }}&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">delegate_to&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">localhost&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">become&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">false&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">register&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">host_cert_signed&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">changed_when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">true&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">not ansible_check_mode&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Install the host cert&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.copy&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/ssh_host_ed25519_key-cert.pub&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">content&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;{{ host_cert_signed.stdout }}\n&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">when&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">not ansible_check_mode&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">rescue&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="kc">No&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Bao login on the control node?&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.fail&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">msg&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="p">&amp;gt;-&lt;/span>&lt;span class="sd">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> Signing the host cert failed ({{ host_cert_signed.stderr | default(&amp;#39;&amp;#39;) | trim }}).
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> Log in on the control node first (bao login -no-print -method=oidc),
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="sd"> or skip host certs with -e ssh_host_cert=false.&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>delegate_to: localhost&lt;/code> with &lt;code>become: false&lt;/code> runs &lt;code>bao&lt;/code> on the control node, as me, with my Bao token. The host needs no access to Bao at all. The &lt;code>rescue&lt;/code> block turns Bao&amp;rsquo;s error into a message that says what to do.&lt;/p>
&lt;p>&lt;code>ssh_host_cert_principals&lt;/code> defaults to &lt;code>[ansible_host]&lt;/code>, the FQDN that I use to connect. A certificate is valid only for the names in it. If you also connect by a short name or an IP address, either put those into the certificate or make the client use the FQDN (see &lt;code>HostKeyAlias&lt;/code> below).&lt;/p>
&lt;h3 id="order-matters-flush-the-handlers">Order matters: flush the handlers&lt;/h3>
&lt;p>In the role, the CA trust and the sshd config come first. Then a &lt;code>meta: flush_handlers&lt;/code> reloads sshd at once, before anything else touches SSH access. In the last post of the series the next tasks remove the plain keys. A host&amp;rsquo;s first run must not take away the key it came in with while certificates do not work there yet.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-yaml" data-lang="yaml">&lt;span class="line">&lt;span class="cl">- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SSH certificates&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.include_tasks&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">ssh_certs.yml&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">SSH hardening&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.template&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">src&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd-hardening.conf.j2&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">dest&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">/etc/ssh/sshd_config.d/00-hardening.conf&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">mode&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="s2">&amp;#34;0644&amp;#34;&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">validate&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd -t -f %s&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">notify&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">Reload sshd&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w">&lt;/span>- &lt;span class="nt">name&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">sshd reloaded before the keys change&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="w"> &lt;/span>&lt;span class="nt">ansible.builtin.meta&lt;/span>&lt;span class="p">:&lt;/span>&lt;span class="w"> &lt;/span>&lt;span class="l">flush_handlers&lt;/span>&lt;span class="w">
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="the-laptop">The laptop&lt;/h2>
&lt;h3 id="ssh-cert-certificates-for-the-day">ssh-cert: certificates for the day&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># ssh-cert: SSH certificates from OpenBao. Logs in through Forgejo (OIDC) when there&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># is no valid Bao token, then signs a public key:&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">#&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-cert both of these (the daily call: logins and Ansible)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-cert admin ~/.ssh/id_ed25519 -&amp;gt; ~/.ssh/id_ed25519-cert.pub (16 h)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-cert ansible ~/.ssh/ansible_ed25519 -&amp;gt; ~/.ssh/ansible_ed25519-cert.pub (16 h)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ssh-cert device &amp;lt;key.pub&amp;gt; prints a 30-day cert for a device without Bao&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="nv">https&lt;/span>&lt;span class="p">://bao.example.net&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sign&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> &lt;span class="c1"># role public-key output (- for stdout)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> &lt;span class="nv">role&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nv">$1&lt;/span> &lt;span class="nv">pub&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nv">$2&lt;/span> &lt;span class="nv">out&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nv">$3&lt;/span> cert
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">cert&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>bao write -field&lt;span class="o">=&lt;/span>signed_key &lt;span class="s2">&amp;#34;ssh-user/sign/&lt;/span>&lt;span class="nv">$role&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="nv">public_key&lt;/span>&lt;span class="o">=&lt;/span>- &amp;lt; &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$pub&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">if&lt;/span> &lt;span class="o">[&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$out&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">=&lt;/span> - &lt;span class="o">]&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">printf&lt;/span> &lt;span class="s1">&amp;#39;%s\n&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$cert&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">else&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">printf&lt;/span> &lt;span class="s1">&amp;#39;%s\n&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$cert&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt; &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$out&lt;/span>&lt;span class="s2">.tmp&amp;#34;&lt;/span> &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> mv &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$out&lt;/span>&lt;span class="s2">.tmp&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$out&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">printf&lt;/span> &lt;span class="s1">&amp;#39;%s\n&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$cert&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="p">|&lt;/span> ssh-keygen -Lf /dev/stdin &lt;span class="p">|&lt;/span> sed -n &lt;span class="s1">&amp;#39;s/^ *\(Valid: .*\)/\1/p&amp;#39;&lt;/span> &lt;span class="p">|&lt;/span> sed &lt;span class="s2">&amp;#34;s|^|&lt;/span>&lt;span class="nv">$role&lt;/span>&lt;span class="s2">: |&amp;#34;&lt;/span> &amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">2&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">what&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">1&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="nv">all&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">case&lt;/span> &lt;span class="nv">$what&lt;/span> in
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> all&lt;span class="p">|&lt;/span>admin&lt;span class="p">|&lt;/span>ansible&lt;span class="p">|&lt;/span>device&lt;span class="o">)&lt;/span> &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> *&lt;span class="o">)&lt;/span> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;usage: ssh-cert [admin|ansible|device &amp;lt;key.pub&amp;gt;]&amp;#34;&lt;/span> &amp;gt;&lt;span class="p">&amp;amp;&lt;/span>2&lt;span class="p">;&lt;/span> &lt;span class="nb">exit&lt;/span> &lt;span class="m">2&lt;/span> &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">esac&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao token lookup &amp;gt;/dev/null 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">1&lt;/span> &lt;span class="o">||&lt;/span> bao login -no-print -method&lt;span class="o">=&lt;/span>oidc
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">case&lt;/span> &lt;span class="nv">$what&lt;/span> in
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> all&lt;span class="o">)&lt;/span> sign admin ~/.ssh/id_ed25519.pub ~/.ssh/id_ed25519-cert.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> sign ansible ~/.ssh/ansible_ed25519.pub ~/.ssh/ansible_ed25519-cert.pub &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> admin&lt;span class="o">)&lt;/span> sign admin ~/.ssh/id_ed25519.pub ~/.ssh/id_ed25519-cert.pub &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> ansible&lt;span class="o">)&lt;/span> sign ansible ~/.ssh/ansible_ed25519.pub ~/.ssh/ansible_ed25519-cert.pub &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> device&lt;span class="o">)&lt;/span> sign device &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">2&lt;/span>&lt;span class="p">:?public key file&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> - &lt;span class="p">;;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">esac&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo dnf install openbao &lt;span class="c1"># the bao CLI&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ln -s ~/ops/client/bin/ssh-cert ~/.local/bin/
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-cert
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># admin: Valid: from 2026-11-10T08:01:00 to 2026-11-11T00:02:00&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ansible: Valid: from 2026-11-10T08:01:00 to 2026-11-11T00:02:00&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The Bao token from the OIDC login is also valid for 16 hours. Run &lt;code>ssh-cert&lt;/code> again during the day, and it signs without the browser.&lt;/p>
&lt;p>ssh picks up &lt;code>id_ed25519-cert.pub&lt;/code> next to &lt;code>id_ed25519&lt;/code> by itself. Nothing else changes in the SSH config. With &lt;code>IdentitiesOnly yes&lt;/code>, ssh offers the plain key first and the certificate second. That is two of the three attempts that &lt;code>MaxAuthTries 3&lt;/code> allows, so do not add more keys to the same host entry.&lt;/p>
&lt;p>Ansible uses its own key, &lt;code>~/.ssh/ansible_ed25519&lt;/code>, and its certificate is picked up the same way.&lt;/p>
&lt;h3 id="one-line-instead-of-known-hosts">One line instead of known_hosts&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl"># ~/.ssh/known_hosts.ca
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">@cert-authority *.example.org,*.example.net ssh-ed25519 AAAA...host-ca-public-key... openbao ssh-host
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl"># ~/.ssh/config
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">Host *
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> # The first file is where ssh writes new keys
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> UserKnownHostsFile ~/.ssh/known_hosts ~/.ssh/known_hosts.ca
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The order matters. ssh writes new host keys into the first file of the list. Keep your normal &lt;code>known_hosts&lt;/code> first, so that ssh never writes into the CA file.&lt;/p>
&lt;p>Now remove the old entries of your managed hosts from &lt;code>known_hosts&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">cp ~/.ssh/known_hosts ~/.ssh/known_hosts.bak
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">for&lt;/span> h in vps1.example.org vps2.example.org home1.example.net&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span> ssh-keygen -R &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$h&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>What happens with other hosts? Nothing changes for them. A server at a friend&amp;rsquo;s place that is not in &lt;code>*.example.org&lt;/code> has no certificate from my CA. ssh asks about its key once, as before, and writes it into &lt;code>~/.ssh/known_hosts&lt;/code>. And the plain key is still offered to hosts that do not trust my user CA.&lt;/p>
&lt;p>An expired or invalid host certificate does not lock you out either. ssh falls back to the plain host key, finds no entry for it and asks, as for a new host. In that situation, do not just say yes: find out why the certificate is not valid.&lt;/p>
&lt;h3 id="short-names-and-hostkeyalias">Short names and HostKeyAlias&lt;/h3>
&lt;p>A host certificate is valid for the names in it. If you connect with &lt;code>ssh v2&lt;/code> or by IP address, the name does not match. In my SSH config every alias of a host sets the FQDN as the name to check:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">Host v2 vps2
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> HostName vps2.example.org
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> HostKeyAlias vps2.example.org
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>HostKeyAlias&lt;/code> also helps when you reach a host over different paths: at home by its LAN address, away through a jump host. The check is always against the same name.&lt;/p>
&lt;h2 id="roll-it-out">Roll it out&lt;/h2>
&lt;ol>
&lt;li>Run Bao&amp;rsquo;s configuration (previous post), and add both CA keys to &lt;code>group_vars/all.yml&lt;/code>.&lt;/li>
&lt;li>Pick one host. Keep a second SSH session open to it, as an emergency way back in:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh -M -S ~/.ssh/cm/standby-vps1 -fN fedora@vps1.example.org
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>&lt;code>bao login -no-print -method=oidc&lt;/code>, then &lt;code>ansible-playbook site.yml -l vps1 -t base --check&lt;/code>, then the same without &lt;code>--check&lt;/code>.&lt;/li>
&lt;li>Test with a fresh connection, not a reused one:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh -o &lt;span class="nv">ControlMaster&lt;/span>&lt;span class="o">=&lt;/span>no -o &lt;span class="nv">ControlPath&lt;/span>&lt;span class="o">=&lt;/span>none -v fedora@vps1.example.org &lt;span class="nb">true&lt;/span> 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">1&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> grep -E &lt;span class="s1">&amp;#39;Server host certificate|Authenticated&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>Expect &amp;ldquo;Server host certificate: &lt;a href="mailto:ssh-ed25519-cert-v01@openssh.com">ssh-ed25519-cert-v01@openssh.com&lt;/a> &amp;hellip; ID &amp;hellip;-host&amp;rdquo; and &amp;ldquo;Authenticated &amp;hellip; using publickey with ED25519-CERT&amp;rdquo;.&lt;/li>
&lt;li>On the host, the log says which certificate logged in:
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo journalctl -u sshd -n &lt;span class="m">20&lt;/span> &lt;span class="p">|&lt;/span> grep &lt;span class="s1">&amp;#39;ED25519-CERT&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Accepted publickey for fedora from 198.51.100.7 ... ED25519-CERT SHA256:... ID oidc-alice-admin (serial ...) CA ED25519 SHA256:...&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/li>
&lt;li>Close the standby session (&lt;code>ssh -S ~/.ssh/cm/standby-vps1 -O exit x&lt;/code>), then the next host.&lt;/li>
&lt;/ol>
&lt;h2 id="negative-tests">Negative tests&lt;/h2>
&lt;p>Test that the host refuses what it should refuse:&lt;/p>
&lt;ul>
&lt;li>A certificate for &lt;code>alice&lt;/code> as &lt;code>ansible&lt;/code> or &lt;code>root&lt;/code>: refused (not in the principals file).&lt;/li>
&lt;li>An expired certificate: sign one with a short TTL in a test role, wait, try. Refused.&lt;/li>
&lt;li>A key from &lt;code>ssh_revoked_keys&lt;/code>: refused, even with a valid certificate.&lt;/li>
&lt;li>The host under a name that is not in its certificate, with only the CA line in &lt;code>known_hosts&lt;/code>: ssh asks or refuses.&lt;/li>
&lt;/ul>
&lt;p>Do these tests sparingly from your own IP. fail2ban counts some of these failures, for example a login as an account that &lt;code>AllowUsers&lt;/code> does not list. Three of them in ten minutes ban you for an hour. I learned that the hard way, more in the last post.&lt;/p>
&lt;p>Next: &lt;a href="../../blog/ssh-04-ci-oidc/">CI deploys without a deploy key&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://man.openbsd.org/sshd_config">sshd_config(5)&lt;/a>: &lt;code>TrustedUserCAKeys&lt;/code>, &lt;code>AuthorizedPrincipalsFile&lt;/code>, &lt;code>HostCertificate&lt;/code> and &lt;code>RevokedKeys&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/ssh_config">ssh_config(5)&lt;/a>: &lt;code>CertificateFile&lt;/code>, &lt;code>HostKeyAlias&lt;/code>, &lt;code>UserKnownHostsFile&lt;/code> and &lt;code>ControlMaster&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd#SSH_KNOWN_HOSTS_FILE_FORMAT">sshd(8), known_hosts format&lt;/a>: the &lt;code>@cert-authority&lt;/code> line on the client.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/ssh-keygen#CERTIFICATES">ssh-keygen(1), section CERTIFICATES&lt;/a>: &lt;code>-L&lt;/code> to inspect a certificate, &lt;code>-h&lt;/code> for host certificates.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/secrets/ssh/signed-ssh-certificates/">OpenBao signed SSH certificates&lt;/a>: the section on host key signing.&lt;/li>
&lt;li>&lt;a href="https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_delegation.html">Ansible delegation&lt;/a>: &lt;code>delegate_to: localhost&lt;/code> and &lt;code>run_once&lt;/code>.&lt;/li>
&lt;/ul></description></item><item><title>OpenBao rootless on Podman</title><link>https://www.linux-lunatix.org/blog/ssh-02-openbao/</link><pubDate>Thu, 05 Nov 2026 20:35:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/ssh-02-openbao/</guid><description>&lt;p>&lt;em>Part 2 of the SSH series that starts with &lt;a href="../../blog/ssh-01-why-certificates/">Why SSH certificates&lt;/a>.&lt;/em>&lt;/p>
&lt;p>At the end of this post OpenBao runs on the home server as a rootless container. It has two SSH CAs (users and hosts), roles for each kind of certificate, and a login through Forgejo in the browser. Its configuration is one script in Git, and a nightly Raft snapshot goes into the backup.&lt;/p>
&lt;h2 id="where-it-runs-and-who-can-reach-it">Where it runs, and who can reach it&lt;/h2>
&lt;p>Bao runs on my home server, &lt;code>home1&lt;/code>. It is reachable from the LAN and through my WireGuard tunnel, never from the internet. The home router does not forward port 443 to it.&lt;/p>
&lt;p>Two DNS records make the same name work in both places:&lt;/p>
&lt;ul>
&lt;li>On the LAN, the local DNS (Pi-hole) answers &lt;code>bao.example.net&lt;/code> with the LAN address, &lt;code>192.168.50.5&lt;/code>.&lt;/li>
&lt;li>The public record of &lt;code>bao.example.net&lt;/code> points to the WireGuard address of the home server, &lt;code>10.99.0.1&lt;/code>. That address is private and only works inside the tunnel. Away from home, I start the tunnel first.&lt;/li>
&lt;/ul>
&lt;p>Traefik gets the certificate with the DNS challenge, so Let&amp;rsquo;s Encrypt never needs to connect to the host. TLS ends at Traefik.&lt;/p>
&lt;p>Hosts do not need Bao to check a certificate. They only need the CA&amp;rsquo;s public key, which they have as a file. Bao must be reachable only to &lt;em>get&lt;/em> a certificate: from my laptop, from the Ansible control node and from the CI runner. If Bao is down, logins with existing certificates keep working.&lt;/p>
&lt;h2 id="step-1-the-quadlet">Step 1: the quadlet&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ~svc/.config/containers/systemd/openbao.container&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Unit]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Description&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">OpenBao&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Container]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ContainerName&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">openbao&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Minor version pinned, patch releases through auto-update&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Image&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">quay.io/openbao/openbao:2.7&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">AutoUpdate&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">registry&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.enable=true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.openbao.rule=Host(`bao.example.net`)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.openbao.entrypoints=https&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.openbao.tls.certresolver=default&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.routers.openbao.middlewares=secureHeaders@file&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Label&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">traefik.http.services.openbao.loadbalancer.server.port=8200&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Exec&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">bao server -config=/openbao/config/openbao.hcl&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># The image&amp;#39;s user openbao (100:1000) is svc on the host: no chown needed&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">UserNS&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">keep-id:uid=100,gid=1000&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Environment&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">SKIP_CHOWN=1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">%h/.config/openbao/openbao.hcl:/openbao/config/openbao.hcl:ro,z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Volume&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">openbao-data.volume:/openbao/file:Z&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">ReadOnly&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Tmpfs&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">/tmp&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">DropCapability&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">ALL&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">NoNewPrivileges&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Network&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">proxy.network:ip=10.89.0.14&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Service]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">Restart&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">always&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Install]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="na">WantedBy&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="s">default.target&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-ini" data-lang="ini">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># openbao-data.volume&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">[Volume]&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Some notes:&lt;/p>
&lt;ul>
&lt;li>&lt;code>UserNS=keep-id:uid=100,gid=1000&lt;/code> maps the image&amp;rsquo;s &lt;code>openbao&lt;/code> user to &lt;code>svc&lt;/code> on the host. The files in the volume then belong to &lt;code>svc&lt;/code>, and the entrypoint does not need to &lt;code>chown&lt;/code> anything. &lt;code>SKIP_CHOWN=1&lt;/code> tells it not to try.&lt;/li>
&lt;li>&lt;code>ReadOnly=true&lt;/code>, &lt;code>DropCapability=ALL&lt;/code> and &lt;code>NoNewPrivileges=true&lt;/code>: Bao needs to write only into its volume and &lt;code>/tmp&lt;/code>.&lt;/li>
&lt;li>Pin the minor version. I want to read the release notes before a minor upgrade of the thing that holds my CAs. Patch releases come through &lt;code>podman auto-update&lt;/code>.&lt;/li>
&lt;/ul>
&lt;h2 id="step-2-the-server-config">Step 2: the server config&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-hcl" data-lang="hcl">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># ~svc/.config/openbao/openbao.hcl
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">&lt;/span>&lt;span class="n">ui&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="kt">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">api_addr&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;https://bao.example.net&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n">cluster_addr&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;http://127.0.0.1:8201&amp;#34;&lt;/span>&lt;span class="c1">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Integrated storage: mlock is not wanted (memory-mapped database)
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">&lt;/span>&lt;span class="n">disable_mlock&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="kt">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">storage&lt;/span> &lt;span class="s2">&amp;#34;raft&amp;#34;&lt;/span> {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n"> path&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;/openbao/file&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n"> node_id&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;home1&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">}
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">listener&lt;/span> &lt;span class="s2">&amp;#34;tcp&amp;#34;&lt;/span> {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n"> address&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;0.0.0.0:8200&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n"> tls_disable&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="kt">true&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">}&lt;span class="c1">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Every request, with secret values HMACed, to stdout: journald through Podman
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1">&lt;/span>&lt;span class="k">audit&lt;/span> &lt;span class="s2">&amp;#34;file&amp;#34; &amp;#34;stdout&amp;#34;&lt;/span> {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="k">options&lt;/span> {
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="n"> file_path&lt;/span> &lt;span class="o">=&lt;/span> &lt;span class="s2">&amp;#34;stdout&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> }
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Integrated storage (Raft) is a single directory and needs no external database. A single node is a valid Raft cluster.&lt;/p>
&lt;p>&lt;code>tls_disable&lt;/code> looks scary, but the listener is reachable only on the Podman network, from Traefik. TLS ends there.&lt;/p>
&lt;p>The audit device is declared in the config file, so it exists from the very first start. The log is in the journal with everything else.&lt;/p>
&lt;h2 id="step-3-initialise">Step 3: initialise&lt;/h2>
&lt;p>Start it, then initialise it once:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">systemctl --user daemon-reload &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> systemctl --user start openbao
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="o">=&lt;/span>https://bao.example.net
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao operator init -key-shares&lt;span class="o">=&lt;/span>&lt;span class="m">1&lt;/span> -key-threshold&lt;span class="o">=&lt;/span>&lt;span class="m">1&lt;/span> -format&lt;span class="o">=&lt;/span>json &amp;gt; /dev/shm/init.json
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>init&lt;/code> prints the unseal key and the initial root token. They go straight into pass, and the file is deleted:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">jq -r &lt;span class="s1">&amp;#39;.unseal_keys_b64[0]&amp;#39;&lt;/span> /dev/shm/init.json &lt;span class="p">|&lt;/span> pass insert -m infra/home1/openbao-unseal-key
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">jq -r &lt;span class="s1">&amp;#39;.root_token&amp;#39;&lt;/span> /dev/shm/init.json &lt;span class="p">|&lt;/span> pass insert -m infra/home1/openbao-root-token
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">shred -u /dev/shm/init.json
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>One key share is enough for one person. Shamir splitting into several shares makes sense when several people must agree to unseal.&lt;/p>
&lt;p>I use &lt;code>/dev/shm&lt;/code> so that the file is in memory and never on disk.&lt;/p>
&lt;h2 id="step-4-unseal-and-why-not-automatically">Step 4: unseal, and why not automatically&lt;/h2>
&lt;p>Bao encrypts everything in its storage. After every start it is sealed: it has the data but not the key. Somebody must give it the unseal key.&lt;/p>
&lt;p>I wanted auto-unseal with the home server&amp;rsquo;s TPM. The plan: encrypt a static seal key with &lt;code>systemd-creds --with-key=tpm2&lt;/code> and give it to the container with &lt;code>LoadCredentialEncrypted=&lt;/code>. Bao would then unseal itself after a reboot, and a stolen disk would be useless without the board.&lt;/p>
&lt;p>It did not work. The board has a TPM 1.2, and &lt;code>systemd-creds&lt;/code> supports only TPM 2.0. So I unseal by hand, from my laptop, with the key from pass:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="cp">#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="cp">&lt;/span>&lt;span class="c1"># bao-unseal: run on the laptop, at home or through the tunnel&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nv">addr&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="nv">https&lt;/span>&lt;span class="p">://bao.example.net&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sealed&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> s
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">s&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>curl -fsS -m &lt;span class="m">15&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$addr&lt;/span>&lt;span class="s2">/v1/sys/seal-status&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span> &lt;span class="o">||&lt;/span> &lt;span class="o">{&lt;/span> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;OpenBao not reachable at &lt;/span>&lt;span class="nv">$addr&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt;&lt;span class="p">&amp;amp;&lt;/span>2&lt;span class="p">;&lt;/span> &lt;span class="nb">exit&lt;/span> 1&lt;span class="p">;&lt;/span> &lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> python3 -c &lt;span class="s1">&amp;#39;import json,sys; sys.exit(0 if json.loads(sys.argv[1])[&amp;#34;sealed&amp;#34;] else 1)&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$s&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> ! sealed&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;OpenBao is not sealed&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">exit&lt;/span> &lt;span class="m">0&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># the key goes to curl on stdin, never onto a command line&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">pass show infra/home1/openbao-unseal-key &lt;span class="p">|&lt;/span> head -1 &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.dumps({&amp;#34;key&amp;#34;: sys.stdin.read().strip()}))&amp;#39;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> curl -fsS -m &lt;span class="m">15&lt;/span> -X PUT -d @- -o /dev/null &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$addr&lt;/span>&lt;span class="s2">/v1/sys/unseal&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> sealed&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;still sealed&amp;#34;&lt;/span> &amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">2&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">exit&lt;/span> &lt;span class="m">1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;OpenBao unsealed&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The key never appears in &lt;code>ps&lt;/code> or in the shell history. A Prometheus blackbox probe watches &lt;code>/v1/sys/health&lt;/code>, which answers 503 while Bao is sealed. An alert tells me after five minutes.&lt;/p>
&lt;p>This is less work than it sounds. The home server reboots only when I reboot it. But remember that a &lt;code>podman auto-update&lt;/code> of the image, or any change to the quadlet, restarts the container, and the restart seals it again.&lt;/p>
&lt;h2 id="step-5-the-configuration-as-a-script">Step 5: the configuration as a script&lt;/h2>
&lt;p>Everything inside Bao (mounts, roles, policies, logins) is set by one script in the repository. It is idempotent: I change the script and run it again. There is no click in the web UI that I would forget.&lt;/p>
&lt;p>The script needs a token with full rights. The first run uses the root token from pass. Later runs use my &lt;code>userpass&lt;/code> login, which the script creates:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nv">BAO_TOKEN&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>pass show infra/home1/openbao-root-token&lt;span class="k">)&lt;/span> ./bao-config.sh &lt;span class="c1"># first run&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao login -no-print -method&lt;span class="o">=&lt;/span>userpass &lt;span class="nv">username&lt;/span>&lt;span class="o">=&lt;/span>alice &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> ./bao-config.sh &lt;span class="c1"># later&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Always use &lt;code>-no-print&lt;/code>. Without it, &lt;code>bao login&lt;/code> prints the token on the screen.&lt;/p>
&lt;p>After the first run, revoke the root token. A new one can be made from the unseal key with &lt;code>bao operator generate-root&lt;/code> when you need it.&lt;/p>
&lt;h3 id="the-ssh-cas">The SSH CAs&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">set&lt;/span> -euo pipefail
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">export&lt;/span> &lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">BAO_ADDR&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="nv">https&lt;/span>&lt;span class="p">://bao.example.net&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">mounted&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> bao &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$1&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> list -format&lt;span class="o">=&lt;/span>json &lt;span class="p">|&lt;/span> python3 -c &lt;span class="s1">&amp;#39;import json,sys; sys.exit(0 if sys.argv[1] + &amp;#34;/&amp;#34; in json.load(sys.stdin) else 1)&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$2&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">secrets&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> mounted secrets &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$2&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">||&lt;/span> bao secrets &lt;span class="nb">enable&lt;/span> -path&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$2&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$1&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">auth&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> mounted auth &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$2&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="o">||&lt;/span> bao auth &lt;span class="nb">enable&lt;/span> -path&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$2&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$1&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="p">;&lt;/span> &lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Two SSH CAs, keys generated inside Bao (ed25519)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">for&lt;/span> m in ssh-user ssh-host&lt;span class="p">;&lt;/span> &lt;span class="k">do&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> secrets ssh &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$m&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> bao &lt;span class="nb">read&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$m&lt;/span>&lt;span class="s2">/config/ca&amp;#34;&lt;/span> &amp;gt;/dev/null 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">1&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="o">||&lt;/span> bao write &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$m&lt;/span>&lt;span class="s2">/config/ca&amp;#34;&lt;/span> &lt;span class="nv">generate_signing_key&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="nb">true&lt;/span> &lt;span class="nv">key_type&lt;/span>&lt;span class="o">=&lt;/span>ed25519 &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">done&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>generate_signing_key=true&lt;/code> creates the key inside Bao. It never leaves it. &lt;code>bao read ssh-user/config/ca&lt;/code> shows the public key, which is what the hosts need.&lt;/p>
&lt;p>The &lt;code>|| bao write&lt;/code> part makes it idempotent. A second run would otherwise replace the CA key, and every certificate would be invalid.&lt;/p>
&lt;h3 id="the-user-roles">The user roles&lt;/h3>
&lt;p>A small function writes a role. The JSON body is easier to read than a long list of &lt;code>key=value&lt;/code> arguments:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># Key ID: who asked (token display name) and with which role; it shows in sshd&amp;#39;s log&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">user_role&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span> &lt;span class="c1"># name principal ttl extensions [critical options as JSON]&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> exts &lt;span class="nv">crit&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">5&lt;/span>&lt;span class="k">:-&lt;/span>&lt;span class="s1">&amp;#39;{}&amp;#39;&lt;/span>&lt;span class="si">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">exts&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.dumps({e: &amp;#34;&amp;#34; for e in sys.argv[1].split(&amp;#34;,&amp;#34;) if e}))&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$4&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> bao write &lt;span class="s2">&amp;#34;ssh-user/roles/&lt;/span>&lt;span class="nv">$1&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> - &amp;gt;/dev/null &lt;span class="s">&amp;lt;&amp;lt;JSON
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">{
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;key_type&amp;#34;: &amp;#34;ca&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allow_user_certificates&amp;#34;: true,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_users&amp;#34;: &amp;#34;$2&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;default_user&amp;#34;: &amp;#34;$2&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;ttl&amp;#34;: &amp;#34;$3&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;max_ttl&amp;#34;: &amp;#34;$3&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_extensions&amp;#34;: &amp;#34;$4&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;default_extensions&amp;#34;: $exts,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_critical_options&amp;#34;: &amp;#34;$(python3 -c &amp;#39;import json,sys; print(&amp;#34;,&amp;#34;.join(json.loads(sys.argv[1])))&amp;#39; &amp;#34;$crit&amp;#34;)&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;default_critical_options&amp;#34;: $crit,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_user_key_lengths&amp;#34;: {&amp;#34;ed25519&amp;#34;: [0], &amp;#34;sk-ed25519&amp;#34;: [0]},
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;key_id_format&amp;#34;: &amp;#34;{{token_display_name}}-{{role_name}}&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">JSON&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">user_role admin alice 16h permit-pty,permit-port-forwarding
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">user_role device alice 720h permit-pty,permit-port-forwarding
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">user_role ansible ansible 16h permit-pty
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">user_role deploy-websites deploy-websites 5m &lt;span class="s2">&amp;#34;&amp;#34;&lt;/span> &lt;span class="s1">&amp;#39;{&amp;#34;force-command&amp;#34;: &amp;#34;podman auto-update&amp;#34;}&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>What each field does:&lt;/p>
&lt;ul>
&lt;li>&lt;code>allowed_users&lt;/code> and &lt;code>default_user&lt;/code>: the role signs exactly one principal. A caller cannot ask for &lt;code>root&lt;/code>.&lt;/li>
&lt;li>&lt;code>ttl&lt;/code> equal to &lt;code>max_ttl&lt;/code>: a caller cannot ask for a longer validity.&lt;/li>
&lt;li>&lt;code>allowed_extensions&lt;/code>: only these. No agent forwarding, no X11.&lt;/li>
&lt;li>&lt;code>allowed_user_key_lengths&lt;/code>: only Ed25519 keys, also on a FIDO2 token (&lt;code>sk-ed25519&lt;/code>).&lt;/li>
&lt;li>&lt;code>key_id_format&lt;/code>: the key ID in the certificate says who asked, for example &lt;code>oidc-alice-admin&lt;/code>. sshd logs it with every login. This is how I see in a host&amp;rsquo;s log which role signed the certificate that just logged in.&lt;/li>
&lt;/ul>
&lt;h3 id="the-host-role">The host role&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">bao write ssh-host/roles/hosts - &amp;gt;/dev/null &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;JSON&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">{
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;key_type&amp;#34;: &amp;#34;ca&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allow_host_certificates&amp;#34;: true,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_domains&amp;#34;: &amp;#34;example.org,example.net&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allow_subdomains&amp;#34;: true,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;ttl&amp;#34;: &amp;#34;8760h&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;max_ttl&amp;#34;: &amp;#34;8760h&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_user_key_lengths&amp;#34;: {&amp;#34;ed25519&amp;#34;: [0]},
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;key_id_format&amp;#34;: &amp;#34;{{token_display_name}}-host&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">JSON&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Host certificates are valid for one year. The Ansible run renews them when less than 60 days are left (next post).&lt;/p>
&lt;h3 id="policies">Policies&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">bao policy write admin - &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;HCL&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"># Daily use (OIDC login): sign user and host certificates
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;ssh-user/sign/admin&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;ssh-user/sign/device&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;ssh-user/sign/ansible&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;ssh-host/sign/hosts&amp;#34; { capabilities = [&amp;#34;update&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">HCL&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao policy write operator - &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;HCL&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"># Running Bao itself (userpass login): everything
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;*&amp;#34; { capabilities = [&amp;#34;create&amp;#34;, &amp;#34;read&amp;#34;, &amp;#34;update&amp;#34;, &amp;#34;patch&amp;#34;, &amp;#34;delete&amp;#34;, &amp;#34;list&amp;#34;, &amp;#34;sudo&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">HCL&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao policy write backup - &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;HCL&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">path &amp;#34;sys/storage/raft/snapshot&amp;#34; { capabilities = [&amp;#34;read&amp;#34;] }
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">HCL&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The daily login can sign and do nothing else. It cannot read the configuration, change roles or create tokens.&lt;/p>
&lt;h2 id="step-6-logins">Step 6: logins&lt;/h2>
&lt;h3 id="oidc-through-forgejo-for-every-day">OIDC through Forgejo, for every day&lt;/h3>
&lt;p>Forgejo is an OpenID Connect provider. In Forgejo, go to Settings, Applications, and create an OAuth2 application with two redirect URIs:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">http://localhost:8250/oidc/callback
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">https://bao.example.net/ui/vault/auth/oidc/oidc/callback
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The first one is for the CLI. &lt;code>bao login -method=oidc&lt;/code> opens the browser and waits on port 8250 for the answer. The second one is for the web UI. Store the client ID and the secret in pass, then:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">auth oidc oidc
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.dumps({&amp;#34;oidc_discovery_url&amp;#34;: &amp;#34;https://git.example.org&amp;#34;, &amp;#34;oidc_client_id&amp;#34;: sys.argv[1], &amp;#34;oidc_client_secret&amp;#34;: sys.stdin.read().strip(), &amp;#34;default_role&amp;#34;: &amp;#34;alice&amp;#34;}))&amp;#39;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">$(&lt;/span>pass show infra/home1/openbao-oidc-client-id &lt;span class="p">|&lt;/span> head -1&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;lt; &amp;lt;&lt;span class="o">(&lt;/span>pass show infra/home1/openbao-oidc-client-secret &lt;span class="p">|&lt;/span> head -1&lt;span class="o">)&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> bao write auth/oidc/config - &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao write auth/oidc/role/alice - &amp;gt;/dev/null &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;JSON&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">{
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;user_claim&amp;#34;: &amp;#34;preferred_username&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;bound_claims&amp;#34;: {&amp;#34;preferred_username&amp;#34;: &amp;#34;alice&amp;#34;},
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;oidc_scopes&amp;#34;: &amp;#34;openid,profile&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;allowed_redirect_uris&amp;#34;: &amp;#34;http://localhost:8250/oidc/callback,https://bao.example.net/ui/vault/auth/oidc/oidc/callback&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_policies&amp;#34;: &amp;#34;admin&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_ttl&amp;#34;: &amp;#34;16h&amp;#34;,
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s"> &amp;#34;token_max_ttl&amp;#34;: &amp;#34;16h&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">}
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">JSON&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>bound_claims&lt;/code> is important. Without it, every account on the Forgejo instance could log in to Bao and get the &lt;code>admin&lt;/code> policy.&lt;/p>
&lt;p>The secret goes to &lt;code>bao&lt;/code> on stdin, inside the JSON, and never onto a command line.&lt;/p>
&lt;h3 id="userpass-for-running-bao">userpass, for running Bao&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">auth userpass userpass
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> ! bao &lt;span class="nb">read&lt;/span> auth/userpass/users/alice &amp;gt;/dev/null 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>1&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> pass show infra/home1/openbao-alice-password &amp;gt;/dev/null 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>&lt;span class="m">1&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="o">||&lt;/span> pass generate -n infra/home1/openbao-alice-password &lt;span class="m">32&lt;/span> &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> pass show infra/home1/openbao-alice-password &lt;span class="p">|&lt;/span> head -1 &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.dumps({&amp;#34;password&amp;#34;: sys.stdin.read().strip(), &amp;#34;token_policies&amp;#34;: &amp;#34;admin,operator&amp;#34;, &amp;#34;token_ttl&amp;#34;: &amp;#34;1h&amp;#34;, &amp;#34;token_max_ttl&amp;#34;: &amp;#34;4h&amp;#34;}))&amp;#39;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> bao write auth/userpass/users/alice - &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This login has the &lt;code>operator&lt;/code> policy for changes to Bao, with a short token. It is also my way in when Forgejo is down, for example during the reinstall of the host it runs on.&lt;/p>
&lt;h3 id="approle-for-the-backup">AppRole, for the backup&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">auth approle approle
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao write auth/approle/role/backup &lt;span class="nv">token_policies&lt;/span>&lt;span class="o">=&lt;/span>backup &lt;span class="nv">token_ttl&lt;/span>&lt;span class="o">=&lt;/span>5m &lt;span class="nv">token_max_ttl&lt;/span>&lt;span class="o">=&lt;/span>10m &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="nv">secret_id_ttl&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="m">0&lt;/span> &lt;span class="nv">token_num_uses&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="m">0&lt;/span> &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">if&lt;/span> ! pass show infra/home1/openbao-backup-approle &amp;gt;/dev/null 2&amp;gt;&lt;span class="p">&amp;amp;&lt;/span>1&lt;span class="p">;&lt;/span> &lt;span class="k">then&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">role_id&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>bao &lt;span class="nb">read&lt;/span> -field&lt;span class="o">=&lt;/span>role_id auth/approle/role/backup/role-id&lt;span class="k">)&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> bao write -f -format&lt;span class="o">=&lt;/span>json auth/approle/role/backup/secret-id &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.dumps({&amp;#34;role_id&amp;#34;: sys.argv[1], &amp;#34;secret_id&amp;#34;: json.load(sys.stdin)[&amp;#34;data&amp;#34;][&amp;#34;secret_id&amp;#34;]}))&amp;#39;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$role_id&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> pass insert -m infra/home1/openbao-backup-approle &amp;gt;/dev/null
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="k">fi&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Ansible writes this JSON to &lt;code>/root/.config/openbao/backup-approle.json&lt;/code> on the home server (mode 0600). The backup script uses it to log in.&lt;/p>
&lt;h2 id="step-7-backup-and-restore">Step 7: backup and restore&lt;/h2>
&lt;p>Do not back up the live Raft directory. It is a database that is being written to. Take a snapshot through the API instead. That is one more step in the backup script from &lt;a href="../../blog/infra-07-monitoring-backup/">the monitoring and backup post&lt;/a>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># a backup_functions entry for home1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">openbao_snapshot&lt;span class="o">()&lt;/span> &lt;span class="o">{&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nb">local&lt;/span> &lt;span class="nv">bao&lt;/span>&lt;span class="o">=(&lt;/span>curl -fsS -m &lt;span class="m">60&lt;/span> --resolve bao.example.net:443:127.0.0.1&lt;span class="o">)&lt;/span> token
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="nv">token&lt;/span>&lt;span class="o">=&lt;/span>&lt;span class="k">$(&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">bao&lt;/span>&lt;span class="p">[@]&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> -d @/root/.config/openbao/backup-approle.json &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> https://bao.example.net/v1/auth/approle/login &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> python3 -c &lt;span class="s1">&amp;#39;import json,sys; print(json.load(sys.stdin)[&amp;#34;auth&amp;#34;][&amp;#34;client_token&amp;#34;])&amp;#39;&lt;/span>&lt;span class="k">)&lt;/span> &lt;span class="o">||&lt;/span> &lt;span class="k">return&lt;/span> &lt;span class="m">1&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="si">${&lt;/span>&lt;span class="nv">bao&lt;/span>&lt;span class="p">[@]&lt;/span>&lt;span class="si">}&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> -H &lt;span class="s2">&amp;#34;X-Vault-Token: &lt;/span>&lt;span class="nv">$token&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> -o &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$DUMP_DIR&lt;/span>&lt;span class="s2">/openbao/raft.snap.tmp&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> https://bao.example.net/v1/sys/storage/raft/snapshot &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> mv &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$DUMP_DIR&lt;/span>&lt;span class="s2">/openbao/raft.snap.tmp&amp;#34;&lt;/span> &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="nv">$DUMP_DIR&lt;/span>&lt;span class="s2">/openbao/raft.snap&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="o">}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>--resolve ...:127.0.0.1&lt;/code> sends the request to the local Traefik, with the right name for the certificate. The live volume is excluded from restic. While Bao is sealed this step fails, and the backup reports it.&lt;/p>
&lt;p>The snapshot is encrypted with Bao&amp;rsquo;s keys. To restore it on any host with Podman:&lt;/p>
&lt;ol>
&lt;li>Start an empty Bao, initialise it with one share, and unseal it with the new key.&lt;/li>
&lt;li>Log in with the new root token, then run &lt;code>bao operator raft snapshot restore -force raft.snap&lt;/code>.&lt;/li>
&lt;li>Bao is sealed again, now with the old key. Unseal it with the old key from pass.&lt;/li>
&lt;li>Everything is back: both CAs, the roles, the policies, the logins. Revoke the new root token.&lt;/li>
&lt;/ol>
&lt;p>Try this once, on a VM, before you need it.&lt;/p>
&lt;h2 id="check">Check&lt;/h2>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">bao login -no-print -method&lt;span class="o">=&lt;/span>oidc &lt;span class="c1"># the browser opens, log in to Forgejo&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao token lookup -format&lt;span class="o">=&lt;/span>json &lt;span class="p">|&lt;/span> jq &lt;span class="s1">&amp;#39;.data | {display_name, policies, ttl}&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao &lt;span class="nb">read&lt;/span> -field&lt;span class="o">=&lt;/span>public_key ssh-user/config/ca
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">bao write -field&lt;span class="o">=&lt;/span>signed_key ssh-user/sign/admin &lt;span class="nv">public_key&lt;/span>&lt;span class="o">=&lt;/span>@&lt;span class="nv">$HOME&lt;/span>/.ssh/id_ed25519.pub &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> &lt;span class="p">|&lt;/span> ssh-keygen -Lf /dev/stdin
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">journalctl --user -u openbao -n &lt;span class="m">3&lt;/span> -o cat &lt;span class="p">|&lt;/span> jq -c &lt;span class="s1">&amp;#39;{type, path: .request.path}&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The certificate shows principal &lt;code>alice&lt;/code>, 16 hours, key ID &lt;code>oidc-alice-admin&lt;/code>. The audit log shows the request.&lt;/p>
&lt;p>Next: &lt;a href="../../blog/ssh-03-hosts/">Hosts that trust the SSH CA&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://openbao.org/docs/">OpenBao documentation&lt;/a>: concepts, configuration, auth methods and secrets engines.&lt;/li>
&lt;li>&lt;a href="https://github.com/openbao/openbao">OpenBao on GitHub&lt;/a>: releases and the container images.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/configuration/storage/raft/">Integrated storage (Raft)&lt;/a>: the storage backend, also for a single node.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/concepts/seal/">Seal and unseal&lt;/a>: Shamir keys, auto unseal, and why the barrier exists.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/secrets/ssh/signed-ssh-certificates/">Signed SSH certificates&lt;/a>: the SSH secrets engine, for user and host keys.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/api-docs/secret/ssh/">SSH secrets engine API&lt;/a>: every field of a role: &lt;code>allowed_users&lt;/code>, &lt;code>ttl&lt;/code>, &lt;code>default_extensions&lt;/code> and more.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/auth/jwt/">JWT/OIDC auth method&lt;/a>: the browser login with Forgejo and, in post 4, the CI login.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/auth/jwt/oidc-providers/">OIDC provider configuration&lt;/a>: examples for many identity providers.&lt;/li>
&lt;li>&lt;a href="https://forgejo.org/docs/latest/user/oauth2-provider/">Forgejo as an OAuth2 provider&lt;/a>: the client registration on the Forgejo side.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/auth/userpass/">Userpass auth method&lt;/a>: the fallback login.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/auth/approle/">AppRole auth method&lt;/a>: the login of the backup job.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/concepts/policies/">Policies&lt;/a>: paths, capabilities and how Bao combines policies.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/audit/file/">File audit device&lt;/a>: the log of every request.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/commands/operator/raft/">bao operator raft&lt;/a>: &lt;code>snapshot save&lt;/code> and &lt;code>snapshot restore&lt;/code>.&lt;/li>
&lt;/ul></description></item><item><title>Why SSH certificates</title><link>https://www.linux-lunatix.org/blog/ssh-01-why-certificates/</link><pubDate>Sun, 01 Nov 2026 17:10:00 +0100</pubDate><guid>https://www.linux-lunatix.org/blog/ssh-01-why-certificates/</guid><description>&lt;p>&lt;em>Part 1 of the SSH series. It builds on the hosts from &lt;a href="../../blog/infra-01-consolidation/">the consolidation series&lt;/a>, but you can read it on its own.&lt;/em>&lt;/p>
&lt;p>This post explains OpenSSH certificates. It has a demo that you can run on one machine in ten minutes, and it explains why I put my CA on OpenBao. The next four posts build the real thing.&lt;/p>
&lt;h2 id="the-problems-with-plain-keys">The problems with plain keys&lt;/h2>
&lt;p>SSH with plain keys has two trust lists, and both are maintained by hand.&lt;/p>
&lt;p>&lt;code>authorized_keys&lt;/code> on the server says which keys may log in. With four hosts, a few accounts each, two laptops, a tablet and a CI job, that is a lot of lines. A new device means editing every host. A lost device means finding its key everywhere and removing it. A CI deploy key lives forever, unless somebody rotates it.&lt;/p>
&lt;p>&lt;code>known_hosts&lt;/code> on the client says which host keys to trust. The first connection asks you to confirm a fingerprint. Almost nobody checks it, which is why this is called &amp;ldquo;trust on first use&amp;rdquo;. After a reinstall, ssh shows a big warning, and you either check the new key over a console or you click it away. In &lt;a href="../../blog/infra-08-rehearsal/">the rehearsal post&lt;/a> I showed how much work a careful check is.&lt;/p>
&lt;p>Both lists work. But they grow, nobody removes old entries, and every change touches many places.&lt;/p>
&lt;h2 id="what-a-certificate-is">What a certificate is&lt;/h2>
&lt;p>An OpenSSH certificate is a public key plus some data, signed by a CA key:&lt;/p>
&lt;ul>
&lt;li>the public key itself,&lt;/li>
&lt;li>the type: user or host,&lt;/li>
&lt;li>the principals: the user names or host names it is valid for,&lt;/li>
&lt;li>the validity: from when until when,&lt;/li>
&lt;li>the critical options, for example &lt;code>force-command&lt;/code> or &lt;code>source-address&lt;/code>,&lt;/li>
&lt;li>the extensions, for example &lt;code>permit-pty&lt;/code> or &lt;code>permit-port-forwarding&lt;/code>,&lt;/li>
&lt;li>a key ID, a free text that appears in the server&amp;rsquo;s log,&lt;/li>
&lt;li>the signature of the CA.&lt;/li>
&lt;/ul>
&lt;p>A server that trusts the CA accepts any user certificate signed by it, if the certificate is still valid and lists a principal that may use the account. A client that trusts a host CA accepts any host that presents a certificate for its name.&lt;/p>
&lt;p>These are not X.509 certificates. There is no chain and no CRL distribution. OpenSSH has its own simple format, and it has supported it since 2010 (OpenSSH 5.4).&lt;/p>
&lt;h2 id="what-this-changes">What this changes&lt;/h2>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Before&lt;/th>
&lt;th>With certificates&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>every host lists every user key&lt;/td>
&lt;td>every host lists one CA key and the allowed principals&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>a new device: edit every host&lt;/td>
&lt;td>a new device: get a certificate&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>a lost device: remove its key everywhere&lt;/td>
&lt;td>a lost device: its certificates expire in hours&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>a CI key that never expires&lt;/td>
&lt;td>a certificate for five minutes, per job&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>known_hosts&lt;/code> per host, TOFU&lt;/td>
&lt;td>one &lt;code>@cert-authority&lt;/code> line on the client&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>reinstall: check the key over a console&lt;/td>
&lt;td>reinstall: sign the new host key, done&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>The short validity is the most important part. Revocation is hard in any PKI. A certificate that expires after 16 hours needs no revocation in most cases.&lt;/p>
&lt;h2 id="try-it-a-ca-in-ten-minutes">Try it: a CA in ten minutes&lt;/h2>
&lt;p>You can try all of this on one machine, with &lt;code>ssh-keygen&lt;/code> and a local sshd. Do it in a VM or a container if you do not want to touch your machine&amp;rsquo;s sshd.&lt;/p>
&lt;h3 id="a-user-ca">A user CA&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">mkdir -p ~/ca-demo &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> &lt;span class="nb">cd&lt;/span> ~/ca-demo
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -t ed25519 -f user_ca -C &lt;span class="s2">&amp;#34;demo user CA&amp;#34;&lt;/span> -N &lt;span class="s1">&amp;#39;&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>user_ca&lt;/code> is the CA&amp;rsquo;s private key and &lt;code>user_ca.pub&lt;/code> its public key. In real life the private key is the most sensitive file of the whole setup. Here it is a demo.&lt;/p>
&lt;h3 id="sign-a-user-key">Sign a user key&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -t ed25519 -f alice -N &lt;span class="s1">&amp;#39;&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -s user_ca -I &lt;span class="s2">&amp;#34;alice-laptop&amp;#34;&lt;/span> -n alice -V +1h alice.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -Lf alice-cert.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">alice-cert.pub:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Type: ssh-ed25519-cert-v01@openssh.com user certificate
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Public key: ED25519-CERT SHA256:Xq2...
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Signing CA: ED25519 SHA256:k9T... (using ssh-ed25519)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Key ID: &amp;#34;alice-laptop&amp;#34;
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Serial: 0
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Valid: from 2026-11-01T17:10:00 to 2026-11-01T18:10:00
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Principals:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> alice
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Critical Options: (none)
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> Extensions:
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> permit-X11-forwarding
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> permit-agent-forwarding
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> permit-port-forwarding
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> permit-pty
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl"> permit-user-rc
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>-I&lt;/code> is the key ID, &lt;code>-n&lt;/code> the principals and &lt;code>-V&lt;/code> the validity. By default &lt;code>ssh-keygen&lt;/code> adds all the extensions. A real CA should add only what you need.&lt;/p>
&lt;p>ssh finds &lt;code>alice-cert.pub&lt;/code> by itself when you use the key &lt;code>alice&lt;/code>: it looks for &lt;code>&amp;lt;key&amp;gt;-cert.pub&lt;/code> next to the key.&lt;/p>
&lt;h3 id="trust-the-ca-on-the-server">Trust the CA on the server&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo install -m &lt;span class="m">0644&lt;/span> user_ca.pub /etc/ssh/user_ca.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo install -d /etc/ssh/principals
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># which principals may log in as the account &amp;#34;demo&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> alice &lt;span class="p">|&lt;/span> sudo tee /etc/ssh/principals/demo
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo tee /etc/ssh/sshd_config.d/10-ca.conf &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;EOF&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">TrustedUserCAKeys /etc/ssh/user_ca.pub
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">AuthorizedPrincipalsFile /etc/ssh/principals/%u
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo useradd -m demo
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo sshd -t &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> sudo systemctl reload sshd
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh -i ~/ca-demo/alice -o &lt;span class="nv">IdentitiesOnly&lt;/span>&lt;span class="o">=&lt;/span>yes demo@localhost
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>You are in, and &lt;code>demo&lt;/code> has no &lt;code>authorized_keys&lt;/code> file. The server log shows who it was:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">sshd[4242]: Accepted publickey for demo from ::1 port 50122 ssh2: ED25519-CERT SHA256:Xq2... ID alice-laptop (serial 0) CA ED25519 SHA256:k9T...
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>AuthorizedPrincipalsFile&lt;/code> is the important line. Without it, sshd accepts a certificate for an account only if the account name itself is one of its principals. With it, you decide per account which principals may use it. One principal can then log in as several accounts, and a certificate for &lt;code>alice&lt;/code> cannot log in as &lt;code>root&lt;/code> or &lt;code>ansible&lt;/code>.&lt;/p>
&lt;h3 id="a-host-ca">A host CA&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -t ed25519 -f host_ca -C &lt;span class="s2">&amp;#34;demo host CA&amp;#34;&lt;/span> -N &lt;span class="s1">&amp;#39;&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo ssh-keygen -s host_ca -I &lt;span class="s2">&amp;#34;&lt;/span>&lt;span class="k">$(&lt;/span>hostname&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> -h -n localhost,&lt;span class="k">$(&lt;/span>hostname -f&lt;span class="k">)&lt;/span> -V +52w &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> /etc/ssh/ssh_host_ed25519_key.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s1">&amp;#39;HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub&amp;#39;&lt;/span> &lt;span class="p">|&lt;/span> sudo tee /etc/ssh/sshd_config.d/11-hostcert.conf
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo systemctl reload sshd
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>-h&lt;/code> makes it a host certificate. The principals are the names that clients use for the host.&lt;/p>
&lt;p>On the client, one line in &lt;code>known_hosts&lt;/code> replaces all the host entries:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">&lt;span class="nb">echo&lt;/span> &lt;span class="s2">&amp;#34;@cert-authority localhost,*.example.org &lt;/span>&lt;span class="k">$(&lt;/span>cat host_ca.pub&lt;span class="k">)&lt;/span>&lt;span class="s2">&amp;#34;&lt;/span> &amp;gt; ~/ca-demo/known_hosts
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh -o &lt;span class="nv">UserKnownHostsFile&lt;/span>&lt;span class="o">=&lt;/span>~/ca-demo/known_hosts -o &lt;span class="nv">StrictHostKeyChecking&lt;/span>&lt;span class="o">=&lt;/span>yes &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -i ~/ca-demo/alice demo@localhost
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>No question about a fingerprint. &lt;code>StrictHostKeyChecking=yes&lt;/code> proves it: ssh would refuse a host it does not know.&lt;/p>
&lt;h3 id="clean-up">Clean up&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">sudo rm /etc/ssh/sshd_config.d/10-ca.conf /etc/ssh/sshd_config.d/11-hostcert.conf &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> /etc/ssh/ssh_host_ed25519_key-cert.pub /etc/ssh/user_ca.pub
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo rm -r /etc/ssh/principals
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">sudo systemctl reload sshd &lt;span class="o">&amp;amp;&amp;amp;&lt;/span> sudo userdel -r demo
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="options-that-the-server-enforces">Options that the server enforces&lt;/h2>
&lt;p>Critical options restrict what a certificate can do:&lt;/p>
&lt;ul>
&lt;li>&lt;code>force-command=&amp;quot;podman auto-update&amp;quot;&lt;/code>: the certificate runs only this command, whatever the client asks for.&lt;/li>
&lt;li>&lt;code>source-address=10.0.0.0/8&lt;/code>: the certificate works only from these addresses.&lt;/li>
&lt;/ul>
&lt;p>There is a catch here. Whoever may ask the CA for a certificate with an option may also ask for another value. If the CA&amp;rsquo;s policy lets a CI job choose its own &lt;code>force-command&lt;/code>, the job can choose &lt;code>bash&lt;/code>. So my hosts enforce the important options themselves, in the principals file. Each line there can carry the same options as an &lt;code>authorized_keys&lt;/code> line:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl"># /etc/ssh/principals/svc
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">alice
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">restrict,command=&amp;#34;podman auto-update&amp;#34; deploy-websites
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>A certificate for &lt;code>alice&lt;/code> logs in normally. A certificate for &lt;code>deploy-websites&lt;/code> gets &lt;code>podman auto-update&lt;/code> and nothing else, even if the CA signed it without the option. &lt;a href="../../blog/ssh-03-hosts/">Post 3 of this series&lt;/a> shows the Ansible role that writes these files.&lt;/p>
&lt;h2 id="more-details-worth-knowing">More details worth knowing&lt;/h2>
&lt;p>The demo covers the basics. These details matter as soon as you run a CA for real.&lt;/p>
&lt;h3 id="sign-only-the-extensions-you-need">Sign only the extensions you need&lt;/h3>
&lt;p>&lt;code>ssh-keygen&lt;/code> adds all five extensions by default. &lt;code>-O clear&lt;/code> removes them, and each further &lt;code>-O&lt;/code> adds one back:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -s user_ca -I &lt;span class="s2">&amp;#34;alice-laptop&amp;#34;&lt;/span> -n alice -V +1h &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -O clear -O permit-pty alice.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This certificate can open a shell, but it cannot forward ports, agents or X11. A certificate for a backup job needs no &lt;code>permit-pty&lt;/code> either. The critical options work the same way:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -s user_ca -I &lt;span class="s2">&amp;#34;backup-job&amp;#34;&lt;/span> -n backup -V +10m &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -O clear -O force-command&lt;span class="o">=&lt;/span>&lt;span class="s2">&amp;#34;/usr/local/bin/backup.sh&amp;#34;&lt;/span> &lt;span class="se">\
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="se">&lt;/span> -O source-address&lt;span class="o">=&lt;/span>192.168.50.0/24 backup.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="validity-and-clocks">Validity and clocks&lt;/h3>
&lt;p>The server checks the validity against its own clock. If the client&amp;rsquo;s clock or the CA&amp;rsquo;s clock runs a little ahead, a fresh certificate can be &amp;ldquo;not yet valid&amp;rdquo; on the server. Let it start a few minutes in the past:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -s user_ca -I &lt;span class="s2">&amp;#34;alice-laptop&amp;#34;&lt;/span> -n alice -V -5m:+16h alice.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>-V&lt;/code> also takes absolute times, for example &lt;code>-V 20261101:20261201&lt;/code>. Keep the clocks of all hosts in sync with NTP. Certificates depend on it.&lt;/p>
&lt;h3 id="always-set-principals">Always set principals&lt;/h3>
&lt;p>&lt;code>-n&lt;/code> is not optional in practice. In the certificate format, an empty principal list means &amp;ldquo;any principal&amp;rdquo;. With &lt;code>TrustedUserCAKeys&lt;/code>, sshd refuses a user certificate without principals, but other programs may handle it differently. A CA that signs whatever it gets is a risk. Give every certificate its principals, and give your CA a list of principals it may sign.&lt;/p>
&lt;h3 id="serial-numbers-and-key-ids">Serial numbers and key IDs&lt;/h3>
&lt;p>&lt;code>-z&lt;/code> sets a serial number:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">ssh-keygen -s user_ca -I &lt;span class="s2">&amp;#34;alice-laptop&amp;#34;&lt;/span> -z &lt;span class="m">1001&lt;/span> -n alice -V +16h alice.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The key ID and the serial appear in the server log for each login. They are also the handles for revocation. Choose key IDs that tell you who and what, for example &lt;code>alice-laptop&lt;/code> or &lt;code>ci-blog-main&lt;/code>. A CA such as OpenBao sets a random serial by itself.&lt;/p>
&lt;h3 id="revocation-when-you-need-it">Revocation, when you need it&lt;/h3>
&lt;p>Short certificates make revocation rare, but not impossible. The certificate of a tablet lives for weeks, and a host certificate for a year. sshd reads a revocation list from &lt;code>RevokedKeys&lt;/code>. That can be a plain list of public keys, or a key revocation list (KRL) in OpenSSH&amp;rsquo;s own compact format. A KRL can revoke by serial, by key ID or by key:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-bash" data-lang="bash">&lt;span class="line">&lt;span class="cl">cat &amp;gt; revoke.txt &lt;span class="s">&amp;lt;&amp;lt;&amp;#39;EOF2&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">serial: 1001
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">id: alice-tablet
&lt;/span>&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="s">EOF2&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -k -f revoked.krl -s user_ca.pub revoke.txt
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">&lt;span class="c1"># test a certificate against the KRL&lt;/span>
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">ssh-keygen -Q -f revoked.krl alice-cert.pub
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;code>-u&lt;/code> adds to an existing KRL instead of replacing it. On the server:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">RevokedKeys /etc/ssh/revoked.krl
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>On the client, a line with &lt;code>@revoked&lt;/code> in &lt;code>known_hosts&lt;/code> refuses a host key, even with a valid host certificate:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">@revoked * ssh-ed25519 AAAA...the-old-host-key...
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="rotate-the-ca-key">Rotate the CA key&lt;/h3>
&lt;p>One day you will want a new CA key, because the old one leaked or because you want a new algorithm. &lt;code>TrustedUserCAKeys&lt;/code> can list several keys, one per line. That makes the change smooth:&lt;/p>
&lt;ol>
&lt;li>Create the new CA key and add its public key to &lt;code>TrustedUserCAKeys&lt;/code> on every host. Both keys are now trusted.&lt;/li>
&lt;li>Sign new certificates with the new key only.&lt;/li>
&lt;li>Wait until the last certificate from the old key has expired.&lt;/li>
&lt;li>Remove the old public key from every host.&lt;/li>
&lt;/ol>
&lt;p>On the client, two &lt;code>@cert-authority&lt;/code> lines do the same for host certificates. With short certificates, step 3 takes one day.&lt;/p>
&lt;h3 id="a-ca-per-account-without-sshd-config">A CA per account, without sshd_config&lt;/h3>
&lt;p>If you cannot change &lt;code>sshd_config&lt;/code>, the CA can go into a user&amp;rsquo;s &lt;code>authorized_keys&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">cert-authority,principals=&amp;#34;alice&amp;#34; ssh-ed25519 AAAA...user-ca-public-key...
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This trusts the CA only for this one account. It is useful on a shared host where you are not root. &lt;code>principals=&lt;/code> has the same role as the principals file.&lt;/p>
&lt;h3 id="host-certificates-and-names">Host certificates and names&lt;/h3>
&lt;p>The client checks that the host certificate lists the name it used to connect. &lt;code>ssh vps1&lt;/code> with a certificate for &lt;code>vps1.example.org&lt;/code> fails, unless your SSH config maps the short name to the full name with &lt;code>HostName&lt;/code> or &lt;code>HostKeyAlias&lt;/code>. Put every name that clients use into the host certificate&amp;rsquo;s principals. &lt;a href="../../blog/ssh-03-hosts/">Post 3&lt;/a> shows how I handle short names.&lt;/p>
&lt;h3 id="see-what-happens">See what happens&lt;/h3>
&lt;p>&lt;code>ssh -v&lt;/code> shows which keys and certificates the client offers and what the server answers:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" class="chroma">&lt;code class="language-text" data-lang="text">&lt;span class="line">&lt;span class="cl">debug1: Offering public key: /home/alice/.ssh/id_ed25519 ED25519-CERT SHA256:... explicit
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">debug1: Server accepts key: /home/alice/.ssh/id_ed25519 ED25519-CERT SHA256:... explicit
&lt;/span>&lt;/span>&lt;span class="line">&lt;span class="cl">debug1: Host &amp;#39;vps1.example.org&amp;#39; is known and matches the ED25519-CERT host certificate.
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>On the server, &lt;code>journalctl -u sshd&lt;/code> shows the key ID, the serial and the CA of each accepted certificate. It also shows the reason for a refusal, for example an expired certificate or a principal that is not allowed.&lt;/p>
&lt;h2 id="where-should-the-ca-key-live">Where should the CA key live?&lt;/h2>
&lt;p>The demo kept the CA key in a file. That works for a homelab, but it has problems:&lt;/p>
&lt;ul>
&lt;li>Whoever has the file can sign anything: any principal, any validity. A stolen laptop is a stolen CA.&lt;/li>
&lt;li>Nothing logs what was signed.&lt;/li>
&lt;li>A CI job cannot get a certificate without a copy of the key or a person in the loop.&lt;/li>
&lt;/ul>
&lt;p>What I wanted from a CA:&lt;/p>
&lt;ul>
&lt;li>The private keys never leave it.&lt;/li>
&lt;li>Signing needs a login, and the login decides what may be signed: which principals, how long, which options.&lt;/li>
&lt;li>My daily login uses my existing single sign-on, my self-hosted Forgejo.&lt;/li>
&lt;li>A CI job proves who it is with the identity token that Forgejo Actions gives it, without a stored secret.&lt;/li>
&lt;li>Every signing is logged.&lt;/li>
&lt;/ul>
&lt;h2 id="why-openbao">Why OpenBao&lt;/h2>
&lt;p>I looked at four options.&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Option&lt;/th>
&lt;th>For it&lt;/th>
&lt;th>Against it&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>ssh-keygen&lt;/code> and a file&lt;/td>
&lt;td>simple, nothing to run&lt;/td>
&lt;td>everything in the list above&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>step-ca (Smallstep)&lt;/td>
&lt;td>built for this, OIDC, good SSH support&lt;/td>
&lt;td>one more tool only for certificates&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>HashiCorp Vault&lt;/td>
&lt;td>the standard, SSH secrets engine&lt;/td>
&lt;td>Business Source License since 2023&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>OpenBao&lt;/td>
&lt;td>Vault&amp;rsquo;s open fork (MPL 2.0, Linux Foundation), same API and SSH engine&lt;/td>
&lt;td>younger project&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>I chose OpenBao. Its SSH secrets engine does exactly what I need. It has OIDC and JWT login, policies, an audit log, and integrated storage with snapshots. It can also hold other secrets later, and step-ca cannot do that. Most Vault documentation and tools work with it unchanged: the CLI is &lt;code>bao&lt;/code> instead of &lt;code>vault&lt;/code>, and the API is the same. Fedora packages the &lt;code>openbao&lt;/code> CLI, and the server runs from the official container image.&lt;/p>
&lt;h2 id="the-design">The design&lt;/h2>
&lt;p>Two CAs, one for users and one for hosts, so that a host key can never log in as a user. Each kind of certificate gets a role in Bao with its own rules:&lt;/p>
&lt;table>
&lt;thead>
&lt;tr>
&lt;th>Role&lt;/th>
&lt;th>Principal&lt;/th>
&lt;th>Valid&lt;/th>
&lt;th>For&lt;/th>
&lt;/tr>
&lt;/thead>
&lt;tbody>
&lt;tr>
&lt;td>&lt;code>admin&lt;/code>&lt;/td>
&lt;td>&lt;code>alice&lt;/code>&lt;/td>
&lt;td>16 h&lt;/td>
&lt;td>my laptop, every day&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>device&lt;/code>&lt;/td>
&lt;td>&lt;code>alice&lt;/code>&lt;/td>
&lt;td>30 days&lt;/td>
&lt;td>a tablet that cannot talk to Bao&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>ansible&lt;/code>&lt;/td>
&lt;td>&lt;code>ansible&lt;/code>&lt;/td>
&lt;td>16 h&lt;/td>
&lt;td>the Ansible key on the control node&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>deploy-websites&lt;/code>&lt;/td>
&lt;td>&lt;code>deploy-websites&lt;/code>&lt;/td>
&lt;td>5 min&lt;/td>
&lt;td>CI jobs; only &lt;code>podman auto-update&lt;/code>&lt;/td>
&lt;/tr>
&lt;tr>
&lt;td>&lt;code>hosts&lt;/code> (host CA)&lt;/td>
&lt;td>the host&amp;rsquo;s FQDN&lt;/td>
&lt;td>1 year&lt;/td>
&lt;td>host certificates, signed during the Ansible run&lt;/td>
&lt;/tr>
&lt;/tbody>
&lt;/table>
&lt;p>16 hours covers a working day. In the morning I run one command, log in through the browser, and have certificates for the day.&lt;/p>
&lt;p>The plain keys go away, except for one break-glass key per host. That key is for the day when Bao is down and all certificates have expired.&lt;/p>
&lt;h2 id="the-series">The series&lt;/h2>
&lt;ol>
&lt;li>This post: why and how it works.&lt;/li>
&lt;li>&lt;a href="../../blog/ssh-02-openbao/">OpenBao rootless on Podman&lt;/a>: the server, its configuration as a script, unseal, backup.&lt;/li>
&lt;li>&lt;a href="../../blog/ssh-03-hosts/">Hosts that trust the SSH CA&lt;/a>: the Ansible side, host certificates, the client.&lt;/li>
&lt;li>&lt;a href="../../blog/ssh-04-ci-oidc/">CI deploys without a deploy key&lt;/a>: Forgejo Actions OIDC and five-minute certificates.&lt;/li>
&lt;li>&lt;a href="../../blog/ssh-05-plain-keys/">Removing the plain SSH keys&lt;/a>: break-glass, devices without Bao, and how not to lock yourself out.&lt;/li>
&lt;/ol>
&lt;p>Next: &lt;a href="../../blog/ssh-02-openbao/">OpenBao rootless on Podman&lt;/a>.&lt;/p>
&lt;h2 id="further-reading">Further reading&lt;/h2>
&lt;ul>
&lt;li>&lt;a href="https://man.openbsd.org/ssh-keygen#CERTIFICATES">ssh-keygen(1), section CERTIFICATES&lt;/a>: signing, &lt;code>-O&lt;/code> options, validity formats and the KRL commands. Start here.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd#AUTHORIZED_KEYS_FILE_FORMAT">sshd(8), authorized_keys format&lt;/a>: &lt;code>cert-authority&lt;/code>, &lt;code>principals=&lt;/code> and the options that a principals file also accepts.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd#SSH_KNOWN_HOSTS_FILE_FORMAT">sshd(8), known_hosts format&lt;/a>: &lt;code>@cert-authority&lt;/code> and &lt;code>@revoked&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://man.openbsd.org/sshd_config">sshd_config(5)&lt;/a>: &lt;code>TrustedUserCAKeys&lt;/code>, &lt;code>AuthorizedPrincipalsFile&lt;/code>, &lt;code>HostCertificate&lt;/code> and &lt;code>RevokedKeys&lt;/code>.&lt;/li>
&lt;li>&lt;a href="https://datatracker.ietf.org/doc/draft-miller-ssh-cert/">SSH certificate format (IETF draft by Damien Miller)&lt;/a>: the exact wire format of user and host certificates, and every option and extension.&lt;/li>
&lt;li>&lt;a href="https://github.com/openssh/openssh-portable/blob/master/PROTOCOL.krl">PROTOCOL.krl&lt;/a>: the format of key revocation lists.&lt;/li>
&lt;li>&lt;a href="https://www.openssh.com/txt/release-5.4">OpenSSH 5.4 release notes&lt;/a>: the release that introduced certificates in 2010.&lt;/li>
&lt;li>&lt;a href="https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Certificate-based_Authentication">OpenSSH Cookbook: certificate-based authentication&lt;/a>: a long tutorial with many examples.&lt;/li>
&lt;li>&lt;a href="https://smallstep.com/blog/use-ssh-certificates/">If you&amp;rsquo;re not using SSH certificates you&amp;rsquo;re doing SSH wrong (Smallstep)&lt;/a>: a good argument for certificates, from the makers of step-ca.&lt;/li>
&lt;li>&lt;a href="https://engineering.fb.com/2016/09/12/security/scalable-and-secure-access-with-ssh/">Scalable and secure access with SSH (Facebook engineering)&lt;/a>: how a large company uses SSH certificates.&lt;/li>
&lt;li>&lt;a href="https://github.com/Netflix/bless">Netflix BLESS&lt;/a>: another CA design: short certificates from a serverless function.&lt;/li>
&lt;li>&lt;a href="https://openbao.org/docs/secrets/ssh/signed-ssh-certificates/">OpenBao signed SSH certificates&lt;/a>: the CA I use in the next posts.&lt;/li>
&lt;li>&lt;a href="https://smallstep.com/docs/step-ca/">step-ca&lt;/a>: the alternative I did not choose.&lt;/li>
&lt;/ul></description></item></channel></rss>