Hosts that trust the SSH CA
Part 3 of the SSH series that starts with Why SSH certificates.
Bao from the previous post can sign certificates. Now the hosts must trust them, and the laptop must use them. At the end of this post:
- every host accepts user certificates for the accounts and principals listed in the inventory,
- every host has a host certificate, renewed by Ansible,
- the laptop gets its certificates with one command and trusts the hosts without
known_hostsentries.
The plain keys stay in place during this step. They go away in the last post of the series, when everything else works.
The inventory
The CA public keys go into group_vars/all.yml. They are public, and a rollout needs no Bao for them:
# group_vars/all.yml
# bao read -field=public_key ssh-user/config/ca
ssh_user_ca: "ssh-ed25519 AAAA...user-ca-public-key... openbao ssh-user"
# bao read -field=public_key ssh-host/config/ca
ssh_host_ca: "ssh-ed25519 AAAA...host-ca-public-key... openbao ssh-host"
bao_addr: https://bao.example.net
Each host says who may log in as which account:
# host_vars/vps1.yml
ssh_principals:
fedora: [alice]
svc: [alice]
ansible: [ansible]
# group_vars/websites.yml
ssh_principals:
fedora: [alice]
svc: [alice, 'restrict,command="podman auto-update" deploy-websites']
ansible: [ansible]
A line may start with options, in the same syntax as in authorized_keys. The host enforces them, whatever the certificate says. The first post explains why that matters.
The base role: trust the CA
# roles/base/tasks/ssh_certs.yml
- name: SSH CA directory
ansible.builtin.file:
path: /etc/ssh/ca
state: directory
mode: "0755"
- name: User CA
ansible.builtin.copy:
dest: /etc/ssh/ca/user_ca.pub
content: "{{ ssh_user_ca }}\n"
mode: "0644"
when: ssh_user_ca | length > 0
notify: Reload sshd
- name: Principals directory
ansible.builtin.file:
path: /etc/ssh/principals
state: directory
mode: "0755"
- name: Principals per account
ansible.builtin.copy:
dest: /etc/ssh/principals/{{ item.key }}
content: "{{ item.value | join('\n') }}\n"
mode: "0644"
loop: "{{ ssh_principals | dict2items }}"
loop_control:
label: "{{ item.key }}"
- name: Principals files in place
ansible.builtin.find:
paths: /etc/ssh/principals
register: principals_files
# Exclusive: an account no longer listed has no principals file, so no cert logs in as it
- name: Principals files of accounts no longer listed
ansible.builtin.file:
path: "{{ item.path }}"
state: absent
loop: "{{ principals_files.files | rejectattr('path', 'in', ssh_principals.keys() | map('regex_replace', '^', '/etc/ssh/principals/') | list) }}"
loop_control:
label: "{{ item.path }}"
- name: Revoked keys
ansible.builtin.copy:
dest: /etc/ssh/revoked_keys
content: "{{ (ssh_revoked_keys | join('\n') ~ '\n') if ssh_revoked_keys else '' }}"
mode: "0644"
notify: Reload sshd
The principals files need no reload. sshd reads them at each login.
The “no longer listed” task makes the list exclusive. If I remove an account from ssh_principals, its file goes away, and no certificate can log in as that account any more. A list that only adds is a list that only grows.
RevokedKeys accepts a plain file of public keys. A key in it is refused even with a valid certificate. That covers a lost device whose certificate is still valid for weeks (the last post of the series has the details). The file must exist, even empty. If RevokedKeys points to a missing file, sshd refuses all public key logins.
sshd’s config
The hardening drop-in from the Ansible groundwork post gets these lines:
HostKey /etc/ssh/ssh_host_ed25519_key
{% if ssh_host_cert_installed | default(false) %}
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
{% endif %}
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
PubkeyAcceptedAlgorithms ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com
# The CA must sign with Ed25519, too
CASignatureAlgorithms ssh-ed25519
{% if ssh_user_ca %}
# Who may log in as an account is the account's principals file (with options),
# not the certificate alone
TrustedUserCAKeys /etc/ssh/ca/user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/principals/%u
{% endif %}
RevokedKeys /etc/ssh/revoked_keys
HostKeyAlgorithms must list the certificate type, or sshd never offers the host certificate. The same applies to PubkeyAcceptedAlgorithms for user certificates.
The HostCertificate line appears only once the certificate exists, so sshd never points to a missing file.
Host certificates from the Ansible run
The host’s key is on the host. The CA is in Bao. The control node has a Bao login. So the playbook reads the host’s public key, has it signed on the control node and installs the certificate on the host.
First the role decides whether a new certificate is needed. A shell task describes the current one as JSON:
- name: Host cert state
ansible.builtin.shell: |
set -o pipefail
c=/etc/ssh/ssh_host_ed25519_key-cert.pub
k=/etc/ssh/ssh_host_ed25519_key.pub
if [ ! -s $c ]; then echo '{"present": false}'; exit 0; fi
info=$(ssh-keygen -Lf $c)
to=$(sed -n 's/^ *Valid: from .* to \(.*\)$/\1/p' <<<"$info")
left=$(( ($(date -d "$to" +%s 2>/dev/null || date +%s) - $(date +%s)) / 86400 ))
ca=$(sed -n 's/^ *Signing CA: [A-Z0-9-]* \(SHA256:[^ ]*\).*/\1/p' <<<"$info")
cert_key=$(sed -n 's/^ *Public key: [A-Z0-9-]* \(SHA256:[^ ]*\).*/\1/p' <<<"$info")
principals=$(sed -n '/^ *Principals:/,/^ *Critical Options:/p' <<<"$info" | sed '1d;$d' | tr -d ' ' | sort | paste -sd,)
printf '{"present": true, "days_left": %d, "ca": "%s", "want_ca": "%s", "cert_key": "%s", "host_key": "%s", "principals": "%s"}\n' \
"$left" "$ca" "$(ssh-keygen -lf /dev/stdin <<<"$HOST_CA" | awk '{print $2}')" \
"$cert_key" "$(ssh-keygen -lf $k | awk '{print $2}')" "$principals"
args:
executable: /bin/bash
environment:
HOST_CA: "{{ ssh_host_ca }}"
register: host_cert_state
changed_when: false
check_mode: false
- name: Host cert needed?
ansible.builtin.set_fact:
host_cert_sign: >-
{{ not state.present
or state.days_left < 60
or state.ca != state.want_ca
or state.cert_key != state.host_key
or state.principals != (ssh_host_cert_principals | sort | join(',')) }}
vars:
state: "{{ host_cert_state.stdout | from_json }}"
A new certificate is signed when there is none, when less than 60 days are left, or when it is from another CA (after a CA rotation). It is also signed when it is for another key (after a reinstall) or for other names. Otherwise the run leaves it alone and stays at changed=0.
Then the signing:
- name: Host cert
when: host_cert_sign
block:
- name: Host public key
ansible.builtin.slurp:
src: /etc/ssh/ssh_host_ed25519_key.pub
register: host_pubkey
# On the control node, with its Bao login (the admin policy may sign host certs)
- name: Sign the host key
ansible.builtin.command:
argv:
- bao
- write
- -field=signed_key
- ssh-host/sign/hosts
- cert_type=host
- valid_principals={{ ssh_host_cert_principals | join(',') }}
- public_key=-
stdin: "{{ host_pubkey.content | b64decode }}"
environment:
BAO_ADDR: "{{ bao_addr }}"
delegate_to: localhost
become: false
register: host_cert_signed
changed_when: true
when: not ansible_check_mode
- name: Install the host cert
ansible.builtin.copy:
dest: /etc/ssh/ssh_host_ed25519_key-cert.pub
content: "{{ host_cert_signed.stdout }}\n"
mode: "0644"
when: not ansible_check_mode
notify: Reload sshd
rescue:
- name: No Bao login on the control node?
ansible.builtin.fail:
msg: >-
Signing the host cert failed ({{ host_cert_signed.stderr | default('') | trim }}).
Log in on the control node first (bao login -no-print -method=oidc),
or skip host certs with -e ssh_host_cert=false.
delegate_to: localhost with become: false runs bao on the control node, as me, with my Bao token. The host needs no access to Bao at all. The rescue block turns Bao’s error into a message that says what to do.
ssh_host_cert_principals defaults to [ansible_host], the FQDN that I use to connect. A certificate is valid only for the names in it. If you also connect by a short name or an IP address, either put those into the certificate or make the client use the FQDN (see HostKeyAlias below).
Order matters: flush the handlers
In the role, the CA trust and the sshd config come first. Then a meta: flush_handlers reloads sshd at once, before anything else touches SSH access. In the last post of the series the next tasks remove the plain keys. A host’s first run must not take away the key it came in with while certificates do not work there yet.
- name: SSH certificates
ansible.builtin.include_tasks: ssh_certs.yml
- name: SSH hardening
ansible.builtin.template:
src: sshd-hardening.conf.j2
dest: /etc/ssh/sshd_config.d/00-hardening.conf
mode: "0644"
validate: sshd -t -f %s
notify: Reload sshd
- name: sshd reloaded before the keys change
ansible.builtin.meta: flush_handlers
The laptop
ssh-cert: certificates for the day
#!/bin/bash
# ssh-cert: SSH certificates from OpenBao. Logs in through Forgejo (OIDC) when there
# is no valid Bao token, then signs a public key:
#
# ssh-cert both of these (the daily call: logins and Ansible)
# ssh-cert admin ~/.ssh/id_ed25519 -> ~/.ssh/id_ed25519-cert.pub (16 h)
# ssh-cert ansible ~/.ssh/ansible_ed25519 -> ~/.ssh/ansible_ed25519-cert.pub (16 h)
# ssh-cert device <key.pub> prints a 30-day cert for a device without Bao
set -euo pipefail
export BAO_ADDR=${BAO_ADDR:-https://bao.example.net}
sign() { # role public-key output (- for stdout)
local role=$1 pub=$2 out=$3 cert
cert=$(bao write -field=signed_key "ssh-user/sign/$role" public_key=- < "$pub")
if [ "$out" = - ]; then
printf '%s\n' "$cert"
else
printf '%s\n' "$cert" > "$out.tmp" && mv "$out.tmp" "$out"
fi
printf '%s\n' "$cert" | ssh-keygen -Lf /dev/stdin | sed -n 's/^ *\(Valid: .*\)/\1/p' | sed "s|^|$role: |" >&2
}
what=${1:-all}
case $what in
all|admin|ansible|device) ;;
*) echo "usage: ssh-cert [admin|ansible|device <key.pub>]" >&2; exit 2 ;;
esac
bao token lookup >/dev/null 2>&1 || bao login -no-print -method=oidc
case $what in
all) sign admin ~/.ssh/id_ed25519.pub ~/.ssh/id_ed25519-cert.pub
sign ansible ~/.ssh/ansible_ed25519.pub ~/.ssh/ansible_ed25519-cert.pub ;;
admin) sign admin ~/.ssh/id_ed25519.pub ~/.ssh/id_ed25519-cert.pub ;;
ansible) sign ansible ~/.ssh/ansible_ed25519.pub ~/.ssh/ansible_ed25519-cert.pub ;;
device) sign device "${2:?public key file}" - ;;
esac
sudo dnf install openbao # the bao CLI
ln -s ~/ops/client/bin/ssh-cert ~/.local/bin/
ssh-cert
# admin: Valid: from 2026-11-10T08:01:00 to 2026-11-11T00:02:00
# ansible: Valid: from 2026-11-10T08:01:00 to 2026-11-11T00:02:00
The Bao token from the OIDC login is also valid for 16 hours. Run ssh-cert again during the day, and it signs without the browser.
ssh picks up id_ed25519-cert.pub next to id_ed25519 by itself. Nothing else changes in the SSH config. With IdentitiesOnly yes, ssh offers the plain key first and the certificate second. That is two of the three attempts that MaxAuthTries 3 allows, so do not add more keys to the same host entry.
Ansible uses its own key, ~/.ssh/ansible_ed25519, and its certificate is picked up the same way.
One line instead of known_hosts
# ~/.ssh/known_hosts.ca
@cert-authority *.example.org,*.example.net ssh-ed25519 AAAA...host-ca-public-key... openbao ssh-host
# ~/.ssh/config
Host *
# The first file is where ssh writes new keys
UserKnownHostsFile ~/.ssh/known_hosts ~/.ssh/known_hosts.ca
The order matters. ssh writes new host keys into the first file of the list. Keep your normal known_hosts first, so that ssh never writes into the CA file.
Now remove the old entries of your managed hosts from known_hosts:
cp ~/.ssh/known_hosts ~/.ssh/known_hosts.bak
for h in vps1.example.org vps2.example.org home1.example.net; do ssh-keygen -R "$h"; done
What happens with other hosts? Nothing changes for them. A server at a friend’s place that is not in *.example.org has no certificate from my CA. ssh asks about its key once, as before, and writes it into ~/.ssh/known_hosts. And the plain key is still offered to hosts that do not trust my user CA.
An expired or invalid host certificate does not lock you out either. ssh falls back to the plain host key, finds no entry for it and asks, as for a new host. In that situation, do not just say yes: find out why the certificate is not valid.
Short names and HostKeyAlias
A host certificate is valid for the names in it. If you connect with ssh v2 or by IP address, the name does not match. In my SSH config every alias of a host sets the FQDN as the name to check:
Host v2 vps2
HostName vps2.example.org
HostKeyAlias vps2.example.org
HostKeyAlias also helps when you reach a host over different paths: at home by its LAN address, away through a jump host. The check is always against the same name.
Roll it out
- Run Bao’s configuration (previous post), and add both CA keys to
group_vars/all.yml. - Pick one host. Keep a second SSH session open to it, as an emergency way back in:
ssh -M -S ~/.ssh/cm/standby-vps1 -fN fedora@vps1.example.org bao login -no-print -method=oidc, thenansible-playbook site.yml -l vps1 -t base --check, then the same without--check.- Test with a fresh connection, not a reused one:Expect “Server host certificate: ssh-ed25519-cert-v01@openssh.com … ID …-host” and “Authenticated … using publickey with ED25519-CERT”.
ssh -o ControlMaster=no -o ControlPath=none -v fedora@vps1.example.org true 2>&1 \ | grep -E 'Server host certificate|Authenticated' - On the host, the log says which certificate logged in:
sudo journalctl -u sshd -n 20 | grep 'ED25519-CERT' # Accepted publickey for fedora from 198.51.100.7 ... ED25519-CERT SHA256:... ID oidc-alice-admin (serial ...) CA ED25519 SHA256:... - Close the standby session (
ssh -S ~/.ssh/cm/standby-vps1 -O exit x), then the next host.
Negative tests
Test that the host refuses what it should refuse:
- A certificate for
aliceasansibleorroot: refused (not in the principals file). - An expired certificate: sign one with a short TTL in a test role, wait, try. Refused.
- A key from
ssh_revoked_keys: refused, even with a valid certificate. - The host under a name that is not in its certificate, with only the CA line in
known_hosts: ssh asks or refuses.
Do these tests sparingly from your own IP. fail2ban counts some of these failures, for example a login as an account that AllowUsers does not list. Three of them in ten minutes ban you for an hour. I learned that the hard way, more in the last post.
Next: CI deploys without a deploy key.
Further reading
- sshd_config(5):
TrustedUserCAKeys,AuthorizedPrincipalsFile,HostCertificateandRevokedKeys. - ssh_config(5):
CertificateFile,HostKeyAlias,UserKnownHostsFileandControlMaster. - sshd(8), known_hosts format: the
@cert-authorityline on the client. - ssh-keygen(1), section CERTIFICATES:
-Lto inspect a certificate,-hfor host certificates. - OpenBao signed SSH certificates: the section on host key signing.
- Ansible delegation:
delegate_to: localhostandrun_once.