Linux Lunatix

Hosts that trust the SSH CA

Stefan Marx
| 11 minutes

Part 3 of the SSH series that starts with Why SSH certificates.

Bao from the previous post can sign certificates. Now the hosts must trust them, and the laptop must use them. At the end of this post:

The plain keys stay in place during this step. They go away in the last post of the series, when everything else works.

The inventory

The CA public keys go into group_vars/all.yml. They are public, and a rollout needs no Bao for them:

# group_vars/all.yml
# bao read -field=public_key ssh-user/config/ca
ssh_user_ca: "ssh-ed25519 AAAA...user-ca-public-key... openbao ssh-user"
# bao read -field=public_key ssh-host/config/ca
ssh_host_ca: "ssh-ed25519 AAAA...host-ca-public-key... openbao ssh-host"
bao_addr: https://bao.example.net

Each host says who may log in as which account:

# host_vars/vps1.yml
ssh_principals:
  fedora: [alice]
  svc: [alice]
  ansible: [ansible]
# group_vars/websites.yml
ssh_principals:
  fedora: [alice]
  svc: [alice, 'restrict,command="podman auto-update" deploy-websites']
  ansible: [ansible]

A line may start with options, in the same syntax as in authorized_keys. The host enforces them, whatever the certificate says. The first post explains why that matters.

The base role: trust the CA

# roles/base/tasks/ssh_certs.yml
- name: SSH CA directory
  ansible.builtin.file:
    path: /etc/ssh/ca
    state: directory
    mode: "0755"

- name: User CA
  ansible.builtin.copy:
    dest: /etc/ssh/ca/user_ca.pub
    content: "{{ ssh_user_ca }}\n"
    mode: "0644"
  when: ssh_user_ca | length > 0
  notify: Reload sshd

- name: Principals directory
  ansible.builtin.file:
    path: /etc/ssh/principals
    state: directory
    mode: "0755"

- name: Principals per account
  ansible.builtin.copy:
    dest: /etc/ssh/principals/{{ item.key }}
    content: "{{ item.value | join('\n') }}\n"
    mode: "0644"
  loop: "{{ ssh_principals | dict2items }}"
  loop_control:
    label: "{{ item.key }}"

- name: Principals files in place
  ansible.builtin.find:
    paths: /etc/ssh/principals
  register: principals_files

# Exclusive: an account no longer listed has no principals file, so no cert logs in as it
- name: Principals files of accounts no longer listed
  ansible.builtin.file:
    path: "{{ item.path }}"
    state: absent
  loop: "{{ principals_files.files | rejectattr('path', 'in', ssh_principals.keys() | map('regex_replace', '^', '/etc/ssh/principals/') | list) }}"
  loop_control:
    label: "{{ item.path }}"

- name: Revoked keys
  ansible.builtin.copy:
    dest: /etc/ssh/revoked_keys
    content: "{{ (ssh_revoked_keys | join('\n') ~ '\n') if ssh_revoked_keys else '' }}"
    mode: "0644"
  notify: Reload sshd

The principals files need no reload. sshd reads them at each login.

The “no longer listed” task makes the list exclusive. If I remove an account from ssh_principals, its file goes away, and no certificate can log in as that account any more. A list that only adds is a list that only grows.

RevokedKeys accepts a plain file of public keys. A key in it is refused even with a valid certificate. That covers a lost device whose certificate is still valid for weeks (the last post of the series has the details). The file must exist, even empty. If RevokedKeys points to a missing file, sshd refuses all public key logins.

sshd’s config

The hardening drop-in from the Ansible groundwork post gets these lines:

HostKey /etc/ssh/ssh_host_ed25519_key
{% if ssh_host_cert_installed | default(false) %}
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
{% endif %}
HostKeyAlgorithms ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
PubkeyAcceptedAlgorithms ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com
# The CA must sign with Ed25519, too
CASignatureAlgorithms ssh-ed25519

{% if ssh_user_ca %}
# Who may log in as an account is the account's principals file (with options),
# not the certificate alone
TrustedUserCAKeys /etc/ssh/ca/user_ca.pub
AuthorizedPrincipalsFile /etc/ssh/principals/%u
{% endif %}
RevokedKeys /etc/ssh/revoked_keys

HostKeyAlgorithms must list the certificate type, or sshd never offers the host certificate. The same applies to PubkeyAcceptedAlgorithms for user certificates.

The HostCertificate line appears only once the certificate exists, so sshd never points to a missing file.

Host certificates from the Ansible run

The host’s key is on the host. The CA is in Bao. The control node has a Bao login. So the playbook reads the host’s public key, has it signed on the control node and installs the certificate on the host.

First the role decides whether a new certificate is needed. A shell task describes the current one as JSON:

- name: Host cert state
  ansible.builtin.shell: |
    set -o pipefail
    c=/etc/ssh/ssh_host_ed25519_key-cert.pub
    k=/etc/ssh/ssh_host_ed25519_key.pub
    if [ ! -s $c ]; then echo '{"present": false}'; exit 0; fi
    info=$(ssh-keygen -Lf $c)
    to=$(sed -n 's/^ *Valid: from .* to \(.*\)$/\1/p' <<<"$info")
    left=$(( ($(date -d "$to" +%s 2>/dev/null || date +%s) - $(date +%s)) / 86400 ))
    ca=$(sed -n 's/^ *Signing CA: [A-Z0-9-]* \(SHA256:[^ ]*\).*/\1/p' <<<"$info")
    cert_key=$(sed -n 's/^ *Public key: [A-Z0-9-]* \(SHA256:[^ ]*\).*/\1/p' <<<"$info")
    principals=$(sed -n '/^ *Principals:/,/^ *Critical Options:/p' <<<"$info" | sed '1d;$d' | tr -d ' ' | sort | paste -sd,)
    printf '{"present": true, "days_left": %d, "ca": "%s", "want_ca": "%s", "cert_key": "%s", "host_key": "%s", "principals": "%s"}\n' \
        "$left" "$ca" "$(ssh-keygen -lf /dev/stdin <<<"$HOST_CA" | awk '{print $2}')" \
        "$cert_key" "$(ssh-keygen -lf $k | awk '{print $2}')" "$principals"
  args:
    executable: /bin/bash
  environment:
    HOST_CA: "{{ ssh_host_ca }}"
  register: host_cert_state
  changed_when: false
  check_mode: false

- name: Host cert needed?
  ansible.builtin.set_fact:
    host_cert_sign: >-
      {{ not state.present
         or state.days_left < 60
         or state.ca != state.want_ca
         or state.cert_key != state.host_key
         or state.principals != (ssh_host_cert_principals | sort | join(',')) }}
  vars:
    state: "{{ host_cert_state.stdout | from_json }}"

A new certificate is signed when there is none, when less than 60 days are left, or when it is from another CA (after a CA rotation). It is also signed when it is for another key (after a reinstall) or for other names. Otherwise the run leaves it alone and stays at changed=0.

Then the signing:

- name: Host cert
  when: host_cert_sign
  block:
    - name: Host public key
      ansible.builtin.slurp:
        src: /etc/ssh/ssh_host_ed25519_key.pub
      register: host_pubkey

    # On the control node, with its Bao login (the admin policy may sign host certs)
    - name: Sign the host key
      ansible.builtin.command:
        argv:
          - bao
          - write
          - -field=signed_key
          - ssh-host/sign/hosts
          - cert_type=host
          - valid_principals={{ ssh_host_cert_principals | join(',') }}
          - public_key=-
        stdin: "{{ host_pubkey.content | b64decode }}"
      environment:
        BAO_ADDR: "{{ bao_addr }}"
      delegate_to: localhost
      become: false
      register: host_cert_signed
      changed_when: true
      when: not ansible_check_mode

    - name: Install the host cert
      ansible.builtin.copy:
        dest: /etc/ssh/ssh_host_ed25519_key-cert.pub
        content: "{{ host_cert_signed.stdout }}\n"
        mode: "0644"
      when: not ansible_check_mode
      notify: Reload sshd
  rescue:
    - name: No Bao login on the control node?
      ansible.builtin.fail:
        msg: >-
          Signing the host cert failed ({{ host_cert_signed.stderr | default('') | trim }}).
          Log in on the control node first (bao login -no-print -method=oidc),
          or skip host certs with -e ssh_host_cert=false.

delegate_to: localhost with become: false runs bao on the control node, as me, with my Bao token. The host needs no access to Bao at all. The rescue block turns Bao’s error into a message that says what to do.

ssh_host_cert_principals defaults to [ansible_host], the FQDN that I use to connect. A certificate is valid only for the names in it. If you also connect by a short name or an IP address, either put those into the certificate or make the client use the FQDN (see HostKeyAlias below).

Order matters: flush the handlers

In the role, the CA trust and the sshd config come first. Then a meta: flush_handlers reloads sshd at once, before anything else touches SSH access. In the last post of the series the next tasks remove the plain keys. A host’s first run must not take away the key it came in with while certificates do not work there yet.

- name: SSH certificates
  ansible.builtin.include_tasks: ssh_certs.yml

- name: SSH hardening
  ansible.builtin.template:
    src: sshd-hardening.conf.j2
    dest: /etc/ssh/sshd_config.d/00-hardening.conf
    mode: "0644"
    validate: sshd -t -f %s
  notify: Reload sshd

- name: sshd reloaded before the keys change
  ansible.builtin.meta: flush_handlers

The laptop

ssh-cert: certificates for the day

#!/bin/bash
# ssh-cert: SSH certificates from OpenBao. Logs in through Forgejo (OIDC) when there
# is no valid Bao token, then signs a public key:
#
#   ssh-cert                      both of these (the daily call: logins and Ansible)
#   ssh-cert admin                ~/.ssh/id_ed25519      -> ~/.ssh/id_ed25519-cert.pub      (16 h)
#   ssh-cert ansible              ~/.ssh/ansible_ed25519 -> ~/.ssh/ansible_ed25519-cert.pub (16 h)
#   ssh-cert device <key.pub>     prints a 30-day cert for a device without Bao
set -euo pipefail
export BAO_ADDR=${BAO_ADDR:-https://bao.example.net}

sign() {  # role public-key output (- for stdout)
    local role=$1 pub=$2 out=$3 cert
    cert=$(bao write -field=signed_key "ssh-user/sign/$role" public_key=- < "$pub")
    if [ "$out" = - ]; then
        printf '%s\n' "$cert"
    else
        printf '%s\n' "$cert" > "$out.tmp" && mv "$out.tmp" "$out"
    fi
    printf '%s\n' "$cert" | ssh-keygen -Lf /dev/stdin | sed -n 's/^ *\(Valid: .*\)/\1/p' | sed "s|^|$role: |" >&2
}

what=${1:-all}
case $what in
    all|admin|ansible|device) ;;
    *) echo "usage: ssh-cert [admin|ansible|device <key.pub>]" >&2; exit 2 ;;
esac

bao token lookup >/dev/null 2>&1 || bao login -no-print -method=oidc
case $what in
    all)     sign admin ~/.ssh/id_ed25519.pub ~/.ssh/id_ed25519-cert.pub
             sign ansible ~/.ssh/ansible_ed25519.pub ~/.ssh/ansible_ed25519-cert.pub ;;
    admin)   sign admin ~/.ssh/id_ed25519.pub ~/.ssh/id_ed25519-cert.pub ;;
    ansible) sign ansible ~/.ssh/ansible_ed25519.pub ~/.ssh/ansible_ed25519-cert.pub ;;
    device)  sign device "${2:?public key file}" - ;;
esac
sudo dnf install openbao        # the bao CLI
ln -s ~/ops/client/bin/ssh-cert ~/.local/bin/
ssh-cert
# admin: Valid: from 2026-11-10T08:01:00 to 2026-11-11T00:02:00
# ansible: Valid: from 2026-11-10T08:01:00 to 2026-11-11T00:02:00

The Bao token from the OIDC login is also valid for 16 hours. Run ssh-cert again during the day, and it signs without the browser.

ssh picks up id_ed25519-cert.pub next to id_ed25519 by itself. Nothing else changes in the SSH config. With IdentitiesOnly yes, ssh offers the plain key first and the certificate second. That is two of the three attempts that MaxAuthTries 3 allows, so do not add more keys to the same host entry.

Ansible uses its own key, ~/.ssh/ansible_ed25519, and its certificate is picked up the same way.

One line instead of known_hosts

# ~/.ssh/known_hosts.ca
@cert-authority *.example.org,*.example.net ssh-ed25519 AAAA...host-ca-public-key... openbao ssh-host
# ~/.ssh/config
Host *
    # The first file is where ssh writes new keys
    UserKnownHostsFile ~/.ssh/known_hosts ~/.ssh/known_hosts.ca

The order matters. ssh writes new host keys into the first file of the list. Keep your normal known_hosts first, so that ssh never writes into the CA file.

Now remove the old entries of your managed hosts from known_hosts:

cp ~/.ssh/known_hosts ~/.ssh/known_hosts.bak
for h in vps1.example.org vps2.example.org home1.example.net; do ssh-keygen -R "$h"; done

What happens with other hosts? Nothing changes for them. A server at a friend’s place that is not in *.example.org has no certificate from my CA. ssh asks about its key once, as before, and writes it into ~/.ssh/known_hosts. And the plain key is still offered to hosts that do not trust my user CA.

An expired or invalid host certificate does not lock you out either. ssh falls back to the plain host key, finds no entry for it and asks, as for a new host. In that situation, do not just say yes: find out why the certificate is not valid.

Short names and HostKeyAlias

A host certificate is valid for the names in it. If you connect with ssh v2 or by IP address, the name does not match. In my SSH config every alias of a host sets the FQDN as the name to check:

Host v2 vps2
    HostName vps2.example.org
    HostKeyAlias vps2.example.org

HostKeyAlias also helps when you reach a host over different paths: at home by its LAN address, away through a jump host. The check is always against the same name.

Roll it out

  1. Run Bao’s configuration (previous post), and add both CA keys to group_vars/all.yml.
  2. Pick one host. Keep a second SSH session open to it, as an emergency way back in:
    ssh -M -S ~/.ssh/cm/standby-vps1 -fN fedora@vps1.example.org
    
  3. bao login -no-print -method=oidc, then ansible-playbook site.yml -l vps1 -t base --check, then the same without --check.
  4. Test with a fresh connection, not a reused one:
    ssh -o ControlMaster=no -o ControlPath=none -v fedora@vps1.example.org true 2>&1 \
        | grep -E 'Server host certificate|Authenticated'
    
    Expect “Server host certificate: ssh-ed25519-cert-v01@openssh.com … ID …-host” and “Authenticated … using publickey with ED25519-CERT”.
  5. On the host, the log says which certificate logged in:
    sudo journalctl -u sshd -n 20 | grep 'ED25519-CERT'
    # Accepted publickey for fedora from 198.51.100.7 ... ED25519-CERT SHA256:... ID oidc-alice-admin (serial ...) CA ED25519 SHA256:...
    
  6. Close the standby session (ssh -S ~/.ssh/cm/standby-vps1 -O exit x), then the next host.

Negative tests

Test that the host refuses what it should refuse:

Do these tests sparingly from your own IP. fail2ban counts some of these failures, for example a login as an account that AllowUsers does not list. Three of them in ten minutes ban you for an hour. I learned that the hard way, more in the last post.

Next: CI deploys without a deploy key.

Further reading

Previous: OpenBao rootless on Podman Next: CI deploys without a deploy key

Related content

Removing the plain SSH keys
CI deploys without a deploy key
OpenBao rootless on Podman