Linux Lunatix

Removing the plain SSH keys

Stefan Marx
| 8 minutes

Part 5 of the SSH series that starts with Why SSH certificates.

Certificates work on every host now. The old plain keys are still in the authorized_keys files, and while they are there, the certificates change nothing about security. A stolen laptop key would still open every door. This post removes the plain keys. Only one break-glass key per account stays, for the day when Bao is gone.

The target

AccountCertificate principalsPlain keys
the admin account (fedora)alicebreak-glass
svc (the apps)alice (+ deploy-websites on the web host)none
ansibleansiblebreak-glass

svc gets no plain key at all. If I need it without Bao, I log in as the admin account and use sudo.

Step 1: the break-glass key

The break-glass key is a normal Ed25519 key. Its private part lives only in pass, never in ~/.ssh. Create it in memory and move it straight into pass:

d=$(mktemp -d -p /dev/shm)
ssh-keygen -q -t ed25519 -N '' -C 'break-glass@infra' -f "$d/key"
pass insert -m infra/ssh-break-glass < "$d/key"
cp "$d/key.pub" ~/ops/client/ssh/break-glass.pub
shred -u "$d/key" && rm -r "$d"

The public part goes into the inventory:

# group_vars/all.yml
ssh_break_glass_key: "ssh-ed25519 AAAA...break-glass-public-key... break-glass@infra"

To use it, a small script loads the key from pass into ssh-agent for 15 minutes. It never writes it to disk:

#!/bin/bash
# ssh-break-glass: only when Bao is gone and the certificates have expired
#
#   ssh-break-glass vps1                       # any ssh arguments
#   ssh-break-glass -- ansible-playbook ...    # a command: Ansible with the key
set -euo pipefail
pub=$(dirname "$(readlink -f "$0")")/../ssh/break-glass.pub
pass show infra/ssh-break-glass | ssh-add -q -t 15m -
if [ "${1:-}" = "--" ]; then
    shift
    exec "$@" --private-key "$pub"
fi
exec ssh -i "$pub" "$@"

ssh-add - reads the private key from stdin. -t 15m removes it from the agent after 15 minutes.

-i with a .pub file looks odd. ssh then selects the matching key in the agent and offers it first. That matters with IdentitiesOnly yes and MaxAuthTries 3. Ansible’s --private-key works the same way.

Test it now, while everything works, and not on the day you need it.

Step 2: authorized_keys from the inventory

Each host lists its plain keys per account:

# host_vars/vps1.yml
ssh_authorized_keys:
  fedora: ["{{ ssh_break_glass_key }}"]
  svc: []
  ansible: ["{{ ssh_break_glass_key }}"]

The rules: a listed account gets exactly these keys, and nothing else stays in its file. An empty list removes the file. An account that is not listed is left alone, so the role cannot touch accounts it does not know.

# roles/base/tasks/ssh_keys.yml
- name: Accounts' home directories
  ansible.builtin.getent:
    database: passwd

- name: authorized_keys with exactly the listed keys
  ansible.posix.authorized_key:
    user: "{{ item.key }}"
    key: "{{ item.value | join('\n') }}"
    exclusive: true
  loop: "{{ ssh_authorized_keys | dict2items | selectattr('value') }}"
  loop_control:
    label: "{{ item.key }}"

- name: No authorized_keys for cert-only accounts
  ansible.builtin.file:
    path: "{{ ansible_facts.getent_passwd[item.key][4] }}/.ssh/authorized_keys"
    state: absent
  loop: "{{ ssh_authorized_keys | dict2items | rejectattr('value') }}"
  loop_control:
    label: "{{ item.key }}"

exclusive: true is what removes the old keys. The home directory comes from getent, because svc and the admin account do not have to live under /home/<name>.

In the role’s main.yml these tasks come after the flush_handlers from the previous post. On a host’s first run, sshd already trusts the CA when the keys go away.

bootstrap.yml from the Ansible post still installs the control nodes’ plain keys for ansible. That is for the first contact with a new host, before it trusts the CA. The first site.yml run then replaces them with the break-glass key.

Step 3: roll it out, one host at a time

I did this first on a throwaway VM, then host by host.

  1. Get fresh certificates: ssh-cert. Check that both are valid: ssh-keygen -Lf ~/.ssh/id_ed25519-cert.pub and the same for ansible_ed25519-cert.pub.
  2. Open a standby session as the admin account and leave it open:
    ssh -M -S ~/.ssh/cm/standby-vps1 -fN fedora@vps1.example.org
    
  3. Close Ansible’s reused connections, then run the base role:
    for s in ~/.ansible/cp/*; do ssh -O exit -o ControlPath="$s" x 2>/dev/null; done
    ansible-playbook playbooks/site.yml -l vps1 -t base --check
    ansible-playbook playbooks/site.yml -l vps1 -t base
    
  4. Test with fresh connections. Each of these must work:
    F="-o ControlMaster=no -o ControlPath=none"
    ssh $F fedora@vps1.example.org true             # certificate
    ssh $F svc@vps1.example.org true                # certificate
    ansible vps1 -m ping                            # the ansible certificate
    ssh-break-glass $F fedora@vps1.example.org true # break-glass key
    
    And this must fail:
    d=$(mktemp -d) && cp ~/.ssh/id_ed25519 "$d/"
    ssh $F -o IdentityAgent=none -o IdentitiesOnly=yes -i "$d/id_ed25519" fedora@vps1.example.org true
    rm -r "$d"
    
    The copy has no -cert.pub next to it, and IdentityAgent=none keeps the agent out, so ssh offers only the plain key. Before this step it was in authorized_keys. Now it is refused.
  5. The second run must report changed=0. Then close the standby session.

Devices that cannot talk to Bao

My tablet runs Blink, an SSH client for iOS. Blink cannot log in to Bao, but it can use a certificate. So the tablet gets a certificate for 30 days, which I sign on the laptop.

  1. In Blink, create an Ed25519 key and copy its public part to the laptop.
  2. Sign it with the device role:
    ssh-cert device tablet.pub
    
  3. In Blink: Keys, the key, Certificate, paste.
  4. Put a reminder into the calendar a few days before the 30 days end. An expired certificate is simply refused.

Blink cannot check host certificates. It asks about each host key once. Compare the fingerprint with the host’s real key before you accept it:

ssh vps1.example.org ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

A second Ansible control node that is not always on gets a 30-day certificate for its ansible key the same way, with a role of its own.

A lost device

For the laptop, the certificates expire within 16 hours. But its Bao token can sign new ones until the token expires. Revoke it at once from another machine:

bao login -no-print -method=userpass username=alice
bao list auth/token/accessors
bao token lookup -accessor <accessor>        # find the oidc-alice tokens
bao token revoke -accessor <accessor>

For a device with a 30-day certificate, put its public key into the revoked list and roll it out:

# group_vars/all.yml
ssh_revoked_keys:
  - "ssh-ed25519 AAAA...the-tablet's-public-key..."
ansible-playbook playbooks/site.yml -t base

sshd then refuses that key with any certificate. The device’s other keys (GitHub, an SFTP backup box) are plain keys and must be removed there, as before.

When Bao is gone

How I nearly locked myself out

Two things went wrong during this work. Each one taught me a rule.

fail2ban banned my home IP

During a negative test I connected to a host by its bare FQDN, without the User that my SSH config sets for its short name. ssh therefore used my local user name. That account is not in the host’s AllowUsers, and fail2ban counts such a login as a failure. Together with my negative tests that made three failures within ten minutes, and fail2ban banned my home IP for an hour.

Interestingly, a refused key alone (“Failed publickey”) does not count in fail2ban’s sshd filter. A login as an account that AllowUsers does not allow does count.

I got in again through another of my hosts, which still had a working connection, and unbanned the IP there:

ssh -J other-host vps1.example.org 'sudo fail2ban-client set sshd unbanip 203.0.113.50'

The rule: test negative cases sparingly, with explicit user names, and know how you get in from another IP.

Ansible’s tests passed without a certificate

After the first host, all Ansible tests passed. Then I noticed that my laptop had no Ansible certificate at all. The ssh-cert script signed only the login key at that point. Ansible had passed because it reused its SSH connections from before the change (ControlPersist). A fresh connection would have failed.

Since then ssh-cert signs both keys by default, and I close ~/.ansible/cp/* before every test.

The rules, in short

Where this leaves me

Every host trusts one CA key and lists, per account, which principals may log in. My laptop gets certificates for the day with one command and needs no known_hosts entries for my hosts. CI jobs get a certificate for five minutes and can run one command. A reinstalled host gets a new host certificate during its first Ansible run, and nobody has to check a fingerprint over a console. The only plain key left lives in pass.

That was the last post of the series. Part 1, about consolidating the servers, starts with Four servers, three distributions: why I consolidated.

Further reading

Previous: CI deploys without a deploy key

Related content

Hosts that trust the SSH CA
CI deploys without a deploy key
OpenBao rootless on Podman