Linux Lunatix

OpenBao rootless on Podman

Stefan Marx
| 11 minutes

Part 2 of the SSH series that starts with Why SSH certificates.

At the end of this post OpenBao runs on the home server as a rootless container. It has two SSH CAs (users and hosts), roles for each kind of certificate, and a login through Forgejo in the browser. Its configuration is one script in Git, and a nightly Raft snapshot goes into the backup.

Where it runs, and who can reach it

Bao runs on my home server, home1. It is reachable from the LAN and through my WireGuard tunnel, never from the internet. The home router does not forward port 443 to it.

Two DNS records make the same name work in both places:

Traefik gets the certificate with the DNS challenge, so Let’s Encrypt never needs to connect to the host. TLS ends at Traefik.

Hosts do not need Bao to check a certificate. They only need the CA’s public key, which they have as a file. Bao must be reachable only to get a certificate: from my laptop, from the Ansible control node and from the CI runner. If Bao is down, logins with existing certificates keep working.

Step 1: the quadlet

# ~svc/.config/containers/systemd/openbao.container
[Unit]
Description=OpenBao

[Container]
ContainerName=openbao
# Minor version pinned, patch releases through auto-update
Image=quay.io/openbao/openbao:2.7
AutoUpdate=registry
Label=traefik.enable=true
Label=traefik.http.routers.openbao.rule=Host(`bao.example.net`)
Label=traefik.http.routers.openbao.entrypoints=https
Label=traefik.http.routers.openbao.tls.certresolver=default
Label=traefik.http.routers.openbao.middlewares=secureHeaders@file
Label=traefik.http.services.openbao.loadbalancer.server.port=8200
Exec=bao server -config=/openbao/config/openbao.hcl
# The image's user openbao (100:1000) is svc on the host: no chown needed
UserNS=keep-id:uid=100,gid=1000
Environment=SKIP_CHOWN=1
Volume=%h/.config/openbao/openbao.hcl:/openbao/config/openbao.hcl:ro,z
Volume=openbao-data.volume:/openbao/file:Z
ReadOnly=true
Tmpfs=/tmp
DropCapability=ALL
NoNewPrivileges=true
Network=proxy.network:ip=10.89.0.14

[Service]
Restart=always

[Install]
WantedBy=default.target
# openbao-data.volume
[Volume]

Some notes:

Step 2: the server config

# ~svc/.config/openbao/openbao.hcl
ui            = true
api_addr      = "https://bao.example.net"
cluster_addr  = "http://127.0.0.1:8201"
# Integrated storage: mlock is not wanted (memory-mapped database)
disable_mlock = true

storage "raft" {
  path    = "/openbao/file"
  node_id = "home1"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = true
}

# Every request, with secret values HMACed, to stdout: journald through Podman
audit "file" "stdout" {
  options {
    file_path = "stdout"
  }
}

Integrated storage (Raft) is a single directory and needs no external database. A single node is a valid Raft cluster.

tls_disable looks scary, but the listener is reachable only on the Podman network, from Traefik. TLS ends there.

The audit device is declared in the config file, so it exists from the very first start. The log is in the journal with everything else.

Step 3: initialise

Start it, then initialise it once:

systemctl --user daemon-reload && systemctl --user start openbao
export BAO_ADDR=https://bao.example.net
bao operator init -key-shares=1 -key-threshold=1 -format=json > /dev/shm/init.json

init prints the unseal key and the initial root token. They go straight into pass, and the file is deleted:

jq -r '.unseal_keys_b64[0]' /dev/shm/init.json | pass insert -m infra/home1/openbao-unseal-key
jq -r '.root_token' /dev/shm/init.json | pass insert -m infra/home1/openbao-root-token
shred -u /dev/shm/init.json

One key share is enough for one person. Shamir splitting into several shares makes sense when several people must agree to unseal.

I use /dev/shm so that the file is in memory and never on disk.

Step 4: unseal, and why not automatically

Bao encrypts everything in its storage. After every start it is sealed: it has the data but not the key. Somebody must give it the unseal key.

I wanted auto-unseal with the home server’s TPM. The plan: encrypt a static seal key with systemd-creds --with-key=tpm2 and give it to the container with LoadCredentialEncrypted=. Bao would then unseal itself after a reboot, and a stolen disk would be useless without the board.

It did not work. The board has a TPM 1.2, and systemd-creds supports only TPM 2.0. So I unseal by hand, from my laptop, with the key from pass:

#!/bin/bash
# bao-unseal: run on the laptop, at home or through the tunnel
set -euo pipefail
addr=${BAO_ADDR:-https://bao.example.net}
sealed() {
    local s
    s=$(curl -fsS -m 15 "$addr/v1/sys/seal-status") || { echo "OpenBao not reachable at $addr" >&2; exit 1; }
    python3 -c 'import json,sys; sys.exit(0 if json.loads(sys.argv[1])["sealed"] else 1)' "$s"
}

if ! sealed; then
    echo "OpenBao is not sealed"
    exit 0
fi
# the key goes to curl on stdin, never onto a command line
pass show infra/home1/openbao-unseal-key | head -1 \
    | python3 -c 'import json,sys; print(json.dumps({"key": sys.stdin.read().strip()}))' \
    | curl -fsS -m 15 -X PUT -d @- -o /dev/null "$addr/v1/sys/unseal"
if sealed; then
    echo "still sealed" >&2
    exit 1
fi
echo "OpenBao unsealed"

The key never appears in ps or in the shell history. A Prometheus blackbox probe watches /v1/sys/health, which answers 503 while Bao is sealed. An alert tells me after five minutes.

This is less work than it sounds. The home server reboots only when I reboot it. But remember that a podman auto-update of the image, or any change to the quadlet, restarts the container, and the restart seals it again.

Step 5: the configuration as a script

Everything inside Bao (mounts, roles, policies, logins) is set by one script in the repository. It is idempotent: I change the script and run it again. There is no click in the web UI that I would forget.

The script needs a token with full rights. The first run uses the root token from pass. Later runs use my userpass login, which the script creates:

BAO_TOKEN=$(pass show infra/home1/openbao-root-token) ./bao-config.sh   # first run
bao login -no-print -method=userpass username=alice && ./bao-config.sh  # later

Always use -no-print. Without it, bao login prints the token on the screen.

After the first run, revoke the root token. A new one can be made from the unseal key with bao operator generate-root when you need it.

The SSH CAs

set -euo pipefail
export BAO_ADDR=${BAO_ADDR:-https://bao.example.net}

mounted() { bao "$1" list -format=json | python3 -c 'import json,sys; sys.exit(0 if sys.argv[1] + "/" in json.load(sys.stdin) else 1)' "$2"; }
secrets() { mounted secrets "$2" || bao secrets enable -path="$2" "$1"; }
auth() { mounted auth "$2" || bao auth enable -path="$2" "$1"; }

# Two SSH CAs, keys generated inside Bao (ed25519)
for m in ssh-user ssh-host; do
    secrets ssh "$m"
    bao read "$m/config/ca" >/dev/null 2>&1 \
        || bao write "$m/config/ca" generate_signing_key=true key_type=ed25519 >/dev/null
done

generate_signing_key=true creates the key inside Bao. It never leaves it. bao read ssh-user/config/ca shows the public key, which is what the hosts need.

The || bao write part makes it idempotent. A second run would otherwise replace the CA key, and every certificate would be invalid.

The user roles

A small function writes a role. The JSON body is easier to read than a long list of key=value arguments:

# Key ID: who asked (token display name) and with which role; it shows in sshd's log
user_role() {  # name principal ttl extensions [critical options as JSON]
    local exts crit=${5:-'{}'}
    exts=$(python3 -c 'import json,sys; print(json.dumps({e: "" for e in sys.argv[1].split(",") if e}))' "$4")
    bao write "ssh-user/roles/$1" - >/dev/null <<JSON
{
  "key_type": "ca",
  "allow_user_certificates": true,
  "allowed_users": "$2",
  "default_user": "$2",
  "ttl": "$3",
  "max_ttl": "$3",
  "allowed_extensions": "$4",
  "default_extensions": $exts,
  "allowed_critical_options": "$(python3 -c 'import json,sys; print(",".join(json.loads(sys.argv[1])))' "$crit")",
  "default_critical_options": $crit,
  "allowed_user_key_lengths": {"ed25519": [0], "sk-ed25519": [0]},
  "key_id_format": "{{token_display_name}}-{{role_name}}"
}
JSON
}
user_role admin alice 16h permit-pty,permit-port-forwarding
user_role device alice 720h permit-pty,permit-port-forwarding
user_role ansible ansible 16h permit-pty
user_role deploy-websites deploy-websites 5m "" '{"force-command": "podman auto-update"}'

What each field does:

The host role

bao write ssh-host/roles/hosts - >/dev/null <<'JSON'
{
  "key_type": "ca",
  "allow_host_certificates": true,
  "allowed_domains": "example.org,example.net",
  "allow_subdomains": true,
  "ttl": "8760h",
  "max_ttl": "8760h",
  "allowed_user_key_lengths": {"ed25519": [0]},
  "key_id_format": "{{token_display_name}}-host"
}
JSON

Host certificates are valid for one year. The Ansible run renews them when less than 60 days are left (next post).

Policies

bao policy write admin - <<'HCL'
# Daily use (OIDC login): sign user and host certificates
path "ssh-user/sign/admin"   { capabilities = ["update"] }
path "ssh-user/sign/device"  { capabilities = ["update"] }
path "ssh-user/sign/ansible" { capabilities = ["update"] }
path "ssh-host/sign/hosts"   { capabilities = ["update"] }
HCL
bao policy write operator - <<'HCL'
# Running Bao itself (userpass login): everything
path "*" { capabilities = ["create", "read", "update", "patch", "delete", "list", "sudo"] }
HCL
bao policy write backup - <<'HCL'
path "sys/storage/raft/snapshot" { capabilities = ["read"] }
HCL

The daily login can sign and do nothing else. It cannot read the configuration, change roles or create tokens.

Step 6: logins

OIDC through Forgejo, for every day

Forgejo is an OpenID Connect provider. In Forgejo, go to Settings, Applications, and create an OAuth2 application with two redirect URIs:

http://localhost:8250/oidc/callback
https://bao.example.net/ui/vault/auth/oidc/oidc/callback

The first one is for the CLI. bao login -method=oidc opens the browser and waits on port 8250 for the answer. The second one is for the web UI. Store the client ID and the secret in pass, then:

auth oidc oidc
python3 -c 'import json,sys; print(json.dumps({"oidc_discovery_url": "https://git.example.org", "oidc_client_id": sys.argv[1], "oidc_client_secret": sys.stdin.read().strip(), "default_role": "alice"}))' \
    "$(pass show infra/home1/openbao-oidc-client-id | head -1)" < <(pass show infra/home1/openbao-oidc-client-secret | head -1) \
    | bao write auth/oidc/config - >/dev/null

bao write auth/oidc/role/alice - >/dev/null <<'JSON'
{
  "user_claim": "preferred_username",
  "bound_claims": {"preferred_username": "alice"},
  "oidc_scopes": "openid,profile",
  "allowed_redirect_uris": "http://localhost:8250/oidc/callback,https://bao.example.net/ui/vault/auth/oidc/oidc/callback",
  "token_policies": "admin",
  "token_ttl": "16h",
  "token_max_ttl": "16h"
}
JSON

bound_claims is important. Without it, every account on the Forgejo instance could log in to Bao and get the admin policy.

The secret goes to bao on stdin, inside the JSON, and never onto a command line.

userpass, for running Bao

auth userpass userpass
if ! bao read auth/userpass/users/alice >/dev/null 2>&1; then
    pass show infra/home1/openbao-alice-password >/dev/null 2>&1 \
        || pass generate -n infra/home1/openbao-alice-password 32 >/dev/null
    pass show infra/home1/openbao-alice-password | head -1 \
        | python3 -c 'import json,sys; print(json.dumps({"password": sys.stdin.read().strip(), "token_policies": "admin,operator", "token_ttl": "1h", "token_max_ttl": "4h"}))' \
        | bao write auth/userpass/users/alice - >/dev/null
fi

This login has the operator policy for changes to Bao, with a short token. It is also my way in when Forgejo is down, for example during the reinstall of the host it runs on.

AppRole, for the backup

auth approle approle
bao write auth/approle/role/backup token_policies=backup token_ttl=5m token_max_ttl=10m \
    secret_id_ttl=0 token_num_uses=0 >/dev/null
if ! pass show infra/home1/openbao-backup-approle >/dev/null 2>&1; then
    role_id=$(bao read -field=role_id auth/approle/role/backup/role-id)
    bao write -f -format=json auth/approle/role/backup/secret-id \
        | python3 -c 'import json,sys; print(json.dumps({"role_id": sys.argv[1], "secret_id": json.load(sys.stdin)["data"]["secret_id"]}))' "$role_id" \
        | pass insert -m infra/home1/openbao-backup-approle >/dev/null
fi

Ansible writes this JSON to /root/.config/openbao/backup-approle.json on the home server (mode 0600). The backup script uses it to log in.

Step 7: backup and restore

Do not back up the live Raft directory. It is a database that is being written to. Take a snapshot through the API instead. That is one more step in the backup script from the monitoring and backup post:

# a backup_functions entry for home1
openbao_snapshot() {
    local bao=(curl -fsS -m 60 --resolve bao.example.net:443:127.0.0.1) token
    token=$("${bao[@]}" -d @/root/.config/openbao/backup-approle.json \
        https://bao.example.net/v1/auth/approle/login \
        | python3 -c 'import json,sys; print(json.load(sys.stdin)["auth"]["client_token"])') || return 1
    "${bao[@]}" -H "X-Vault-Token: $token" -o "$DUMP_DIR/openbao/raft.snap.tmp" \
        https://bao.example.net/v1/sys/storage/raft/snapshot \
        && mv "$DUMP_DIR/openbao/raft.snap.tmp" "$DUMP_DIR/openbao/raft.snap"
}

--resolve ...:127.0.0.1 sends the request to the local Traefik, with the right name for the certificate. The live volume is excluded from restic. While Bao is sealed this step fails, and the backup reports it.

The snapshot is encrypted with Bao’s keys. To restore it on any host with Podman:

  1. Start an empty Bao, initialise it with one share, and unseal it with the new key.
  2. Log in with the new root token, then run bao operator raft snapshot restore -force raft.snap.
  3. Bao is sealed again, now with the old key. Unseal it with the old key from pass.
  4. Everything is back: both CAs, the roles, the policies, the logins. Revoke the new root token.

Try this once, on a VM, before you need it.

Check

bao login -no-print -method=oidc       # the browser opens, log in to Forgejo
bao token lookup -format=json | jq '.data | {display_name, policies, ttl}'
bao read -field=public_key ssh-user/config/ca
bao write -field=signed_key ssh-user/sign/admin public_key=@$HOME/.ssh/id_ed25519.pub \
    | ssh-keygen -Lf /dev/stdin
journalctl --user -u openbao -n 3 -o cat | jq -c '{type, path: .request.path}'

The certificate shows principal alice, 16 hours, key ID oidc-alice-admin. The audit log shows the request.

Next: Hosts that trust the SSH CA.

Further reading

Previous: Why SSH certificates Next: Hosts that trust the SSH CA

Related content

Removing the plain SSH keys
CI deploys without a deploy key
Hosts that trust the SSH CA