Enabling encryption after install

Vaks PM · Day-2 encryption · August 2026

Overview

Each encryption layer is enabled independently, with its own procedure, risk level and impact on the service. Two layers are driven from the application (they are not treated here as host operations):

• Application secrets (the KEK/DEK keyring) and key rotation — from the admin interface (see Key rotation).
• Encrypted backups — a configuration variable on the backup hosts.

The infrastructure layers (disk LUKS, internal PostgreSQL TLS, overlay IPsec, etcd) touch the hosts and their system services. These are the ones the two methods below address.

The layers, by risk level

Which method?

Both methods produce exactly the same result. They differ in where you type the commands.

Pick one or the other; you do not need to do both. etcd TLS and Key rotation are identical regardless of the method and are described once, at the end.

Method A — orchestrated from a control station

The infra/crypto/retrofit.mjs tooling runs on the administration machine (the same host, with the same key-based SSH access, as deployment). It connects to each host itself, applies the layer, handles the node-by-node sequencing and verifies. You do no manual SSH. Every command in this section is run from the control station.

Data at rest (LUKS)

# control station
node infra/crypto/retrofit.mjs enable-luks --pg --force     # add --garage for the object store

The orchestrator encrypts the replicas first, switches the primary to an encrypted replica, then encrypts the former primary — without downtime. The operation is idempotent (re-running it skips already-encrypted nodes).

Internal PostgreSQL TLS

# control station
node infra/crypto/retrofit.mjs enable-pg-tls --force

Sets ssl=on in the cluster configuration and restarts the members one at a time (replicas first). The application stays available (PostgreSQL still accepts non-TLS connections). Moving the application to TLS is a separate second pass.

Encrypted backups (age)

Prerequisite: age installed on the backup hosts and a key generated (see the manual method for age-keygen); keep the private key in escrow off the cluster.

# control station
node infra/crypto/retrofit.mjs enable-backup-encryption --recipient age1yourpublickey --force

Overlay network encryption (IPsec)

First make sure the stack file declares the overlay as encrypted, then:

# control station
node infra/crypto/retrofit.mjs enable-overlay-ipsec --window --force

Method B — manual, host by host

No tooling: the administrator connects to the relevant hosts over SSH and runs the commands as root (sudo -i, or sudo before each command). Each command block below states which host to connect to. The "database hosts" are the nodes carrying the "db" role; the "manager" hosts drive the container orchestrator.

Data at rest (LUKS)

Migration one node at a time, without downtime: encrypt the replicas first, switch the primary to an already-encrypted replica, encrypt the former primary last. On each node the clear is moved aside, the encrypted volume is mounted empty, and the database re-synchronises onto it.

Step 1 — prepare the Tang key servers (once, on every database host):

# SSH into EACH database host, as root
apt-get install -y tang clevis clevis-luks cryptsetup
mkdir -p /etc/systemd/system/tangd.socket.d
printf '[Socket]\nListenStream=\nListenStream=7500\n' > /etc/systemd/system/tangd.socket.d/override.conf
systemctl daemon-reload
systemctl enable --now tangd.socket

Step 2 — identify the roles. Never start with the primary.

# SSH into any database host
patronictl -c /etc/patroni.yml list        # "Role" column: Leader = primary, Replica = replica

Step 3 — encrypt one replica. Replace PEER1/PEER2 with the IPs of the two other database hosts.

# SSH into the replica to encrypt, as root
systemctl stop patroni

# a) move the clear aside and prepare an empty mount point
mv /var/lib/postgresql/17/main /var/lib/postgresql/17/main.preluks
mkdir -p /var/lib/postgresql/17/main

# b) create the LUKS2 container and bind it to the peers' Tang servers (Clevis SSS, threshold 1)
fallocate -l 4G /var/lib/vaks-pgdata.luks
PASS=$(openssl rand -base64 32)
printf '%s' "$PASS" | cryptsetup luksFormat --type luks2 --batch-mode /var/lib/vaks-pgdata.luks -
printf '%s' "$PASS" | clevis luks bind -d /var/lib/vaks-pgdata.luks -k - -y sss \
  '{"t":1,"pins":{"tang":[{"url":"http://PEER1:7500"},{"url":"http://PEER2:7500"}]}}'
printf '%s\n' "$PASS" > /root/vaks-pgdata.luks.recovery.key   # RECOVERY key: move off-cluster, then delete from here
chmod 600 /root/vaks-pgdata.luks.recovery.key

# c) format, mount, hand ownership back to postgres
clevis luks unlock -d /var/lib/vaks-pgdata.luks -n pgdata_crypt
mkfs.ext4 -q /dev/mapper/pgdata_crypt
mount /dev/mapper/pgdata_crypt /var/lib/postgresql/17/main
rm -rf /var/lib/postgresql/17/main/lost+found
chown -R postgres:postgres /var/lib/postgresql/17/main
chmod 700 /var/lib/postgresql/17/main

Step 4 — automatic unlock at boot (same replica). Without this unit the volume would stay locked after a reboot.

# SSH into the same replica, as root
cat > /usr/local/sbin/vaks-cryptmount.sh <<'EOF'
#!/bin/bash
set -e
F="$1"; N="$2"; M="$3"
for i in $(seq 1 30); do
  [ -e "/dev/mapper/$N" ] && break
  clevis luks unlock -d "$F" -n "$N" 2>/dev/null && break || true
  sleep 2
done
mountpoint -q "$M" || mount "/dev/mapper/$N" "$M"
EOF
chmod 755 /usr/local/sbin/vaks-cryptmount.sh

cat > /etc/systemd/system/vaks-cryptmount-pgdata.service <<'EOF'
[Unit]
Description=Unlock+mount pgdata (Clevis/Tang)
DefaultDependencies=no
Wants=network-online.target
After=network-online.target
Before=patroni.service
ConditionPathExists=/var/lib/vaks-pgdata.luks
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/vaks-cryptmount.sh /var/lib/vaks-pgdata.luks pgdata_crypt /var/lib/postgresql/17/main
[Install]
WantedBy=multi-user.target
EOF

mkdir -p /etc/systemd/system/patroni.service.d
printf '[Unit]\nRequires=vaks-cryptmount-pgdata.service\nAfter=vaks-cryptmount-pgdata.service\n' \
  > /etc/systemd/system/patroni.service.d/10-cryptmount.conf

systemctl daemon-reload
systemctl enable vaks-cryptmount-pgdata.service

# restart Patroni: the replica re-clones (basebackup) onto the encrypted volume
systemctl start patroni
watch patronictl -c /etc/patroni.yml list   # wait for "streaming", lag 0, then Ctrl-C

Step 5 — repeat steps 3–4 on each replica. Then switch the primary and encrypt the former primary last:

# SSH into any database host
patronictl -c /etc/patroni.yml switchover --candidate <already-encrypted-replica> --force

Step 6 — verify, then clean up the clear once validated.

# SSH into each database host
mount | grep pgdata_crypt                                  # encrypted volume mounted
cryptsetup luksDump /var/lib/vaks-pgdata.luks | grep -i cipher   # aes-xts-plain64
patronictl -c /etc/patroni.yml list                        # 1 primary + replicas caught up
rm -rf /var/lib/postgresql/17/main.preluks                 # DELETE the clear AFTER validation

Internal PostgreSQL TLS

Step 1 — install the server certificate, on every database host:

# SSH into EACH database host, as root
install -D -o postgres -g postgres -m 644 /etc/vaks/server.crt /etc/vaks/pg/server.crt
install -D -o postgres -g postgres -m 600 /etc/vaks/server.key /etc/vaks/pg/server.key

Step 2 — turn on ssl=on (once) then restart the members one at a time. Replace vakspm-cluster with the scope: from /etc/patroni.yml.

# SSH into any database host, as root
patronictl -c /etc/patroni.yml edit-config --force \
  -s postgresql.parameters.ssl=on \
  -s postgresql.parameters.ssl_cert_file=/etc/vaks/pg/server.crt \
  -s postgresql.parameters.ssl_key_file=/etc/vaks/pg/server.key

# restart one at a time — replicas first, primary LAST
patronictl -c /etc/patroni.yml restart vakspm-cluster <replica>  --force
patronictl -c /etc/patroni.yml restart vakspm-cluster <primary> --force

# verify (on the primary)
sudo -u postgres psql -p 5432 -tAc 'SHOW ssl'    # must return: on

Encrypted backups (age)

Additive and reversible. Prerequisite: the host backups already installed (/etc/vaks-backup.env exists).

# SSH into EACH backup host, as root
apt-get install -y age
age-keygen -o /root/vaks-backup-age.key       # prints the PUBLIC key age1… ; keep the PRIVATE one in escrow off-cluster
echo 'BACKUP_AGE_RECIPIENT=age1yourpublickey' >> /etc/vaks-backup.env

Overlay network encryption (IPsec)

Step 1 — declare the overlay as encrypted in the deployed stack file (on a manager host), under the vakspm network:

# SSH into a manager host — edit docker-stack.gen.yml
#   networks:
#     vakspm:
#       driver: overlay
#       attachable: true
#       driver_opts:            <-- add
#         encrypted: "true"     <-- add

Step 2 — recreate the network (flag not switchable while live):

# SSH into a manager host, as root
docker stack rm vaks-pm                       # stops the services (~1-2 min)
sleep 20                                       # let the overlay network disappear
docker network ls | grep vaks-pm_vakspm || echo "network removed, ready to redeploy"
docker stack deploy -c ~/vakspm-cluster/docker-stack.gen.yml vaks-pm

docker network inspect vaks-pm_vakspm -f '{{index .Options "encrypted"}}'   # true

etcd TLS (common to both methods)

Encrypts the cluster coordination-store traffic. This layer is the most delicate — it is always done by manual rebuild, whichever method you chose for the other layers.

Under a maintenance window (the database goes briefly read-only). On each etcd host:

# SSH into EACH etcd host, as root — maintenance window
systemctl stop patroni
systemctl stop etcd
mv /var/lib/etcd /var/lib/etcd.old && mkdir -p /var/lib/etcd

# re-bootstrap etcd in TLS from the first boot, using the internal certificate /etc/vaks/server.*
ETCD_INTERNAL_TLS=1 bash ~/vakspm-cluster/bootstrap/10-etcd.sh

# once all 3 etcd nodes are back over https, restart Patroni; it rebuilds its state
systemctl start patroni
patronictl -c /etc/patroni.yml list          # cluster reformed, ssl re-captured

KEK custody — local keyring, KMS or HSM

Recall the envelope model: each secret is encrypted by a DEK (per organisation), and each DEK is wrapped by the master key, the KEK. By default the KEK lives in a file keyring on the hosts. Its custody can be moved to a KMS or an on-premise HSM, live and without re-encrypting any data — only the DEKs are re-wrapped.

On-premise HSM (PKCS#11). The application never speaks PKCS#11 directly: a "key-broker" sidecar (image infra/hsm-broker/) isolates the vendor library and the PIN, and only exposes /wrap and /unwrap on the internal network. The KEK never leaves the HSM (wrap/unwrap runs inside it).

1. Deploy the broker sidecar on the internal overlay, with the vendor PKCS#11 library mounted read-only and the PIN + a shared token mounted as secrets (see infra/hsm-broker/README.md).
2. On the api, worker and controlplane services, set the variables then roll:

   # on a manager host — application service env
   CRYPTO_HSM_BROKER_URL=http://hsm-broker:8081
   CRYPTO_HSM_KEY_LABEL=<the KEK's label in the HSM>
   CRYPTO_HSM_BROKER_TOKEN=<shared token>
   CRYPTO_KEY_PROVIDER=hsm          # NEW DEKs are now wrapped by the HSM

3. Re-wrap existing DEKs under the HSM key (per organisation), from Admin → Security & compliance → Encryption keys → Rotate KEK, or via the API:

   POST /api/v1/admin/crypto/rewrap-keks

4. Verify: GET /api/v1/admin/crypto/status returns every DEK as hsm:<label> ("activeKek":"hsm:…").

A KMS (AWS KMS or a compatible endpoint) follows exactly the same shape, with CRYPTO_KEY_PROVIDER=kms and CRYPTO_KMS_KEY_ID / _REGION / _ENDPOINT. The local, kms and hsm blobs coexist (routed by the kekRef scheme): a mixed state during migration is normal.

Data key rotation (DEK)

Independent of the infrastructure and identical for both methods. Vaks PM encrypts sensitive secrets at rest (multi-factor secrets, SMTP password, single sign-on and connector secrets) with a per-organisation data key (DEK), itself wrapped by the master key (KEK) — see the model under Security. DEK rotation re-encrypts all of those secrets under a fresh key, without downtime.

• Add or activate a real KEK (instead of the default master key): a live operation, applied by simple re-wrapping, the encrypted data never being touched.
• Rotate the DEK (for example after a suspected compromise) from Admin → Security & compliance → Encryption keys, or via the admin API. The operation is transactional, resumable, and covers the entirety of the stored secrets.

See also: Encryption at install (Docker Swarm) · Security, keys & DORA · Operations (day 2) · Documentation home