Deploy on Docker Swarm

Vaks PM · From-scratch deployment · Docker Swarm · June 2026

Overview

The Swarm install is driven by a guided installer run from a control workstation, the machine the administrator works from, which is not part of the cluster. The installer connects to the target hosts over SSH (Secure Shell, an encrypted remote-administration protocol) and does everything remotely. No Docker is required on the control workstation: by default the container images are pulled from the public registry at deploy time (or, in build mode, built on a target host — never locally). The wizard asks Images (ghcr|build); ghcr is the default.

A single declarative manifest (infra/cluster.json) describes the target: its mode, hosts, roles and tenants. A generator (infra/cluster/generate.mjs) turns that manifest into ready-to-run deployment artifacts. From there:

• Single host runs everything on one machine in a single-node Swarm. PostgreSQL runs as a container, the same orchestrator script (out/deploy-ssh.sh) performs the bring-up over SSH, and the result is well suited to evaluation and small sites.
• High availability (HA) spreads across several hosts with a replicated database and no single point of failure. One orchestrator script (out/deploy-ssh.sh) performs the entire bring-up over SSH.

The running system is the same in both cases: stateless application services (web, api, worker, the mcp AI connector, and a shared controlplane front door) backed by three durable stores: PostgreSQL, Redis, and an on-premise S3 (Simple Storage Service) object store. A Traefik reverse proxy terminates TLS and exposes only ports 80 and 443.

Glossary

Acronyms used throughout this guide, defined once here.

Prerequisites & initial state

The lists below describe the expected starting point before any command is run.

Control workstation

The machine the administrator drives the install from. It does not join the cluster.

• Windows, Linux or macOS with an SSH client (OpenSSH) and Node.js installed (Node is required by the generator).
• No local Docker. By default the container images are pulled from the public registry (ghcr.io/vaks-fr/vakspm-*) by each host at deploy time; in build mode they are built on a target host instead. Either way the control workstation needs no Docker.
• A copy of the project repository — git clone https://github.com/vaks-fr/Vaks-Pm.git (or the forge's "Download ZIP" button). The download does not contain secrets: .env.local and the keys/ folder are intentionally excluded and must be provided by hand (below); node_modules/ and dist/ are also absent (the build happens on a host).

From a Windows control workstation

The install scripts come in two equivalent forms: the .sh ones (POSIX shell) and their native PowerShell twins. On a Windows workstation — including a bare Windows Server with no Git Bash and no WSL — the PowerShell route depends on no bash at all. The scripts that run on the hosts stay in bash, but they run on the Debian hosts, never on the control workstation.

The generator (infra/cluster/generate.mjs, plain Node.js) emits both orchestrators side by side in out/ — the .sh route is left strictly unchanged, the PowerShell route is additive. Full sequence from PowerShell:

.\setup.ps1          # wizard: same questions -> writes the manifest -> generates out\ (incl. out\deploy.ps1)
.\out\deploy.ps1     # native orchestrator: drives the whole bring-up over SSH
# or in one command:
.\setup.ps1 -Deploy

Target hosts

• Debian 13 (or compatible), reachable over SSH by key, with sudo available.
• Single 1 host. HA 3 hosts is the reference; an odd number satisfies the etcd quorum the database coordination layer needs.
• The administrator's public SSH key already present in ~/.ssh/authorized_keys of the install user on each host. Without it the installer cannot connect.
• Known system host names, for example host1, host2, host3, or real names such as srv-prod-01. The name must match the value returned by uname -n, because it is used to set Swarm node labels and to identify Patroni/etcd members.

Network, DNS & TLS

• Inbound: only ports 80 and 443 need to be reachable from clients. Traefik redirects 80 to 443 and terminates TLS; every other service port stays on the internal/overlay network.
• DNS: each organization's host name (<slug>.<baseDomain>) must resolve to the entry point, which is the manager host's address or the shared VIP in HA. A single host name is enough for a single-organization install. For a quick test before DNS is ready, the request can be forced to an address with curl --resolve.
• TLS: a self-signed certificate is generated automatically during the install (sufficient for a test or LAN, though browsers show a warning). For production, either drop your own certificate at infra/certs/wildcard.{crt,key} before deploying, or enable Let's Encrypt: set tls.acme.email in the manifest (the wizard also asks for a Let's Encrypt email), and Traefik issues a real, browser-trusted certificate for the tenant domain(s) via the TLS-ALPN-01 challenge. Let's Encrypt requires the tenant host name to resolve publicly to the cluster and ports 80/443 reachable from the internet. Traefik is automatically pinned to a single node in this mode (the acme.json store is single-writer).

The configuration file .env.local

A file at the repository root on the control workstation, never committed to source control. It carries the SSH access and the host addresses. The infra block is required for both modes:

VM_USER=ops                 # SSH and sudo user on the hosts
VM_PASS=********            # real sudo password of VM_USER (dev/test convenience)
VM_HOST1=10.0.0.1
VM_HOST2=10.0.0.2          # HA only
VM_HOST3=10.0.0.3          # HA only
SSH_KEY=keys/id_ed25519    # path to the OpenSSH private key

Copy the OpenSSH private key of the install user into the repository (by convention keys/id_ed25519, which is git-ignored) and set its permissions:

cp .env.example .env.local       # then edit it
mkdir -p keys && cp /path/to/my_key keys/id_ed25519 && chmod 600 keys/id_ed25519

Manifest & generator

A guided wizard (setup.sh) asks for the mode, the image source (ghcr pull or build), the hosts and their roles, the root domain, the tenants, the at-rest encryption, and optionally a Let's Encrypt email for a trusted certificate, then writes the manifest (infra/cluster.json, git-ignored) and calls the generator. Run it from the repository root. There are three invocations:

bash setup.sh                                          # interactive wizard, generate only
bash setup.sh --manifest infra/cluster.json            # non-interactive, generate only (replays a saved manifest)
bash setup.sh --manifest infra/cluster.json --deploy   # generate, then deploy

Before it prompts, setup.sh runs a preflight check on .env.local and lists exactly what is missing (the SSH access, and in HA the database/JWT secrets) rather than failing cryptically later. It then asks, in order:

By default setup.sh only generates. It writes the manifest, renders the out/ artifacts listed below, and stops. Nothing is installed on any host, which leaves the output reviewable before anything runs. The non-interactive form does the same from a manifest that already exists, which is how a known-good configuration is replayed without answering the prompts again.

--deploy adds a single action after generation: it runs bash out/deploy-ssh.sh, the orchestrator that performs the whole bring-up over SSH. In HA that covers the ordered steps detailed in High-availability deployment: the database tier, Swarm initialisation and join, node labels and secrets, the image build, the stack deploy, the Garage initialisation, and tenant provisioning. In single mode it covers the same sequence minus the native database tier — PostgreSQL runs as a container inside the stack (see Single host deployment). Omitting --deploy does not skip the deployment, it only separates it: the same orchestrator is then launched by hand as the explicit next step.

bash out/deploy-ssh.sh   # run the orchestrator when ready

The generator (infra/cluster/generate.mjs) renders the following artifacts under out/:

Single host deployment 1 machine

In single mode everything runs on one machine in a single-node Swarm (the same secrets mechanism as production, upgradeable to HA later). PostgreSQL runs in a container, so there is no native Patroni/etcd/HAProxy tier. As in HA, the bring-up is orchestrated over SSH from the control workstation by the generated out/deploy-ssh.sh:

bash setup.sh                  # choose "single", enter the IP, the domain, the database name
bash out/deploy-ssh.sh         # orchestrates the ENTIRE bring-up
# or in one go:
bash setup.sh --manifest infra/cluster.json --deploy

Only VM_USER and SSH_KEY (plus VM_PASS when sudo asks for a password) are required in .env.local. Every password and key the stack needs is generated automatically when not provided: the Postgres password is persisted to .pg-creds and the S3 key pair to .s3-creds (both git-ignored), which makes the script idempotent — re-running it reuses the same values and skips what already exists.

The orchestrator runs these ordered steps over SSH, all automatic:

Manual bring-up (fallback)

The same sequence, step by step, run directly on the target machine — for when no control workstation with SSH access is available, or to understand exactly what the orchestrator does. Generate out/ anywhere, transfer it to the host or clone the repo there, then walk the steps on that machine.

Manual step 1: install Docker on the machine

Single mode has no native database tier, but Docker Engine is required for the single-node Swarm. The generator produces a ready-to-run script named after the host. Run it on the target machine:

sudo bash out/host-node1.sh     # installs Docker Engine + enables it
docker --version                # check

Manual step 2: build the images and prepare the deployment

The stack runs on three application images built on the machine, plus a couple of files it expects: the two generated configs and a self-signed TLS certificate.

# 1) Build the 3 application images (from the repository root)
docker build -t vaks-pm/api:latest    -f api/Dockerfile .
docker build -t vaks-pm/web:latest    -f web/Dockerfile .
docker build -t vaks-pm/worker:latest -f worker/Dockerfile .

# 2) Files the stack expects (the ./infra paths resolve relative to the compose file)
mkdir -p out/infra/certs
cp out/traefik-dynamic.gen.yml out/garage.gen.toml out/infra/
openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=*.local" \
  -keyout out/infra/certs/wildcard.key -out out/infra/certs/wildcard.crt

Manual step 3: Swarm + secrets

Initialize the Swarm, then create the secrets. Copy-paste the block as-is, changing only the two cleartext passwords (Postgres and S3):

docker swarm init

# 1) Random secrets: nothing to choose
openssl rand -base64 48 | docker secret create jwt_access_secret -
openssl rand -base64 48 | docker secret create jwt_refresh_secret -
openssl rand -base64 32 | docker secret create mfa_encryption_key -
openssl rand -hex 32   | docker secret create garage_rpc_secret -
openssl rand -hex 32   | docker secret create garage_admin_token -

# 2) Postgres password: choose ONCE, reused in database_url
PGPASS='ChangeMe-strong-Postgres-password'
printf '%s' "$PGPASS" | docker secret create postgres_password -
printf 'postgresql://vakspm:%s@postgres:5432/vakspm?schema=public' "$PGPASS" | docker secret create database_url -

# 3) S3 (Garage) key, formats imposed by Garage: access = "GK"+24 hex, secret = 64 hex
S3_ACCESS=GK$(openssl rand -hex 12)
S3_SECRET=$(openssl rand -hex 32)
echo "S3 access=$S3_ACCESS"; echo "S3 secret=$S3_SECRET"   # NOTE these two values
printf '%s' "$S3_SECRET" | docker secret create s3_secret_key -

# 4) SMTP password (leave a dummy value if no email / anonymous relay)
printf 'ChangeMe-or-x' | docker secret create smtp_password -

# 5) Internal secrets — REQUIRED by the stack even when the feature is unused
#    (they are declared external: the deploy fails if any is missing)
openssl rand -hex 32 | docker secret create worker_api_token -
openssl rand -hex 16 | docker secret create sso_oidc_secret -
printf 'none' | docker secret create syslog_client_key -

docker secret ls   # check: 12 secrets created

Manual step 4: deploy the stack

Carry the S3 access key from manual step 3 (same shell) into the stack, then deploy:

sed -i -E "s|S3_ACCESS_KEY:.*|S3_ACCESS_KEY: $S3_ACCESS|g" out/docker-stack.gen.yml
docker stack deploy -c out/docker-stack.gen.yml vaks-pm
docker service ls   # garage/postgres/redis/api/controlplane at 1/1

Manual step 5: initialize the Garage storage (S3)

Garage starts blank. Assign it a storage layout, then register the S3 key noted in manual step 3. The API then creates its buckets on restart:

CID=$(docker ps -q -f name=vaks-pm_garage | head -1)
g(){ docker exec "$CID" /garage "$@"; }

# 1) Layout: read the node id, assign a zone + capacity, apply
g status                                  # note the <node_id>
g layout assign -z dc1 -c 100G <node_id>
g layout apply --version 1

# 2) Register the S3 key (same values as manual step 3) + bucket-creation right
g key import --yes -n vakspm-app "$S3_ACCESS" "$S3_SECRET"
g key allow --create-bucket "$S3_ACCESS"

# 3) Restart the API → it creates the buckets on startup
docker service update --force vaks-pm_api
g bucket list   # → charter-blobs, document-templates, vaks-pm-attachments, vaks-pm-branding, vaks-pm-audit-archive, vaks-pm-exports

Manual step 6: create the first organization and admin

In dedicated mode the manifest creates no organization, so bootstrap one after the stack is up. The simplest path is to declare a tenant slug at the wizard (the slugs prompt) — step 7 then provisions it automatically and prints the admin password, and you can skip this section entirely.

To do it by hand, mind where each command runs: the template org-bootstrap.example.json lives in the repository (the infra/ folder, on your control workstation), while the docker commands need the api container and therefore run on the target host. So you edit the file on the workstation, copy it to the host, then run the bootstrap there:

# 1) On the control workstation (repository root): copy the template and edit it
cp infra/org-bootstrap.example.json infra/org-bootstrap.json
#    edit inside the file — REQUIRED:
#      organization.id     stable key for the org (slug-like, e.g. acme; usually = slug)
#      organization.slug   subdomain → <slug>.<baseDomain>  ([a-z0-9-])
#      organization.name   display name
#      admin.email         first admin's login email
#    optional:
#      organization.primaryDomain   custom domain (e.g. pm.acme.com), or null
#      organization.timezone        default Europe/Paris
#      admin.name                   default "Org Admin"
#    leave admin.password = null  → generated randomly, shown ONCE below.

# 2) Copy it to the target host, then bootstrap inside the api container
scp -i <ssh-key> infra/org-bootstrap.json <user>@<host>:~/
ssh -i <ssh-key> <user>@<host>
API=$(sudo docker ps -q -f name=vaks-pm_api | head -1)
sudo docker cp ~/org-bootstrap.json "$API":/app/api/org-bootstrap.json
sudo docker exec "$API" sh -c 'export DATABASE_URL=$(cat /run/secrets/database_url) \
  && cd /app/api && node prisma/bootstrap-org.js'

If you ran the fully-manual bring-up directly on the target host, the repository is already there — skip the scp/ssh and run the docker commands locally with infra/org-bootstrap.json.

High availability N hosts

The HA topology removes every single point of failure: a replicated PostgreSQL cluster, a redundant control plane, and an S3 store spread across hosts. The whole bring-up is orchestrated over SSH from the control workstation by a single command, with no manual step on the hosts.

Host roles

The wizard asks which roles each host carries. Several roles can be combined on one host.

HA topology

• Control plane: Traefik + controlplane replicated on the cp hosts. A single entry point is provided by a keepalived VIP on those hosts (the controlPlane.vip manifest field) or by DNS round-robin.
• Data plane: the api pool is spread across the app hosts. The control plane's JWT is required, and its embedded tenant must match the requested host.
• PostgreSQL HA: Patroni + etcd (odd quorum ≥ 3); a local HAProxy routes writes to the leader on port 5000 and reads to replicas on 5001. The backend reaches the database through HAProxy, never directly.

What goes into .env.local for HA

In addition to the infra variables, the HA orchestrator reads the database and JWT values. Generate each once in the shell and paste the literal result. Do not write $(openssl …) into the file: it is sourced on every run and would regenerate a different value each time.

# DB / Patroni
POSTGRES_USER=vakspm
POSTGRES_PASSWORD=<value>
POSTGRES_DB=vakspm
PATRONI_REPLICATION_PASSWORD=<value>
PATRONI_SUPERUSER_PASSWORD=<value>
# Auth (JWT signing keys, high entropy required)
JWT_ACCESS_SECRET=<value>
JWT_REFRESH_SECRET=<value>

# Generate the values once, then paste each result above:
openssl rand -hex 24     # DB passwords (hex: no @ : / ? # & % which would break the DSN)
openssl rand -base64 48  # JWT keys (read as Docker secrets, not embedded in a URL)

Run the orchestrator

bash setup.sh                  # choose "ha", N hosts, roles, tenants
bash out/deploy-ssh.sh         # orchestrates the ENTIRE bring-up
# or in one go:
bash setup.sh --manifest infra/cluster.json --deploy

The orchestrator runs these ordered steps over SSH, all automatic:

Provisioning tenants

In pooled mode each tenant (customer organization) has its own database <dbName>_<slug> inside the shared PostgreSQL cluster, and all tenants share the api pool, resolved by host name. A slug is a short, url-safe identifier ([a-z0-9-]+: lowercase, digits and hyphens only; no dot, space, uppercase or accent). Tenants can be declared in the manifest or provisioned later without touching it:

# Run on the target host (single) or an "app" host (HA), where Docker runs:
API=$(sudo docker ps -q -f name=vaks-pm_api | head -1)     # id of the running api container
sudo docker exec "$API" sh -c \
  'export DATABASE_URL=$(cat /run/secrets/database_url) && \
   cd /app/api && node prisma/provision-tenant.js <SLUG> <ADMIN-EMAIL>'

Replace the two arguments before running:

• <SLUG> — the organization's short url-safe id ([a-z0-9-]+, e.g. acme). It becomes the subdomain: sign-in is at <slug>.<baseDomain>.
• <ADMIN-EMAIL> — the first administrator's login email (e.g. admin@acme.com). Its password is generated and printed once in the output; a change is forced at first login.

Provisioning creates the tenant database (<dbName>_<slug>), applies the schema and bootstraps that admin. A dedicated tenant additionally gets its own api-<slug> / worker-<slug> cell with isolated resources, routed by its host name.

Encryption layers

A Swarm install has several independent encryption layers. Internal TLS, at-rest volume encryption (on a high-availability install) and the application key ring's fallback master key are active out of the box. The remaining hardening, a custom certificate authority, strict database verification, the versioned key ring and backup encryption, is turned on by a stack-file change or an admin action. The layers are documented together in this section, but they are not all applied at the same moment; the last column records when each one happens.

Internal TLS

A fresh install provisions internal TLS automatically with self-signed certificates: the internal traffic is encrypted, but peer identity is not verified. Three flows are covered:

• Database: PostgreSQL TLS for application connections and for replication between nodes (20-patroni.sh).
• Coordination store: etcd peer and client mutual TLS (mTLS), where each node is both a server and a client (10-etcd.sh). etcd starts over HTTPS from the first boot, and a retrofit of etcd TLS onto a live cluster is risky and is deliberately avoided.
• Overlay network: the Swarm overlay uses IPsec symmetric-key encryption managed by Docker (no certificate involved).

The bootstrap places one shared certificate at /etc/vaks/server.{crt,key} (the same bytes on every host, which is what lets the auto-anchored etcd peer mTLS work: each node trusts the certificate its peers present by using that same certificate as its own trust anchor). Per-service copies for postgres and etcd are created once the system users exist.

Supplying a custom Certificate Authority (optional)

To add identity verification (protection against an active man-in-the-middle), replace the self-signed certificates with ones issued by the organization's internal CA (Certificate Authority). The decisive requirement: issue one certificate whose Extended Key Usage carries both serverAuth and clientAuth (etcd mTLS needs both, and a server-only certificate is rejected on the client side), and whose Subject Alternative Name (SAN) lists every host IP plus 127.0.0.1 and localhost. Produce three PEM files: the leaf certificate (server.crt, plus any intermediate chain), the matching unencrypted private key (server.key, mode 600, owned by the service user), and the CA certificate (ca.crt). Install them on each host one at a time (replicas first), point the trust anchors at ca.crt (etcd *_TRUSTED_CA_FILE, Patroni's etcd3.cacert), then restart the services and confirm cluster health before moving to the next node.

By default the application opens a TLS connection to PostgreSQL but does not verify the server certificate. Strict verification (protection against an active man-in-the-middle on the app-to-database hop) is controlled by three environment variables read by the api and controlplane services, the only two that open database connections:

The CA certificate is not secret, so it is supplied as a Docker config, the same mechanism the stack already uses for Traefik's certificate. The procedure edits the stack file: docker-stack.yml on a single host, or out/docker-stack.gen.yml on a high-availability cluster (regenerated by the manifest generator, so re-apply the edit after any regeneration). Run the steps on a Swarm manager.

1. Place the CA on the manager and declare it as a config. Copy the issuing CA certificate to infra/certs/pg-ca.crt next to the stack file, then add an entry under the top-level configs: block:

configs:
  pg_ca_v1:
    file: ./infra/certs/pg-ca.crt

2. Mount the config and set the variables on both services. Add the same four lines to the api service and to the controlplane service (both connect to the database; the worker does not and is left unchanged):

    environment:
      # ...existing variables...
      PG_TLS: "1"
      PG_TLS_CA: /etc/vaks/pg-ca/ca.crt
      PG_TLS_MODE: verify-ca
    configs:
      - source: pg_ca_v1
        target: /etc/vaks/pg-ca/ca.crt

3. Redeploy and confirm. Configs are immutable once created, so a later change to the CA needs a new name (pg_ca_v2) referenced in both places.

docker stack deploy --with-registry-auth -c <stack-file> vaks-pm
# the api/controlplane tasks restart; a bad CA path or chain shows up as a
# database-connection error in the logs:
docker service logs --since 3m vaks-pm_api
docker service logs --since 3m vaks-pm_controlplane

Rotating the public TLS certificate

A fresh install serves the browser-facing site with a self-signed certificate, which is fine for a test or LAN but makes browsers show a warning. This is the certificate Traefik presents at the edge; the install stores it as two immutable Swarm configs, tls_cert_v1 and tls_key_v1. There are two ways to move to a trusted certificate after the instance is already running, and which one applies depends on whether the hostname is reachable from the public internet.

Option A — switch to Let's Encrypt (automatic, renewing)

Enable the ACME resolver by adding the contact email to the manifest, then regenerate and redeploy. The generator wires the resolver, the acme.json store and, on a multi-node cluster, pins Traefik to a single node (the store is single-writer):

// infra/cluster.json
"tls": { "acme": { "email": "ops@example.com" } },

bash setup.sh --manifest infra/cluster.json          # regenerate the stack with ACME
cd <deploy-dir> && docker stack deploy --with-registry-auth -c docker-stack.gen.yml vaks-pm

Traefik then obtains a browser-trusted certificate for the tenant domain(s) and renews it automatically. On a single host there is no node-pinning to worry about.

Option B — supply your own certificate (internal CA or purchased)

Use this when the hostname is not public, or when the organisation issues certificates from its own CA. Because the certificate lives in immutable Swarm configs, you cannot overwrite tls_cert_v1 in place — a redeploy that reuses the same config name with different bytes is silently ignored. Bump the version suffix instead, exactly as for traefik_dynamic and pg_ca.

1. Put the real certificate next to the stack file. Replace infra/certs/wildcard.crt and infra/certs/wildcard.key with your leaf certificate (plus any intermediate chain) and its unencrypted private key. On an orchestrated deploy these go into your local infra/certs/ (the deploy script copies that folder to the manager and wipes the manager-side copy each run); on a hand-edited stack, place them in infra/certs/ next to the stack file on the manager.

2. Bump the config version in the stack file (docker-stack.yml on a single host, docker-stack.gen.yml on HA). Rename in the top-level configs: block and in the Traefik service's config mounts:

configs:
  tls_cert_v2:                 # was tls_cert_v1
    file: ./infra/certs/wildcard.crt
  tls_key_v2:                  # was tls_key_v1
    file: ./infra/certs/wildcard.key

# ...in the traefik service:
    configs:
      - source: tls_cert_v2    # was tls_cert_v1
        target: /certs/wildcard.crt
      - source: tls_key_v2     # was tls_key_v1
        target: /certs/wildcard.key
        mode: 0400

3. Redeploy and clean up. Traefik rolls onto the new certificate; then remove the old, now-unreferenced configs:

cd <deploy-dir> && docker stack deploy --with-registry-auth -c <stack-file> vaks-pm
docker config rm vaks-pm_tls_cert_v1 vaks-pm_tls_key_v1   # once nothing references them

Disk encryption (LUKS2 + Tang)

The at-rest disk-encryption layer protects the data volumes (the PostgreSQL data and the Garage object store) with LUKS2 (aes-xts-plain64, 512-bit). It encrypts only those volumes, not the host OS, and is unlocked using network-bound key release: a stolen disk on its own cannot be decrypted. Two bootstrap steps set it up. On a high-availability install the generated per-host script runs them automatically, ahead of the database tier, so the volumes are mounted empty before PostgreSQL initialises. The settings below are filled in by the generator and are documented here to explain what runs, not as a manual step. To skip this layer, for example when full-disk encryption is already provided at the infrastructure level, the wizard accepts a no answer, recorded as encryption.diskLuks: false in the manifest.

Tang server (15-tang.sh)

Tang is a small, stateless network key server (it holds no per-client secret). The bootstrap installs it alongside Clevis (the client that binds a LUKS volume to one or more Tang servers) and listens on port 7500. In a multi-host install each host runs a Tang server, and the hosts bind to each other's Tang servers, never to their own.

LUKS volumes & Clevis binding (18-luks-data.sh)

This step creates file-backed LUKS2 containers for the PostgreSQL and Garage data, then binds them to the peer Tang servers with a Clevis SSS (Shamir Secret Sharing) policy of threshold t=1: the volume unlocks automatically as long as at least one peer Tang server is reachable. A boot-time helper retries the unlock and mounts the volume. The generated host script passes the peer Tang addresses (the other database hosts' IPs) automatically:

PEER_TANG_IPS="ip1,ip2"   # IPs of the PEER Tang hosts (never the host's own)
SETUP_PG=1                # prepare the pgdata volume   (default)
SETUP_GARAGE=1            # prepare the garage volume   (default)

Local key ring (KEK/DEK)

The application encrypts certain sensitive fields (SMTP password, SSO (single sign-on) client secrets, MFA (multi-factor authentication) seeds, ticketing-connector secrets) with envelope encryption: a per-tenant DEK (Data Encryption Key, AES-256-GCM) encrypts the fields, and a KEK (Key Encryption Key) wraps the DEK. The KEK never touches the data; it only locks and unlocks DEKs. Rotating the KEK re-wraps a handful of small DEKs and re-encrypts no field data, so it is fast and causes no downtime.

Default state: no key ring

Out of the box no versioned key files are mounted. The application reads a single 32-byte master key from the Docker secret mfa_encryption_key (created during the install) and registers it as the KEK reference env. Encryption and decryption work immediately; there is simply nothing to rotate yet. In production at least one KEK must be loadable, either a kek_v* file or the mfa_encryption_key master key, or the api/controlplane services refuse to start.

Add and activate a versioned KEK

Mounting a versioned key ring is what enables rotation. Each KEK version is one Docker secret named kek_v<N> holding base64 text that decodes to exactly 32 bytes. Run on a Swarm manager:

# 1) Create the secret directly from generated material (never written to disk)
openssl rand -base64 32 | docker secret create kek_v1 -

Declare the secret external: true in the stack file and mount it on both the api and controlplane services (the only services that decrypt, since the worker does not), setting the active version:

environment:
  CRYPTO_KEYS_DIR: /run/secrets   # directory scanned for kek_v* files
  CRYPTO_ACTIVE_KEK: v1           # new wraps use this version
secrets:
  - kek_v1

docker stack deploy --with-registry-auth -c <stack-file> vaks-pm
docker service logs --since 3m vaks-pm_api | grep Keyring
# expected: Keyring loaded: v1, env (active=v1)

Rotate

To rotate the KEK: add a new kek_v<N+1> secret keeping the previous one mounted, set CRYPTO_ACTIVE_KEK to the new version, redeploy, then trigger the re-wrap from Admin → Security & Compliance → Encryption keys → Rotate KEK. Each tenant's DEK is unwrapped with its old KEK and re-wrapped under the new one; no field data is re-encrypted. Retire the old KEK only once every DEK references the new one and no retained backup can still reference the old KEK.

Encryption checklist

Confirm each layer is active after the install:

Swarm secrets

Sensitive values are never baked into images or committed. They live as Swarm secrets, encrypted in the Raft log on the managers and mounted read-only into containers under /run/secrets/. In both modes the orchestrator creates/generates them all automatically; the manual single-host fallback creates them by hand at manual step 3.

Post-install checks

Confirm the instance is healthy before handing it over:

# cluster state
docker stack ps vaks-pm                  # all tasks Running
patronictl -c /etc/patroni.yml list      # 1 leader + N replicas (HA)

# control plane: sign-in on a tenant (401 on bad creds = the chain works)
curl -sk --resolve acme.vaks-pm.com:443:<manager-ip> \
  -X POST https://acme.vaks-pm.com/api/v1/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"admin@acme.com","password":"..."}'

# data plane: api liveness/readiness
curl -sk --resolve acme.vaks-pm.com:443:<manager-ip> https://acme.vaks-pm.com/api/v1/health/live    # 200
curl -sk --resolve acme.vaks-pm.com:443:<manager-ip> https://acme.vaks-pm.com/api/v1/health/ready   # 200

• Liveness / readiness: the api exposes /health/live and /health/ready. Swarm itself does not currently probe them for api/web (no HEALTHCHECK is declared in docker-stack.yml — only the mcp service has one); use them for external monitoring and the manual curl checks above, not as an automatic rollout gate.
• Sign-in: the seeded administrator account can sign in and reach the admin area. /api/v1/auth/* is served by the controlplane; everything else by the api. A JWT issued for another tenant is rejected.
• Object store: uploading a task attachment succeeds (this exercises the S3 path end to end).
• Email (if configured): the admin area's Send test email action delivers a message.

Failures & recovery

Business data lives in PostgreSQL (HA via Patroni); the other services carry soft state that can be rebuilt. When a host crashes, Swarm reacts according to each service's placement type.

Updates & upgrades

An upgrade replaces the container images with a newer version; the customer never builds or edits code, and has no migration command to run — the database schema reconciles itself at api start-up. Because api/web/worker are stateless and run several replicas, Swarm performs a rolling update — it swaps replicas one at a time while the surviving ones carry the traffic, so the operation is zero-downtime.

1 · Get the new images

Images are published publicly on GitHub Container Registry, no credentials required:

ghcr.io/vaks-fr/vakspm-api
ghcr.io/vaks-fr/vakspm-web
ghcr.io/vaks-fr/vakspm-worker
ghcr.io/vaks-fr/vakspm-mcp

Servers with outbound Internet access — on every node of the cluster:

TAG=1.4.0   # the version being rolled out

docker pull ghcr.io/vaks-fr/vakspm-api:$TAG
docker pull ghcr.io/vaks-fr/vakspm-web:$TAG
docker pull ghcr.io/vaks-fr/vakspm-worker:$TAG
docker pull ghcr.io/vaks-fr/vakspm-mcp:$TAG

Air-gapped environment (the servers have no outbound Internet) — on a third-party machine that has Internet access (not necessarily a cluster node):

TAG=1.4.0

docker pull ghcr.io/vaks-fr/vakspm-api:$TAG
docker pull ghcr.io/vaks-fr/vakspm-web:$TAG
docker pull ghcr.io/vaks-fr/vakspm-worker:$TAG
docker pull ghcr.io/vaks-fr/vakspm-mcp:$TAG

docker save ghcr.io/vaks-fr/vakspm-api:$TAG \
            ghcr.io/vaks-fr/vakspm-web:$TAG \
            ghcr.io/vaks-fr/vakspm-worker:$TAG \
            ghcr.io/vaks-fr/vakspm-mcp:$TAG \
            -o vaks-pm-$TAG.tar

Carry the vaks-pm-$TAG.tar file (USB key or whatever transfer method is available) to every node of the cluster, then load the images there:

docker load -i vaks-pm-$TAG.tar

2 · Rolling-update the services

Swarm does not re-pull an unchanged tag without --force. Deploy service by service. The recommended order puts api (and its twin controlplane) first — its start-up is what reconciles the database schema, no separate migration command is needed:

TAG=1.4.0   # the version being rolled out

docker service update --force --image ghcr.io/vaks-fr/vakspm-api:$TAG    vaks-pm_api
docker service update --force --image ghcr.io/vaks-fr/vakspm-api:$TAG    vaks-pm_controlplane
docker service update --force --image ghcr.io/vaks-fr/vakspm-worker:$TAG vaks-pm_worker
docker service update --force --image ghcr.io/vaks-fr/vakspm-web:$TAG    vaks-pm_web

docker service ls          # every service back to its replica count, image = the new tag

Each command waits for the replicas to converge before returning. Because api/web/worker have no Swarm-level HEALTHCHECK today (unlike mcp), the rollout does not automatically pause on a failing /health/ready — run the verification curls from step 4 after each service update before moving to the next. In pooled mode, the per-tenant dedicated cells (vaks-pm_api-<slug> / vaks-pm_worker-<slug>) update the same way, one at a time.

3 · Configuration or stack changes exception

The large majority of upgrades stop at step 2 — nothing more to do. This step applies only if the release notes explicitly call out a change to docker-stack.gen.yml or traefik-dynamic.gen.yml (for example a new service, a newly required secret, or a new routing rule). In that case, regenerate then redeploy:

bash setup.sh --manifest infra/cluster.json
docker stack deploy --with-registry-auth -c out/docker-stack.gen.yml vaks-pm

If a new secret is required (rare, explicitly called out in the release notes):

openssl rand -hex 32 | docker secret create <secret-name> -
# then reference this secret in docker-stack.gen.yml before the docker stack deploy above

4 · Verify after the upgrade

Confirm the new version is healthy before treating the operation as done:

docker stack ps vaks-pm                 # all tasks Running on the new image
docker service logs --since 5m vaks-pm_api | grep -Ei 'schema|prisma|error'   # schema applied cleanly

# liveness / readiness on a tenant hostname
curl -sk --resolve acme.vaks-pm.com:443:<manager-ip> https://acme.vaks-pm.com/api/v1/health/ready   # 200

• All tasks are Running on the new tag; no restart loop on api (the tell-tale sign of a failed schema push).
• Sign-in on a tenant works and an admin page opens; an attachment upload succeeds (exercises the S3 path).

Related: Product overview · Architecture · Deploy on Kubernetes · Security · Operations · Demo session