Deploy on Docker Swarm

Vaks PM · From-scratch deployment · Docker Swarm · June 2026

Overview

The Swarm install is driven by a guided installer run from a control workstation, the machine the administrator works from, which is not part of the cluster. The installer connects to the target hosts over SSH (Secure Shell, an encrypted remote-administration protocol) and does everything remotely. No Docker is required on the control workstation, since the container images are built on a target host rather than locally.

A single declarative manifest (infra/cluster.json) describes the target: its mode, hosts, roles and tenants. A generator (infra/cluster/generate.mjs) turns that manifest into ready-to-run deployment artifacts. From there:

• Single host runs everything on one machine in a single-node Swarm. PostgreSQL runs as a container, the same orchestrator script (out/deploy-ssh.sh) performs the bring-up over SSH, and the result is well suited to evaluation and small sites.
• High availability (HA) spreads across several hosts with a replicated database and no single point of failure. One orchestrator script (out/deploy-ssh.sh) performs the entire bring-up over SSH.

The running system is the same in both cases: stateless application services (web, api, worker, and a shared controlplane front door) backed by three durable stores: PostgreSQL, Redis, and an on-premise S3 (Simple Storage Service) object store. A Traefik reverse proxy terminates TLS and exposes only ports 80 and 443.

Glossary

Acronyms used throughout this guide, defined once here.

Prerequisites & initial state

The lists below describe the expected starting point before any command is run.

Control workstation

The machine the administrator drives the install from. It does not join the cluster.

• Windows, Linux or macOS with an SSH client (OpenSSH) and Node.js installed (Node is required by the generator).
• No local Docker. Container images are built on a target host.
• A copy of the project repository, via the forge's "Download ZIP" button or git clone. The download does not contain secrets: .env.local and the keys/ folder are intentionally excluded and must be provided by hand (below); node_modules/ and dist/ are also absent (the build happens on a host).

Target hosts

• Debian 13 (or compatible), reachable over SSH by key, with sudo available.
• Single 1 host. HA 3 hosts is the reference; an odd number satisfies the etcd quorum the database coordination layer needs.
• The administrator's public SSH key already present in ~/.ssh/authorized_keys of the install user on each host. Without it the installer cannot connect.
• Known system host names, for example host1, host2, host3, or real names such as srv-prod-01. The name must match the value returned by uname -n, because it is used to set Swarm node labels and to identify Patroni/etcd members.

Network, DNS & TLS

• Inbound: only ports 80 and 443 need to be reachable from clients. Traefik redirects 80 to 443 and terminates TLS; every other service port stays on the internal/overlay network.
• DNS: each organization's host name (<slug>.<baseDomain>) must resolve to the entry point, which is the manager host's address or the shared VIP in HA. A single host name is enough for a single-organization install. For a quick test before DNS is ready, the request can be forced to an address with curl --resolve.
• TLS: a self-signed certificate is generated automatically during the install (sufficient for a test or LAN, though browsers show a warning). For production, replace it with a real certificate, or enable Let's Encrypt issuance on Traefik.

The configuration file .env.local

A file at the repository root on the control workstation, never committed to source control. It carries the SSH access and the host addresses. The infra block is required for both modes:

VM_USER=ops                 # SSH and sudo user on the hosts
VM_PASS=********            # real sudo password of VM_USER (dev/test convenience)
VM_HOST1=10.0.0.1
VM_HOST2=10.0.0.2          # HA only
VM_HOST3=10.0.0.3          # HA only
SSH_KEY=keys/id_ed25519    # path to the OpenSSH private key

Copy the OpenSSH private key of the install user into the repository (by convention keys/id_ed25519, which is git-ignored) and set its permissions:

cp .env.example .env.local       # then edit it
mkdir -p keys && cp /path/to/my_key keys/id_ed25519 && chmod 600 keys/id_ed25519

Manifest & generator

A guided wizard (setup.sh) asks for the mode, the hosts and their roles, the root domain and the tenants, then writes the manifest (infra/cluster.json, git-ignored) and calls the generator. Run it from the repository root. There are three invocations:

bash setup.sh                                          # interactive wizard, generate only
bash setup.sh --manifest infra/cluster.json            # non-interactive, generate only (replays a saved manifest)
bash setup.sh --manifest infra/cluster.json --deploy   # generate, then deploy

By default setup.sh only generates. It writes the manifest, renders the out/ artifacts listed below, and stops. Nothing is installed on any host, which leaves the output reviewable before anything runs. The non-interactive form does the same from a manifest that already exists, which is how a known-good configuration is replayed without answering the prompts again.

--deploy adds a single action after generation: it runs bash out/deploy-ssh.sh, the orchestrator that performs the whole bring-up over SSH. In HA that covers the ordered steps detailed in High-availability deployment: the database tier, Swarm initialisation and join, node labels and secrets, the image build, the stack deploy, the Garage initialisation, and tenant provisioning. In single mode it covers the same sequence minus the native database tier — PostgreSQL runs as a container inside the stack (see Single host deployment). Omitting --deploy does not skip the deployment, it only separates it: the same orchestrator is then launched by hand as the explicit next step.

bash out/deploy-ssh.sh   # run the orchestrator when ready

The generator (infra/cluster/generate.mjs) renders the following artifacts under out/:

Single host deployment 1 machine

In single mode everything runs on one machine in a single-node Swarm (the same secrets mechanism as production, upgradeable to HA later). PostgreSQL runs in a container, so there is no native Patroni/etcd/HAProxy tier. As in HA, the bring-up is orchestrated over SSH from the control workstation by the generated out/deploy-ssh.sh:

bash setup.sh                  # choose "single", enter the IP, the domain, the database name
bash out/deploy-ssh.sh         # orchestrates the ENTIRE bring-up
# or in one go:
bash setup.sh --manifest infra/cluster.json --deploy

Only VM_USER and SSH_KEY (plus VM_PASS when sudo asks for a password) are required in .env.local. Every password and key the stack needs is generated automatically when not provided: the Postgres password is persisted to .pg-creds and the S3 key pair to .s3-creds (both git-ignored), which makes the script idempotent — re-running it reuses the same values and skips what already exists.

The orchestrator runs these ordered steps over SSH, all automatic:

Manual bring-up (fallback)

The same sequence, step by step, run directly on the target machine — for when no control workstation with SSH access is available, or to understand exactly what the orchestrator does. Generate out/ anywhere, transfer it to the host or clone the repo there, then walk the steps on that machine.

Manual step 1: install Docker on the machine

Single mode has no native database tier, but Docker Engine is required for the single-node Swarm. The generator produces a ready-to-run script named after the host. Run it on the target machine:

sudo bash out/host-node1.sh     # installs Docker Engine + enables it
docker --version                # check

Manual step 2: build the images and prepare the deployment

The stack runs on three application images built on the machine, plus a couple of files it expects: the two generated configs and a self-signed TLS certificate.

# 1) Build the 3 application images (from the repository root)
docker build -t vaks-pm/api:latest    -f api/Dockerfile .
docker build -t vaks-pm/web:latest    -f web/Dockerfile .
docker build -t vaks-pm/worker:latest -f worker/Dockerfile .

# 2) Files the stack expects (the ./infra paths resolve relative to the compose file)
mkdir -p out/infra/certs
cp out/traefik-dynamic.gen.yml out/garage.gen.toml out/infra/
openssl req -x509 -newkey rsa:2048 -nodes -days 365 -subj "/CN=*.local" \
  -keyout out/infra/certs/wildcard.key -out out/infra/certs/wildcard.crt

Manual step 3: Swarm + secrets

Initialize the Swarm, then create the secrets. Copy-paste the block as-is, changing only the two cleartext passwords (Postgres and S3):

docker swarm init

# 1) Random secrets: nothing to choose
openssl rand -base64 48 | docker secret create jwt_access_secret -
openssl rand -base64 48 | docker secret create jwt_refresh_secret -
openssl rand -base64 32 | docker secret create mfa_encryption_key -
openssl rand -hex 32   | docker secret create garage_rpc_secret -
openssl rand -hex 32   | docker secret create garage_admin_token -

# 2) Postgres password: choose ONCE, reused in database_url
PGPASS='ChangeMe-strong-Postgres-password'
printf '%s' "$PGPASS" | docker secret create postgres_password -
printf 'postgresql://vakspm:%s@postgres:5432/vakspm?schema=public' "$PGPASS" | docker secret create database_url -

# 3) S3 (Garage) key, formats imposed by Garage: access = "GK"+24 hex, secret = 64 hex
S3_ACCESS=GK$(openssl rand -hex 12)
S3_SECRET=$(openssl rand -hex 32)
echo "S3 access=$S3_ACCESS"; echo "S3 secret=$S3_SECRET"   # NOTE these two values
printf '%s' "$S3_SECRET" | docker secret create s3_secret_key -

# 4) SMTP password (leave a dummy value if no email / anonymous relay)
printf 'ChangeMe-or-x' | docker secret create smtp_password -

# 5) Internal secrets — REQUIRED by the stack even when the feature is unused
#    (they are declared external: the deploy fails if any is missing)
openssl rand -hex 32 | docker secret create worker_api_token -
openssl rand -hex 16 | docker secret create sso_oidc_secret -
printf 'none' | docker secret create syslog_client_key -

docker secret ls   # check: 12 secrets created

Manual step 4: deploy the stack

Carry the S3 access key from manual step 3 (same shell) into the stack, then deploy:

sed -i -E "s|S3_ACCESS_KEY:.*|S3_ACCESS_KEY: $S3_ACCESS|g" out/docker-stack.gen.yml
docker stack deploy -c out/docker-stack.gen.yml vaks-pm
docker service ls   # garage/postgres/redis/api/controlplane at 1/1

Manual step 5: initialize the Garage storage (S3)

Garage starts blank. Assign it a storage layout, then register the S3 key noted in manual step 3. The API then creates its buckets on restart:

CID=$(docker ps -q -f name=vaks-pm_garage | head -1)
g(){ docker exec "$CID" /garage "$@"; }

# 1) Layout: read the node id, assign a zone + capacity, apply
g status                                  # note the <node_id>
g layout assign -z dc1 -c 100G <node_id>
g layout apply --version 1

# 2) Register the S3 key (same values as manual step 3) + bucket-creation right
g key import --yes -n vakspm-app "$S3_ACCESS" "$S3_SECRET"
g key allow --create-bucket "$S3_ACCESS"

# 3) Restart the API → it creates the buckets on startup
docker service update --force vaks-pm_api
g bucket list   # → charter-blobs, document-templates, vaks-pm-attachments, vaks-pm-branding

Manual step 6: create the first organization and admin

In dedicated mode the manifest creates no organization, so bootstrap one after the stack is up:

# 1) Prepare the config from the template, then edit it (slug, name, admin email)
cp infra/org-bootstrap.example.json infra/org-bootstrap.json
#    admin.password = null → generated randomly, shown ONCE in the output.

# 2) Inject it into the api container and run the bootstrap
CID=$(docker ps -q -f name=vaks-pm_api | head -1)
docker cp infra/org-bootstrap.json "$CID":/app/api/org-bootstrap.json
docker exec "$CID" sh -c 'export DATABASE_URL=$(cat /run/secrets/database_url) \
  && cd /app/api && node prisma/bootstrap-org.js'

High availability N hosts

The HA topology removes every single point of failure: a replicated PostgreSQL cluster, a redundant control plane, and an S3 store spread across hosts. The whole bring-up is orchestrated over SSH from the control workstation by a single command, with no manual step on the hosts.

Host roles

The wizard asks which roles each host carries. Several roles can be combined on one host.

HA topology

• Control plane: Traefik + controlplane replicated on the cp hosts. A single entry point is provided by a keepalived VIP on those hosts (the controlPlane.vip manifest field) or by DNS round-robin.
• Data plane: the api pool is spread across the app hosts. The control plane's JWT is required, and its embedded tenant must match the requested host.
• PostgreSQL HA: Patroni + etcd (odd quorum ≥ 3); a local HAProxy routes writes to the leader on port 5000 and reads to replicas on 5001. The backend reaches the database through HAProxy, never directly.

What goes into .env.local for HA

In addition to the infra variables, the HA orchestrator reads the database and JWT values. Generate each once in the shell and paste the literal result. Do not write $(openssl …) into the file: it is sourced on every run and would regenerate a different value each time.

# DB / Patroni
POSTGRES_USER=vakspm
POSTGRES_PASSWORD=<value>
POSTGRES_DB=vakspm
PATRONI_REPLICATION_PASSWORD=<value>
PATRONI_SUPERUSER_PASSWORD=<value>
# Auth (JWT signing keys, high entropy required)
JWT_ACCESS_SECRET=<value>
JWT_REFRESH_SECRET=<value>

# Generate the values once, then paste each result above:
openssl rand -hex 24     # DB passwords (hex: no @ : / ? # & % which would break the DSN)
openssl rand -base64 48  # JWT keys (read as Docker secrets, not embedded in a URL)

Run the orchestrator

bash setup.sh                  # choose "ha", N hosts, roles, tenants
bash out/deploy-ssh.sh         # orchestrates the ENTIRE bring-up
# or in one go:
bash setup.sh --manifest infra/cluster.json --deploy

The orchestrator runs these ordered steps over SSH, all automatic:

Provisioning tenants

In pooled mode each tenant (customer organization) has its own database <dbName>_<slug> inside the shared PostgreSQL cluster, and all tenants share the api pool, resolved by host name. A slug is a short, url-safe identifier ([a-z0-9-]+: lowercase, digits and hyphens only; no dot, space, uppercase or accent). Tenants can be declared in the manifest or provisioned later without touching it:

# In an api container (on an "app" host):
docker exec <api-cid> sh -c \
  'export DATABASE_URL=$(cat /run/secrets/database_url) && \
   cd /app/api && node prisma/provision-tenant.js acme admin@acme.com'

Provisioning creates the tenant database, applies the schema and bootstraps its admin. Sign-in is then done at acme.<baseDomain> via the control plane. A dedicated tenant additionally gets its own api-<slug> / worker-<slug> cell with isolated resources, routed by its host name.

Encryption layers

A Swarm install has several independent encryption layers. Internal TLS, at-rest volume encryption (on a high-availability install) and the application key ring's fallback master key are active out of the box. The remaining hardening, a custom certificate authority, strict database verification, the versioned key ring and backup encryption, is turned on by a stack-file change or an admin action. The layers are documented together in this section, but they are not all applied at the same moment; the last column records when each one happens.

Internal TLS

A fresh install provisions internal TLS automatically with self-signed certificates: the internal traffic is encrypted, but peer identity is not verified. Three flows are covered:

• Database: PostgreSQL TLS for application connections and for replication between nodes (20-patroni.sh).
• Coordination store: etcd peer and client mutual TLS (mTLS), where each node is both a server and a client (10-etcd.sh). etcd starts over HTTPS from the first boot, and a retrofit of etcd TLS onto a live cluster is risky and is deliberately avoided.
• Overlay network: the Swarm overlay uses IPsec symmetric-key encryption managed by Docker (no certificate involved).

The bootstrap places one shared certificate at /etc/vaks/server.{crt,key} (the same bytes on every host, which is what lets the auto-anchored etcd peer mTLS work: each node trusts the certificate its peers present by using that same certificate as its own trust anchor). Per-service copies for postgres and etcd are created once the system users exist.

Supplying a custom Certificate Authority (optional)

To add identity verification (protection against an active man-in-the-middle), replace the self-signed certificates with ones issued by the organization's internal CA (Certificate Authority). The decisive requirement: issue one certificate whose Extended Key Usage carries both serverAuth and clientAuth (etcd mTLS needs both, and a server-only certificate is rejected on the client side), and whose Subject Alternative Name (SAN) lists every host IP plus 127.0.0.1 and localhost. Produce three PEM files: the leaf certificate (server.crt, plus any intermediate chain), the matching unencrypted private key (server.key, mode 600, owned by the service user), and the CA certificate (ca.crt). Install them on each host one at a time (replicas first), point the trust anchors at ca.crt (etcd *_TRUSTED_CA_FILE, Patroni's etcd3.cacert), then restart the services and confirm cluster health before moving to the next node.

By default the application opens a TLS connection to PostgreSQL but does not verify the server certificate. Strict verification (protection against an active man-in-the-middle on the app-to-database hop) is controlled by three environment variables read by the api and controlplane services, the only two that open database connections:

The CA certificate is not secret, so it is supplied as a Docker config, the same mechanism the stack already uses for Traefik's certificate. The procedure edits the stack file: docker-stack.yml on a single host, or out/docker-stack.gen.yml on a high-availability cluster (regenerated by the manifest generator, so re-apply the edit after any regeneration). Run the steps on a Swarm manager.

1. Place the CA on the manager and declare it as a config. Copy the issuing CA certificate to infra/certs/pg-ca.crt next to the stack file, then add an entry under the top-level configs: block:

configs:
  pg_ca_v1:
    file: ./infra/certs/pg-ca.crt

2. Mount the config and set the variables on both services. Add the same four lines to the api service and to the controlplane service (both connect to the database; the worker does not and is left unchanged):

    environment:
      # ...existing variables...
      PG_TLS: "1"
      PG_TLS_CA: /etc/vaks/pg-ca/ca.crt
      PG_TLS_MODE: verify-ca
    configs:
      - source: pg_ca_v1
        target: /etc/vaks/pg-ca/ca.crt

3. Redeploy and confirm. Configs are immutable once created, so a later change to the CA needs a new name (pg_ca_v2) referenced in both places.

docker stack deploy --with-registry-auth -c <stack-file> vaks-pm
# the api/controlplane tasks restart; a bad CA path or chain shows up as a
# database-connection error in the logs:
docker service logs --since 3m vaks-pm_api
docker service logs --since 3m vaks-pm_controlplane

Disk encryption (LUKS2 + Tang)

The at-rest disk-encryption layer protects the data volumes (the PostgreSQL data and the Garage object store) with LUKS2 (aes-xts-plain64, 512-bit). It encrypts only those volumes, not the host OS, and is unlocked using network-bound key release: a stolen disk on its own cannot be decrypted. Two bootstrap steps set it up. On a high-availability install the generated per-host script runs them automatically, ahead of the database tier, so the volumes are mounted empty before PostgreSQL initialises. The settings below are filled in by the generator and are documented here to explain what runs, not as a manual step. To skip this layer, for example when full-disk encryption is already provided at the infrastructure level, the wizard accepts a no answer, recorded as encryption.diskLuks: false in the manifest.

Tang server (15-tang.sh)

Tang is a small, stateless network key server (it holds no per-client secret). The bootstrap installs it alongside Clevis (the client that binds a LUKS volume to one or more Tang servers) and listens on port 7500. In a multi-host install each host runs a Tang server, and the hosts bind to each other's Tang servers, never to their own.

LUKS volumes & Clevis binding (18-luks-data.sh)

This step creates file-backed LUKS2 containers for the PostgreSQL and Garage data, then binds them to the peer Tang servers with a Clevis SSS (Shamir Secret Sharing) policy of threshold t=1: the volume unlocks automatically as long as at least one peer Tang server is reachable. A boot-time helper retries the unlock and mounts the volume. The generated host script passes the peer Tang addresses (the other database hosts' IPs) automatically:

PEER_TANG_IPS="ip1,ip2"   # IPs of the PEER Tang hosts (never the host's own)
SETUP_PG=1                # prepare the pgdata volume   (default)
SETUP_GARAGE=1            # prepare the garage volume   (default)

Local key ring (KEK/DEK)

The application encrypts certain sensitive fields (SMTP password, SSO (single sign-on) client secrets, MFA (multi-factor authentication) seeds, ticketing-connector secrets) with envelope encryption: a per-tenant DEK (Data Encryption Key, AES-256-GCM) encrypts the fields, and a KEK (Key Encryption Key) wraps the DEK. The KEK never touches the data; it only locks and unlocks DEKs. Rotating the KEK re-wraps a handful of small DEKs and re-encrypts no field data, so it is fast and causes no downtime.

Default state: no key ring

Out of the box no versioned key files are mounted. The application reads a single 32-byte master key from the Docker secret mfa_encryption_key (created during the install) and registers it as the KEK reference env. Encryption and decryption work immediately; there is simply nothing to rotate yet. In production at least one KEK must be loadable, either a kek_v* file or the mfa_encryption_key master key, or the api/controlplane services refuse to start.

Add and activate a versioned KEK

Mounting a versioned key ring is what enables rotation. Each KEK version is one Docker secret named kek_v<N> holding base64 text that decodes to exactly 32 bytes. Run on a Swarm manager:

# 1) Create the secret directly from generated material (never written to disk)
openssl rand -base64 32 | docker secret create kek_v1 -

Declare the secret external: true in the stack file and mount it on both the api and controlplane services (the only services that decrypt, since the worker does not), setting the active version:

environment:
  CRYPTO_KEYS_DIR: /run/secrets   # directory scanned for kek_v* files
  CRYPTO_ACTIVE_KEK: v1           # new wraps use this version
secrets:
  - kek_v1

docker stack deploy --with-registry-auth -c <stack-file> vaks-pm
docker service logs --since 3m vaks-pm_api | grep Keyring
# expected: Keyring loaded: v1, env (active=v1)

Rotate

To rotate the KEK: add a new kek_v<N+1> secret keeping the previous one mounted, set CRYPTO_ACTIVE_KEK to the new version, redeploy, then trigger the re-wrap from Admin → Security & Compliance → Encryption keys → Rotate KEK. Each tenant's DEK is unwrapped with its old KEK and re-wrapped under the new one; no field data is re-encrypted. Retire the old KEK only once every DEK references the new one and no retained backup can still reference the old KEK.

Encryption checklist

Confirm each layer is active after the install:

Swarm secrets

Sensitive values are never baked into images or committed. They live as Swarm secrets, encrypted in the Raft log on the managers and mounted read-only into containers under /run/secrets/. In both modes the orchestrator creates/generates them all automatically; the manual single-host fallback creates them by hand at manual step 3.

Post-install checks

Confirm the instance is healthy before handing it over:

# cluster state
docker stack ps vaks-pm                  # all tasks Running
patronictl -c /etc/patroni.yml list      # 1 leader + N replicas (HA)

# control plane: sign-in on a tenant (401 on bad creds = the chain works)
curl -sk --resolve acme.vaks-pm.com:443:<manager-ip> \
  -X POST https://acme.vaks-pm.com/api/v1/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"email":"admin@acme.com","password":"..."}'

# data plane: api liveness/readiness
curl -sk --resolve acme.vaks-pm.com:443:<manager-ip> https://acme.vaks-pm.com/api/v1/health/live    # 200
curl -sk --resolve acme.vaks-pm.com:443:<manager-ip> https://acme.vaks-pm.com/api/v1/health/ready   # 200

• Liveness / readiness: the api exposes /health/live and /health/ready (both used by Swarm's own health checks).
• Sign-in: the seeded administrator account can sign in and reach the admin area. /api/v1/auth/* is served by the controlplane; everything else by the api. A JWT issued for another tenant is rejected.
• Object store: uploading a task attachment succeeds (this exercises the S3 path end to end).
• Email (if configured): the admin area's Send test email action delivers a message.

Failures & recovery

Business data lives in PostgreSQL (HA via Patroni); the other services carry soft state that can be rebuilt. When a host crashes, Swarm reacts according to each service's placement type.

Updates & upgrades

An upgrade replaces the container images with a newer version; the customer never builds or edits code. Because api/web/worker are stateless and run several replicas, Swarm performs a rolling update — it swaps replicas one at a time while the surviving ones carry the traffic, so the operation is zero-downtime. The database schema travels with the image (see Database schema below), so there is no separate SQL migration pack to run in the routine case.

1 · Get the new images

Three ways to bring the new version to the hosts, depending on how the instance was delivered:

2 · Rolling-update the services

Swarm does not re-pull an unchanged tag without --force. Deploy service by service. The recommended order puts api (and its twin controlplane) first, because that is what aligns the schema at start-up:

TAG=1.4.0   # the version being rolled out

docker service update --force --with-registry-auth --image vaks-pm/api:$TAG    vaks-pm_api
docker service update --force --with-registry-auth --image vaks-pm/api:$TAG    vaks-pm_controlplane
docker service update --force --with-registry-auth --image vaks-pm/worker:$TAG vaks-pm_worker
docker service update --force --with-registry-auth --image vaks-pm/web:$TAG    vaks-pm_web

docker service ls          # every service back to its replica count, image = the new tag

Each command waits for the replicas to converge before returning; the rollout stops on its own if the new tasks do not become healthy (the /health/ready health check), leaving the old tasks serving. In pooled mode, the per-tenant dedicated cells (vaks-pm_api-<slug> / vaks-pm_worker-<slug>) update the same way, one at a time.

3 · Database schema

The schema travels in the image: at start-up the api container reads the new image's schema.prisma and aligns the database automatically (prisma db push). For a routine upgrade there is therefore no manual migration step — simply rolling-updating api in step 2 applies the schema.

# from an api image, pointed at the target database with a DDL-capable role
docker run --rm -e DATABASE_URL='postgresql://<owner>:…@<host>:5432/<db>' \
  vaks-pm/api:$TAG sh -c 'cd /app/api && npx prisma db push --accept-data-loss'

4 · Configuration or stack changes

• Stack file / Traefik routing. If docker-stack.gen.yml or traefik-dynamic.gen.yml change, regenerate (bash setup.sh --manifest infra/cluster.json) then docker stack deploy --with-registry-auth -c out/docker-stack.gen.yml vaks-pm. The dynamic Traefik config is an immutable Swarm config: bump its version suffix (traefik_dynamic_v2 → v3) in the stack file before redeploying, otherwise the new config is ignored.
• Secrets. Swarm secrets are immutable: to rotate a value, create a new secret (versioned name), reference it in the stack file, then docker stack deploy. Never remove a KEK that still wraps a live DEK (see Local key ring).

5 · Verify after the upgrade

Confirm the new version is healthy before treating the operation as done:

docker stack ps vaks-pm                 # all tasks Running on the new image
docker service logs --since 5m vaks-pm_api | grep -Ei 'schema|prisma|error'   # schema applied cleanly

# liveness / readiness on a tenant hostname
curl -sk --resolve acme.vaks-pm.com:443:<manager-ip> https://acme.vaks-pm.com/api/v1/health/ready   # 200

• All tasks are Running on the new tag; no restart loop on api (the tell-tale sign of a failed schema push).
• Sign-in on a tenant works and an admin page opens; an attachment upload succeeds (exercises the S3 path).

Related: Product overview · Architecture · Deploy on Kubernetes · Security · Operations · Demo session