Operations (day-2)

Vaks PM · Day-2 operations · June 2026

Purpose of this guide. This document covers how to run a Vaks PM instance after it has been installed and handed over: the recurring tasks of operating it in production. That means backing it up and restoring it, rolling out new versions without an outage, confirming it is healthy, adding an organization, and collecting the right data when support is needed. It is written for a system administrator who is new to the product and assumes no prior context. The one-time installation itself is covered separately, in Docker Swarm deployment and Kubernetes deployment.

Overview

Vaks PM is a self-hosted, multi-user project-management web application. Once it is live, the day-to-day responsibility of running it falls into a small number of recurring activities, each covered by a section of this guide:

Backup & restore: protecting the data against logical errors, corruption and disaster, and being able to bring it back. This is the single most important day-2 task and should be set up before the instance carries real data.
Updates: rolling out a new image version with zero downtime.
Health: knowing, at any moment, whether the instance is serving traffic correctly.
Tenants: adding another organization when one instance serves several.
Support: gathering diagnostics and logs when something needs investigation.

The architecture matters for all of these. The application services, api, web and worker, are stateless: they hold no durable data and can be started, stopped and replicated freely. All durable state lives in three backing stores: PostgreSQL (the relational database), Redis (cache and job queue) and an S3-compatible object store (uploaded files). Operations therefore divide cleanly into stateless concerns such as updates, scaling and health, and stateful concerns such as backup and restore. See Architecture for the full picture.

Glossary

Acronyms used in this guide, defined once here for reference.

Term	Meaning
RPO	Recovery Point Objective: the maximum amount of data loss an organization is willing to tolerate after an incident, expressed as a span of time (for example "at most 15 minutes of work"). A smaller RPO means more frequent backups.
RTO	Recovery Time Objective: the target duration of a restore, that is, how long it may take to bring the service back after an incident. RPO is about how much is lost; RTO is about how long recovery takes.
PITR	Point-In-Time Recovery: restoring the database to a chosen instant (for example the second before a bad command), rather than to the moment of the last full backup.
WAL	Write-Ahead Log: PostgreSQL's journal, in which every change is recorded before the tables are touched. Continuously archiving the WAL is what makes PITR possible.
S3	Simple Storage Service: an object-storage interface, originally from Amazon, now a de-facto industry standard. Vaks PM stores uploaded files in an S3-compatible store it hosts itself.
SPA	Single-Page Application: the browser front end (the `web` service), a set of static assets that runs the user interface in the browser and talks to the API.
JWT	JSON Web Token: a signed session token issued after sign-in; it carries the user and tenant identity so the API can authorize each request.

Other abbreviations used in passing: SSH (Secure Shell, an encrypted remote-administration protocol), HA (High Availability, a topology that survives the loss of a single host), DB (database), DR (disaster recovery), and KEK/DEK (the key-encryption / data-encryption keys described under Encrypted backups).

Backup tiers

Vaks PM does not impose a single backup policy. It offers a set of tiers, each accepting a different amount of data loss (a different RPO) at a different storage cost. An organization picks the tier that matches the data loss it can tolerate. Every tier relies only on tools that ship with PostgreSQL, so no third-party backup software is required.

The database tier in production is replicated across several hosts for availability. Replication is not a backup: a mistaken DELETE, a corrupting change or ransomware propagates to every replica within milliseconds, so three healthy copies become three copies of the damage. Real backups, retained over time with at least one copy outside the cluster, are what protect against logical errors. The tiers below provide exactly that.

The three tiers are cumulative, each building on the one before. Most organizations start at Tier 1 and only move up if losing a day's work is unacceptable.

Tier	Mechanism	RPO (max data loss)	Cost / complexity
Tier 1 Logical dump	A compressed SQL snapshot (`pg_dump`) taken every night and shipped off-cluster.	~24 hours (since the last nightly dump).	Low. A single self-contained file per day; trivial retention; the recommended starting point.
Tier 2 PITR via WAL archiving	A daily full base backup (`pg_basebackup`) plus continuous archiving of the WAL stream.	~15 minutes, adjustable (set by `archive_timeout`; a smaller value lowers the RPO).	Medium. Restores to a precise instant. ~25 GB for a 10 GB database over 7 days.
Tier 2b Incremental (PostgreSQL 17)	A weekly full base backup plus daily incremental base backups (a native PostgreSQL 17 feature), still with continuous WAL.	~15 minutes (same as Tier 2).	Medium+. Same recovery precision as Tier 2 at roughly one-third the storage: about 10 to 12 GB for the same 7-day window.

The lower the RPO, the more is captured and the more storage is consumed. Tier 1 is the simplest and is enough for most sites. Tier 2 adds point-in-time precision. Tier 2b gives Tier 2's precision while keeping the storage bill down at scale, which makes it the appealing choice for a 200-user instance, where the cost is dominated by full database copies rather than by the WAL. Switching tiers is a configuration change, not a code change.

WAL archiving must be monitored. Under Tier 2 / 2b, if the archive destination fills up or becomes unreachable, PostgreSQL refuses to recycle the WAL and the primary's disk eventually saturates. Monitor the free space of the archive directory and the failure counters in PostgreSQL's pg_stat_archiver view. The tier-by-tier procedures and scheduling are described in the sections below.

Sizing

The figures below model an instance for about 200 users (the product's reference target), assuming roughly 72,000 database writes per day and a database of about 10 GB. Uploaded files live in the object store and do not inflate the database or the WAL; they are sized separately (see Object store).

Item	Raw	Compressed	Note
One full base backup	~10 GB	~3 GB	Tracks the database size.
7 full bases (Tier 2)	~70 GB	~21 GB	The dominant cost at this scale.
WAL per day	~2 GB	~0.5 to 0.7 GB	Forced, nearly-empty segments compress heavily.
WAL over 7 days	~14 GB	~3.5 to 5 GB	Cheap relative to the full bases.
Tier 2 total (7 days)	n/a	~25 GB	For a 10 GB database.
Tier 2b total (7 days)	n/a	~10 to 12 GB	Weekly full + daily increments.

Rule of thumb. Tier 1: days × compressed_database. Tier 2: days × (compressed_base + ~0.6 GB WAL/day). Tier 2b: 1 full + days × (increment + ~0.6 GB). Because the WAL is cheap and the full bases are heavy, Tier 2b is markedly more economical at the 200-user scale. Always provision the off-cluster destination with comfortable headroom.

Object store

Uploaded files, including task attachments, organization branding, document templates, project charter documents and compressed audit archives, are stored in the S3-compatible object store, outside PostgreSQL. The database holds only references to those files. The consequence is direct: a database-only restore loses every uploaded file. The object store must be backed up as its own concern.

Two equivalent approaches work for backing up the object store:

Synchronize the buckets to an off-cluster destination with a standard S3 client (for example aws s3 sync pointed at the store's endpoint). This is portable and incremental.
Archive the data directories on each storage node directly (for example a tar of the underlying data), which is useful when filesystem-level copies are preferred.

The application uses a small, fixed set of buckets, all of which must be included in the backup: attachments, branding, document templates, project charter blobs, and the cold audit archive. Keep the object-store backup on the same schedule and retention as the database backup, so that a restore brings back a consistent pair: the database rows and the files they reference.

A full disaster-recovery restore also needs the deploy configuration, the TLS certificates and the encryption key material, none of which live in the database or the object store. The complete "what else to back up" checklist is given in the Restore section.

Encrypted backups

By default, backup artifacts leave the host in clear text. For regulated environments, where encryption is required both at rest and in transit, every artifact (the logical dump, each base backup, and each WAL segment) can be encrypted at creation, before it is ever shipped off the host. The mechanism is age, a modern file-encryption tool, used asymmetrically: the host holds only the public key (the "recipient"), so there is nothing on the host worth stealing, while the matching private key needed to decrypt is kept in escrow off-cluster. Each encrypted artifact gains an .age suffix, and restore procedures decrypt with the escrowed private key as their first step.

Key custody equals recoverability. An encrypted backup is only as recoverable as the key that unlocks it. Two distinct keys must be preserved, both outside the backups they protect and with a retention at least as long as the backups themselves:

The age private key, which decrypts the backup artifacts.
The application's KEK (Key Encryption Key). Vaks PM encrypts sensitive fields at rest with a per-tenant DEK (Data Encryption Key) that is itself wrapped by the KEK. A database dump contains only the wrapped keys plus ciphertext, so without the KEK a restored instance runs but its encrypted secrets (multi-factor secrets, SMTP password, single-sign-on and connector secrets) are permanently unreadable.

Never retire a KEK while any retained backup can still reference it. The full key model and the key-management policy are described in Security: backups & key custody.

Restore

The restore procedure depends on the tier the backup was taken at. In every case, if the artifacts were encrypted (an .age suffix), decrypt them first with the escrowed private key.

Tier	How a restore works
Tier 1 Logical dump	Verify the dump's checksum, then re-import it with `pg_restore`, into a fresh isolated database for a test, or over the live one for a real recovery. Individual tables can be restored selectively. The restored database recovers to the state of the last nightly dump.
Tier 2 PITR	Deploy the latest base backup onto the target, then configure PostgreSQL to replay the archived WAL forward to a chosen instant (for example the second before a bad command) and stop there. A PITR restore rebuilds the database cluster. In an HA topology it is performed with the cluster manager stopped, then the replicas are re-seeded afterwards.
Tier 2b Incremental	First recombine the weekly full base with the daily increments into a single complete copy (`pg_combinebackup`), then follow the same point-in-time WAL replay as Tier 2.

An untested backup is not a backup. Schedule a periodic restore drill: restore the latest backup into a disposable database or VM and confirm an expected row count and that a field-encrypted secret decrypts. A row count alone passes even when the KEK is missing; only a successful decrypt proves the key is present on the restored cluster. Record the time the restore takes, since that figure is the real RTO. The exact restore commands, the disaster-recovery checklist and the drill procedure are given in the Restore section.

Rolling updates

New image versions are rolled out without downtime. Because the api, web and worker services are stateless and run as several replicas, the orchestrator replaces them one (or a few) at a time while the remaining replicas keep serving traffic, so clients see no interruption.

Platform	How the rollout is performed
Docker Swarm	A rolling service update instructs Swarm to replace the running tasks of a service with the new image, respecting the configured update parallelism and delay. Replicas are cycled gradually; if a new task fails its health check the rollout can be paused or rolled back.
Kubernetes	A `helm upgrade` applies the new image; the Deployment's rolling-update strategy brings up new pods and only retires old ones once the new pods report ready.

Database schema changes are applied automatically at API start-up. When a new version requires a schema change, the api reconciles the database schema as it boots, so there is no separate, manual migration command to run for a routine upgrade. Combined with the stateless rollout, this makes a version upgrade a single action on either platform.

Verify a rollout the same way the installer was verified: confirm the health endpoints return success (see Health checks) and that the sign-in page loads and an account can authenticate. In multi-tenant (pooled) deployments where each organization has its own database, schema reconciliation is applied to each tenant database. See Adding an organization and the deployment guides for the per-tenant procedure.

Health checks

The api service exposes two HTTP health endpoints. The orchestrator's own health checks call them, and an external monitoring system can poll them too. Both should return a success response on a healthy instance.

Endpoint	Indicates	Used for
`/health/live` liveness	The process is alive and able to serve requests, meaning it has not hung or deadlocked.	The orchestrator restarts a replica whose liveness probe fails.
`/health/ready` readiness	The replica is ready to accept traffic: its dependencies (such as the database) are reachable.	The orchestrator withholds traffic from a replica whose readiness probe fails (for example during start-up), and only sends requests to ready replicas during a rolling update.

The distinction matters during a rollout: a freshly started replica may be live but not yet ready. Routing traffic only to ready replicas is what keeps the rolling update from dropping requests. If a replica is repeatedly restarted, a failing liveness probe is the first signal; if traffic is not flowing to new replicas after an update, check the readiness probe and the dependency it reports on.

Adding an organization

Vaks PM can run in two tenancy modes. In dedicated mode the instance serves a single organization, the typical on-premise case. In pooled mode one instance serves several organizations, each isolated in its own database and reached at its own host name.

Adding an organization in pooled mode is an additive, isolated operation: it provisions a new database for the tenant and routes a new host name to it, without disturbing the existing tenants. There is no downtime for organizations already on the instance, and their data is never touched. In outline, provisioning a tenant involves:

Creating the tenant's database (named after the tenant) and applying the schema to it.
Routing the tenant's public host name to the instance entry point, with a TLS certificate for that name.
Seeding the organization and its first administrator account.

Because each tenant has its own database, an upgrade that changes the schema reconciles every tenant database, not just one. Keep this in mind when planning an upgrade on a busy pooled instance. The authoritative, platform-specific provisioning commands are in Docker Swarm deployment and Kubernetes deployment.

Diagnostics & logs

When something needs investigation, support data is collected at two tiers, reflecting a deliberate security boundary: the application runs as a hardened, stateless container that writes its logs to standard output only, has no access to the Docker daemon, and is walled off from the hosts. Because of this, the application cannot read infrastructure logs (those of the reverse proxy, the database cluster, the cache or the object store); those live on the hosts. The two tiers are complementary.

Tier 1: the in-app diagnostics bundle

An organization administrator can download a one-click diagnostics bundle from the admin area (Admin → Security & Compliance → Support & diagnostics → Download bundle). It is produced by an admin-only endpoint and returns a redacted JSON file that covers application-level problems:

Dependency health: live probes of the database, Redis and the object store, plus whether SMTP (email) is configured, each with its latency.
Job queues: the depth (waiting / active / failed) of the email, webhook and audit-forwarding queues.
Email delivery: a status breakdown of recent sends and the last failures with their raw SMTP error (recipients masked), usually enough to diagnose an email problem on its own.
Application info: version, runtime version, uptime, memory and tenancy mode.
Effective settings, with every secret masked.
Recent critical audit entries and the last errors/warnings held in memory by the API replica that served the request.

No secrets leave the application. Passwords, API keys, SMTP credentials and encryption/signing keys are masked or omitted, using the same redaction as the audit log. The file still carries snapshot labels (names and emails of who did what), so treat it as confidential, though it contains no credentials.

The in-memory error buffer is per API replica, and the instance runs several. The bundle is an excellent first signal, but for a complete trace add the host logs below: the service-level log command spans every replica.

Tier 2: host logs

Infrastructure problems require logs gathered on the hosts over SSH by an operator. On Docker Swarm these are read with service-level log commands on the manager node, scoped to the time window of the incident; on Kubernetes the equivalent is reading pod logs. Match the log source to the symptom:

Symptom	Tier 1 bundle	Host logs to add
Feature broken, server errors, wrong data	Primary signal	`api` (and `worker`) service logs
Sign-in / single-sign-on / MFA failures	Yes (audit + config)	`api` service logs
Gateway errors, TLS / certificate errors	Not covered	Reverse-proxy (Traefik) logs
Site down, database unreachable, failover	Yes (DB probe)	Database cluster status & logs (Patroni / etcd / HAProxy)
Emails / webhooks not sent	Yes (queue depths)	`worker` and Redis logs
External ticketing connector failing (ServiceNow, Jira…): test fails, or ticket search returns nothing in the timesheet	Partial (effective settings)	`api` service logs — filter on `ticketing` / `servicenow`. The connector Test button also shows the live error (401 invalid credentials, 403 insufficient Table API rights, timeout, unreachable host).
Attachments failing to upload / download	Yes (S3 probe)	Object-store (Garage / MinIO) logs
Diffuse slowness / random failures	Yes (memory, uptime)	Host resources (disk, memory) and cluster task status

To raise a support request, download the Tier 1 bundle. If the issue is infrastructure-related, collect the matching Tier 2 host logs. Note the approximate incident time (with timezone) and what was being done, then attach everything. The full set of host commands, with examples per service, is in the dedicated support guide referenced from Security.