Operations Runbook

Day-2 operations for a production AVA deployment: what to back up, how to
upgrade, how to roll back, and how to check that everything is healthy. This
page assumes you deployed with the provided templates
(docker-compose.production.template.yml + nginx/default.conf.template) as
described in the Quickstart; adjust paths and names if your
host layout differs.

AVA is deliberately simple to operate: one app container, one nginx
container, one certbot container, all state in two Docker volumes plus one
host-side config file. There is no external database, no Redis, no message
queue — nothing else to back up or monitor.

1. What holds state — the complete list

Everything else (the app image, .next cache, /tmp) is rebuilt from the
repo or regenerated at runtime and needs no backup.

Volume names: Docker Compose prefixes volume names with the project
name (usually the directory name), so the actual volume may be
ava_ava-data rather than ava-data. Confirm with docker volume ls
before scripting against them. The examples below use the bare names —
substitute what docker volume ls shows you.

2. Backups

2.1 One-shot backup (tar via a throwaway container)

Run from any directory on the Docker host. This produces timestamped
tarballs of both volumes plus a copy of the config file:

STAMP=$(date +%Y%m%d-%H%M%S)
BACKUP_DIR=/opt/ava-backups
mkdir -p "$BACKUP_DIR"

# SQLite database volume
docker run --rm \
  -v ava-data:/data:ro \
  -v "$BACKUP_DIR":/backup \
  alpine tar czf "/backup/ava-data-$STAMP.tgz" -C /data .

# Uploads volume
docker run --rm \
  -v ava-uploads:/data:ro \
  -v "$BACKUP_DIR":/backup \
  alpine tar czf "/backup/ava-uploads-$STAMP.tgz" -C /data .

# Runtime config (contains secrets — keep permissions tight)
cp /opt/ava-config/env-production.conf "$BACKUP_DIR/env-production.conf-$STAMP"
chmod 600 "$BACKUP_DIR"/env-production.conf-*

Notes:

• SQLite is a single file; a file-level copy taken while the app is running
  is usually fine at helpdesk write volumes, but for a guaranteed-consistent
  snapshot take the backup during a quiet window, or briefly stop the app
  container first (docker compose -f docker-compose.production.yml stop ava,
  back up, start ava).
• Always take a database backup immediately before every upgrade (see
  §3). It is the cheapest insurance you will ever buy.
• Ship backups off the host (object storage, another machine). A backup on
  the same disk as the database protects against nothing but fat fingers.
• The config-file backup contains live secrets. Store it encrypted or in
  a secrets manager, never in a git repo or shared drive.

2.2 Restore

# Stop the app first — never restore under a running writer
docker compose -f docker-compose.production.yml stop ava

# Restore the database volume
docker run --rm \
  -v ava-data:/data \
  -v /opt/ava-backups:/backup:ro \
  alpine sh -c "rm -rf /data/* && tar xzf /backup/ava-data-STAMP.tgz -C /data"

# Restore uploads the same way against ava-uploads, then:
docker compose -f docker-compose.production.yml start ava
docker logs -f ava   # watch for "Environment validation passed" + schema sync

2.3 Retention

A nightly cron running §2.1 plus a simple
find /opt/ava-backups -mtime +30 -delete gives you 30 days of point-in-time
recovery for a few hundred MB. Uploaded files are additionally capped
server-side (2 GB total by default, 30-day retention for closed sessions —
see UPLOAD_* in the Configuration Reference), so
backup growth stays bounded.

3. Upgrades — build-before-swap

The golden rule: build the new image FIRST, while the old stack keeps
serving. Never docker compose down before a successful build. A failed
build must leave production untouched; the swap itself is a single container
recreate measured in seconds.

cd /path/to/ava            # your repo checkout on the Docker host

# 0. Back up the database (§2.1). Every time. No exceptions.

# 1. Pre-flight: build cache accumulates fast with --no-cache builds.
docker builder prune -af
df -h /                    # need >= 8 GB free before building

# 2. Fetch and check out the release tag
git fetch --tags --force
git checkout vX.Y.Z

# 3. Build the new image — the OLD stack keeps serving during this step
docker build --no-cache -t ava:latest .

# 4. Swap — compose recreates only the changed container(s)
docker compose -f docker-compose.production.yml up -d

# 5. Clean up superseded images and watch the boot
docker image prune -f
docker logs -f ava

What a healthy boot looks like in the logs:

1. Environment validation passed. — the entrypoint checked every required
   variable (it refuses to start on a missing one — see
   Troubleshooting).
2. Syncing database schema... — the entrypoint runs npx prisma db push
   on every start. It is idempotent (a no-op when nothing changed) and
   applies additive schema changes automatically, so releases that add
   columns or tables need no manual migration step.
3. The server "running on port" banner.

Then verify:

curl -fsS https://chat.example.com/api/health     # expect {"status":"ok"}

…and do a human smoke test: version string in the dashboard footer, engineer
sign-in works, one end-to-end test chat.

nginx config changes deserve one extra caution: nginx bind-mounts
nginx/default.conf as a single file, and a git checkout replaces the
file's inode — a running container can keep serving the old content until
recreated. After any release that touches the nginx config, force it:

docker compose -f docker-compose.production.yml up -d --force-recreate nginx

(~2–3 seconds of connection blip; the app container is untouched.)

4. Rollback

Rollback is the upgrade flow pointed at the previous tag:

git fetch --tags --force
git checkout vX.Y.(Z-1)                       # the previous release tag
docker builder prune -af && df -h /
docker build --no-cache -t ava:latest .
docker compose -f docker-compose.production.yml up -d

Two cases:

• No schema change between the tags (the common case — the
  changelog notes when a release touches the schema):
  rollback is a pure code swap. The database is untouched and nothing
  else is required.
• The newer release added schema (columns/tables): AVA's schema changes
  are additive, so the older code simply ignores the new columns and still
  runs. If you want the database bit-identical to its pre-upgrade state,
  restore the backup you took in step 0 (§2.2) — this is the reason that
  backup is non-negotiable.

The entrypoint's prisma db push never removes anything on its own, so
rolling code backward cannot destroy data.

5. Health checking

5.1 Built-in container health check

The image ships a Docker HEALTHCHECK (every 30 s, 5 s timeout, 3 retries,
15 s start grace) that probes /api/health. docker ps shows the status:

docker ps --format 'table {{.Names}}\t{{.Status}}'
# ava          Up 2 days (healthy)

5.2 The /api/health endpoint

• Unauthenticated (what the Docker health check and any external monitor
  sees): a minimal 200 {"status":"ok"} — safe to expose to an uptime
  monitor, leaks nothing.
• Authenticated as an engineer (hit it from a signed-in dashboard
  browser): adds uptime, memory usage (RSS / heap), queued-session count,
  active-session count, and online-engineer count — a quick one-URL vitals
  panel.

5.3 What to watch

TLS renewal is fully automatic (certbot attempts renewal every 12 h — a
no-op until the cert is within 30 days of expiry; nginx gracefully reloads
every 6 h to pick up renewed certs without dropping WebSockets). A manual
renewal, if you ever need one:

docker compose -f docker-compose.production.yml run --rm --entrypoint certbot certbot renew

6. Logs

All three services log to the Docker json-file driver with rotation
pre-configured in the compose template (10 MB per file, 3 files per
container) — logs cannot fill the disk.

docker logs -f ava            # the app: boot validation, schema sync, per-component
                              #   [component]-prefixed runtime logs, errors
docker logs -f ava-nginx      # access + error logs (client PII is minimized
                              #   in app logs by design)
docker logs -f ava-certbot    # renewal loop output

docker logs --since 1h ava    # time-windowed
docker logs ava 2>&1 | grep -i error

App log lines are prefixed by component (e.g. [closure], [socket],
[chat-upload]), which makes grepping for a subsystem straightforward when
working a problem — see Troubleshooting for what the
common failure signatures look like.

See also

• Quickstart — first-time setup and deploy
• Configuration Reference — every environment variable
• Troubleshooting — symptom → cause → fix
• Support & Licensing — how to reach the vendor