Troubleshooting

Every known failure mode, in symptom → cause → fix form, grouped by
subsystem. When working a problem, start with the app logs
(docker logs ava — lines are [component]-prefixed) and the matching
section below. Environment variables referenced here are all documented in
the Configuration Reference.

Container won't start

Symptom: container exits immediately with `ERROR: Missing required environment variables`

Cause: the entrypoint validates the mounted env file
(/app/.env.local) before anything else runs and refuses to boot
half-configured. The always-required variables are:

NEXTAUTH_SECRET
AUTOTASK_API_USER
AUTOTASK_API_SECRET
AUTOTASK_API_INTEGRATION_CODE
AUTOTASK_API_ZONE_URL
AZURE_AD_CLIENT_ID
AZURE_AD_CLIENT_SECRET
AZURE_AD_TENANT_ID
ENGINEERING_GROUP_ID
RECAPTCHA_SECRET_KEY
SMTP2GO_USER
SMTP2GO_PASSWORD
SUPPORT_FALLBACK_EMAIL

…plus the API key matching your configured AI provider(s): the
entrypoint parses LLM_FAST / LLM_QUALITY (default provider: anthropic)
and requires the corresponding key — ANTHROPIC_API_KEY, OPENAI_API_KEY,
GOOGLE_GENERATIVE_AI_API_KEY, or OPENROUTER_API_KEY. The error message
names exactly which variables are missing and which LLM tier demanded which
key.

Fix: add the named variables to your runtime config file and
docker compose ... up -d again.

Symptom: `ERROR: NEXTAUTH_SECRET is too short` or `looks like a placeholder value`

Cause: NEXTAUTH_SECRET signs every engineer JWT and client session
token, so the entrypoint enforces a minimum of 32 characters and rejects
obvious placeholder strings (anything containing changeme, secret,
password, placeholder, …).

Fix: generate a real secret and set it:

openssl rand -base64 48

Note that rotating this value invalidates all existing sessions — engineers
sign in again, in-flight client chats get a fresh token on their next visit.

Symptom: `ERROR: /app/.env.local not found`

Cause: the compose file's config mount doesn't point at a real file on
the host, or the host path moved.

Fix: check the volumes: entry in your compose file — the host-side path
(e.g. /opt/ava-config/env-production.conf) must exist and be readable.

Symptom: `ERROR: prisma db push failed — refusing to start the server`

Cause: the entrypoint syncs the SQLite schema on every start and
deliberately refuses to serve requests against a database it couldn't bring
up to date. Common triggers: the ava-data volume is missing/unwritable, or
the disk is full.

Fix: the lines just above the error contain the actual SQL/prisma
complaint. Check df -h /, check the volume mounts, then restart.

Symptom: `WARNING: ... is group- or world-readable and holds live secrets`

Cause: your runtime config file's permissions are too open. This is a
warning, not a boot failure.

Fix: chmod 600 /opt/ava-config/env-production.conf on the host.

AutoTask integration

Symptom: every AutoTask call returns 401

Cause (check in this order):

The API user account is locked. Repeated 401s from a mistyped or
mangled secret lock the AutoTask API user — after which even the
correct credentials return 401. Always rule this out first.
Wrong credentials: AUTOTASK_API_USER, AUTOTASK_API_SECRET, or
AUTOTASK_API_INTEGRATION_CODE incorrect.
A $ in the secret got mangled. The env loader treats $x as
variable expansion. AVA reads AUTOTASK_API_SECRET raw from the file to
avoid this — but the value must be wrapped in double quotes in the
env file:
```
AUTOTASK_API_SECRET="abc$def123"
```
Wrong zone: AUTOTASK_API_ZONE_URL pointing at a different zone than
the one AutoTask assigned your API user.

Fix: check the API user's lock status in AutoTask admin
(Admin → Resources (API users)) and unlock it; verify the three
credentials; re-discover your zone:

GET https://webservices.autotask.net/atservicesrest/v1.0/zoneInformation?user=<AUTOTASK_API_USER>

then validate end-to-end with npx tsx scripts/test-autotask.ts.

Symptom: one specific AutoTask operation returns 403 while others work

Cause: the API user's security level is missing a grant for that one
entity. AVA needs read/write across Tickets, Ticket Notes, Ticket
Attachments, Time Entries, Contacts, and read on Companies, Resources, and
Billing Codes — see Requirements §1.2 for the exact
entity/operation matrix.

Fix: edit the API user's security level in AutoTask admin to grant the
missing entity, matching the requirements table.

Symptom: closing a chat fails with an AutoTask 500 mentioning `Picklist value [N] does not exist for publish`

Cause: the publish field on AutoTask ticket notes is a per-instance
picklist — valid values differ between AutoTask customers. AVA posts
client-facing notes with the value from AUTOTASK_NOTE_PUBLISH_EXTERNAL
(default 1 = "Internal Only", the universally valid value); if you set it
to a value your instance doesn't define, note creation 500s.

Fix: discover your instance's valid publish values via
GET /TicketNotes/entityInformation/fields (or your AT admin UI) and set
AUTOTASK_NOTE_PUBLISH_EXTERNAL accordingly. Never assume picklist IDs
transfer between AutoTask instances — the same applies to status, priority,
queue, source, and issue-type IDs (see the Quickstart
picklist-discovery step).

Symptom: a chat from a personal email address (gmail.com, etc.) produced no AutoTask ticket

Cause: the sender's email domain matched no AutoTask company, and
AUTOTASK_CATCHALL_COMPANY_ID is unset — so ticket creation was skipped at
session start. The chat itself still works; the session just has no ticket
behind it.

Fix and built-in recovery:

Set AUTOTASK_CATCHALL_COMPANY_ID to the numeric ID of a catchall
company in your AutoTask instance (see
Requirements §1.4). This prevents the whole
situation.
For a session already in this state, nothing is lost: the engineer's
wrap-up flow retries full ticket creation at close time (once the
catchall variable is set, closing the session creates the ticket, notes,
transcript, and time entry normally). The dashboard marks such sessions
with a "No AT ticket" badge so they're easy to spot.
Only if late creation also fails does the wrap-up offer an explicit
engineer-confirmed "close without AutoTask sync" escape hatch — the
transcript is always retained in AVA either way.

Symptom: AutoTask calls intermittently slow or failing, then succeeding

Cause/behavior: transient AutoTask 5xx/429 responses. The client retries
these automatically (up to 3 retries with exponential backoff) and logs
AutoTask's response body on each attempt. Deterministic errors that AutoTask
mislabels as 5xx (e.g. picklist complaints) are detected and not
retried, and non-idempotent creates are never blindly retried after a
timeout — so retries cannot produce duplicate tickets or notes.

Fix: none needed for transient blips. If a specific call fails every
attempt, the logged AT response body tells you the real complaint.

Engineer sign-in (Microsoft Entra ID)

Symptom: engineer completes Microsoft login but lands back on the sign-in page with `access_denied`

Cause: the signed-in user did not pass the group-membership check
against ENGINEERING_GROUP_ID. Two variants:

The groups claim isn't configured on your app registration — the ID
token arrives with no group information at all, so every user is
denied. The app registration must emit the groups claim in the ID
token (see Requirements §2.2).
The user simply isn't a member of the security group whose object ID
is in ENGINEERING_GROUP_ID.

The app log records which it was:
Access denied for <user>: <reason>.

Fix: configure the groups claim on the app registration (Token
configuration → Add groups claim → Security groups, for ID tokens), verify
ENGINEERING_GROUP_ID is the group's object ID (a GUID), and add the
engineer to that group. Group membership changes can take a few minutes to
appear in fresh tokens.

Symptom: redirect goes to the wrong URL or Microsoft shows a redirect-URI mismatch error

Cause: AZURE_AD_REDIRECT_URI doesn't exactly match a redirect URI
registered on the app registration.

Fix: both must be exactly https://<your-domain>/api/auth/callback —
scheme, host, and path all matter.

Client chat page

Symptom: the chat form says live chat is unavailable outside your actual business hours (or you can't test off-hours)

Cause: the business-hours gate. The schedule comes from
BUSINESS_HOURS_TZ / BUSINESS_HOURS_OPEN / BUSINESS_HOURS_CLOSE /
BUSINESS_HOURS_DAYS (default Mon–Fri, 06:00–18:00, America/Los_Angeles).
Both the server (/api/chat/start returns 403 off-hours) and the client UI
(via GET /api/chat/availability) enforce it.

Fix: set the BUSINESS_HOURS_* variables to your real schedule. For
testing outside hours only, set BUSINESS_HOURS_BYPASS=true — and
remove it before real clients use the page. Verify with:

curl -s https://chat.example.com/api/chat/availability
# {"available":true,"scheduleText":"..."}

Symptom: clients report "Too many sessions created" (HTTP 429) — or, worse, unrelated clients block each other

Cause: the per-IP session rate limit (3 new chats per IP per hour). If
unrelated clients block each other, the limiter is seeing every request as
one IP — which happens when the app runs behind a reverse proxy but
NGINX_TRUSTED_PROXY is not set to 1: for safety, AVA never trusts
X-Forwarded-For unless explicitly told to, and every request falls into a
single shared "unknown" bucket.

Fix: when (and only when) the app always sits behind your own reverse
proxy — as in the provided compose template — set NGINX_TRUSTED_PROXY=1
in the runtime config. Never set it on a deployment reachable without a
proxy, or clients could spoof their IP via the header.

Symptom: chat form submissions rejected with a CAPTCHA/verification error

Cause: reCAPTCHA verification failed — wrong
NEXT_PUBLIC_RECAPTCHA_SITE_KEY / RECAPTCHA_SECRET_KEY pair, the site key
not registered for your domain, or Google unreachable from the host.
Verification fails closed by design: if the check can't complete, the
session is not created. (RECAPTCHA_DEV_BYPASS=true exists for
non-production development only and is refused outright when
NODE_ENV=production.)

Fix: confirm both keys belong to the same reCAPTCHA v3 site, that
the site's domain list includes your chat domain, and that outbound HTTPS to
Google works from the container.

Symptom: chat page loads but never connects (spinner, no messages flowing)

Cause: the WebSocket can't reach the server — most often
NEXT_PUBLIC_SOCKET_URL not matching the public domain (it must be
wss://<your-domain>), or a proxy in front of nginx stripping the
Upgrade/Connection headers.

Fix: verify NEXT_PUBLIC_APP_URL / NEXT_PUBLIC_SOCKET_URL in the
compose file's environment: block (these are also CORS-enforced — a
mismatch rejects the connection), and keep the provided nginx template,
which forwards WebSocket upgrade headers correctly.

File uploads

Symptom: upload rejected with HTTP 413

Cause: the file exceeds the per-file cap (5 MB). The declared size is
rejected before the body is even buffered, so large files fail fast.

Fix: this cap is intentional (memory protection on a lean container) and
not configurable. Have the client compress or split the file, or share it
through your normal file channel and reference it in the chat.

Symptom: upload rejected with HTTP 507

Cause: the server-wide upload disk quota is exhausted
(UPLOAD_TOTAL_MAX_BYTES, default 2 GB across all sessions combined).

Fix: old uploads from closed sessions are reclaimed automatically by the
retention job (UPLOAD_RETENTION_DAYS, default 30 days). To recover
immediately, lower the retention window or raise the quota — see the
UPLOAD_* variables in the Configuration Reference.

Symptom: a specific file type is rejected

Cause: the upload allowlist. Accepted: PNG, JPEG, GIF, HEIC, PDF, DOCX,
XLSX, EML, ZIP, TXT, CSV. Content is additionally verified by magic-byte
inspection, so renaming a disallowed file's extension does not get it
through.

Fix: ZIP the file if it's a legitimate type outside the list.

Symptom: upload rejected with HTTP 429

Cause: the per-IP upload budget (default 30 uploads/IP/hour,
UPLOAD_MAX_PER_IP_PER_HOUR). If it triggers far too eagerly for unrelated
users, check the NGINX_TRUSTED_PROXY issue above — same shared-bucket
mechanics.

Engineer dashboard

Symptom: notification sounds don't play when the dashboard tab is in the background

Cause: browsers suspend a background tab's AudioContext. AVA already
mitigates this — it creates a persistent audio context on load and resumes
it on tab-visibility changes, clicks, and keypresses — but the browser still
requires at least one user interaction with the page (a click or
keypress) after load before any audio may play. An engineer who signs in and
immediately backgrounds the tab without interacting may miss the first
sound.

Fix: click anywhere in the dashboard once after loading it. Desktop
(system) notifications are not affected — allow browser notifications for
the dashboard origin as the reliable channel.

Symptom: dashboard shows stale data / engineers appear offline after a deploy

Cause: normal — the container recreate drops all WebSockets; clients and
dashboards reconnect automatically within a few seconds.

Fix: none needed. If a specific browser stays disconnected, a page
refresh re-establishes the socket.

Host & platform

Symptom: `docker build` fails with a disk-space error

Cause: --no-cache builds accumulate build cache aggressively.

Fix: this is why the upgrade flow front-loads
docker builder prune -af && df -h / — prune, confirm ≥ 8 GB free, rebuild.
Because of build-before-swap, a failed build never touches the running
stack. See Operations §3.

Symptom: HTTPS suddenly failing / certificate expired

Cause: the certbot renewal loop isn't running, or its initial issuance
never happened.

Fix: docker ps should show ava-certbot up;
docker logs ava-certbot shows the 12-hourly renewal attempts. Force a
renewal check:

docker compose -f docker-compose.production.yml run --rm --entrypoint certbot certbot renew

Then remember nginx only reads certs on reload — the template reloads every
6 h automatically, or docker exec ava-nginx nginx -s reload immediately.

Symptom: nginx serving old behavior after a config change

Cause: single-file bind mounts pin the file's inode — a git checkout
that replaces nginx/default.conf leaves the running container holding the
old inode.

Fix:
docker compose -f docker-compose.production.yml up -d --force-recreate nginx.

Still stuck?

Collect docker logs --since 1h ava (plus the nginx log if it's a
connectivity problem), your app version (dashboard footer), and the
timestamps involved, then see Support.