Troubleshooting

Every known failure mode, in symptom → cause → fix form, grouped by subsystem. When working a problem, start with the app logs (docker logs ava — lines are [component]-prefixed) and the matching section below. Environment variables referenced here are all documented in the Configuration Reference.

Container won't start

Symptom: container exits immediately with `ERROR: Missing required environment variables`

Cause: the entrypoint validates the mounted env file (/app/.env.local) before anything else runs and refuses to boot half-configured. The always-required variables are:

NEXTAUTH_SECRET
AUTOTASK_API_USER
AUTOTASK_API_SECRET
AUTOTASK_API_INTEGRATION_CODE
AUTOTASK_API_ZONE_URL
AZURE_AD_CLIENT_ID
AZURE_AD_CLIENT_SECRET
AZURE_AD_TENANT_ID
ENGINEERING_GROUP_ID
RECAPTCHA_SECRET_KEY
SMTP2GO_USER
SMTP2GO_PASSWORD
SUPPORT_FALLBACK_EMAIL

…plus the API key matching your configured AI provider(s): the entrypoint parses LLM_FAST / LLM_QUALITY (default provider: anthropic) and requires the corresponding key — ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_GENERATIVE_AI_API_KEY, or OPENROUTER_API_KEY. The error message names exactly which variables are missing and which LLM tier demanded which key.

Fix: add the named variables to your runtime config file and docker compose ... up -d again.

Symptom: `ERROR: NEXTAUTH_SECRET is too short` or `looks like a placeholder value`

Cause: NEXTAUTH_SECRET signs every engineer and client session, so the entrypoint enforces a minimum of 32 characters and rejects obvious placeholder strings (for example anything containing changeme).

Fix: generate a real secret and set it:

openssl rand -base64 48

Note that rotating this value invalidates all existing sessions — engineers sign in again, in-flight client chats get a fresh token on their next visit.

Symptom: `ERROR: /app/.env.local not found`

Cause: the compose file's config mount doesn't point at a real file on the host, or the host path moved.

Fix: check the volumes: entry in your compose file — the host-side path (e.g. /opt/ava-config/env-production.conf) must exist and be readable.

Symptom: `ERROR: prisma db push failed — refusing to start the server`

Cause: the entrypoint syncs the SQLite schema on every start and deliberately refuses to serve requests against a database it couldn't bring up to date. Common triggers: the ava-data volume is missing/unwritable, or the disk is full.

Fix: the lines just above the error contain the actual SQL/prisma complaint. Check df -h /, check the volume mounts, then restart.

Symptom: `WARNING: ... is group- or world-readable and holds live secrets`

Cause: your runtime config file's permissions are too open. This is a warning, not a boot failure.

Fix: chmod 600 /opt/ava-config/env-production.conf on the host.

AutoTask integration

Symptom: every AutoTask call returns 401

Cause (check in this order):

The API user account is locked. Repeated 401s from a mistyped or mangled secret lock the AutoTask API user — after which even the correct credentials return 401. Always rule this out first.
Wrong credentials: AUTOTASK_API_USER, AUTOTASK_API_SECRET, or AUTOTASK_API_INTEGRATION_CODE incorrect.
A $ in the secret got mangled. The env loader treats $x as variable expansion. AVA reads AUTOTASK_API_SECRET raw from the file to avoid this — but the value must be wrapped in double quotes in the env file:
```
AUTOTASK_API_SECRET="abc$def123"
```
Wrong zone: AUTOTASK_API_ZONE_URL pointing at a different zone than the one AutoTask assigned your API user.

Fix: check the API user's lock status in AutoTask admin (Admin → Resources (API users)) and unlock it; verify the three credentials; re-discover your zone:

GET https://webservices.autotask.net/atservicesrest/v1.0/zoneInformation?user=<AUTOTASK_API_USER>

then validate end-to-end with npx tsx scripts/test-autotask.ts.

Symptom: one specific AutoTask operation returns 403 while others work

Cause: the API user's security level is missing a grant for that one entity. AVA needs read/write across Tickets, Ticket Notes, Ticket Attachments, Time Entries, Contacts, and read on Companies, Resources, and Billing Codes — see Requirements §1.2 for the exact entity/operation matrix.

Fix: edit the API user's security level in AutoTask admin to grant the missing entity, matching the requirements table.

Symptom: closing a chat fails with an AutoTask 500 mentioning `Picklist value [N] does not exist for publish`

Cause: the publish field on AutoTask ticket notes is a per-instance picklist — valid values differ between AutoTask customers. AVA posts client-facing notes with the value from AUTOTASK_NOTE_PUBLISH_EXTERNAL (default 1 = "Internal Only", the universally valid value); if you set it to a value your instance doesn't define, note creation 500s.

Fix: discover your instance's valid publish values via GET /TicketNotes/entityInformation/fields (or your AT admin UI) and set AUTOTASK_NOTE_PUBLISH_EXTERNAL accordingly. Never assume picklist IDs transfer between AutoTask instances — the same applies to status, priority, queue, source, and issue-type IDs (see the Quickstart picklist-discovery step).

Symptom: a chat from a personal email address (gmail.com, etc.) produced no AutoTask ticket

Cause: the sender's email domain matched no AutoTask company, and AUTOTASK_CATCHALL_COMPANY_ID is unset — so ticket creation was skipped at session start. The chat itself still works; the session just has no ticket behind it.

Fix and built-in recovery:

Set AUTOTASK_CATCHALL_COMPANY_ID to the numeric ID of a catchall company in your AutoTask instance (see Requirements §1.4). This prevents the whole situation.
For a session already in this state, nothing is lost: the engineer's wrap-up flow retries full ticket creation at close time (once the catchall variable is set, closing the session creates the ticket, notes, transcript, and time entry normally). The dashboard marks such sessions with a "No AT ticket" badge so they're easy to spot.
Only if late creation also fails does the wrap-up offer an explicit engineer-confirmed "close without AutoTask sync" escape hatch — the transcript is always retained in AVA either way.

Symptom: AutoTask calls intermittently slow or failing, then succeeding

Cause/behavior: transient AutoTask 5xx/429 responses. The client retries these automatically (up to 3 retries with exponential backoff) and logs AutoTask's response body on each attempt. Deterministic errors that AutoTask mislabels as 5xx (e.g. picklist complaints) are detected and not retried, and non-idempotent creates are never blindly retried after a timeout — so retries cannot produce duplicate tickets or notes.

Fix: none needed for transient blips. If a specific call fails every attempt, the logged AT response body tells you the real complaint.

Engineer sign-in (Microsoft Entra ID)

Symptom: engineer completes Microsoft login but lands back on the sign-in page with `access_denied`

Cause: the signed-in user did not pass the group-membership check against ENGINEERING_GROUP_ID. Two variants:

The groups claim isn't configured on your app registration — the ID token arrives with no group information at all, so every user is denied. The app registration must emit the groups claim in the ID token (see Requirements §2.2).
The user simply isn't a member of the security group whose object ID is in ENGINEERING_GROUP_ID.

The app log records which it was: Access denied for <user>: <reason>.

Fix: configure the groups claim on the app registration (Token configuration → Add groups claim → Security groups, for ID tokens), verify ENGINEERING_GROUP_ID is the group's object ID (a GUID), and add the engineer to that group. Group membership changes can take a few minutes to appear in fresh tokens.

Symptom: redirect goes to the wrong URL or Microsoft shows a redirect-URI mismatch error

Cause: AZURE_AD_REDIRECT_URI doesn't exactly match a redirect URI registered on the app registration.

Fix: both must be exactly https://<your-domain>/api/auth/callback — scheme, host, and path all matter.

Client chat page

Symptom: the chat form says live chat is unavailable outside your actual business hours (or you can't test off-hours)

Cause: the business-hours gate. The schedule comes from BUSINESS_HOURS_TZ / BUSINESS_HOURS_OPEN / BUSINESS_HOURS_CLOSE / BUSINESS_HOURS_DAYS (default Mon–Fri, 06:00–18:00, America/Los_Angeles). Both the server (/api/chat/start returns 403 off-hours) and the client UI (via GET /api/chat/availability) enforce it.

Fix: set the BUSINESS_HOURS_* variables to your real schedule. For testing outside hours only, set BUSINESS_HOURS_BYPASS=true — and remove it before real clients use the page. Verify with:

curl -s https://chat.example.com/api/chat/availability
# {"available":true,"scheduleText":"..."}

Symptom: clients report "Too many sessions created" (HTTP 429) — or, worse, unrelated clients block each other

Cause: the per-IP session rate limit (3 new chats per IP per hour). If unrelated clients block each other, AVA is seeing every request as coming from one address — which happens when the app runs behind a reverse proxy but NGINX_TRUSTED_PROXY is not set to 1: AVA only reads X-Forwarded-For when explicitly told to.

Fix: when (and only when) the app always sits behind your own reverse proxy — as in the provided compose template — set NGINX_TRUSTED_PROXY=1 in the runtime config. Leave it unset on a deployment that clients can reach without going through your proxy.

Symptom: chat form submissions rejected with a CAPTCHA/verification error

Cause: reCAPTCHA verification failed — wrong NEXT_PUBLIC_RECAPTCHA_SITE_KEY / RECAPTCHA_SECRET_KEY pair, the site key not registered for your domain, or Google unreachable from the host. Verification fails closed by design: if the check can't complete, the session is not created. (RECAPTCHA_DEV_BYPASS=true exists for non-production development only and is refused outright when NODE_ENV=production.)

Fix: confirm both keys belong to the same reCAPTCHA v3 site, that the site's domain list includes your chat domain, and that outbound HTTPS to Google works from the container.

Symptom: chat page loads but never connects (spinner, no messages flowing)

Cause: the WebSocket can't reach the server — most often NEXT_PUBLIC_SOCKET_URL not matching the public domain (it must be wss://<your-domain>), or a proxy in front of nginx stripping the Upgrade/Connection headers.

Fix: verify NEXT_PUBLIC_APP_URL / NEXT_PUBLIC_SOCKET_URL in the compose file's environment: block (these are also CORS-enforced — a mismatch rejects the connection), and keep the provided nginx template, which forwards WebSocket upgrade headers correctly.

File uploads

Symptom: upload rejected with HTTP 413

Cause: the file exceeds the per-file cap (5 MB).

Fix: this cap is intentional (memory protection on a lean container) and not configurable. Have the client compress or split the file, or share it through your normal file channel and reference it in the chat.

Symptom: upload rejected with HTTP 507

Cause: the server-wide upload disk quota is exhausted (UPLOAD_TOTAL_MAX_BYTES, default 2 GB across all sessions combined).

Fix: old uploads from closed sessions are reclaimed automatically by the retention job (UPLOAD_RETENTION_DAYS, default 30 days). To recover immediately, lower the retention window or raise the quota — see the UPLOAD_* variables in the Configuration Reference.

Symptom: a specific file type is rejected

Cause: the upload allowlist. Accepted: PNG, JPEG, GIF, HEIC, PDF, DOCX, XLSX, EML, ZIP, TXT, CSV. The file's content must also match its type, so a file whose extension doesn't match its content is rejected.

Fix: ZIP the file if it's a legitimate type outside the list.

Symptom: upload rejected with HTTP 429

Cause: the per-IP upload budget (default 30 uploads/IP/hour, UPLOAD_MAX_PER_IP_PER_HOUR). If it triggers far too eagerly for unrelated users, check the NGINX_TRUSTED_PROXY issue above — same cause.

Engineer dashboard

Symptom: notification sounds don't play when the dashboard tab is in the background

Cause: browsers suspend a background tab's AudioContext. AVA already mitigates this — it creates a persistent audio context on load and resumes it on tab-visibility changes, clicks, and keypresses — but the browser still requires at least one user interaction with the page (a click or keypress) after load before any audio may play. An engineer who signs in and immediately backgrounds the tab without interacting may miss the first sound.

Fix: click anywhere in the dashboard once after loading it. Desktop (system) notifications are not affected — allow browser notifications for the dashboard origin as the reliable channel.

Symptom: dashboard shows stale data / engineers appear offline after a deploy

Cause: normal — the container recreate drops all WebSockets; clients and dashboards reconnect automatically within a few seconds.

Fix: none needed. If a specific browser stays disconnected, a page refresh re-establishes the socket.

Engineer console

How these features work for engineers is described in engineer-console.md.

Symptom: Priority is read-only ("Priority editing isn't configured — ask your administrator")

Cause: priority editing needs a complete P1–P4 mapping: AUTOTASK_PRIORITY_P1..P4 must all be set, all different, and each must be an active value of your AutoTask priority picklist. /priority is greyed out with "Priority mapping is not configured" for the same reason.

Fix: discover your priority IDs and set all four — see autotask-setup.md Step 6 — then restart.

Symptom: "Complete" isn't in the Status list, or picking it says "Use /wrapup to complete the ticket"

Cause: by design. The ticket is completed by the wrap-up flow, which also writes the time entry, notes and transcript. If your instance renamed the Complete status and it still appears mid-chat, its ID isn't AutoTask's default 5.

Fix: close the chat with Close Session or /wrapup and pick the final status there. For a non-default Complete status ID, set AUTOTASK_COMPLETE_STATUS_ID.

Symptom: status or priority change fails with "Slow down — too many ticket changes"

Cause: the per-engineer limit on ticket changes from the console (TICKET_WRITE_RATE_LIMIT_PER_MIN, default 10 per minute).

Fix: wait a minute and try again. Raise the variable only if your engineers routinely need more.

Symptom: AI suggestion unavailable — "This chat has used its AI allowance" or "You've reached your hourly AI limit"

Cause: the engineer AI usage limits: AI_MAX_CALLS_PER_SESSION (default 30 per chat) and AI_MAX_CALLS_PER_ENGINEER_PER_HOUR (default 60 per engineer per hour). Counters reset when the container restarts. "An AI request is already running for this chat" means a request for that command is still in progress — wait for it to finish.

Fix: wait for the hourly window, or raise the limits in the runtime config and restart. If the AI commands are greyed out entirely, one of the two limits is set to 0. If requests fail with "The AI took too long to answer" or "The AI request failed", check your AI provider key, model strings and ANTHROPIC_TIMEOUT_MS (see ai-providers.md).

Symptom: an internal note shows "Not synced — Retry" or "Not saved — Retry"

Cause: "Not saved" means the note didn't reach AVA (usually a dropped connection). "Not synced" means AVA has the note but posting it to the AutoTask ticket failed — typically AutoTask being briefly unavailable, or an AUTOTASK_NOTE_PUBLISH_INTERNAL value your instance doesn't define. If the card warns that the last attempt timed out, the note may already be on the ticket.

Fix: click Retry. After a timeout warning, check the ticket in AutoTask first so you don't add a duplicate. If every note fails to sync, check the app logs for the AutoTask error and verify AUTOTASK_NOTE_PUBLISH_INTERNAL (see autotask-setup.md Step 7). Unsynced notes are still included in the chat transcript posted at close.

Symptom: notes show "Will sync at close" even though the chat has a ticket

Cause: live note sync is turned off (AUTOTASK_SYNC_INTERNAL_NOTES=false). Notes are kept in AVA and reach AutoTask at close.

Fix: none needed if that was intended (for example because an AutoTask workflow rule emails contacts when a note is added). Otherwise remove the variable and restart.

Symptom: ticket numbers in the dashboard aren't links, or open the wrong AutoTask host

Cause: AVA builds ticket links from your AutoTask web address, derived from AUTOTASK_API_ZONE_URL when it follows the webservicesN pattern.

Fix: set AUTOTASK_WEB_URL to the webUrl from the zoneInformation call (an https:// origin, e.g. https://ww5.autotask.net) — see autotask-setup.md Step 3.

Symptom: a reply won't send — "Replace `{{ticketNumber}}` before sending"

Cause: a snippet placeholder couldn't be filled for this chat (for example {{ticketNumber}} on a chat with no ticket yet). AVA won't send a message with an unfilled placeholder.

Fix: replace the placeholder text in the draft, then send.

Symptom: a message starting with `/` won't send

Cause: a draft that starts with / is treated as a command, never sent to the client.

Fix: start the message with // to send text that begins with / (for example //shrug sends /shrug). Text whose first word is a built-in command name still can't be sent this way — rephrase it.

Symptom: Ctrl+Shift+X switches text direction instead of Reply / Internal note (Firefox on Windows/Linux)

Cause: Firefox uses the same shortcut to switch text direction.

Fix: use the Reply | Internal note toggle above the reply box, or type /note.

Host & platform

Symptom: `docker build` fails with a disk-space error

Cause: --no-cache builds accumulate build cache aggressively.

Fix: this is why the upgrade flow front-loads docker builder prune -af && df -h / — prune, confirm ≥ 8 GB free, rebuild. Because of build-before-swap, a failed build never touches the running stack. See Operations §3.

Symptom: HTTPS suddenly failing / certificate expired

Cause: the certbot renewal loop isn't running, or its initial issuance never happened.

Fix: docker ps should show ava-certbot up; docker logs ava-certbot shows the 12-hourly renewal attempts. Force a renewal check:

docker compose -f docker-compose.production.yml run --rm --entrypoint certbot certbot renew

Then remember nginx only reads certs on reload — the template reloads every 6 h automatically, or docker exec ava-nginx nginx -s reload immediately.

Symptom: nginx serving old behavior after a config change

Cause: single-file bind mounts pin the file's inode — a git checkout that replaces nginx/default.conf leaves the running container holding the old inode.

Fix: docker compose -f docker-compose.production.yml up -d --force-recreate nginx.

Still stuck?

Collect docker logs --since 1h ava (plus the nginx log if it's a connectivity problem), your app version (dashboard footer), and the timestamps involved, then see Support.