Troubleshooting¶
Start with the banner. It prints the resolved protocol, auth style, model, paths and CLI versions — and never a secret.
docker compose logs --tail=50 runtime
OpenCode reports Unexpected server error¶
A heartbeat is not evidence that an agent can execute. Check the model catalogue
inside the affected worker as its runtime user (opencode models <provider>),
not only the router's /v1/models endpoint. Never dump provider configuration:
it contains credentials.
One possible cause is a failed startup model-catalogue probe. A router can recover while an existing worker still has an incomplete local catalogue. Compare the requested model with the models available inside that worker; this generic error alone does not establish the cause.
When the probe fails, the fallback includes the configured main, subagent and tier models. No provider-specific model names are hardcoded. Successful probes remain authoritative; unconfigured aliases are not added automatically. The fallback does not provide inference while the upstream service itself is unavailable.
Deploy an image containing the fix using your platform's deployment process. Preserve credentials, unrelated settings and in-flight work when replacing or reconfiguring existing workers. Verify an actual agent response and successful run completion afterward, not merely a heartbeat or container health check.
Regression: bash scripts/test-opencode-catalogue.sh (also in make lint).
The container exits immediately¶
Run the entrypoint with a dry run to see the configuration error:
docker compose run --rm -e MULTICA_DRY_RUN=1 runtime
| Log line | Meaning |
|---|---|
MULTICA_TOKEN is required |
Token missing or empty |
AI_BASE_URL (or ANTHROPIC_BASE_URL) is required |
Endpoint missing |
AI_API_KEY (or ANTHROPIC_AUTH_TOKEN / ANTHROPIC_API_KEY) is required |
Key missing |
AI_PROTOCOL must be auto, openai or anthropic |
Typo in AI_PROTOCOL |
AI_PROVIDER_ID must match … |
Provider id has uppercase or punctuation |
multica login fails¶
- Token expired or revoked → create a new one.
- Self-hosted Multica → set
MULTICA_SERVER_URL(including scheme). - Corporate proxy → the container needs
HTTPS_PROXYandNO_PROXY; the Multica CLI and both agent CLIs honour them.
docker compose run --rm -e HTTPS_PROXY=http://proxy:3128 runtime
The runtime does not appear in Multica¶
- Check the logs for
Starting Multica daemon (foreground)…. - Check
MULTICA_DAEMON_WORKSPACE_ALLOWLIST— if the token can reach several workspaces, the runtime may have registered in another one. - Check the dashboard's Runtimes filter (workspace + status).
- A container with an old
MULTICA_DAEMON_IDand a new one are two runtimes. Agents bound to the old id do not follow automatically.
Issues wait / runtime looks stale¶
Diagnose the workspace-specific bound runtime, not another workspace's runtime with the same daemon id:
python3 scripts/check-runtime.py --workspace-id <workspace-id> --agent-id <agent-id>
This read-only check paginates agent runs, compares heartbeat and recent run
messages, and prints only status metadata. Exit codes: 0 no pickup anomaly,
1 investigation needed, 2 incomplete diagnosis. Run it from an authenticated
CLI context; never put a token in the command. It does not restart or cancel work.
SATURATED:agent.max_concurrent_tasksis full. This is independent ofMULTICA_DAEMON_MAX_CONCURRENT_TASKS; increasing daemon capacity alone cannot unblock the agent. Wait for active work, or review an explicit agent-limit increase against available CPU/memory and API capacity.ACTIVITY: recent events exist; useful progress is not verified. Read tool results and changes, not just timestamps or message counts.STALLED: last tool result timed out, even if that event is fresh.FAILED: last event is an execution error despite serverrunningstatus.- Zero messages on a running task is unverified execution, not healthy progress.
CHECK: no recent messages is a warning, not proof of a dead process. Inspect the run before cancelling; a long tool call may be legitimate.OFFLINE: verify service capacity, outbound connectivity and registration. A process-only container healthcheck is not proof of server connectivity.api_invalid_requestwith unmatchedtool_use_id/tool_result: malformed provider conversation history, not heartbeat failure. Review the run error and router protocol. After confirming no live execution for that issue, usemultica issue rerun <issue>for a fresh run; do not resume the broken session, delete session files, or restart unrelated active runs. Reruns can repeat side effects, so check previous changes first.
Frontend investigation — 2026-11-01¶
At approximately 11:55 UTC, GFS frontend's bound OpenCode runtime was online;
ECS service had one healthy task. Agent concurrency was 2, daemon concurrency
3. Two running tasks had recent tool activity; two more tasks were queued.
The initial interpretation (only agent saturation) was incomplete: recent tool
events did not prove sustained progress. GFS-250 subsequently failed
idle_watchdog at 12:44:20 UTC after its last message at 12:33:22. Its replacement
produced a 300-second npm ci timeout with no output. GFS-249's next run initially
had zero messages. Both replacement runs then emitted Cannot connect to API
at approximately 12:55 UTC despite running status.
Live ECS inspection found the shared gfs-ai-9router service at desired 1,
running 0. Its essential headroom container exited 137 with
OutOfMemoryError: container killed due to memory usage; its hard memory limit
was 1024 MiB. Essential-container coupling took the router down too. Router logs
also showed compression timeouts and upstream free-provider 429 responses before
the outage. These are separate observations; they do not prove the cause of the
earlier idle-watchdog failure or npm timeout.
Recovery: forced a new deployment of only gfs-ai-9router, keeping task
definition revision 4 and desired count 1. Frontend runs, credentials, persistent
files and agent limits were not changed. The ECS stable waiter subsequently succeeded and the ALB target returned
healthy. End-to-end agent execution recovery remains unverified; deployment
acceptance or target health alone is not proof of successful task completion.
Permanent router work belongs in its infrastructure/source repository: review
headroom request concurrency and memory, then decide whether compression should
fail open rather than killing an essential router dependency. Do not silently
weaken controls or increase billing commitments here.
One earlier run also failed api_invalid_request (unmatched tool-result history);
a subsequent run for that issue was already queued. The live image used Multica
0.6.1 / OpenCode 1.18.34, older than this repository's pins; updating the repo does
not update ECS automatically. Deploy through the infrastructure review process,
then verify task pickup and run output. This check is diagnostic, not an automatic
recovery controller or a guarantee against upstream session-history errors.
Claude Code returns 401¶
The credential style is wrong. See Configuration → Credential style.
auth=bearer → Authorization: Bearer ... (routers)
auth=api-key → x-api-key: ... (api.anthropic.com)
Set ANTHROPIC_AUTH_TOKEN or ANTHROPIC_API_KEY explicitly to force one.
Claude Code returns 404 / “model not found”¶
AI_MODEL is not what the router calls the model. List the catalogue:
curl -s "$AI_BASE_URL/v1/models" -H "Authorization: Bearer $AI_API_KEY" | jq
Then set AI_MODEL to an id from that list. If the router has no /v1/models,
set AI_MODELS_PROBE=0 and AI_PROTOCOL explicitly.
OpenCode starts but every session fails¶
OpenCode needs provider/model. The entrypoint qualifies a bare AI_MODEL as
<AI_PROVIDER_ID>/<model>. If you set a fully qualified name, keep it consistent
with AI_PROVIDER_ID:
AI_PROVIDER_ID=multica
AI_MODEL=multica/general
The entrypoint also warns when a pinned model is not in the probed catalogue.
git commit fails: Author identity unknown¶
Git refuses to commit without user.name and user.email. On each
multica repo checkout, the daemon copies the global/system identity into the
worktree's multica-identity.config and masks anything missing with empty
values, so an unset global identity becomes fatal: empty ident name.
The entrypoint sets the global identity from GIT_AUTHOR_NAME /
GIT_AUTHOR_EMAIL, or Multica Agent <agent@multica.invalid> when both are
unset. Set the variables to attribute commits to a real account:
GIT_AUTHOR_NAME=ProReka Bot
GIT_AUTHOR_EMAIL=<id>+<login>@users.noreply.github.com
Check inside a task checkout:
git config --global --get user.email
git var GIT_AUTHOR_IDENT
A checkout created before the fix keeps its empty identity until the next
multica repo checkout refreshes it. To commit now, override per command:
git -c user.name="ProReka Bot" -c user.email=<email> commit ....
Sessions disappear after restart¶
- The volume is not mounted at
/workspaces. - A bind mount is not writable by uid 1000.
CLAUDE_CONFIG_DIRorXDG_DATA_HOMEwas overridden to an ephemeral path.
docker compose exec runtime sh -c 'echo $CLAUDE_CONFIG_DIR; echo $XDG_DATA_HOME; ls -la /workspaces'
See Persistence.
PHP image: database connection refused¶
The embedded databases start before the daemon, but an application that connects during a custom startup hook may be too early. Check:
docker compose exec runtime pgrep -a mariadbd
docker compose exec runtime mariadb -h 127.0.0.1 -u laravel -psecret -e 'select 1'
Point the app at 127.0.0.1, not localhost (MariaDB is bound to IPv4). The
entrypoint does not export DB_*, so your .env decides the credentials.
Design image: Penpot MCP tools fail¶
The MCP server is useless until a Penpot file is open in Chromium with the plugin connected. Check the three processes:
docker compose exec runtime sh -c 'pgrep -a Xvfb; pgrep -af penpot'
docker compose exec runtime curl -sI http://127.0.0.1:4401/mcp
docker compose exec runtime curl -sI http://127.0.0.1:4400/manifest.json
Logs are in /workspaces/.penpot/. The browser profile persists the login; delete
/workspaces/.penpot-browser to force a fresh login.
Chromium 142+ enforces Private Network Access: loading a plugin from
localhost into design.penpot.app may show a permission prompt. The agent must
approve it, or the browser needs the flag for that origin.
no space left on device¶
The workspace volume fills up with task checkouts and build caches. Prune inside the container:
docker compose exec runtime sh -c '
rm -rf /workspaces/*/.git/modules 2>/dev/null
du -sh /workspaces/* 2>/dev/null | sort -h | tail'
Then remove finished task directories, or recreate the volume if you do not need
the session history. The image itself is read-only at runtime; only /workspaces
and /tmp grow.
ARM64: Flutter build fails¶
The Flutter image is amd64-only. On Apple Silicon, run it with:
docker run --platform linux/amd64 ...
or build in CI. Flutter web/Linux work on arm64 if you build a custom image that
skips the Android SDK, but flutter build apk does not.
Getting help¶
Open an issue with:
- the banner lines (they contain no secrets),
- the exact command,
docker compose config(redact credentials),- the relevant log lines.
See SUPPORT.md.