Troubleshooting

Common failures, what they mean, and how to fix them. Setup steps live in Getting Started; every environment variable referenced here is defined in Runtime Configuration.

Quick Index

Install And Build Failures

• pnpm install fails or picks the wrong pnpm. The repo pins pnpm@10.x through the packageManager field. Use Corepack instead of a globally installed pnpm: corepack enable, then corepack pnpm install.
• Engine errors during install. Assembly Line requires Node >=22.19.0 (declared in engines). Check node --version and upgrade before reinstalling.
• pnpm assembly-line fails with a module-not-found error. The CLI runs from built output; run pnpm build first. This also applies after pulling changes that touch any packages/*/src.
• Typecheck or tests fail on a fresh clone. Run in order: pnpm install, pnpm build, pnpm check, pnpm test.

CLI Errors

• assembly-line: command not found. The global assembly-line binary comes from the published @assemblyline-agents/sdk npm package. When it is not installed, including whenever you work from a source checkout, run the same commands through the repo workspace instead: pnpm assembly-line <command> resolves the built @assemblyline-agents/cli binary (run pnpm build first).
• pnpm assembly-line prints nothing useful. Run pnpm assembly-line help for the full command list or pnpm assembly-line help <command> (or pnpm assembly-line <command> --help) for per-command flags.
• Unknown command: <cmd>. Run "assembly-line help" for usage. Check the spelling against pnpm assembly-line help; the help output is canonical. Provider-specific hosts and state command namespaces no longer exist.
• Agent root not found: <path>. The positional argument (or --root) must point at an agent folder. Run assembly-line init <path> to scaffold one.

Model And Provider Key Errors

• A model run fails before the first response. The model prefix must resolve an installed and locked model plugin. Inspect its declared authentication and preflight requirements: the scaffolded openrouter/* needs OPENROUTER_API_KEY, while openai/* needs OPENAI_API_KEY.
• openai-codex/* reports an authentication failure. Run assembly-line auth openai-codex <agentRoot> --status, then authenticate with the same command without --status. This provider uses Pi-native OAuth, not OPENAI_API_KEY, a Codex CLI, or a copied CLI login cache.
• A build cannot resolve model capabilities. Check the provider/model spelling and provider connectivity. Providers with dynamic discovery resolve the ID directly. OpenRouter uses GET /api/v1/models. If discovery is offline, only that plugin's own static fallback can resolve the model; Pi has no bundled cross-provider catalog.
• A turn rejects an image or video. The selected model's build-frozen metadata does not advertise the required input modality. Choose a model that supports that modality and rebuild so the manifest records its capabilities.
• assembly-line models cannot load a provider. Verify that the model plugin package is installed, selected by the agent, and unchanged from its plugins.lock entry. Discovery belongs to that package, not Pi.

Agent Composition Errors

• validate reports invalid-agent-graph. Inspect the nested message for the invalid agent.md rule or plugin contribution. Declarative composition must reference compiled capabilities and stay within maxReasoning.
• A run fails with agent.hook_evaluation_failed. Inspect GET /runs/:id/events for the error and evaluation reason. Common causes include a dynamic plugin composition handler that does not select a model, conflicting singular selections, attempting a state write during composition, and exceeding the 50-snapshot limit. Keep composition handlers synchronous and side-effect free. Move I/O and state writes into tools or event handlers.
• The host logs state.degraded with agentState or conversationTurns. The configured state backend omitted those optional facets, so the runtime is using in-memory fallbacks. Runs still work, but hook state or queued conversation turns will not survive a restart. Implement those facets or use the file or Postgres state adapter before production.

Build Artifacts And .assembly-line Churn

• A command fails while rebuilding an example's .assembly-line/ directory. Point the artifact somewhere disposable with --out /private/tmp/assembly-line-minimal (supported by build, run, dev, serve, and deploy), or delete the generated .assembly-line/ directory and rebuild.
• Example artifacts pile up. pnpm clean:artifacts removes ignored examples/**/.assembly-line directories; pnpm clean removes package dist/ output; pnpm clean:tmp removes assembly-line-* temp directories.
• Generated files show up in git status. .assembly-line/, dist/, node_modules/, and env files are local build output and should stay out of commits.
• A rebuild did not restart dev --watch. Restarts only happen when the agent revision changes; the CLI prints Rebuilt: no manifest change; keeping the running server. for cosmetic edits. While the folder is invalid, the previous server keeps serving and the CLI prints the validation issues until the folder is valid again.

401, 403, And 429 From The HTTP API

• Production boot fails with Production Assembly Line Node runtime requires ASSEMBLY_LINE_ADMIN_TOKEN or a host-provided auth policy to protect control-plane routes. Set ASSEMBLY_LINE_ADMIN_TOKEN (or pass an auth policy from host code). Local assembly-line serve runs in dev mode, where inspection endpoints are open; a deployed artifact boots in production mode.
• 401 admin_auth_required / 403 admin_auth_invalid. /manifest, /routes, and the /runs* inspection endpoints require Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN> in production.
• 403 API run creation is disabled. Production POST /runs is off by default. Set ASSEMBLY_LINE_ENABLE_API_RUNS=true and authenticate with the admin token; leave it off unless API-triggered runs are intended.
• 429 { "error": "Rate limited." }. Ingress rate limiting is enabled (ASSEMBLY_LINE_INGRESS_RATE_LIMIT / ASSEMBLY_LINE_RUNS_RATE_LIMIT or the host rateLimit option). The Retry-After header says when to retry.
• 429 Run capacity exhausted. Retry later. The run concurrency cap (ASSEMBLY_LINE_MAX_CONCURRENT_RUNS, production default 16) rejected a direct run such as POST /runs. Accepted provider webhooks are durably queued per conversation and do not return this capacity error.
• Load balancer keeps routing during deploys. Point readiness at GET /readyz. It returns 503 { "draining": true } during graceful shutdown. Use /health or /healthz for liveness; both stay 200. Deliver SIGTERM so in-flight runs drain within ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS (default 30s); a second signal exits immediately.

Connection Store Secrets

File-backed connection credential stores are encrypted, and production boot validates the secret:

• File-backed connection credential stores require ASSEMBLY_LINE_CONNECTION_STORE_SECRET or ASSEMBLY_LINE_SECRET outside dev mode. Set one of the two, or use Postgres state (the Postgres adapter implements the grant stores directly and needs no file secret).
• ... must be at least 32 characters outside dev mode. The secret has a hard 32-character minimum in production.
• ... cannot use the local development secret outside dev mode. The built-in dev fallback value is rejected in production; generate a real secret.

Rotating the secret makes previously encrypted grant files unreadable, plan rotation as a re-authorization event.

Deploy Preflight Failures

• deploy --dry-run reports missing setup. Each preflight item names a required env var or provider setup step inferred from agent.md profiles, channels, selected plugin connections and tools, and the model prefix. Satisfy every required item before publishing.
• Railway: requires RAILWAY_TOKEN (or an authenticated Railway CLI) and a linked project/service, provider environment IDs, or deploy.railway.project and deploy.railway.service.
• Docker: requires Docker locally; set an explicit image with deploy.docker.image when needed.
• Fly: requires FLY_API_TOKEN and deploy.fly.app or FLY_APP_NAME.
• VPS: requires a named assembly-line.hosts.json entry, a verified SHA256: SSH host-key fingerprint, the inventory-selected identity-file environment variable, supported AMD64 Ubuntu 24.04/26.04 or Debian 12, Docker Compose, and available ports 80/443. Create or adopt a Hetzner host through the VPS plugin's bootstrapHetznerHost() operator API, or register a pre-hardened host directly. The package verifies cloud-init, Docker, UFW, fail2ban, and unattended upgrades. First deploys need --sync-secrets unless a complete remote runtime environment already exists.
• SSH host-key fingerprint mismatch: verify the new fingerprint through the provider console or another trusted path. Do not replace the pin based only on the key returned by the same network connection.
• A VPS release rolls back after readiness: inspect the blue/green runtime containers and assembly-line-caddy on the host. Assembly Line restores the previous route when container health, Caddy validation/reload, or public HTTPS /readyz fails.
• A derived hostname is already owned: another deployment has the host-local ownership claim. Confirm that agent IDs and environments are unique within the host namespace, then run deploy --ingress-only; never delete ownership files merely to bypass the collision. Ingress reconciliation retains the old claim until the new Caddy route passes public readiness and rolls the new claim back on failure.
• A prepared release cannot activate: --activate is revision-fenced. Run it from the same agent revision that produced --prepare-only. If another prepare superseded it, prepare the intended revision again.
• Postgres image mismatch: ordinary deployment refuses to recreate the cluster under a different image. Review downtime and backups, then execute the VPS plugin's reviewed hostPostgresUpgradeScript() operator workflow.
• Postgres transfer verification failed: Assembly Line compares normalized schema and exact per-table counts and restores the pre-transfer target on failure. Keep the source quiesced, inspect the reported failure, and do not resume traffic until a later verified transfer succeeds.
• <bin> not found. The deploy path shells out to railway, docker, flyctl, or the VPS SSH client; install the binary or set the selected deploy plugin's typed *Bin YAML option.
• Migration files are present, but no migration runner was configured. The artifact contains .assembly-line/migrations; pass --migration-command or set ASSEMBLY_LINE_MIGRATION_COMMAND.
• Unknown provider profile: agent.md accepts only the reviewed profile catalog. Use a scalar profile or its documented single-key mapping from the Declarative Reference.

Webhooks And Scheduled Runs Rejected

• 403 scheduler_unauthorized on /assembly-line/automations/tick. Set ASSEMBLY_LINE_SCHEDULER_SECRET and send it as Authorization: Bearer <secret> or x-assembly-line-scheduler-secret. Without a configured secret the endpoint only accepts dev-mode requests.
• Production boot fails with Production provider ingress requires webhook authentication: .... A channel declared ingress secrets (ingress.requiredSecretEnv, any-of groups) and none of its groups is fully set, for example Telegram needs TELEGRAM_WEBHOOK_SECRET, and Photon needs a signing secret or a bearer token. Dev mode logs a warning instead of failing. Set the secrets named in the error.
• Webhooks return 401 even though boot succeeded. Boot checks that the secrets exist; each request is still verified by the channel module (for example Telegram compares x-telegram-bot-api-secret-token constant-time). Make sure the provider-side webhook config sends the same secret.

Runs Stuck, Deliveries Missing, Sandbox Writes Lost

• Deliveries sit in pending and never send. The delivery worker drains the durable queue; check that ASSEMBLY_LINE_DELIVERY_WORKER is not set to false/0 and that the host called startBackgroundWorkers() (listenNodeRuntime does this automatically).
• A run completed but the user got no message yet. A retryable send failure defers the delivery instead of failing the run: the run event log shows delivery.deferred with the error, attempt count, and nextAttemptAt, and the worker retries with backoff up to ASSEMBLY_LINE_DELIVERY_QUEUE_MAX_ATTEMPTS (default 5) before a terminal delivery.failed.
• Runs stuck in running after a crash or redeploy. Orphan recovery sweeps runs stale past max(5min, 4x heartbeat): already-delivered runs complete, runs with a continuation checkpoint get one resume attempt, runs with a model response get a real pending delivery, and everything else is marked failed. It runs at boot and every ASSEMBLY_LINE_RUN_RECOVERY_INTERVAL_MS; the kill-switch is ASSEMBLY_LINE_RUN_RECOVERY=false.
• A tool module or other run dependency fails before the first model request. Current runtimes fail the run immediately with run.initialization_failed and queue a failure notice unless the harness already saved a resumable continuation. If an ordinary initialization error instead remains running until orphan recovery, update the deployed runtime package.
• A sandboxed tool reports ENOENT for a neighboring template, script, stylesheet, schema, or binary. Import the required asset from the tool's module so it is included in the self-contained sandbox bundle; undeclared neighboring files are intentionally excluded. See Tools: Execution Model.
• Sandbox writes (memory, skills, /workspace files) not persisting. The sandbox-sync worker (ASSEMBLY_LINE_SANDBOX_SYNC_WORKER) performs the writeback; use sandboxSyncDiagnostics() and inspectSandboxSyncJob(jobId) to see due/leased/expired/blocked jobs, and retrySandboxSyncJob(jobId) after fixing a blocked one.
• A sandbox reports /home/user, /root, or another cwd instead of /workspace. Current built-in hosted adapters reject that session during initialization; do not rewrite command strings or add a symlink in agent code. Confirm the deployment contains the current adapter packages and inspect its provider metadata for assembly-line.filesystemContractVersion. Sandboxes without the current version are deliberately not reconnected.
• workingDirectory must be /workspace. Remove the alias or set it to /workspace. The field cannot remap absolute paths embedded in shell commands. For local testing of those absolute paths, use Docker instead of the Local adapter.
• Kill-switches for debugging: ASSEMBLY_LINE_DELIVERY_WORKER, ASSEMBLY_LINE_SANDBOX_SYNC_WORKER, and ASSEMBLY_LINE_RUN_RECOVERY each accept false/0. Every background behavior has one; see the tables in Runtime And Deployment.

Still Stuck?

Inspect the durable record: .assembly-line/manifest.json, .assembly-line/preflight.json, and .assembly-line/route-table.json for compile-time surprises. Use GET /runs/:id/events (or the state file in development) for runtime behavior. Every model step, tool call, pause, delivery attempt, and recovery action is recorded as an event.