Runtime and Deployment

Assembly Line separates authoring from execution:

agent folder -> compiler -> .assembly-line artifact -> runtime host -> durable runs

The compiler validates a folder and emits a deterministic manifest. The runtime loads the compiled artifact, executes runs, records events, stores context and artifacts, manages approvals and human input pauses, and delivers final responses idempotently.

Contents:

• CLI Commands and Project environment
• Configuration and credential boundary
• Build Artifact
• Runtime Lifecycle
• Evolution History
• Node Runtime HTTP API and Direct conversations
• Operator Controls
• Durability Workers
• Concurrency And Rate Limiting
• Graceful Shutdown
• State And Blob Storage and Sandbox Sync
• Deploy Targets
• Model Provider Authentication
• Migrations
• Preflight
• Production Checklist

CLI Commands

Run the CLI against an agent root:

assembly-line <command> <agent-root>

(For a monorepo checkout, Getting Started explains the in-repo pnpm assembly-line form.)

Common flags:

Run assembly-line help <command> for the full per-command flag list. See Coding Agents for the progressive authoring workflow and MCP configuration.

Project environment

Commit shared, non-secret deployment values in agent/config.production.ts:

import { defineProductionConfig } from "@assemblyline-agents/core";

export default defineProductionConfig({
  CATALOG_API_URL: "https://catalog.example.com/v1",
  ASSEMBLY_LINE_LOG_LEVEL: "info"
});

The compiler statically reads this exact default-export shape, accepts only uppercase string-literal fields, and never executes the module. It rejects secret-like names, credential-bearing URLs, private keys, bearer values, spreads, and dynamic expressions. The validated map and source hash are part of the artifact. Runtime resolution is: explicit shell/host value, then committed production config, then the framework default.

The CLI no longer loads an agent-root .env. Put local non-secret overrides in the invoking shell. Keep runtime credentials in the configured secret store; keep model bootstrap and deploy credentials in host secrets. --secrets-from remains an explicit private input to deploy --sync-secrets; without that flag, no dotenv file is read. Committed configuration is never uploaded as a secret.

The resolved non-secret host view is available as runtime.config. Connections and authored tools still receive only the names they declared through their scoped ctx.config; ordinary sandbox processes do not inherit the host view.

Configuration and credential boundary

Compilation classifies declared names as non-secret configuration, gateway/bootstrap credentials, connection credentials, or explicit sandbox materialization. At runtime, connections receive only their configuration and resolve their own declared credentials just in time through the host broker. Authored tools receive declared non-secret ctx.config and run in the sandbox by default in production. Trusted channel media processors receive the credentials declared by their selected audio provider through the gateway boundary; those credentials remain absent from ordinary runtime config.

Deploy-provider policy classifies every environment requirement as config or credential. The CLI evaluates conditional policy for the selected target before resolving credentials, then gives that trusted deploy capability only its active declared credential scope. Conditional credentials can therefore come from the configured secret store without becoming ambient host config; inactive conditional credentials are not requested.

Configuration may be contextual. Credentials are capability-scoped. See Configuration And Credentials for HTTP, SDK, stdio, CLI lease, 1Password, and secure browser-fill examples and the migration from ambient environment APIs.

Build Artifact

assembly-line build emits:

.assembly-line/
  manifest.json
  agent-revision.json
  build-revision.json
  Dockerfile
  package.json
  preflight.json
  route-table.json
  automations.json
  server/
    boot.json
    boot.js
    sources.json
    source-metadata.json
    source-map.json
  assets/
  migrations/
  resources/

Builds always write .assembly-line/. The directory is generated output: .gitignore excludes it and pnpm clean:artifacts removes it.

Important files:

• manifest.json - complete compiled agent contract.
• server/sources.json - deterministic executable wrappers and provider modules plus the exact selected local/published plugin sources. Declarative composition is stored directly in manifest.json; only plugin-provided executable behavior requires a source module.
• agent-revision.json - deterministic source/config revision.
• build-revision.json - deterministic identity of the complete immutable runtime artifact, including packaged framework code. Deployment images and cache reuse use this identity so a framework-only change cannot reuse a stale image while the authored agent source is unchanged.
• package.json - artifact dependency declaration and start script (node server/boot.js).
• preflight.json - required env and provider setup.
• route-table.json - HTTP channel routes.
• automations.json - canonical schedule- and event-triggered automation registrations.
• resources/ - byte-for-byte copies of every root or recursive-subagent skill resource plus every non-UTF-8 (binary) file from any agent folder, so all files hashed into the manifest actually ship; server/sources.json carries UTF-8 text only.
• migrations/ - adapter-generated migrations plus any agent-authored migrations/ folder, copied verbatim.
• server/boot.js - production Node runtime boot entrypoint. It loads manifest.json and server/sources.json, constructs the resolved adapters, and serves the full HTTP API, not a health-only stub.
• deployment.json - created by assembly-line deploy after a local or provider publish/prepare operation, not by plain build.

The artifact is generated output and should not be committed.

Build artifacts support two package modes:

• Local mode is the default inside a monorepo checkout. It writes file:./vendor dependencies for Assembly Line workspace packages, copies those packages once under vendor/, links them into node_modules, and copies the runtime dependency closure needed for no-install local artifact smoke tests. Optional native dependencies are filtered to the generated Docker target (Linux x64 with glibc) plus the current build host, so the same local artifact remains runnable for smoke tests without copying every platform binary.
• Release mode is for published package deployments and is the default when the toolchain is installed from npm (i.e. when no packages/ workspace layout is present). You can also force it with packageMode: "release" on await buildAgent({ ..., packageMode: "release" }) or ASSEMBLY_LINE_ARTIFACT_PACKAGE_MODE=release. The artifact package.json pins the published @assemblyline-agents/* versions (all share one fixed version) and omits local vendor/ and node_modules/ copies; each published package's own external dependencies are resolved transitively by the package manager. Referenced channel and gateway-adapter packages are pinned even when the compiler's own install cannot resolve them locally (for example under pnpm's isolated layout), so the deploy install, not a silent drop, is what decides whether they exist. The generated Dockerfile then installs dependencies through normal package-manager semantics.

Runtime Lifecycle

For each run, the runtime:

1. Persists the run before execution starts.
2. Resolves or creates a conversation and materializes safe attachment metadata.
3. Loads the bounded conversation control-state snapshot and synchronously evaluates the compiled composition plan, invoking a selected plugin composition handler only when the plan names one.
4. Validates the declaration against static ceilings and the compiled catalog, persists a complete capability checkpoint, then emits run.capabilities_resolved before applying it.
5. Checks the active plugin-declared connections and builds context from permanent instructions plus the resolved instructions, skills, tools, connections, subagents, sandbox metadata, and output schema.
6. Starts the model loop or forced tool call with that exact snapshot.
7. Sends each Pi model request with an explicit maximum output budget of 128,000 tokens (or the lower caller/model limit), then observes provider responses and attempts to persist source-backed usage without local spend or cumulative-usage gates.
8. Creates and sends final delivery obligations idempotently.
9. Marks the run completed, failed, suspended, cancelled, waiting for input, or waiting for approval.

At every model-iteration boundary the harness checks the state revision. ctx.agentState or a usePersistentState() setter atomically increments that revision and emits agent.state_changed without values. A dirty run re-evaluates before the next request, records the new snapshot, atomically replaces visible tools, and refreshes model/reasoning/prompt/sandbox/schema selection. Nothing changes during an in-flight provider request or tool call.

Recovery compares the current state revision and declaration hash with the last hydrated agent.capability_snapshot checkpoint. It reuses an exact match and re-evaluates otherwise; it never reconstructs capabilities from partial events. Hook-evaluation failure emits agent.hook_evaluation_failed and fails the run. Event-observer failure emits agent.event_handler_failed and is non-terminal.

Failed final deliveries are durable. When a channel send keeps failing retryably, the obligation is deferred onto a durable delivery queue (delivery.deferred, status pending with backoff) instead of going terminal, and a delivery worker drains it later, including after a crash or in another replica (Postgres leases use for update skip locked). Exhausted or non-retryable deliveries end as delivery.failed with failedAt.

Channel adapters must make their delivery boundary explicit. The Slack adapter keeps final text and explicitly selected files in one completion transaction. An attachment read, upload, or completion failure therefore cannot leave a misleading final message claiming that a file was attached. After the durable retry budget is exhausted, the runtime sends a separate text-only failure notice naming the preserved files and carrying the transport error.

Recovery is staleness-guarded and conservative. Executing runs heartbeat updatedAt (default 30s, ASSEMBLY_LINE_RUN_HEARTBEAT_MS); only runs stuck in created/running past max(5min, 4x heartbeat) are swept, and each candidate is claimed through an idempotency key so concurrent replicas never double-recover. The sweep completes already-delivered runs without re-sending, cancels tool calls before side effects start, attempts one in-place resume when a harness continuation checkpoint exists, enqueues a real pending delivery for runs that reached a model response but not delivery, and marks everything else failed (with run.failed) instead of pretending it completed. listenNodeRuntime runs recoverIncompleteRuns() once at boot and startBackgroundWorkers() keeps the delivery, sandbox-sync, conversation-turn mailbox, and orphan-recovery workers running until the server closes.

Evolution History

When an agent or background review changes a skill or durable memory, the runtime mirrors the accepted result into a private Git repository. The default location is <artifactRoot>/evolution, created lazily on the first change. The model never receives this path or a Git tool. Existing skill and memory stores remain the live, authoritative state; Git is a readable audit trail.

The repository uses this layout:

evolution/
  skills/<name>/SKILL.md
  subagents/<path>/skills/<name>/SKILL.md
  subagents/<path>/memory/<path>
  memory/<path>
  memory/users/<hashed-id>/conversations/<hashed-id>/projects/<hashed-id>/<path>

User, conversation, and project identifiers are hashed before they appear in paths. Each non-empty commit records the runtime-selected actor and reason, and includes a run or background-review job identifier when available. Each memory or skill sandbox-sync batch becomes one commit even when it changes several files. Approval-gated skill changes are committed only after approval; rejected and pending changes do not enter the repository.

Inspect local history with ordinary read-only Git commands:

git -C .assembly-line/evolution log --stat
git -C .assembly-line/evolution show HEAD

Successful run-associated commits emit evolution.commit_created; tracking failures emit evolution.commit_failed and a warning. Tracking is deliberately fail-open so an audit-storage problem cannot corrupt or undo accepted live state. Set ASSEMBLY_LINE_EVOLUTION_TRACKING=false to disable the projection, or ASSEMBLY_LINE_EVOLUTION_ROOT to place it on a writable durable volume. Generated deployment images include Git but exclude local evolution/ history from the image build context. A hosted deployment therefore needs a persistent mount at the selected root if its history must survive replacement. Embedders may supply RuntimeOptions.evolutionTracker for another audit sink or set it to false explicitly. Because commits contain complete skill and memory bodies, protect this repository with the same access controls as the underlying stores. The built-in tracker serializes one runtime process; do not point concurrent replicas at the same Git working tree. Use one root per replica or a custom tracker backed by shared storage.

Usage Accounting And Observability

Usage accounting is behavior-neutral observability. It never reserves expected tokens or cash, estimates a request, rejects a provider call, or changes an agent response based on local cost state. A response_completed event is normalized into the ledger using the provider response ID when available, so retries are idempotent. If ledger persistence fails, the provider response still completes; the durable model event and runtime warning expose the observation gap.

Every record carries the run, parent run, stable agent/revision, subagent, iteration, provider, requested and response model, provider request ID, billing mode, UTC occurrence time, native input/output/cache-write/cached/ reasoning tokens, currency, receipt hash, and sanitized provider receipt. Actual cash uses integer cost_micros and only accepts provider_reported/provider_reconciled provenance. Local catalog-price math is discarded. Unavailable cash remains null, never $0 or an estimate.

The ledger has three non-additive record kinds:

• transaction: one attributable model response; this is the default report.
• control_total: a provider organization/activity bucket used to verify completeness.
• adjustment: reserved for explicit accounting corrections.

Provider plugins own response accounting and optional reconciliation. The kernel accepts only normalized records; it contains no OpenAI, OpenRouter, or Codex response branches. OpenRouter's plugin, for example, follows its generation ID to a settled receipt and can import activity control totals with OPENROUTER_MANAGEMENT_KEY. OpenAI's plugin can import organization usage and cost control totals with ASSEMBLY_LINE_OPENAI_ADMIN_KEY or OPENAI_ADMIN_KEY. The openai-codex plugin labels Pi's standard provider as subscription billing and records provider-reported tokens; unavailable per-turn cash remains null.

A third-party model plugin may implement the same hooks without a framework edit. Reconciliation loads the selected package's assemblyLineUsageReconciler registration, and the harness calls the selected model registration's accounting hook. Plugins may attach sanitized native receipts, but neither plugins nor the kernel may turn observability into a request gate.

The reconciliation report returns transaction totals, provider control totals, signed variances, separate unreconciled token/cash counts, and exact run-token/run-cost coverage for any [from,to) USD period. Schedule the authenticated reconciliation operation after the provider's settlement delay (48 hours by default for OpenAI). Control totals are excluded from ordinary transaction summaries, so reconciling never doubles reported spend.

A transport failure after the durable request_started event produces an unobserved transaction with zero measured tokens, null cash, and unavailable provenance. Crash recovery does the same for interrupted requests. These records express uncertainty; they are not charged against a local quota. A missing or failing usage facet emits degradation warnings but never blocks provider execution or response delivery.

Node Runtime HTTP API

Auth model

In dev mode, local inspection and API-run endpoints are open for quick iteration. In production, the Node host fails boot unless control-plane auth is configured with a host-provided auth policy or ASSEMBLY_LINE_ADMIN_TOKEN. /manifest, /routes, /conversations, /conversations/:id/messages, /runs, /usage, /runs/:id, /runs/:id/events, /runs/:id/timeline, and /runs/:id/stream require Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN> when the built-in token policy is used. Conversation rename/archive routes use the same authenticated agent-control policy. POST /conversations/:id/turns uses the authenticated run-create policy and, like POST /runs, is disabled in production by default; set ASSEMBLY_LINE_ENABLE_API_RUNS=true only for deployments that intentionally expose authenticated API-triggered runs. The resume and operator endpoints use the same admin auth but are not gated by ASSEMBLY_LINE_ENABLE_API_RUNS, paused and active runs must remain operable even when API-triggered run creation is off. Resumes are safe to replay: each one claims the run's per-pause idempotency key, so a double-submit executes the gated tool at most once, even across replicas.

Compiled channel routes, such as /slack/events or /message, are also mounted from the manifest route table. In production, first-party provider helper routes remain public so the provider can call them, but their normalizers must verify signatures or shared secrets before accepting a turn. Generic defineChannel() HTTP routes require a host auth policy or Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN>; the raw message fallback is dev-only unless the host has authenticated the request.

Endpoints

These internal routes are served only under the /assembly-line/* prefix. Callback-URL construction always emits /assembly-line/connections/callback.

Connection event callbacks are public because providers must reach them, but each adapter verifies the provider's signature, token, challenge, or channel secret before accepting data. The runtime checks the normalized source, event, connection, and JSON-subset filter against explicit automations before durable enqueue. Unmatched events are acknowledged and discarded without a run or a stored payload. Matching events enter the durable connection event inbox; the worker deduplicates by connection, principal, and provider event ID, leases deliveries, and retries transient failures with bounded backoff. A settled inbox payload is deleted immediately; the automation idempotency ledger remains the durable defense against a later provider replay. Postgres uses 024_assembly_line_connection_events; local file storage encrypts registration state and pending inbox records with the runtime connection secret.

The connection-event worker starts with the other durability workers. Set ASSEMBLY_LINE_CONNECTION_EVENT_WORKER=false only when another process owns that queue. Active runtimes reconcile registrations at boot and every minute; an OAuth callback also immediately reconciles that user's connection. Hosted deploys reconcile once more after activation using the receipt's deploymentUrl. If a provider requires manual console setup, the stored registration remains needs_setup with the exact callback URL and instructions instead of pretending it is active.

POST /runs/:id/replay guarantees equality of the model-visible input snapshot, not equality of the resulting output. Provider behavior, model sampling, live connections, tool results, and external state can change between executions. Every new replay records run.replay_started with the source run/revision and input, context, and attachment digests so operators can audit what was held constant. The replay uses a new conversation record to avoid appending duplicate turns to the source conversation; the frozen context bundle supplies the exact original history and memory snapshot to the model. The source run remains immutable; replay progress and results are recorded only on the new run.

Accepted provider channels may request a bounded composition window when one user action arrives as multiple webhooks. The durable conversation mailbox keeps every event independently idempotent, delays the head turn until the window closes, and atomically folds matching pending parts into one run. A coalesced run receives the ordered text and every attachment together; sibling mailbox rows are marked coalesced with the head turn id recorded in their private queue payload.

Direct conversations

direct is Assembly Line's built-in, provider-neutral client transport. It is the right channel name for a first-party app, internal console, custom web UI, or mobile client chatting with an agent. custom remains appropriate only for a developer-defined channel adapter with its own route, normalization, delivery, and trust boundary.

The direct API is control-plane HTTP, not a public provider webhook. Production boot requires host authentication, transcript reads require admin-read authority, mutations require agent-control or run-create authority, and conversation ids are checked against the runtime's stable agent scope before messages can be read or added. A client should send Authorization: Bearer <ASSEMBLY_LINE_ADMIN_TOKEN> (or credentials accepted by the host-provided auth policy) over HTTPS and must never embed an admin token in a public browser bundle.

Conversations persist across agent revisions when agent.id is stable. Their messages and metadata live in the configured state adapter. Inbound attachment bytes are normalized into the configured private blob adapter before transcript metadata is returned; generated files use the same private storage boundary. Object storage is not made public unless application code explicitly writes a public blob.

Create and stream a turn:

curl -N https://agent.example/conversations/01JTHREAD/turns \
  -H "Authorization: Bearer $ASSEMBLY_LINE_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"message":"Summarize the attached brief","stream":true}'

The SSE response includes durable lifecycle events, ephemeral model deltas, a final run_result, and stream_end. If the connection drops after the run id is known, reattach through GET /runs/:id/stream with Last-Event-ID. Use an Idempotency-Key header or a stable body eventId when a client may retry the initial POST.

Operator Controls

Cancellation is terminal at the durable write boundary. POST /runs/:id/cancel atomically moves any non-terminal run to cancelled, emits run.cancelled, settles the conversation turn, and returns 200; it does not wait for a model, tool, sandbox-sync, or recovery boundary. On the owning process it also aborts the active model request immediately. A queued tool call re-checks the durable run and cannot start after cancellation. An authored tool already executing cannot be safely unwound, but its result cannot revive, complete, or deliver the cancelled run. The endpoint is idempotent: retrying after the run is already cancelled returns the same terminal result with 200. A different executing replica observes the terminal row on its next heartbeat, bounded by ASSEMBLY_LINE_RUN_HEARTBEAT_MS (30 seconds by default). Cancel wins a race with suspend.

Suspension remains cooperative at the model-request boundary. Its intent is written atomically to the durable run row and the executing replica persists a continuation before moving the run to suspended.

Suspension applies only to running runs. It persists a harness continuation, moves the run to suspended, and keeps it outside orphan recovery while still bounding its active checkpoints. POST /runs/:id/resume claims a per-generation idempotency key and re-enters the normal continuation path. Pending tool records become cancelled when their run is cancelled.

The agent control is an ingress kill switch, not a process kill switch. Disabled channel ingress returns 503 without Retry-After and consumes neither capacity nor the provider event's idempotency key. In-flight runs, explicit resumes, schedules, and operator access continue. The setting is scoped by stable agent.id when present (otherwise the compiled revision). Use FileStateAdapter, PostgresStateAdapter, or another durable RuntimeSettingsStore; a missing settings facet falls back to memory and will not survive restart.

Remote CLI equivalents use the same authenticated HTTP API:

assembly-line runs suspend <runId> --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"
assembly-line runs resume <runId> --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"
assembly-line runs cancel <runId> --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"
assembly-line agent disable --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"
assembly-line agent enable --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"
assembly-line workspaces status <workspaceId> --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"
assembly-line workspaces retention <workspaceId> --tail 50 --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"
assembly-line workspaces gc --min-age-ms 86400000 --url https://agent.example --token "$ASSEMBLY_LINE_ADMIN_TOKEN"

ASSEMBLY_LINE_URL and ASSEMBLY_LINE_ADMIN_TOKEN are the flag fallbacks. Retention and garbage collection are dry runs unless --apply is present.

Request bodies are capped before parsing. The default limit is 10 MiB; set ASSEMBLY_LINE_HTTP_MAX_BODY_BYTES or NodeRuntimeServerOptions.maxRequestBodyBytes only when a deployment intentionally accepts larger webhook payloads.

Provider channel routes remain public HTTP routes because providers such as Slack, Telegram, Teams, and Discord interactions call them directly. Those routes must rely on their channel adapter's signature or token verification before accepting a turn. Long-lived provider ingress such as Discord Gateway is started by the Node host through startIngress() and feeds the same idempotent accepted-turn path as HTTP channels.

Trusted hosts and agent-to-agent gateways may pass principal and initiator to runtime.run(). Forward canonical identity claims only after authenticating the caller; do not forward provider tokens. Runs that provide only userId receive a compatibility principal scoped to their channel.

/assembly-line/automations/tick accepts optional now and lookbackMs values in the query string or JSON body. In production, set ASSEMBLY_LINE_SCHEDULER_SECRET and send it as Authorization: Bearer <secret> or x-assembly-line-scheduler-secret. Without a configured secret, the endpoint only accepts dev-mode runtime requests.

Scheduling is kernel behavior and has no agent.md provider selector. The Node host starts the in-process loop by default; set ASSEMBLY_LINE_SCHEDULER_ENABLED=false when an external scheduler owns calls to the tick endpoint. Duplicate workers coordinate through the selected state adapter's durable scheduling operations and idempotency keys. This is why the Postgres plugin exposes scheduling-store capability without pretending to be a second scheduler implementation.

Durability Workers

listenNodeRuntime runs recoverIncompleteRuns() once at boot, then calls runtime.startBackgroundWorkers() to keep the delivery worker, sandbox-sync worker, conversation-turn mailbox worker, background-subagent worker, background-review worker, and periodic orphan sweep running; they stop when the server closes. Each worker has an env kill-switch (ASSEMBLY_LINE_DELIVERY_WORKER, ASSEMBLY_LINE_SANDBOX_SYNC_WORKER, ASSEMBLY_LINE_CONVERSATION_TURN_WORKER, ASSEMBLY_LINE_BACKGROUND_REVIEW_WORKER, ASSEMBLY_LINE_BACKGROUND_SUBAGENT_WORKER, ASSEMBLY_LINE_RUN_RECOVERY), the heartbeat and sweep cadence are tunable (ASSEMBLY_LINE_RUN_HEARTBEAT_MS, ASSEMBLY_LINE_RUN_RECOVERY_INTERVAL_MS), and the delivery queue has lease, batch, attempt, and interval knobs. The canonical tables are in the Configuration Reference: Durability workers and recovery and Delivery queue.

Model-Call Resilience

Every model request runs inside a retry-and-deadline envelope. Retryable failures (408/429/5xx, provider overload, network errors) are retried with jittered exponential backoff, Retry-After hints are honored and capped, while fatal failures (invalid API key, authentication, invalid request, exhausted provider usage limits) fail the run immediately with the durable reason model.request_failed and enqueue a user-visible failure notice carrying the real error. Async channels (Slack, Discord, …) would otherwise never learn the outcome. A stream that stops producing events is aborted by an inactivity watchdog and retried. Retries are visible as model.request_retried events and warn-level log lines. When the retry budget is exhausted on a transient error, the run is left running with a durable run.execution_error event so the orphan sweep resumes it from the latest continuation checkpoint, one transient outage never terminally fails a run. Knobs: ASSEMBLY_LINE_MODEL_MAX_RETRIES, ASSEMBLY_LINE_MODEL_TIMEOUT_MS, ASSEMBLY_LINE_MODEL_MAX_RETRY_DELAY_MS, ASSEMBLY_LINE_MODEL_STREAM_IDLE_TIMEOUT_MS.

Progress Lease And Tool Timeouts

The run heartbeat supervises a renewable progress lease. An active run may execute for any total duration while it continues crossing durable progress boundaries: persisted checkpoints, completed model responses, settled tool executions, and explicit authored-tool ctx.reportProgress() calls all renew the lease. Foreground subagent progress also renews each active parent waiting on that child. Ordinary heartbeats, request starts, retries, and streamed response deltas do not. If a run makes no durable progress for ASSEMBLY_LINE_RUN_STALL_TIMEOUT_MS (default 1 h), its in-flight work is aborted via AbortSignal and the run fails with reason run.stalled. Parked runs (approvals, human input) hold no lease and start a fresh one on resume. The complete tool operation -- sandbox acquisition, workspace hydration, credential projection, authored execution, and model output conversion -- is bounded by ASSEMBLY_LINE_TOOL_TIMEOUT_MS (per-tool timeoutMs on the definition overrides it). The runtime passes an AbortSignal through the harness and abandons an unfinished acquisition, so a provider call that never settles cannot retain the run or later publish a stale sandbox. Model-supplied bash timeouts are clamped to ASSEMBLY_LINE_BASH_TIMEOUT_MAX_MS, timeoutMs: 0 falls back to the default rather than disabling the timeout.

Sandbox retain/dispose calls made after terminal ownership or by the sync worker are independently bounded by ASSEMBLY_LINE_SANDBOX_CLEANUP_TIMEOUT_MS (default 30 s). A cleanup timeout releases run admission and records failure evidence; it does not interrupt a sandbox owned by an actively executing run. Long-running tools remain supported by setting their definition's timeoutMs to the required duration or to 0 to disable the tool deadline intentionally.

Terminal Outcomes And Recovery Fidelity

Every run that reaches failed records a machine-readable terminalReason and human-readable terminalError on the run record (queryable without scanning the event log; also on the run.failed event payload). Reasons include model.request_failed, run.max_iterations_exceeded, run.stalled, run.initialization_failed, output.validation_exhausted, trigger.*_failed, run.orphaned, connection.unavailable (the recovery sweep terminalized a run parked on a required connection that never became available, unblocking its conversation), and run.execution_failed (the orphan sweep terminalized a run whose last recorded outcome was a durable execution error). Terminal outcomes (run.completed/run.failed/ run.cancelled) are always logged at a single choke point, whichever code path produced them; response content is never logged, only its size.

An error that escapes before the first durable model.request_started event and before the harness has persisted a resumable continuation is an initialization failure, not a resumable outage. The runtime marks it failed immediately with run.initialization_failed and queues the normal user-visible failure notice. A saved continuation still takes the normal recovery path; errors after a model request retain the existing fatal/transient classification and recovery behavior.

tool.execution_failed is a failed tool-call event, not a terminal run outcome. Model-invoked local, connection, delegation, and sandbox failures are returned to the model while the run remains active. Historical run records may still carry the retired tool.execution_failed or connection.tool_failed terminal reasons, which remain readable for replay and diagnostics.

Crash recovery reads the newest continuation checkpoint when reconciling a run that died after its final model response: the actual answer is delivered (recovered: true, contentRecovered: true) and the "response was interrupted" notice is reserved for genuinely missing checkpoints. A run whose last recorded outcome is a durable execution error (a run.execution_error or failed recovery resume after the last model response, or a response whose finishReason is failed) did not crash. It failed, so the sweep terminalizes it with reason run.execution_failed and delivers a notice carrying the recorded error instead of the misleading restart notice. When a crash interrupted a tool batch, the resume surfaces already-completed tool outputs and interrupted-tool warnings to the model as a recovery report so completed side effects are not blindly re-executed. Dynamic automations that fail repeatedly back off exponentially and are auto-disabled after ASSEMBLY_LINE_SCHEDULE_MAX_FAILURES consecutive failures (an operator-visible schedule.disabled_after_failures control event is recorded).

Concurrency And Rate Limiting

Accepted provider turns are first written to a durable per-conversation FIFO mailbox. One turn per (agent, conversation) may be running; later turns wait, while distinct conversations can use the full global concurrency budget in parallel. Channel normalization defines the boundary, so separate Slack thread roots are separate conversations and can run in parallel while one DM or one thread remains serialized. Provider routes therefore acknowledge valid durable work even when all run slots are busy instead of relying on webhook redelivery for backpressure. Postgres enforces the active-turn exclusion across replicas. The active dispatcher renews its mailbox ownership lease while the run executes; if that process disappears, lease expiry hands settlement to recovery without imposing a maximum run duration.

Approval and explicit operator suspension can deliberately retain the active mailbox position. A reply-capable authorization wait releases it; when consent completes, the callback places a continuation at the end of the same FIFO. This preserves one active turn per conversation without letting an external browser wait block later messages indefinitely.

Direct brand-new runs still pass through the bounded semaphore before run state is written. When its in-memory admission queue is full the runtime rejects direct work with RunCapacityError, and the Node host maps that to HTTP 429 with a whole-second Retry-After on POST /runs. In-place resumes (approvals, protocol-owned human input, orphan recovery, and scheduler continuations of an existing run) never queue behind the limit. Queueing a resume behind the run it unblocks would deadlock, but continuations still count toward the drain performed by graceful shutdown. Reply-channel authorization callbacks instead enqueue a new continuation turn after releasing the old mailbox position. A terminal in-place resume also releases the next mailbox turn for that conversation.

Ingress rate limiting is a token bucket applied after auth on provider-channel, run-create, and scheduler routes (health and admin routes are exempt). It is off by default and enabled either through NodeRuntimeServerOptions.rateLimit (see the customization guide) or through ASSEMBLY_LINE_INGRESS_RATE_LIMIT and ASSEMBLY_LINE_RUNS_RATE_LIMIT (capacity/refillPerSecond form, e.g. 60/10); ASSEMBLY_LINE_RUNS_RATE_LIMIT also seeds the run-resume and run-control buckets unless those are configured separately. ASSEMBLY_LINE_MAX_CONCURRENT_RUNS bounds simultaneously executing brand-new runs (createProductionRuntimeOptions defaults it to 16 outside dev) and ASSEMBLY_LINE_MAX_QUEUED_RUNS (default 0) lets excess runs wait for a slot. The canonical table is Concurrency and rate limiting.

Rate-limited requests receive 429 { "error": "Rate limited." } with a Retry-After header.

Rate-limit buckets are keyed by route class and client address. Behind a reverse proxy or load balancer, set ASSEMBLY_LINE_TRUST_PROXY=true so the first X-Forwarded-For hop is used as the client address; without it, every proxied request shares one bucket keyed by the proxy's address, so a single noisy client can exhaust the limit for everyone.

Graceful Shutdown

listenNodeRuntime returns a NodeRuntimeHandle with an idempotent shutdown({ timeoutMs? }) and a closed promise. The shutdown sequence: mark draining (/readyz starts answering 503 so load balancers stop routing new traffic; /health//healthz stay 200 for liveness) -> close the HTTP listener and stop channel ingress -> stop the scheduler -> stop background workers -> wait for in-flight runs up to the timeout -> flush the telemetry sink -> close the state adapter (Postgres ends its pool when it created it). Runs still executing at the timeout are abandoned safely: orphan recovery repairs them on the next boot.

assembly-line serve and assembly-line deploy --serve install SIGTERM/SIGINT handlers (installSignalHandlers from @assemblyline-agents/node): the first signal drains gracefully and exits 0; a second signal exits 1 immediately.

The drain timeout is ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS (default 30000), and ASSEMBLY_LINE_SIGNAL_HANDLERS=false prevents handler installation; see Graceful shutdown.

State And Blob Storage

Local development uses file-backed state and local blob storage. Production state should use Postgres:

state: neon
blob: r2

Postgres stores runs and atomic control intents, events, messages, conversations, tool calls, approvals, delivery obligations, schedules, schedule run lifecycle status, memory indexes, workspace-scoped file catalog records, sandbox leases, idempotent usage receipts, exact micro-dollar/token aggregates, run queues, idempotency keys, learned skills, dynamic automations including trigger metadata, dynamic connections, runtime settings, conversation-scoped agent hook state with atomic aggregate revisions, capability checkpoints, workspace identities, immutable version metadata, checkpoint and fork pointers, search chunks, and control-plane audit events. PostgreSQL provides durable multi-replica usage observability; file/in-memory accounting is process-local. Conversation message text has a Postgres full-text index for attributed history search; no separate Slack history database is required.

Connection grants and OAuth authorization sessions use the durable state adapter when it implements those stores. The Postgres adapter does, so Postgres-backed production deployments do not need a separate file encryption secret for connection credentials. Tool discovery never creates authorization sessions; explicit authorization reuses an unexpired pending session and prunes expired sessions before creating a replacement.

When a production Node deployment uses file-backed connection credential stores, set ASSEMBLY_LINE_CONNECTION_STORE_SECRET or ASSEMBLY_LINE_SECRET to a stable secret of at least 32 characters. The built-in local development fallback is accepted only with devMode: true; production boot rejects missing, short, or known development secrets.

Blob storage stores context bundles, attachments, extracted text, generated artifacts, and immutable workspace file and manifest objects. Workspace content is addressed by SHA-256 and shared safely across versions and forks. Blob records are private by default. S3/R2 *_PUBLIC_BASE_URL is used only when code explicitly writes a blob with visibility: "public"; context bundles, memory files, inbound attachments, and workspace objects should remain private.

The database decides which workspace version is current. The blob store supplies the bytes named by that version's manifest. Back up both systems on a coordinated schedule:

1. Take a Postgres backup or point-in-time recovery marker.
2. Preserve object versions or a bucket snapshot that covers the same or an earlier point.
3. Restore Postgres first, then restore missing blob objects.
4. Run workspaces verify-all, then workspaces reachability.
5. Keep garbage collection in dry-run mode until verification is healthy.

Restoring only Postgres can leave referenced objects missing. Restoring only R2 or S3 cannot recover head pointers, checkpoint names, fork ownership, or idempotency records. Never run garbage collection while a database restore, object restore, or workspace sync is in progress. A corrupt or missing object can be repaired only with bytes that match its recorded immutable hash. A stuck head repair uses compare-and-set so it cannot overwrite a concurrent update.

Sandbox Sync

Sandbox-backed file tools write through a provider working copy, but durable production persistence remains the versioned Assembly Line workspace in state and blob storage. A new sandbox hydrates the current complete manifest before use. When a run finishes with dirty sandbox files and async sync is enabled, the runtime retains the dirty sandbox instead of deleting it, queues a sync job, and lets final delivery complete. Every session and sync job carries immutable agent scope, logical session key, and physical provider session key metadata. The sync worker first queries provider inventory and requires an exact ownership match, then calls adapter connect() and wake() so paused, stopped, detached, or otherwise retained warm sessions can still be synced. After a successful sync the session is marked clean and disposed through the adapter's clean lifecycle path. Sync compares the working tree with its base version, uploads changed content, records deleted paths, and advances the head only if that base is still current. Conflicts and exhausted failures retain the dirty sandbox for operator recovery.

Dirty generations that have missing or mismatched ownership are quarantined under a separate durable session-record key. Their provider identity is never rewritten, their pending sync job follows the quarantined record, and the next run receives a collision-resistant fresh provider generation.

Hosted sandbox adapters do not use provider snapshots as the normal persistence path. Snapshots are created only when the configured sandbox snapshot policy requests them and the provider SDK exposes snapshot creation.

Deploy Targets

Every non-local target follows one CLI sequence: resolve the publisher, run optional target preflight, reconcile or reuse the selected content-addressed sandbox environment, run preparation hooks, sync secrets when requested, run artifact migrations once (locally or through the publisher), publish, then write the final receipt with migration status and sandbox artifact resolution. Environment reconciliation also runs for --prepare-only, but control-only activate, rollback, ingress, and destroy operations skip it. Built-in option precedence is CLI flag, environment variable, selected agent.md gateway profile, then provider default.

Release history

Hosted deploys automatically record the exact compiled source as an immutable Git snapshot when the agent is inside a Git worktree. The CLI allocates the agent's next rev-N tag only after activation and health checks succeed; a failed or prepare-only deploy does not create a deployed version. Agents in a shared repository use identity-scoped tags, so each agent has its own rev-1, rev-2, and so on. Redeploying the same agentRevision reuses its source version while the receipt still records the new deployment event and current buildRevision.

Successful receipts include releaseVersion, releaseGitSha, and releaseActor. Assembly Line Builder imports those fields automatically, which means deploys started in Builder, a terminal, or a coding-agent session all populate the same Versions history. Direct provider changes made below the CLI do not create this evidence and remain out of band until the next Assembly Line deploy.

Environments

--env <name> selects the deploy environment (default: development). Every environment holds an isolated deployment of the same agent:

• Identity. The default environment uses unscoped provider resource names. Any other environment gets environment-scoped resources: Docker suffixes container and volume names (assembly-line-<agent>-test) and requires an explicit --port in serve mode; Fly derives <app>-<environment> when the app name comes from provider defaults (an explicit deploy.fly.app is respected verbatim); VPS uses the host inventory's baseDomain and derives <agent-id>.<baseDomain> for the default environment or <agent-id>-<environment>.<baseDomain> for any other environment; Railway passes the environment through natively, so the named Railway environment must exist. Configure one wildcard DNS record for the VPS host namespace. Explicit Docker, Fly, and Railway plugin options win over derived names; VPS hostnames remain host-owned and deterministic.
• Receipts. Each environment's receipt is written to .assembly-line/deployments/<environment>.json. A successful hosted receipt also carries the immutable release version and Git commit described above.
• Teardown. assembly-line deploy <agentRoot> --env <name> --destroy removes the environment's provider resources on every built-in target (docker, fly, railway, hetzner, local). Durable data, volumes, databases, the VPS deployment directory, survives unless --purge-data is passed; Fly and Railway require it explicitly, because destroying a Fly app or deleting a Railway environment always removes the volumes and databases inside it. VPS destroy never touches shared host infrastructure (the Caddy edge, the shared Postgres cluster) or customer-owned external databases. Destroying the default environment additionally requires --force.

Composed with assembly-line eval --url, this is the ephemeral test-environment recipe:

assembly-line deploy agent --env test --sync-secrets --secrets-from .env.test
assembly-line eval agent --url https://<test-gateway> --token $ASSEMBLY_LINE_ADMIN_TOKEN
assembly-line deploy agent --env test --destroy --purge-data

Local

Use local deploy for developer machines or long-lived VMs:

assembly-line deploy agent --target local --serve --port 3000

Railway

deploy: railway publishes the built artifact through the @assemblyline-agents/railway deploy publisher and Railway CLI.

Required:

• RAILWAY_TOKEN or authenticated Railway CLI
• A linked project/service, RAILWAY_PROJECT_ID and RAILWAY_SERVICE_ID, or plugin-owned deploy.railway.project and deploy.railway.service options

deploy:
  railway:
    project: prj_x
    service: svc_y

assembly-line deploy agent --target railway

When the agent uses state: railway, deployment first inspects the selected Railway environment. It reuses the Postgres database service when present, otherwise provisions one with railway add --database postgres, and sets the Assembly Line service's DATABASE_URL to ${{Postgres.DATABASE_URL}} before railway up. The deploy receipt records whether the database was created or reused. Use this mapping to require a specific existing database service without automatic creation:

state:
  railway:
    databaseService: name
    provision: false

Automatic creation uses Railway's default Postgres service name. A local DATABASE_URL is not required for this auto-provisioned path; the ordinary Postgres, Neon, Supabase, and provision: false paths still require one during deployment preflight.

Syncing secrets to the target

By default, deploy sets nothing on the remote service. You configure variables in the provider dashboard. Pass --sync-secrets to push your local secrets as part of the deploy: the CLI uses declared credentials from the host environment and optionally reads an explicitly named --secrets-from <path>. It never loads a project .env implicitly. The publisher receives those credentials before publishing, so the first deploy boots with them. Undeclared host variables and names compiled into config.production.ts are never copied. Only key names are logged, never values; empty keys are skipped, and removing a local key does not delete the existing remote value. Variable names must use portable shell identifier syntax. Values containing line breaks or null bytes are rejected before a provider command runs. Supported on railway (one railway variable set KEY --stdin --skip-deploys call per key), fly (one flyctl secrets import stream for all keys), hetzner (an atomic remote 0600 environment file), and docker (held in memory for the deploy and handed to serve-mode docker run through the child process environment with value-less --env KEY flags, never on argv or disk; image-only builds never bake secrets). The local target reports that sync is unsupported and leaves secrets to you.

Private dotenv files (.env and .env.*, except .env.example and .env.*.example) remain excluded from compiled artifacts. An explicit --secrets-from dotenv input is repaired to owner-only permissions (0600). Railway and Fly receive runtime values through child-process stdin; values do not appear in provider argv, deployment logs, or receipts. A failed secret sync aborts the deployment before publish.

Credentials required by the selected deploy adapter remain local and are not copied into the agent runtime. For example, RAILWAY_TOKEN authorizes the Railway CLI and FLY_API_TOKEN authorizes flyctl; runtime requirements such as model, channel, state, blob, and connection credentials are eligible for remote sync.

assembly-line secrets diff separates required, optional, provider-managed, missing-local, missing-remote, and extra names. Provider-managed values include the VPS public URL and host-database URL, plus the AWS_* backup aliases the VPS publisher derives from ASSEMBLY_LINE_VPS_BACKUP_*; these do not appear as misleading extras. Secret values are never read into the report.

assembly-line deploy agent --target railway --sync-secrets

The deploy receipt is written to .assembly-line/deployments/<environment>.json. deploymentUrl is the reachable service URL when the provider CLI reports one; otherwise it is null. dashboardUrl points to the provider's management console, so the two are never conflated. Every built-in provider receipt records both agentRevision and buildRevision; health endpoints expose the same pair.

Docker

Preview: this surface may change without notice.

deploy: docker builds the compiled artifact as a Docker image through the @assemblyline-agents/docker deploy publisher and can run it locally.

Served deployments use a stable assembly-line-<agent-slug> container name. When a selected model plugin declares persistent credential storage, the artifact also uses a stable assembly-line-<agent-slug>-data volume mounted at /data; the slug comes from agent id, then name, then the agent folder.

Served containers run with --restart unless-stopped, matching the VPS compose default, so the agent comes back after daemon or host restarts. Redeploys stop the outgoing container with a 30-second shutdown grace before removing it; the old release is never SIGKILLed mid-run.

deploy:
  docker:
    image: assembly-line/my-agent
    serve: true
    port: 3000

assembly-line deploy agent --target docker

Fly

Preview: this surface may change without notice.

deploy: fly writes a minimal fly.toml and deploys the artifact with flyctl through the @assemblyline-agents/fly deploy publisher.

Artifacts whose selected plugins require persistent /data storage provision the app-scoped assembly_line_data volume. When that volume already exists, deploys reuse it from whichever region it lives in and align the provider CLI's primary region to the volume; an explicit conflicting deploy.fly.region or FLY_REGION fails loudly instead of creating a second empty volume that would fork durable state. Fly deploys are limited to single-Machine apps because Fly volumes are Machine-local; deploy and auth fail clearly when an existing app has more than one Machine.

The generated fly.toml defaults to always-on (auto_stop_machines = "off", min_machines_running = 1): durable agents run schedules and background work that a stopped Machine cannot make progress on, so scale-to-zero is an explicit opt-in via the autoStop (and optional minMachinesRunning) deploy options.

Required:

• FLY_API_TOKEN
• deploy.fly.app or FLY_APP_NAME

deploy:
  fly:
    app: my-agent
    region: iad

assembly-line deploy agent --target fly

The deployment contract has credential-free smoke checks: pnpm smoke:deploy:docker exercises a real local build, run, recreation, remote command, and persistent volume; pnpm smoke:deploy:fly parses generated configuration with the installed Fly CLI and verifies the publisher's CLI surface using an intentionally invalid token. Neither check creates hosted resources. pnpm smoke:deploy:fly:live is the opt-in hosted exit gate: it creates a uniquely named app, deploys through the Assembly Line publisher, verifies HTTP health, secret sync, SSH execution, and /data persistence across a redeploy, then destroys the app and volume and verifies their absence. It requires an authenticated flyctl session and may incur brief provider usage.

Hetzner VPS

Supported.

deploy: hetzner, implemented by @assemblyline-agents/vps, deploys to an AMD64 Ubuntu 24.04, Ubuntu 26.04, or Debian 12 Hetzner host. The package contains the reusable VPS publisher; the public provider identity is hetzner.

Hetzner hosts can be created or adopted through the plugin's exported bootstrapHetznerHost() operator API. It verifies that the public key matches the private key selected by identityFileEnv, creates an assembly-line sudo user, installs Docker Engine and Compose, enables UFW, fail2ban, unattended upgrades, provider backups, delete and rebuild protection, and a Hetzner Firewall. The provider firewall restricts SSH to the API's reviewed source-CIDR allowlist unless global SSH is explicitly authorized; UFW admits port 22 behind that provider edge so a changing workstation address cannot create a second, stale allowlist. If a native deploy times out before a host key is returned, Assembly Line replaces stale SSH source rules on its Hetzner firewall with the native process's current public IPv4 /32 and retries once. It never changes firewall access after a host-key mismatch. This keeps SSH key-only and host-key pinned while tolerating network or full-tunnel VPN changes. The plugin waits for cloud-init and the security services, pins the SSH host key, then writes the inventory entry. Existing servers are never rebuilt implicitly. Adoption of an existing named server additionally requires an expected host-key SHA-256 fingerprint obtained from the provider console or another trusted path; Assembly Line will not establish trust from an in-band key scan alone.

Host provisioning is intentionally not a framework CLI namespace. Use the package API from a reviewed operator program or register an already hardened host directly in inventory; the deploy plugin owns this operational surface.

The server must have Docker Engine, Docker Compose, flock, curl, ss, seccomp, AppArmor, and root SSH or passwordless sudo. Host Postgres mode also requires OpenSSL and systemd. Register it by name in assembly-line.hosts.json:

{
  "version": 2,
  "hosts": {
    "production-eu": {
      "address": "203.0.113.10",
      "ingress": {
        "baseDomain": "agents.example.com",
        "defaultVisibility": "public"
      },
      "ssh": {
        "user": "deploy",
        "port": 22,
        "identityFileEnv": "ASSEMBLY_LINE_PRODUCTION_EU_SSH_KEY",
        "hostKeySha256": "SHA256:replace-with-the-pinned-fingerprint"
      },
      "provider": {
        "kind": "hetzner",
        "resourceId": "optional-server-id",
        "region": "ash"
      }
    }
  }
}

Inventory precedence is deploy.hetzner.hostsFile, ASSEMBLY_LINE_HETZNER_HOSTS_FILE, then the nearest assembly-line.hosts.json found upward from the agent root. The SSH private key path comes from identityFileEnv; the key and host address are never written to deployment receipts. Host-key scanning must match the pinned SHA-256 fingerprint before strict SSH is allowed.

deploy:
  hetzner:
    host: production-eu
    environment: production
    expectedRegion: ash
    database:
      mode: host
state: postgres
blob: r2
sandbox: e2b

Set deploy.hetzner.environment when an agent's established unscoped ingress belongs to an environment other than development. That environment remains the default and keeps <agent-id>.<baseDomain>; other environments receive the normal environment suffix.

Hetzner deployment requires a stable agent.id, Node runtime, Postgres state, S3/R2 blobs, a hosted sandbox, ASSEMBLY_LINE_ADMIN_TOKEN, and one wildcard DNS record for the host inventory's ingress base domain. Local state/blob storage and local or Docker-socket sandboxes are hard preflight failures. ASSEMBLY_LINE_VPS_ALERT_WEBHOOK_URL is an optional notification destination; health checks continue to run and record failures in systemd/journald when it is unset. expectedRegion compares the configured intent with inventory and warns about likely user, Photon, database, or sandbox latency.

Use --sync-secrets on the first deploy. Repeat deploys can reuse the complete remote 0600 environment without copying runtime credentials back to the operator machine; the publisher validates required keys remotely before database setup, migrations, and activation. For public agents, the VPS publisher sets ASSEMBLY_LINE_PUBLIC_URL to the derived HTTPS hostname on every secret sync so callbacks and generated public links cannot retain a prior provider's hostname. Private agents receive no public URL.

assembly-line secrets diff agent --target hetzner --env production compares required and configured key names without returning remote values. --sync-secrets reads declared credentials from the command environment; an explicitly requested missing --secrets-from file is an error.

assembly-line deploy agent \
  --target hetzner \
  --sync-secrets \
  --env production

The host owns agents.example.com. The default environment for agent ID support receives support.agents.example.com; an alternate staging environment receives support-staging.agents.example.com. This requires one *.agents.example.com DNS record, not per-agent DNS configuration.

Each agent receives a dedicated hardened non-root container, ingress network, data network, /data volume, secret file, hostname ownership record, and database identity. Runtime containers are read-only, drop all capabilities, set no-new-privileges, carry CPU/memory/PID/log limits, and never mount the Docker socket. A bounded, non-executable /app/.assembly-line/module-cache tmpfs holds generated runtime module-cache files without making the application root or the rest of the Assembly Line artifact namespace writable. Runtime startup probes that exact cache path and fails readiness with runtime_module_cache_unwritable when the deployment did not provide it. A trusted shared Caddy container joins each ingress network but agents do not join one another's networks.

For an internal-only agent, declare ingress: { visibility: "private" }. Private activation does not bootstrap or attach Caddy, request a certificate, publish a hostname, or run public readiness checks. Changing visibility removes or attaches the route and hostname claim transactionally.

Releases use immutable revision-labelled images and inactive blue/green slots. Preparation, activation, rollback, and ingress reconciliation are separate operations:

# Build the inactive slot, sync secrets, run migrations, but do not route traffic.
assembly-line deploy agent --target hetzner --env production --sync-secrets --prepare-only

# Authenticate any interactive model plugin on the prepared release if needed.
assembly-line auth openai-codex agent --target hetzner --env production --prepared

# Activate exactly the persisted prepared revision.
assembly-line deploy agent --target hetzner --env production --activate

# Restore the previous runtime/route without changing durable state.
assembly-line deploy agent --target hetzner --env production --rollback

# Reconcile public/private ingress without rebuilding.
assembly-line deploy agent --target hetzner --env production --ingress-only

Activation rejects stale prepared metadata, waits for container /readyz, transactionally claims the derived hostname for public agents, reloads Caddy, verifies public readiness, records the prior slot as the rollback target, and only then removes the old runtime. Private activation transactionally removes any old route and hostname claim. Failures restore the prior ingress state and leave both the live runtime and the recorded previous release untouched, so a failed deploy never redefines the rollback target as the release still serving traffic. --rollback refuses with a clear error when no distinct previous release exists instead of stopping the live container. The host retains the active and previous build-revision directories and prunes older managed release directories and images. A repeated immutable build reuses the host image cache and skips artifact upload. The deploy lifecycle ends with an explicit cleanup stage after activation (or after preparation for --prepare-only). Cleanup also runs when any post-preflight stage fails, while preserving the original deploy error. Explicit activation, rollback, and ingress reconciliation use the same final cleanup path.

VPS cleanup is serialized with image builds and only reclaims exited or dead inactive Assembly Line runtime containers for that deployment, obsolete tagged release images, dangling deployment-labelled images across the host, and abandoned artifact-upload directories older than 24 hours. Active, rollback, prepared, and container-referenced images are protected, and an intentionally stopped active-slot container is left in place. The generated runtime image cleans npm's download cache and applies /app ownership within existing filesystem layers instead of copying the application into a second ownership-only layer. Image builds use Docker's failed-intermediate-container cleanup, and uploads remove their remote staging directory even when installation fails. Cleanup ignores newly created containers and shared edge/database containers, never runs an unfiltered Docker prune, deletes volumes or databases, stops active containers, or removes tagged active and rollback images. A cleanup failure marks the cleanup receipt as degraded without changing whether the deployment itself succeeded or failed.

Before a state cutover, use the durable maintenance fence:

assembly-line agent quiesce --url https://agent.example.com
assembly-line agent status --url https://agent.example.com
# perform the verified transfer
assembly-line agent resume --url https://agent.example.com

Quiescence stops new ingress, schedules, delivery work, sandbox-sync work, and run recovery, then waits for the reported in-flight run count to reach zero. The state survives process restarts.

database.mode: "external" requires DATABASE_URL and writes a deployment ownership marker before migrations, refusing reuse by another agent. database.mode: "host" runs one private Postgres cluster and creates a separate database/login role per agent. Host mode requires non-secret ASSEMBLY_LINE_VPS_BACKUP_BUCKET and ASSEMBLY_LINE_VPS_BACKUP_REGION configuration plus ASSEMBLY_LINE_VPS_BACKUP_ACCESS_KEY_ID and ASSEMBLY_LINE_VPS_BACKUP_SECRET_ACCESS_KEY credentials. Optional non-secret ASSEMBLY_LINE_VPS_BACKUP_ENDPOINT selects a custom S3-compatible endpoint and ASSEMBLY_LINE_VPS_BACKUP_RETENTION_DAYS defaults to 30. A daily systemd timer creates a compressed dump, verifies the uploaded object, and enforces retention. A weekly timer downloads the newest backup and restores it into a scratch database. A deployment-scoped restore script remains on the VPS; it requires the literal --replace-confirmed argument, takes a fresh backup, and automatically restores the pre-restore database if the requested restore fails. Migration commands connect to the private cluster through a temporary fingerprint-pinned SSH tunnel; the tunnel closes as soon as the migration command finishes.

To move an external database such as Neon into host mode, quiesce the source first and call the VPS plugin's exported transferPostgresToVps() operator API from a reviewed administrative program. The framework CLI has no provider-name state migration branch.

The transfer uses version-matched containerized clients, rejects an older target major, takes an offsite and local pre-transfer backup, verifies the uploaded dump checksum, restores into the isolated role/database, and compares normalized schema plus exact per-table row counts. Any restore or verification failure automatically restores the pre-transfer target. Receipts contain the source environment-variable name and dump hash, never the URL.

Postgres images require explicit numeric tags and default to postgres:17.10-alpine. PostgreSQL 18+ uses the official image's /var/lib/postgresql volume layout; 17 and earlier use /var/lib/postgresql/data. A reviewed administrative program performs a major upgrade with the VPS package's hostPostgresUpgradeScript() workflow; there is no provider-specific assembly-line state command.

The upgrade pulls and verifies the image major, creates logical globals and per-database custom dumps, restores into a new versioned volume, compares exact row counts, and retains the stopped previous container and volume for rollback. Ordinary deploys refuse an image mismatch instead of silently upgrading.

The host monitor runs every five minutes and checks the active container, public /readyz, Postgres, backup/restore-verification units and timers, and free disk space. Failures are recorded by systemd/journald and are also posted to ASSEMBLY_LINE_VPS_ALERT_WEBHOOK_URL when it is configured.

This is trusted-owner process isolation, not hostile multi-tenant isolation. Use separate VMs or microVMs for mutually untrusted tenants. V1 schedules one AMD64 replica per agent; ARM64, high availability, multi-replica scheduling, automatic workload deletion, and non-Hetzner provider bootstrapping are deferred.

Model Provider Authentication

Interactive authentication is a model-plugin capability. The generic command loads the selected provider registration and writes its returned credential to the same durable credential store used by the runtime:

assembly-line auth <provider> agent
assembly-line auth <provider> agent --status --json
assembly-line auth <provider> agent --logout

Local file-backed credentials are encrypted outside the manifest. A Postgres state plugin stores them durably with the rest of the runtime's private state. Root and subagent calls using the same provider reuse that store. Credential values never enter agent.md, plugins.lock, artifacts, receipts, logs, or model context.

For a hosted target, the deploy plugin must advertise remote-exec and implement the generic remote-command contract. Persistent storage is derived from the model plugin's runtime requirements rather than from a provider name. Railway, Docker, Fly, and Hetzner can therefore run the same provider-auth entrypoint without CLI branches for any model vendor.

OpenAI Codex subscriptions

@assemblyline-agents/openai-codex registers Pi's standard openaiCodexProvider(). Pi owns browser and device-code OAuth, refresh, message phases, and ChatGPT subscription semantics. Assembly Line supplies only the generic credential store and durable harness loop. There is no custom Codex app-server harness, bundled Codex CLI, CODEX_HOME, or provider-prefix branch in the Node host.

assembly-line add openai-codex agent --role model
assembly-line auth openai-codex agent
assembly-line run agent --message "hello from my ChatGPT plan"

Remote authentication prefers the plugin-declared device_code method. You may select another declared method explicitly with --method. Usage is labeled subscription; provider-reported token counts are exact, while unavailable cash stays null. For API-key traffic, select the independent openai or openrouter model plugin instead.

The runtime service and its credential store are part of the credential trust boundary. Restrict operator and remote-command access, protect backups, and use assembly-line auth openai-codex agent --logout when retiring a deployment.

Migrations

If the build artifact contains migration files under .assembly-line/migrations, hosted deploys require a migration runner:

assembly-line deploy agent \
  --migration-command ./scripts/run-assembly-line-migrations

The migration process receives:

• ASSEMBLY_LINE_ARTIFACT_ROOT
• ASSEMBLY_LINE_AGENT_REVISION
• ASSEMBLY_LINE_DEPLOY_ENV
• ASSEMBLY_LINE_MIGRATION_FILES

The Postgres adapter records schema migrations with id, checksum, description, package version, and applied time.

Preflight

Use dry-run deploys before publishing:

assembly-line deploy agent --target railway --dry-run

Preflight requirements are inferred from:

• Resolved agent.md infrastructure profiles.
• Selected plugin connections and their bindings.
• Selected plugin tools and their declared required or optional environment.
• Explicit channel providers.
• The model provider prefix in agent.md.
• Artifact deployment requirements such as persistent directories and remote execution.

For a new release, the CLI also runs live installation checks declared by channel plugins. Slack requires app_mentions:read, channels:history, chat:write, files:read, files:write, im:history, im:write, users:read, and users:read.email; assistant:write is optional for Agent Messages and groups:history is optional for private-channel context. Missing required scopes or an invalid available token stop the release. If no local token is available, the check is reported as skipped because the remote secret value is not read. Control-only operations (--activate, --rollback, --ingress-only, and --destroy) are never blocked by this release preflight. Run the same check directly with assembly-line channels check <agentRoot>.

For every non-local target, planning refuses sandbox: local unless ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_IN_PRODUCTION=true explicitly acknowledges the unconfined execution risk. Local state and local blob storage are allowed, but the plan warns that container replacement can lose them.

Static stdio MCP connections launch their configured binary on the Node runtime host and are closed during graceful runtime shutdown. The command is never routed through a shell. Ensure the binary, working directory, and OS permissions exist on every replica. In particular, @assemblyline-agents/peekaboo only operates when the Node host itself is a permitted macOS 15+ machine; deploying that agent to a Linux container does not create remote access back to the developer's Mac. Peekaboo declares local and darwin host requirements, so an incompatible deployment plan is rejected before publishing and a non-macOS runtime rejects the connection before process launch. Remote computer access uses the separate @assemblyline-agents/computer-use connection, Assembly Line Builder's Mac Computer Host, and an end-to-end encrypted relay; it is not a mode of this stdio plugin. The hosted runtime requires ASSEMBLY_LINE_COMPUTER_USE_BINDING; a self-hosted relay can also set ASSEMBLY_LINE_COMPUTER_USE_RELAY_URL. See Remote Computer Use.

Official model-provider authentication:

Other prefixes compile only when an installed and locked third-party model plugin declares them. Its contract, not this table or a core registry, defines the required environment and interactive methods.

LiveKit voice-call tools and connections use LIVEKIT_URL, LIVEKIT_API_KEY, and LIVEKIT_API_SECRET. Outbound phone-call defaults can also use LIVEKIT_OUTBOUND_TRUNK_ID and LIVEKIT_VOICE_AGENT_NAME. defineLiveKitConnection() contributes the required LiveKit env to preflight.

Provider setup details are in Adapters.

Production Checklist

• Use a stable agent.id for agents with learned skills or durable state.
• Use Postgres for hosted durable state.
• Set ASSEMBLY_LINE_CONNECTION_STORE_SECRET or ASSEMBLY_LINE_SECRET if production uses file-backed connection credential stores instead of Postgres-backed stores.
• Use S3-compatible blob storage or R2 for attachments and artifacts.
• Use Docker or a hosted sandbox for untrusted code and shell work. The production Node helper refuses the local sandbox unless ASSEMBLY_LINE_ALLOW_LOCAL_SANDBOX_IN_PRODUCTION=true.
• Shared hosts that accept untrusted agent artifacts should set RuntimeOptions.authoredToolExecution to "sandbox" and use a sandbox image with Node.js 22. The default is direct execution for trusted, latency-sensitive agents.
• Use deploy --dry-run and resolve all required preflight items.
• Keep model provider, channel, database, blob, sandbox, and deploy credentials out of the agent folder.
• For openai-codex/*, authenticate through the generic provider command and protect the resulting model-credential store; no Codex CLI installation or CODEX_HOME is involved.
• Set ASSEMBLY_LINE_ADMIN_TOKEN or provide a host auth policy before production boot.
• Set ASSEMBLY_LINE_ENABLE_API_RUNS=true only when authenticated API-created runs are intended.
• Set ASSEMBLY_LINE_BASH_TOOL_MODE=approval or disabled for agents that should not get direct shell access. The default is enabled in every runtime mode. Embedders can gate any tool by name via RuntimeOptions.coreToolPolicy (e.g. { write: "disabled" }).
• Set TELEGRAM_WEBHOOK_SECRET for Telegram channels and either PHOTON_WEBHOOK_SIGNING_SECRET or PHOTON_INGRESS_TOKEN for Photon channels before production boot.
• Bound run concurrency (ASSEMBLY_LINE_MAX_CONCURRENT_RUNS; createProductionRuntimeOptions defaults to 16 outside dev) and enable ingress rate limiting (ASSEMBLY_LINE_INGRESS_RATE_LIMIT, ASSEMBLY_LINE_RUNS_RATE_LIMIT) on internet-facing hosts.
• Set ASSEMBLY_LINE_TRUST_PROXY=true when the host sits behind a reverse proxy or load balancer so rate limits key on the real client address.
• Point load-balancer readiness at GET /readyz (drains to 503 during shutdown) and liveness at /health; deliver SIGTERM for deploys so in-flight runs drain within ASSEMBLY_LINE_SHUTDOWN_TIMEOUT_MS.
• Verify /health, /readyz, authenticated /manifest, channel routes, and authenticated /runs after deploy.
• Make side-effect tools idempotent and approval-gated where appropriate.