Framework Guide

Assembly Line is a filesystem-first framework for durable AI agents. An agent folder declares what the agent is; the compiler turns that folder into a manifest and runtime artifact; the runtime executes runs durably through pluggable adapters.

This page explains the concepts and contracts. For command tables, the artifact tree, the HTTP API, and deploy targets, see Runtime And Deployment. For every define* shape and ASSEMBLY_LINE_* variable, see the Configuration Reference.

Contents:

• Agent Folder Convention
• agent.ts
• Agent Engine
• context.ts
• gateway.ts
• Tools
• Skills And Self-Improvement
• Channels
• Connections
• Automations
• Sandbox
• Compiler Output
• Durability Guarantees
• Observability
• State And Blob Adapters

Agent Folder Convention

Only instructions.md and agent.ts are required.

agent/
  instructions.md
  agent.ts
  context.ts
  gateway.ts
  skills/
  tools/
  channels/
  automations/
  automation-handlers/
  connections/
  evals/
  sandbox/
  subagents/
  instrumentation.ts

lib/ and playbooks/ are not reserved Assembly Line conventions. App helpers can live wherever the app normally keeps source code.

agent.ts

agent.ts exports static identity/policy and a synchronous setup() that selects runtime capabilities through hooks.

import { defineAgent, useModel, useReasoning, useTool } from "@assemblyline-agents/core";

export default defineAgent({
  description: "A portable Assembly Line agent.",
  maxReasoning: "medium",
  setup() {
    useModel("openai/gpt-5.4-mini");
    useReasoning("medium");
    useTool("echo");
  }
});

The compiler extracts static policy, follows the local import graph, records literal hook declarations and possible models, and packages the source needed to evaluate custom hooks. Tool implementations, secrets, databases, deployment providers, and blob stores do not belong in agent.ts.

Assembly Line reads manifest-facing config through the TypeScript AST. This applies to agent.ts, gateway.ts, tools, channels, automations, automation handlers, connections, sandbox definitions, and subagent agent.ts files. Manifest-affecting values must be statically readable: a default-exported define*({ ... }) helper call, an exported identifier that resolves to that call or object, top-level const indirection for literal strings, arrays, and objects, shorthand properties, imported helper aliases, as const, and satisfies are supported. The compiler never executes config code during manifest generation, so dynamic expressions cannot affect the manifest.

Agent Engine

Pi is the primary engine. Assembly Line adapts the pi model loop in @assemblyline-agents/pi and constructs it directly in the runtime. Agents never select a harness: agent.ts has no harness: slot (declaring one fails validation with harness-not-configurable). The Node host does have a built-in preview provider route: selecting openai-codex/* uses @assemblyline-agents/codex and OpenAI's official Codex app-server, authenticated by the Codex CLI. Subagents also run through Pi. Their hooks select the model and active capabilities, and their static definitions can set workspace and connection limits. They cannot select another engine.

The runtime talks to Pi only through the internal AgentHarness contract in @assemblyline-agents/core. This seam is not a public extension point. The runtime imports no Pi types. Continuation state is opaque JSON owned by the engine adapter, so persisted runs never depend on Pi internals. All durability (tool records, approvals, checkpoints, events, usage, spans) stays in the runtime. Embedders and tests can replace the engine for every run through RuntimeOptions.agentHarness, or register host-owned provider-prefix routes through RuntimeOptions.modelHarnesses. The durability test suite drives scripted engines through exactly this seam. A sibling seam, RuntimeOptions.toolStubs, replaces named tool executions with canned implementations (approval gates and tool-call recording still apply); the eval runner's case-level mocks build on it. The resolved model spec is stored with the continuation so paused and recovered runs resume through the same route. Subagents reuse the Pi loop, including follow-ups via persisted continuations; there is no public subagent-harness extension point.

Tool batches run in parallel by default: when the model emits several tool calls in one response, parallel-safe tools execute concurrently. Pause-capable tools include approval-gated tools, ask_question, connection tools, and the deferred tool_call bridge. They carry execution: "sequential" on their descriptors, which serializes any batch that contains one. A pause stops the batch wherever it happens: later sequential calls are skipped with an explicit { skipped: true } result instead of executing after the run parked, and a dynamic pause inside a parallel batch (a custom tool calling ctx.askQuestion) still ends the active loop execution without another model request.

The engine also reports token deltas (response_delta events). They are ephemeral: the runtime fans them out to live run-stream subscribers (runtime.subscribeRunStream(runId, listener), or the Node host's GET /runs/:id/stream SSE endpoint) and never writes them to the durable event log.

context.ts

Agents do not need context.ts. When it is absent, Assembly Line uses defaultContext().

defaultContext() builds a Flue-shaped context bundle with trusted instructions, active event, bounded recent history, memory filesystem shape, file manifest, current attachments, a compact skill and capability catalog, always-on tool summaries, channel metadata, and trust boundaries. Deferred tool schemas and skill bodies are not injected up front. Files, screenshots, webpages, tool output, search results, memory, and attachments are context, not instructions.

Image and supported video attachments are model-native by default. After attachment intake stores the private bytes, the runtime hydrates bounded PNG, JPEG, GIF, WebP, MP4, MPEG, MOV, and WebM inputs at the harness boundary. Pi sends native image blocks when the selected model advertises image input. For OpenRouter models, Pi verifies the live model metadata and sends complete videos through OpenRouter's native video_url content type only when input_modalities includes video.

Assembly Line never silently substitutes sampled frames for native video. If the selected model or harness does not support video, the turn returns an explicit limitation without making a model request or extracting frames. Frame-based review remains an explicit user-approved FFmpeg operation. The Codex app-server input protocol accepts text and images, but not video, so openai-codex/* follows that explicit limitation path. Image-bearing MCP tool results stay typed rather than being flattened into JSON text, and base64 payloads are omitted from observability content events.

Every runtime model prompt also receives a small stable Assembly Line filesystem contract immediately after instructions.md. It names resource paths and their mutability: /memory is durable memory accessed directly through memory tools by default, /workspace is writable workspace for generated artifacts and modified copies when a sandbox exists, /history is read-only conversation history, and /files is read-only input context whose manifest.json should be inspected before reading file contents. Hosted sandboxes establish /workspace as a physical shell cwd, so core file tools and shell commands may use the same absolute paths. The trusted Local adapter maps those paths onto a temporary host directory; Docker is the local parity path for absolute shell semantics. These paths are projected only when sandbox-backed tools need them; the contract does not inject memory or file contents into every turn.

Prompt layout and prompt caching

The system prompt is cache-stable by construction. It contains only conversation-invariant content: instructions, the filesystem contract, and compact JSON for the skill index, core tool summaries, capabilities, channels, and trust boundaries. This content stays byte-identical across the turns of a conversation.

Per-turn data travels with the turn's user message as an OpenEve turn context: block, followed by the user's text. This data includes the active event's channel context, prompt context, automation target, and attachment metadata. Timestamps never enter the model-visible prompt. This keeps the invariant prefix eligible for provider prompt caching. The cacheReadRatio attribute on each ai.streamText span reports the result.

Conversation transcript resume and compaction

After a successful conversation run, the runtime checkpoints the harness's final continuation as conversation.transcript and stores a pointer on the conversation record. The next turn resumes the full transcript, including assistant turns and tool calls, instead of rebuilding context from flattened recent history. The pointer is an optimization, not the source of truth. A load, harness-version, or trim failure falls back to flattened history and emits context.transcript_fallback. Durable conversation messages and /history remain unchanged.

Before resume the harness trims the stored transcript in two layers. First, tool-result bodies outside the recent ~20k-token tail are capped with a restorable marker (re-run the tool or read the sandbox file to recover the full output; no model call). Second, when the transcript still exceeds the model context window minus a reserve, older full turns are summarized into a structured context checkpoint (pi-agent-core's summarizer), keeping the recent tail verbatim and never separating a tool result from its call; re-compactions update the previous summary instead of stacking summaries, and the run is notified to persist durable facts under /memory. Compactions are evented as context.transcript_compacted with the pre-compaction token estimate. Stored media (base64 images/video) never replays on conversation resumes. Trimming runs at resume time between runs; mid-run growth is bounded by maxIterations.

Custom context can extend the default:

import { defaultContext, defineContext } from "@assemblyline-agents/core";

export const customContext = defineContext({
  kind: "custom",
  name: "customContext",
  extends: defaultContext({ recentHistory: { maxMessages: 5 } })
});

Context policy is trusted app-runtime code and is recorded in the manifest with source attribution.

gateway.ts

gateway.ts is the portable stack declaration. It is tiny and declarative:

import { adapter, defineGateway } from "@assemblyline-agents/core";

export default defineGateway({
  deploy: adapter("railway"),
  runtime: adapter("node"),
  state: adapter("postgres"),
  blob: adapter("r2"),
  sandbox: adapter("daytona"),
  scheduler: adapter("gateway")
});

Runtime host, state, blobs, sandbox, scheduler, connections, and observability are independent choices. Deploy adapters host the runtime process; they do not force a state/blob/sandbox provider.

Like agent.ts, gateway.ts is parsed as TypeScript syntax rather than executed. Use statically readable defineGateway({ ... }) declarations, adapter("kind"), or known provider helper calls such as railwayDeploy(), vpsDeploy(), neonPostgres(), railwayPostgres(), supabasePostgres(), r2Blob(), and dockerSandbox(). Dynamic expressions are ignored unless they resolve to top-level literals the compiler can validate.

Scheduler choices are explicit. adapter("local") starts an in-process polling loop for development or single-process hosts. adapter("gateway") does not start a loop; a cloud scheduler, platform cron, or gateway worker calls the runtime tick endpoint or runDueAutomations(). adapter("postgres") starts the same polling loop but expects Postgres state so multiple workers coordinate through shared idempotency and dynamic-automation leases.

Production state is Postgres. Neon is the default hosted path. Railway, Supabase, and local or custom Postgres are presets; each uses adapter("postgres") because it exposes standard Postgres. On Railway, railwayPostgres() provisions or reuses a managed database and wires its private DATABASE_URL before publishing the runtime. Blob storage is S3-compatible, with R2 as a first-class preset and wrapper.

Railway and generic VPS deploy providers are supported. Docker and Fly deploy providers are preview. All four implement the same artifact-level persistent storage and remote execution contract. Deploy choice does not imply a state, blob, or sandbox vendor. See Deploy Targets.

See Adapters for the current adapter list, helper functions, and provider environment variables.

Tools

Each file in tools/ becomes one model-facing tool. The filename is the tool name. Tool descriptors include name, description, input schema, optional output schema, approval policy, and model-output projection.

Tool code runs in the trusted app runtime by default. It only uses the sandbox through ctx.getSandbox().

agent.ts selects the exact visible tool set for each capability snapshot with useTool(). Core tools such as read, write, edit, delete, list, grep, bash, ask_question, load_skill, tool_search, tool_describe, and tool_call are available to select, but none becomes visible merely because it was compiled. useSubagent("researcher") adds only delegate_researcher; there is no generic subagent-spawn escape hatch. Host tool policy remains the final ceiling and can remove a selected tool before the snapshot applies.

Every core harness tool is a replaceable slot: an authored tools/<name>.ts with a built-in's name overrides it (spread builtInToolDefaults from @assemblyline-agents/runtime to wrap instead of rewrite), and a disableTool() default export removes it, with unknown names failing the build. See Customizing Agents.

Tools can optionally declare capability metadata, but most apps should rely on inference:

capability: {
  visibility: "deferred",
  execution: "direct",
  namespace: "billing",
  tags: ["invoice", "customer"],
  aliases: ["receivables"]
}

visibility can be auto, always, deferred, skill, or hidden; execution can be auto, direct, sandbox, or both. auto defers to inference. These fields feed deferred discovery metadata. Core harness tools are not configurable by agent authors.

All file, memory, skill, workspace, and artifact work goes through the sandbox filesystem. /history and /files are read-only; /memory, /skills, and /workspace are writable according to policy. The core file tools lazily acquire and hydrate the sandbox for requested paths.

import { approvalRequired, defineTool } from "@assemblyline-agents/core";

export default defineTool({
  description: "Record a note after approval.",
  inputSchema: { type: "object", properties: { note: { type: "string" } }, required: ["note"] },
  needsApproval: approvalRequired("Recording a note is a durable side effect."),
  async execute(input, ctx) {
    await ctx.emit("note.recorded", { idempotencyKey: ctx.idempotencyKey("note") });
    return { recorded: true, note: input.note };
  }
});

toModelOutput can expose a bounded, safe projection while the rich result remains available in the durable tool log. See tools/*.ts for the full field reference.

Skills And Self-Improvement

Skills live under skills/<name>/SKILL.md. Assembly Line parses skill frontmatter and builds compact skill capabilities in the manifest. useSkill() selects the skills present in the current snapshot. If load_skill is also selected with useTool(), the model may load the body of those selected skills only; loading never widens the skill or tool set. If a sandbox already exists, a selected skill may also be projected into /skills/<name>/SKILL.md for filesystem inspection.

Self-improvement means the agent authors and edits its own skills. Configure it with selfImprovement in agent.ts; it is on by default. Runtime automations and connections use separate config blocks (dynamicAutomations and dynamicConnections) and separate tool APIs. They are not part of self-improvement. Turn off all three blocks for static, manifest-only behavior.

export default defineAgent({
  // Stable logical identity for durable learned state across revisions/deploys:
  id: "travel-concierge",
  // The agent learning by writing/editing skills:
  selfImprovement: { writable: true, writeApproval: false },
  // Separate, clearly-named concerns:
  dynamicAutomations: { dynamic: true, approval: false },
  dynamicConnections: { dynamic: false, approval: true, allowedHosts: [] },
  setup() { useModel("openai/gpt-5.4-mini"); }
});

Skills are a durable folder (the self-improvement surface). On boot, compiled skills/ seed a durable SkillStore, scoped by stable agent.id when configured and tracked by a content hash so redeploys upgrade pristine seeded skills and preserve anything the agent or user changed. Each run puts only the skill index in the prompt: name, description, and path. Full bodies live at /skills/<name>/SKILL.md and are read, edited, created, or deleted with the normal file tools. When selfImprovement.writable is off, /skills writeback is blocked.

Tools reach these concerns through three distinct context APIs, ctx.selfImprovement (skills), ctx.automationManager, and ctx.connectionManager, and the runtime exposes saveSkill, listSkills, dispatchAutomationEvent, runDueSchedules, saveConnectionDefinition, and related methods for host-driven use. Durable stores are provided by the state adapter (Postgres) or fall back to local JSON files under the artifact root. See examples/self-improving-agent and Customizing Agents.

Channels

Channels normalize platform entrypoints and delivery behavior. HTTP-capable channels declare a route and methods; the compiler emits a route table.

import { defineChannel } from "@assemblyline-agents/core";

export default defineChannel({
  transport: "http",
  route: "/message",
  methods: ["POST"]
});

The default raw HTTP message fallback is dev-only unless the Node host has authenticated the request. Production provider routes should use a helper or custom normalizeHttp() that verifies the provider request before accepting a turn.

Channel files own platform event semantics, not durable state schema or sandbox lifecycle.

Assembly Line ships one-line helpers for Slack, Discord, Telegram, Microsoft Teams, and Photon-style agent communication channels. Provider helpers preserve the same channel contract: verify the incoming event, normalize to ChannelTurn, use provider delivery IDs for idempotency where available, and send replies through provider APIs. GitHub is available separately as an authenticated connection package for repository tooling; it is not an inbound channel.

For retried webhook providers, channel modules can return kind: "accepted" to acknowledge the HTTP request before the model turn completes. Use the provider's stable delivery id as the idempotency key. For Slack Events API channels, verify the request signature, normalize the event, return a 2xx response immediately, and set idempotencyKey to Slack's event_id so retries do not start duplicate turns.

Accepted turns enter a durable FIFO mailbox keyed by stable agent identity and normalized conversation id. Only one turn in a conversation can be running or parked at a time; later messages wait. Different conversations lease independently and consume the ordinary global run-concurrency budget in parallel. A parked approval, input, connection, or suspended run keeps its conversation closed until it reaches a terminal state. This rule is enforced by the runtime after normalization, so channel modules define conversation boundaries but do not implement their own queues.

Channel modules can also augment context after ACK and before default context bundle construction. Slack uses this hook to bridge a bounded set of recent Assembly Line Slack conversations for the same user, so a user can follow up from a thread in a DM without merging every Slack surface into one transcript or fetching workspace-wide Slack context.

Channel modules can also export startIngress(ctx, emit) for long-lived provider listeners. The Node host starts these listeners beside the scheduler, restarts provider-owned listeners through adapter code, and stops them on server shutdown. emit.accepted({ turn, idempotencyKey, idempotencyScope }) feeds Gateway-style events into the same durable, idempotent run path used by accepted HTTP webhooks. Discord uses this for Gateway DMs, mentions, and thread messages.

Connections

Connections declare required external capabilities, scopes, subject mapping, and whether they are required. Raw secrets and refresh tokens stay outside the agent folder and model context. Live MCP, OpenAPI, HTTP, and sandbox CLI connection tools are searched through tool_search; concrete schemas are not visible until tool_describe selects them. MCP supports request-policy-governed Streamable HTTP and static, directly spawned stdio processes. Sandbox CLI connections preserve the same connection policy and scoping while invoking reviewed arguments in the active run sandbox, where short-lived materialized credentials may be projected when explicitly configured. Dynamic connections are URL-only and cannot launch host or sandbox processes. Tools and channels consume connection handles from runtime context.

Dynamic connections are separate and gated. When dynamicConnections.dynamic is on (it is off by default), a tool can use ctx.connectionManager to persist an MCP, HTTP, or OpenAPI connection in a ConnectionDefinitionStore. Stored definitions join the connection registry and become discoverable through tool_search. Credentials flow through host APIs or authorization into the encrypted grant store. They never enter model-visible tool input, the agent folder, the sandbox, or model context. Saving requires approval and a host in dynamicConnections.allowedHosts. Remote tool descriptions remain untrusted data. Agents cannot author trusted tool code. Run a user-supplied CLI in the sandbox and wrap it in a skill; new typed tools remain reviewed source changes.

Automations

Automations declare durable work started by either a schedule trigger or a normalized provider event. Schedule triggers use the existing cron dispatcher. Event triggers enter through runtime.dispatchAutomationEvent(), a verified channel normalizer, a long-lived channel listener, or authenticated POST /assembly-line/automations/events. Both paths reserve stable idempotency keys, honor run capacity, and execute the same target and lifecycle contract. Time-based triggers are dispatched by runtime.runDueAutomations() and /assembly-line/automations/tick.

Trusted prepare/finalize code lives in automation-handlers/ and is referenced through lifecycle.handler. Dynamic time-based automations use dynamicAutomations and ctx.automationManager; dynamic event subscriptions remain reviewed source because they own provider authentication and subscription policy. See automations/.

Sandbox

Sandbox files declare the agent computer selection. The core contract supports file reads/writes, shell execution, and optional provider snapshots. The local adapter is for trusted dev/test work; Docker is the supported local isolation baseline; Daytona, E2B, and Modal are supported hosted sandbox choices. Sandboxes are acquired lazily when a sandbox-backed tool or capability asks for one.

Hosted sandbox paths share one physical, versioned namespace rooted at /workspace. Providers validate shell cwd and file-API agreement after create, connect, and wake; runtime manifests fence older contract versions from reuse. Providers reject traversal and return canonical absolute paths such as /workspace/report.txt from listings. /runtime is retired and rejected. Local sandbox execution emulates the logical namespace in a temporary host directory and is trusted developer or self-managed execution only; use Docker, Daytona, or E2B when untrusted code needs an isolation boundary.

Snapshots are a scarce infrastructure checkpoint, not the normal turn persistence mechanism. Production runs should use Assembly Line state/blob sync for memory, messages, tool traces, and /workspace artifacts, and keep fresh run sandboxes ephemeral. The default snapshot policy is never, so a hosted sandbox run does not create a remote snapshot unless the agent explicitly opts in.

Sandbox files may declare a snapshot policy:

export default defineSandbox({
  adapter: "daytona",
  image: "node:22",
  snapshot: {
    mode: "manual",
    retainLast: 3,
    reason: "operator-requested checkpoint"
  }
});

Supported modes are never, manual, on_failure, and always. manual only captures when run metadata includes an explicit sandbox snapshot request. always is intended for short-lived debugging or controlled checkpoint jobs, not chat turns. ASSEMBLY_LINE_SANDBOX_SNAPSHOT_MODE, ASSEMBLY_LINE_SANDBOX_SNAPSHOT_RETAIN_LAST, and ASSEMBLY_LINE_SANDBOX_SNAPSHOT_REASON can override policy at runtime.

Compiler Output

assembly-line build emits the .assembly-line/ artifact; the canonical file tree and per-file descriptions live in Runtime And Deployment → Build Artifact. Two contracts matter conceptually:

• agentRevision is a deterministic hash of source paths, source hashes, and relevant config. Rebuilding unchanged source produces the same revision; changing source changes it.
• The manifest contains instructions, the agent definition, context policy, gateway config, tools, capabilities, skills, channels, automations, automation handlers, connections, sandbox declarations, subagents, instrumentation source, the route table, preflight requirements, validation results, package versions, and source hashes.

For the CLI commands that produce and run the artifact, see the CLI reference.

Durability Guarantees

Persistence model

The runtime persists a run before execution starts. It records context creation, sandbox acquisition, model and tool events, approval pauses, memory sync, delivery, and the final status. Replay rebuilds the run from durable events, recovery checkpoints, tool calls, and delivery records.

Checkpoints support recovery; they are not the permanent audit log:

• Active runs keep a bounded tail of harness.continuation checkpoints while preserving pause and durable-step data needed by the live run.
• Events, messages, tool calls, delivery obligations, idempotency keys, files, memory, schedules, and approvals remain separate durable records.
• Large checkpoints are gzip-compressed into blob storage. SQL keeps a small content-addressed record with the pointer, hash, and size. Resume hydrates the payload without exposing this split to the harness.
• assembly-line checkpoints compact <agentRoot> reports terminal-run cleanup by default. Add --apply to delete rows older than the configured TTLs.

See the model loop reference for checkpoint cadence, retention, and TTL settings.

HITL resume

Approvals, ask_question, and suspension states are durable. Resume re-enters the model loop:

• resumeApproval(runId) executes the approved tool and returns its result to the harness as the pending tool result.
• resumeInput(runId, answer) returns the human answer to the agent that asked the question.
• resumeSuspended(runId) restores the latest compatible continuation.

Each path claims a per-generation idempotency key first. Repeated approvals, answers, OAuth callbacks, or suspension resumes cannot execute twice across replicas. If no usable continuation exists, the runtime completes the run directly and emits harness.resume_degraded. Final delivery remains an idempotent delivery obligation.

Durable steps

Authored tools and automation handlers can wrap expensive or side-effect-adjacent substeps in ctx.step(key, fn). A completed step stores its JSON result as a durable_step.completed checkpoint. If the same run reaches the key again, the runtime emits durable_step.replayed and returns the stored result instead of running fn. Steps are scoped to one run and use the existing checkpoint store. If a process dies inside fn before the result is stored, the body may run again. External writes still need ctx.idempotencyKey(...) or a destination-level deduplication key.

Delivery queue

A delivery reports success only after a real channel sender runs. The two documented no-sender results are dev-no-sender in development and local-no-sender for a local channel with no required environment. If a channel module fails to load, delivery fails as retryable and the runtime logs the error.

Final delivery uses a durable queue:

1. The completion path creates the delivery obligation in sending with a lease token held by the inline sender. Another worker cannot send the same obligation concurrently.
2. The inline sender retries transient failures first. The default is two retries (ASSEMBLY_LINE_DELIVERY_RETRY_ATTEMPTS).
3. A remaining retryable failure returns the obligation to pending with exponential backoff and emits delivery.deferred. The run completes with deliveryDeferred metadata.
4. runDueDeliveries() or startDeliveryWorker() recovers expired leases and leases due work. Postgres uses for update skip locked for safe parallel workers.
5. The worker sends the persisted payload and records delivery.sent, delivery.retrying, or terminal delivery.failed. Non-retryable errors and exhausted attempts fail immediately.

See the delivery queue reference for lease, batch, attempt, and interval settings.

Orphan recovery

Every executing run updates updatedAt through the guarded touchRun method. The default heartbeat is 30 seconds (ASSEMBLY_LINE_RUN_HEARTBEAT_MS). This write never changes status or revives a terminal run.

recoverIncompleteRuns() considers only runs left in created or running past max(5 minutes, 4 x heartbeat). staleAfterMs can override that window. The runtime claims each candidate with a run:recovery idempotency key, then applies this policy in order:

1. Complete a run that already has delivery.sent, without sending again.
2. Cancel tool calls requested but not started.
3. Mark a tool interrupted after tool.execution_started when no settle event exists. A parked continuation receives { interrupted: true } instead of silently executing the tool again.
4. Give a run with a harness.continuation checkpoint one in-place resume attempt and emit run.recovery_resume_attempted.
5. For a run with a model response but no delivery, adopt the existing delivery obligation or create one with the original final-delivery key.
6. Mark any other candidate failed and emit run.failed.

The sweep runs at host boot and through startBackgroundWorkers(). Its default interval is 60 seconds (ASSEMBLY_LINE_RUN_RECOVERY_INTERVAL_MS).

Resume restarts from a checkpoint; it does not replay events deterministically. Tool execution is therefore at least once. A crash after an external side effect but before the next continuation may lead the model to request the tool again. Per-iteration checkpoints, interrupted-tool guards, and completed ctx.step(...) results reduce this window but cannot close it for arbitrary external writes. Use ctx.idempotencyKey(...) or a destination-level key for non-idempotent writes. A crashed in-flight model request restarts from the last continuation, which may add token cost but does not lose durable state.

runtime.startBackgroundWorkers() starts the delivery, sandbox-sync, conversation-turn mailbox, and orphan-recovery workers, and returns a controller with stop(). The Node host (listenNodeRuntime) wires all of this automatically and each worker has an env kill-switch; see the durability workers reference.

Sandbox sync

Assembly Line does not acquire a sandbox before every turn. Channel lifecycle events and final text delivery run without sandbox hydration. The first core file or shell tool lazily acquires the sandbox, then hydrates only requested paths or bounded candidate sets:

/memory
/history
/files
/workspace

The local adapter materializes these paths under a temporary sandbox root. /memory, /skills, and /workspace are writable; /history and /files become read-only after hydration. The runtime hydrates indexes, bounded history, file manifests, runtime context, and selected resources instead of every skill or memory document. Write generated artifacts and modified copies under /workspace. Use Docker when a development test needs a physical absolute /workspace shell path.

Sandbox acquisition follows the same order for every provider:

1. Connect to the current live sandbox for the agent, conversation, and project.
2. Wake the warm sandbox recorded in Postgres.
3. Create a new sandbox.

The runtime skips reconnect, provider lookup, and snapshot restore when the recorded filesystem contract is obsolete. Versioned provider names prevent a replacement from colliding with the old resource. Before a mutating side effect, the foreground path creates a durable session row and sync obligation. Delivery and the next message do not wait for filesystem scanning or writeback.

Sandbox sync is a durable queue. A mutating sandbox tool enqueues a sandbox_sync_job before the side effect. The worker:

• leases jobs, retries with backoff, and recovers expired leases;
• writes /memory/** into durable memory documents;
• writes or deletes /skills/*/SKILL.md through the durable skill store;
• stores /workspace/** blobs and file catalog records; and
• records blocked failures as blocked_requires_operator without undoing final delivery.

The runtime retains or pauses dirty sandboxes until writeback completes. Operators can use sandboxSyncDiagnostics(), inspectSandboxSyncJob(jobId), and retrySandboxSyncJob(jobId) to inspect and retry jobs. See the sandbox sync reference for inline mode, lease, batch, and attempt settings.

Security boundaries

Memory, history, files, webpages, search results, and tool output are untrusted context, not instructions. /files and /history are read-only projections; write generated or transformed outputs under /workspace. Skills are trusted instructions and load one selected skill at a time. A sandbox receives an allowlisted copy of selected resources, never ambient host filesystem access. See Architecture for the full trust-boundary map.

Observability

Agent-authored useEvent() observers run after the matching event persists; failures emit agent.event_handler_failed and are isolated from the run. Every setup evaluation is stored as a complete capability checkpoint and summarized by run.capabilities_resolved before it applies. See Customizing Agents.

Run, event, tool, checkpoint, delivery, usage, subagent, and grouped-run query contracts work without instrumentation.ts. The Node host exposes GET /runs, GET /runs/:id, GET /runs/:id/events, GET /runs/:id/timeline, and arbitrary-period GET /usage. Dashboards and CLI tools can inspect persisted runs without replaying provider calls.

Usage accounting stores provider-reported or reconciled cash only. Unavailable values remain null, and aggregate control totals are compared with transactions instead of added to them. Observation failures produce warnings but do not block model execution or delivery.

Every terminal outcome produces one structured log entry. The runtime logs run.completed and run.cancelled at info, and run.failed at warn with its machine-readable reason. Failed run records also carry terminalReason and terminalError, so operators can list failures without scanning events. Model retries emit model.request_retried; errors that escape a run emit the non-terminal run.execution_error for crash recovery. Logs contain response sizes, not response content. Content capture is a separate opt-in telemetry setting.

Optional OpenTelemetry-shaped sinks receive a parent-child span hierarchy. Each completed turn emits ai.assembly-line.turn as the parent span, with children for model steps (ai.streamText), tool calls (ai.toolCall), subagents, sandbox commands, memory sync, and delivery sends. The runtime also emits assembly-line.run and assembly-line.tool for compatibility. Spans carry trace and span IDs plus agent revision, run, session or conversation, turn, channel, model, tool, sandbox, delivery, status, usage, cost, and error attributes where available.

Configure telemetry in agent/instrumentation.ts. The runtime discovers this file and runs it once at startup. @assemblyline-agents/otlp provides createOtlpSink and createOtlpSinkFromEnv for OTLP/HTTP export to Langfuse, Phoenix, Grafana, Honeycomb, or another OTLP backend. See Customizing Agents: Observability.

When instrumentation.ts exports defineInstrumentation({ setup }), the runtime calls setup({ agentName, manifest, env }) before the first turn. recordInputs, recordOutputs, captureContent, and functionId control capture and export. The default usage level records tokens, cost, and model metadata without message bodies. A sink returned by setup() is used unless the host supplied one directly. An exported telemetry value remains a compatibility fallback.

State And Blob Adapters

The state contract stores runs, run events, checkpoints, tool calls, delivery obligations, replay data, FIFO conversation turns, runtime settings, and conversation-scoped agent state. The Postgres package provides migrations plus a driver-neutral query(sql, params) adapter. The Node production host wires that adapter to DATABASE_URL through pg; Neon, Railway, Supabase, local Postgres, and custom Postgres use the same schema. The schema covers agents, revisions, conversations, messages, conversation-turn mailboxes, runs, run events, capability snapshots, hook state, checkpoints, tool calls, approval gates, delivery obligations, schedules, memory, file records, sandbox leases, usage receipts and aggregates, runtime controls, and idempotency keys.

Postgres memory search uses indexed full-text search for keyword/exact/hybrid modes. Semantic search is optional and needs a MemoryEmbeddingProvider: embedding-backed search turns on automatically when a provider is configured only when the state adapter reports that its vector backend is available, and ASSEMBLY_LINE_MEMORY_EMBEDDINGS_ENABLED can disable the feature. Postgres enables that capability only when optionalMigrations includes 003_openeve_memory_embeddings_pgvector (or is true); the database must provide pgvector. Run runMemoryEmbeddingBackfill() after enabling it to populate stale or missing embeddings. Deployments without the optional backend keep deterministic full-text and portable lexical fallback behavior without querying pgvector tables.

Postgres migrations are recorded in openeve_schema_migrations with id, checksum, description, package version, and applied time. PostgresStateAdapter.migrate() is idempotent and rejects checksum drift; planMigrations() reports pending/applied/skipped-optional/checksum-mismatch state without applying SQL. Existing memory file indexes can be promoted into state-backed memory documents with backfillMemoryDocumentsFromFileIndexes() when the blob adapter can read the indexed blob keys.

The blob contract stores context bundles, attachments, extracted text, generated artifacts, and sandbox sync bundles. The S3 package implements the blob contract against S3-compatible storage and ships R2/AWS/MinIO-style helpers. The R2 package is a compatibility wrapper and in-memory test bucket.

Related Docs

• Configuration Reference: every define* shape and ASSEMBLY_LINE_* variable.
• Runtime And Deployment: CLI, artifact tree, HTTP API, lifecycle, deploy targets.
• Architecture: system diagram, trust boundaries, and package boundaries.
• Adapters: provider matrix and per-adapter environment.
• Plugins: the extension model and plugin catalog.