Engines & backends

Some Xe apps are just files — HTML, JS, CSS — served over HTTP. Others carry a backend: one or more containers that must actually run somewhere. This chapter is about that “somewhere” (the engine) and about how a running backend is reached, started, gated, and updated without ever handing untrusted code a port on your machine.

The headline architecture is d/094’s third path: the launcher owns a single connector that consumes a unix socket published by the engine itself — no per-app sidecar container, no loopback TCP publication of backend ports. That one sentence drives most of this chapter.

Citations are code at xenon-launcher main @56e94575 unless noted; decisions cite d/NNN in the ledger. For what an app is and how it installs, read app model & packaging first.

Primer: the words this chapter assumes

• Engine — the thing that actually runs containers. On the Xe Computer the productized default is a pinned Colima/Lima/Docker stack; an experimental alternative is smolvm.
• microVM — a lightweight virtual machine that boots in well under a second and isolates a workload far more strongly than a bare container. Both Colima and smolvm run Docker inside a microVM, so an app’s containers never touch the host kernel directly.
• smolvm — an upstream open-source microVM engine (smol-machines/smolvm) Xe forks and extends. Its role here is to publish a unix socket the launcher’s connector can consume.
• docker compose — a compose.yaml file describing a multi-container service. See the compose spec.
• unix socket — a file on disk that programs use to talk to each other locally. Unlike a TCP port, it is not reachable over the network and its access is governed by ordinary file permissions — which is exactly why d/094 chose it.
• docker exec — running a command inside an already-running container. Xe uses it to reach a container’s port from the outside without publishing that port anywhere.
• route worker (workerd) — a small sandboxed JS worker (workerd) that implements the serving plane’s HTTP routing. It decides which app a request belongs to and whether to proxy it to a backend. See the serving plane.
• consent gate — a point where a human must approve something before the system proceeds. Xe’s are cryptographically bound to exactly what was approved.

Zoom 1 — the shape of the whole thing

An app with a backend runs like this, top to bottom:

1. The browser (Prism) makes an ordinary HTTP request to the app’s plane origin, http://<app>.localhost:5454.
2. The serving plane’s route worker recognizes a backend route and, rather than serving a file, forwards the request to a launcher-owned connector.
3. The connector reaches the app’s container through the engine’s unix socket, using docker exec to touch the container’s port — never a published port, never a sidecar.
4. If the backend is not running yet, the launcher’s lifecycle engine starts it on the first request, holding replayable requests briefly and fast-failing the rest with honest HTTP status.

Two design commitments make this trustworthy, both from d/094:

• The launcher owns every boundary. The engine socket, the connector, the route worker’s proxy arm — all launcher-written, launcher-held. Untrusted app code never gets a host port and never holds the socket.
• No sidecar, no loopback publication. The P-141 connector question was resolved against both escalation options (a per-app sidecar container, or publishing backend ports on loopback TCP). The chosen “third path” lives at the engine layer, where the isolation already is.

The rest of this chapter zooms into each of those pieces.

Zoom 2 — subsystems

The engine socket contract (d/094)

This is the load-bearing mechanism, and it is live at HEAD (shipped in P-203). Four rules define it:

Walking it through as a sequence — from a browser request to bytes flowing to the container:

sequenceDiagram
    participant Browser as Browser (Prism)
    participant Worker as Route worker (workerd)
    participant Conn as BackendConnectorService 127.0.0.1:5199
    participant Bridge as ExecBridge 127.0.0.1:0
    participant Engine as Engine (smolvm/colima)
    participant Ctr as App container

    Browser->>Worker: GET http://app.localhost:5454/api/…
    Note over Worker: recognizes BackendProxy route;<br/>grants never cover the backend prefix (403)
    Worker->>Conn: fetch external "backend" service
    Conn->>Bridge: connect (launcher-resolved svc+port)
    Bridge->>Engine: docker compose exec -T svc  nc 127.0.0.1 port
    Engine->>Ctr: pipe bytes to container port
    Ctr-->>Browser: response (proxied back up the chain)

Two details that matter for trust:

• The connector trusts no request input. ComposeBackendController resolves the project, route, and service entirely from launcher-written registry state — never from the HTTP request (backend.rs:416). The x-xe-app header is validated to [a-z0-9-]{1,63} (backend.rs:1091), and the route worker’s grants never cover the backend prefix (a 403, assets/route-worker.js:304) — so a cross-origin grant can never be used to reach a backend.
• The socket path is refused if ambiguous. SmolvmBackend refuses zero or more-than-one sockets (smolvm.rs:276), so no other engine can be silently substituted for the one the launcher expects.

The lifecycle engine: lazy start, honest failure

Backends are started on first request, not eagerly. The LifecycleEngine (crates/launcher/src/backend.rs:188) makes that safe and honest:

stateDiagram-v2
    [*] --> NotStarted
    NotStarted --> Starting: first request claims the driver (singleflight)
    Starting --> Ready: probe healthy
    Starting --> Failed: total_start_deadline (120s) OR start error
    Ready --> NotStarted: invalidate() on transport failure
    Failed --> NotStarted: stop() clears the slot
    Ready --> NotStarted: stop()
    note right of Starting
      Wait class held up to per_request_wait (15s) then 503 Retry-After
      Immediate class 503 at once (never held then lost)
      waiters capped at 256
    end note
    note right of Failed
      distinct 502 - a failed start is not a slow one
    end note

The design choices, plainly:

• Singleflight. The first request wins a compare-exchange and spawns the one start-and-poll task per cold-start episode (backend.rs:236). A test fires 200 concurrent first requests and asserts exactly one start (backend.rs:1573).
• Wait class. A HEAD request, or a manifest-declared idempotent-GET prefix (matched on segment boundaries), is safe to hold while the backend warms → Wait. Everything else is Immediate fail-fast, so a non-replayable request (a POST) is never held and then lost (classify, backend.rs:96).
• Two deadlines, two truths. per_request_wait (15 s) → an honest, retryable 503 with a jittered Retry-After. total_start_deadline (120 s) → a 502 Failed — because a backend that failed to start is a different fact from one that is merely slow, and the status code should say so. The 502 names xe doctor as the next step (backend.rs:1110).
• Self-healing edges. invalidate() drops a stale Ready on transport failure; stop() clears a failed slot for a clean retry.

Compose apps: the safe subset

Most backends today are docker compose apps. The compose adapter does not run arbitrary compose files: it enforces a safe subset and fails the install closed if a compose.yaml steps outside it.

The full refusal list is normative and lives in the spec candidate, but the shape is: anything that would puncture the microVM isolation boundary is refused — privileged, raw devices:, host networking, namespace joins, bind mounts of host paths, mounting the docker socket, publishing host ports (use expose), unpinned images, and more. Each refusal yields a stable diagnostic code with a source:line:col pointer (compatibility_diagnostics, compose.rs:907). Config interpolation is likewise constrained: only keys the app declared in its config_schema may interpolate, and shell-style default operators are forbidden (interpolate_declared, compose.rs:856).

A known-holes caveat. The refusal list has three documented gaps tracked by P-174 (open): short-form host ports (ports: ["8080"]), non-UTF-8 corruption in the interpolation path, and network_mode: container:. These matter because P-174 gates running compose on user-supplied manifests — the review treats it as a sequencing question before third-party apps.

Git-sourced compose is handled inside this same adapter (GitComposeSource + stage_source_tree), so a compose app can be fetched from a pinned git commit rather than a registry.

Consent gates: two gates, cryptographically bound

Installing or updating an app that runs code requires human consent, and Xe’s consent design is deliberately strict. It is shared between the install pipeline and backends, so it lives here.

There are two gates, and they are not interchangeable:

• Gate 1 — Plan Review. Subject is sha256(plan). Approving it mints a BuildGrant that is unforgeable by construction: BuildGrant::from_consent requires a gate-1 consent whose subject_digest == sha256(plan) (adapter.rs:93), and every adapter re-checks it at materialize (compose.rs:249, static_v0.rs:155, swbn.rs:145).
• Gate 2 — Grant Consent. Subject is the exact locked release digest. Approving the plan does not approve the built result; the second gate binds to the bytes that were actually produced.

sequenceDiagram
    actor User
    participant CLI as xe app install
    participant Adapter
    participant Queue as Consent queue (pending/claimed/decided)
    participant Surface as Launcher surface (xe consent / tray)
    participant Origin as origin::generate

    CLI->>Adapter: probe → resolve → analyze
    Adapter-->>CLI: Plan (unsupported_features must be empty)
    CLI->>Queue: file PlanReview request (inert; subject = sha256(plan))
    Surface->>Queue: claim (atomic rename) + decide
    Surface-->>CLI: gate-1 consent → BuildGrant
    CLI->>Adapter: materialize (under BuildGrant) → compile → locked release
    CLI->>Queue: file GrantConsent request (subject = locked digest)
    Surface->>Queue: claim + decide
    Surface-->>CLI: gate-2 consent
    CLI->>CLI: transition Staging → Installed (signed audit)
    CLI->>Origin: generate RouteTable at app.localhost:5454

The custody model is the trustworthy part (crates/launcher/src/consent.rs):

• A consent request is an inert JSON record in pending/<id>.json. It mints no authority by merely existing (ConsentAction::AppInstallGate “mints no runtime authority”, consent.rs:75).
• Deciding it means claiming it by an atomic rename to claimed/ (a single executor), then decided/. There is a bound: MAX_PENDING=8, REQUEST_TTL_SECS=15 min.
• The Console / web bridge has no approval endpoint, by construction. Only a launcher-owned surface — xe consent or the macOS tray — can mint authority (consent.rs:1-28). A web app, however privileged it renders, cannot approve its own install.
• The runtime is the final choke point. ManagedCompose::authorized refuses to construct a runnable project unless gate-2 consent matches the locked release digest and the state is Staging/Installed (runtime.rs:138). Private fields make the ungated path unreachable to callers — you cannot forget to check.

This install consent is distinct from the cross-origin Grant table (grants.rs, d/086) used for one app to reach another — the review keeps them from being conflated.

Zoom 3 — module detail

smolvm’s actual scope, plainly

The docs and the code disagree slightly, and the honest version matters:

• The default engine is not smolvm. docs/smolvm-engine.md:1 states “the smolvm substrate is an explicit macOS experiment.” The productized default is AppEngineBackend — Colima + Lima + Docker, pinned and verified, macOS (crates/substrate/src/backends/app_engine.rs) — which downloads and verifies a guest image and never touches host engine installs.
• The code has moved past the doc. SmolvmBackend now carries a Linux x86_64 path (smolvm.rs:66, pin 1.6.1-agent54-engine-socket) that publishes the unix socket the d/094 connector consumes. So the honest statement is: smolvm remains an opt-in experiment (mac-first per the doc), and is now also the Linux engine that publishes the d/094 socket. It verifies a pinned launcher + binary sha256 before exec (tamper-refused, smolvm.rs:301) and brand-sanitizes its output to “app engine.”

The d/094 upstream story: the socket capability builds on smolvm upstream PR #656 (socket relay, merged upstream), extended on Xe’s fork (agent54/smolvm on the forge) to publish a guest TCP port as a host unix socket. Submitting that upstream is a separate human-gated step (d/013, d/094).

The exec bridge, precisely

The connector’s ExecBridge (backend.rs:521) binds 127.0.0.1:0 (an ephemeral loopback port the launcher owns, not the app) and pipes each connection through docker compose exec -T <service> sh -c 'nc 127.0.0.1 <port>' (compose_exec_command, backend.rs:827). On Linux the engine is reached via DOCKER_HOST=unix://<data-root>/engine/smolvm/**/docker.sock; on macOS/colima via tcp://127.0.0.1:2375 or docker --context colima (backend.rs:854-891). The connector service itself binds 127.0.0.1:5199 (BACKEND_CONNECTOR_ADDR, origin.rs:708) and starts only when the app actually has a backend (manifest.rs:333, origin.rs:1010).

The update engine: built, but with no user loop yet

App updates are a complete engine in crates/apps/src/update.rs, and this chapter has to be honest that the machinery is done while the user-facing loop is not:

• discover_update (semver-gated) finds a candidate; compute_diff detects escalation across permissions, endpoints, config, and source.
• apply_update takes a pre-update snapshot, stages, activates, and rolls back on failure; rollback_update restores artifacts and digest-verified volumes.
• UpdatePolicy is Auto | Notify | Manual; auto_update_action only auto-applies patch, non-escalating updates. An escalating update forces fresh gate-2 consent (update.rs:239); a non-escalating one carries the prior consent forward (update.rs:242).

Honest status. The update mechanism is library-complete and correct, but the recon surfaced no CLI or consent-surface wiring driving it. There is no proven user loop for app updates today — see the review.

Where engines & backends do not reach yet

• Windows backend apps are unsolved — the exec bridge exists only for smolvm and colima; WSL2 and provisioned-VM backends have no bridge (backend.rs:878-889).
• GPU / ML backends collide with the isolation subset — the safe-compose list forbids exactly the devices: + privileged a GPU needs; Immich dropped its ML service for this reason (a Luke ruling, per the review).
• Resource limits are declared, not enforced — deploy.resources.limits in a compose file is not mapped to container cgroups.
• The update engine has no user loop (above).

Links

• Engines & backends review · App model & packaging · The serving plane
• Emerging specs: engine socket contract, compose refusal list
• d/094 (engine socket) · d/086 (cross-origin grants) · d/013 (upstream push gate) — decisions ledger
• smolvm · smolvm PR #656 · compose spec · workerd
• Forge: xenon-launcher · agent54/smolvm