WI 1 (SPIKE, gates the rest) — hypervisor pick + refraction-forger rootfs with sccache inside #433

Merged

toasterson merged 1 commit from claude/wi-019f9079-wi-1-spike-gates-the-rest-hypervisor-pic into main

2026-08-18 19:48:55 +00:00

toasterson commented

2026-07-29 14:29:47 +00:00

Owner

Anima work item 019f9079-6122-7273-9898-255d99a43ba2.

De-risk the whole plan before committing. Pick the hypervisor and prove the base mechanics on archibald. This is a SPIKE — the deliverable is a decision + a working demo, not production code.

UPDATE 2026-07-27 — the hypervisor question is largely answered; re-scope this spike

Till surfaced AgentENV (https://github.com/kvcache-ai/AgentEnv, MIT, Rust) — kvcache-ai's production sandbox platform for Kimi K3 agentic RL training. It is this exact design, already built and running at scale: Firecracker microVMs, snapshot-backed resume under 50 ms, pause under 100 ms, incremental snapshot under 100 ms, and native fork of a running VM into N independent sandboxes — which is precisely the "warm AND isolated" combination this plan has been chasing.

Harvest analysis: docs/design/2026-07-27-microvm-harvest-agentenv.md (branch claude/microvm-architecture-harvest-c5698c). Read it before starting this spike.

The split decided there: vm-manager owns the host layer, Anima owns policy only. The hypervisor machinery is now filed as work in the vm-manager project — plan 019fa4aa-96db-7e93-95e3-106e09de824f "Snapshot-native VM lifecycle (AgentENV harvest)": Firecracker backend (WI 2), snapshot/restore on the Hypervisor trait + local POSIX snapshot store (WI 3), tap/slot networking (WI 4), warm pools (WI 5), fork() (WI 6). Solstice CI consumes the same layer via its own plan 019fa4aa-cfce-7c12-bc96-ef1ee31121ab.

So this spike no longer has to answer "Firecracker vs cloud-hypervisor" or "is vm-manager the orchestration layer" from scratch. The verdict, unless this spike disproves it on our hardware, is: Firecracker, via vm-manager. Two consumers now need the same thing, and building it twice is the failure mode to avoid.

Also settled by Till 2026-07-27, so do not re-open:

ssh2 STAYS. AgentENV replaces SSH with an in-guest envd daemon; we are NOT taking that. Snapshot resume already removes the boot-and-wait-for-sshd cost without touching the transport.
NET_ADMIN is accepted — tap/slot networking is faster than user-mode SLIRP and the capability is a smaller cost than the latency.
Archibald runs kernel 6.8.0 or better and can move to 7.x, so ublk and io_uring zero-copy are available if the layered-storage path is ever pursued.
QCOW2 overlays on a backing image already give us disk templating. The missing pieces are memory snapshot, fork, and layered on-demand image transport — not disk templates.

What remains for this spike, on archibald, with our hardware: the rootless-KVM privilege question, the refraction-forger rootfs (now Firecracker-shaped: kernel + raw/ext4 rootfs, not QCOW2), sccache-in-guest reachability, vsock, memory caps, and the warm-build number. Everything below stands.

ROOTFS: CloudNebulaProject/refraction-forger is the client-OS image builder (https://code.aopc.cloud/CloudNebulaProject/refraction-forger) — use it to build the guest image, do not roll a bespoke one. It already produced the arm64 VM image whose opencode config the container runner mirrors. Note the shape change: Firecracker needs a kernel image plus a raw/ext4 rootfs, not a QCOW2 — coordinate with vm-manager WI 2, which has the same dependency.

Prove (deliverable = a reproducible demo on archibald)

A micro-VM boots as a NON-ROOT user via the kvm group. Document exactly what privilege is still required and why true container-rootless is unreachable with KVM.
A refraction-forger-built rootfs carrying the agent toolchain (git, rust + node) + ONE ACP CLI (opencode first — no Anthropic auth) boots and the CLI runs inside.
sccache INSIDE the guest, wired to the shared S3 cache. Non-negotiable: without it every VM pays a cold Rust compile and per-session VMs are unusably slow. Bake sccache in and inject the same env the container runner uses today. Verify the guest can REACH the S3 endpoint from its network position — a guest that cannot reach the cache degrades SILENTLY to full compiles (it will look like "VMs are slow", not "cache unreachable"), so assert on a cache-hit metric, not on wall time alone.
A vsock channel host↔guest is up (substrate for WI 2 — just prove reachability).
Hard memory cap enforced. Measure: cold boot time, idle footprint, and a WARM-CACHE build time vs the current container runner. The sccache-hit build number is the figure that decides whether per-session VMs are viable at all — if it is not close to the container runner's, stop and report before WI 2 starts.
New: confirm on our hardware that a snapshot restore of an agent VM lands anywhere near AgentENV's published numbers. If restore is slow on archibald's storage, WI 4 (warm sessions via snapshot/restore) needs rethinking before it is released.

Files to read (Anima repo)

docs/design/2026-07-27-microvm-harvest-agentenv.md — the harvest analysis and the three-project split. Start here.
Dockerfile.runner — what the current executor environment contains (toolchain, sccache install, ACP CLIs) — the guest image must match this capability set.
docker-compose.runner.yml — the x-runner-common env block is the authoritative list of SCCACHE_* / AWS_* / RUSTC_WRAPPER vars to reproduce in-guest, plus OPENCODE_MODEL / OLLAMA_API_KEY.
deploy/runner/opencode.json and deploy/runner/anima-opencode-acp — the baked opencode provider config + the wrapper the runner spawns. The guest needs the equivalent.
docs/design/2026-06-11-acp-runner-architecture.md §4.2, §7.

Tools

mcp__anima__find to locate the sccache/executor wiring; mcp__anima__get_document for the design doc; mcp__anima__post_message to report the numbers to the WI thread. Bash/SSH to archibald for the actual boot tests.

Output

A short decision doc (hypervisor choice + rationale, vm-manager verdict, rootless-KVM privilege caveat, boot/mem/warm-build/restore numbers) committed to docs/design/, plus the refraction-forger image recipe committed reproducibly. Everything downstream (WI 2-7) assumes this choice — post the decision to the thread before those are released.

Host constraint: a VM host must be Linux/KVM. archibald (amd64) works now; the future arm64 Linux Solstice host is the arm64 VM host. armini is macOS and can NEVER host guests.

Anima work item `019f9079-6122-7273-9898-255d99a43ba2`. De-risk the whole plan before committing. Pick the hypervisor and prove the base mechanics on archibald. **This is a SPIKE — the deliverable is a decision + a working demo, not production code.** ## UPDATE 2026-07-27 — the hypervisor question is largely answered; re-scope this spike Till surfaced **AgentENV** (https://github.com/kvcache-ai/AgentEnv, MIT, Rust) — kvcache-ai's production sandbox platform for Kimi K3 agentic RL training. It is this exact design, already built and running at scale: Firecracker microVMs, snapshot-backed resume under 50 ms, pause under 100 ms, incremental snapshot under 100 ms, and **native fork of a running VM into N independent sandboxes** — which is precisely the "warm AND isolated" combination this plan has been chasing. Harvest analysis: `docs/design/2026-07-27-microvm-harvest-agentenv.md` (branch `claude/microvm-architecture-harvest-c5698c`). **Read it before starting this spike.** The split decided there: **vm-manager owns the host layer, Anima owns policy only.** The hypervisor machinery is now filed as work in the vm-manager project — plan `019fa4aa-96db-7e93-95e3-106e09de824f` "Snapshot-native VM lifecycle (AgentENV harvest)": Firecracker backend (WI 2), `snapshot`/`restore` on the `Hypervisor` trait + local POSIX snapshot store (WI 3), tap/slot networking (WI 4), warm pools (WI 5), `fork()` (WI 6). Solstice CI consumes the same layer via its own plan `019fa4aa-cfce-7c12-bc96-ef1ee31121ab`. **So this spike no longer has to answer "Firecracker vs cloud-hypervisor" or "is vm-manager the orchestration layer" from scratch.** The verdict, unless this spike disproves it on our hardware, is: **Firecracker, via vm-manager.** Two consumers now need the same thing, and building it twice is the failure mode to avoid. Also settled by Till 2026-07-27, so do not re-open: - **`ssh2` STAYS.** AgentENV replaces SSH with an in-guest `envd` daemon; we are NOT taking that. Snapshot resume already removes the boot-and-wait-for-sshd cost without touching the transport. - **`NET_ADMIN` is accepted** — tap/slot networking is faster than user-mode SLIRP and the capability is a smaller cost than the latency. - **Archibald runs kernel 6.8.0 or better** and can move to 7.x, so ublk and io_uring zero-copy are available if the layered-storage path is ever pursued. - **QCOW2 overlays on a backing image already give us disk templating.** The missing pieces are memory snapshot, fork, and layered on-demand image transport — not disk templates. **What remains for this spike, on archibald, with our hardware:** the rootless-KVM privilege question, the refraction-forger rootfs (now Firecracker-shaped: kernel + raw/ext4 rootfs, not QCOW2), sccache-in-guest reachability, vsock, memory caps, and the warm-build number. Everything below stands. --- ROOTFS: **CloudNebulaProject/refraction-forger is the client-OS image builder** (https://code.aopc.cloud/CloudNebulaProject/refraction-forger) — use it to build the guest image, do not roll a bespoke one. It already produced the arm64 VM image whose opencode config the container runner mirrors. Note the shape change: Firecracker needs a kernel image plus a raw/ext4 rootfs, not a QCOW2 — coordinate with vm-manager WI 2, which has the same dependency. ## Prove (deliverable = a reproducible demo on archibald) - A micro-VM boots as a NON-ROOT user via the kvm group. Document exactly what privilege is still required and why true container-rootless is unreachable with KVM. - A refraction-forger-built rootfs carrying the agent toolchain (git, rust + node) + ONE ACP CLI (opencode first — no Anthropic auth) boots and the CLI runs inside. - **sccache INSIDE the guest, wired to the shared S3 cache.** Non-negotiable: without it every VM pays a cold Rust compile and per-session VMs are unusably slow. Bake sccache in and inject the same env the container runner uses today. Verify the guest can REACH the S3 endpoint from its network position — a guest that cannot reach the cache degrades SILENTLY to full compiles (it will look like "VMs are slow", not "cache unreachable"), so assert on a cache-hit metric, not on wall time alone. - A vsock channel host↔guest is up (substrate for WI 2 — just prove reachability). - Hard memory cap enforced. Measure: cold boot time, idle footprint, and a WARM-CACHE build time vs the current container runner. **The sccache-hit build number is the figure that decides whether per-session VMs are viable at all** — if it is not close to the container runner's, stop and report before WI 2 starts. - **New:** confirm on our hardware that a snapshot restore of an agent VM lands anywhere near AgentENV's published numbers. If restore is slow on archibald's storage, WI 4 (warm sessions via snapshot/restore) needs rethinking before it is released. ## Files to read (Anima repo) - `docs/design/2026-07-27-microvm-harvest-agentenv.md` — the harvest analysis and the three-project split. Start here. - `Dockerfile.runner` — what the current executor environment contains (toolchain, sccache install, ACP CLIs) — the guest image must match this capability set. - `docker-compose.runner.yml` — the `x-runner-common` env block is the authoritative list of SCCACHE_* / AWS_* / RUSTC_WRAPPER vars to reproduce in-guest, plus OPENCODE_MODEL / OLLAMA_API_KEY. - `deploy/runner/opencode.json` and `deploy/runner/anima-opencode-acp` — the baked opencode provider config + the wrapper the runner spawns. The guest needs the equivalent. - `docs/design/2026-06-11-acp-runner-architecture.md` §4.2, §7. ## Tools `mcp__anima__find` to locate the sccache/executor wiring; `mcp__anima__get_document` for the design doc; `mcp__anima__post_message` to report the numbers to the WI thread. Bash/SSH to archibald for the actual boot tests. ## Output A short decision doc (hypervisor choice + rationale, vm-manager verdict, rootless-KVM privilege caveat, boot/mem/warm-build/restore numbers) committed to `docs/design/`, plus the refraction-forger image recipe committed reproducibly. **Everything downstream (WI 2-7) assumes this choice — post the decision to the thread before those are released.** Host constraint: a VM host must be Linux/KVM. archibald (amd64) works now; the future arm64 Linux Solstice host is the arm64 VM host. armini is macOS and can NEVER host guests.

toasterson added 1 commit

2026-07-29 14:29:48 +00:00

docs(design): WI 1 spike — Firecracker-via-vm-manager verdict + refraction-forger rootfs recipe 24bf3e8fb3

Decision doc (docs/design/2026-07-29-firecracker-spike-wi1-findings.md):
- Hypervisor: Firecracker, via vm-manager (AgentENV harvest already decided
  this; this spike confirms rather than re-litigates).
- vm-manager owns the host layer; Anima owns policy only. Snapshot/restore,
  tap/slot networking, warm pools, fork land in vm-manager, consumed by this
  plan's WI 4-6.
- Rootless-KVM caveat: non-root via kvm group is the achievable bar; true
  container-rootless is unreachable with KVM. kvm group is a strictly smaller
  privilege than the docker socket we use today. NET_ADMIN accepted.
- Rootfs shape: kernel (vmlinux) + raw ext4, NOT qcow2.
- sccache-in-guest non-negotiable, asserted on the cache-hit metric.
- WI 2-7 gated on the §6 hands-on test plan on archibald.

Reproducible recipe (images/anima-runner-vm/):
- README.md, anima-runner-vm.kdl (KDL), cloud-init.example.yaml — Firecracker-
  shaped guest matching Dockerfile.runner's capability set.

Replaces round-1 drafts lost in the rebase onto origin/main.

toasterson changed title from ~~WIP: WI 1 (SPIKE, gates the rest) — hypervisor pick + refraction-forger rootfs with sccache inside~~ to WI 1 (SPIKE, gates the rest) — hypervisor pick + refraction-forger rootfs with sccache inside

2026-08-18 19:48:33 +00:00

toasterson merged commit d97bb92ffb into main

2026-08-18 19:48:54 +00:00

toasterson referenced this pull request from a commit

2026-08-18 19:48:56 +00:00

Merge pull request 'WI 1 (SPIKE, gates the rest) — hypervisor pick + refraction-forger rootfs with sccache inside' (#433) from claude/wi-019f9079-wi-1-spike-gates-the-rest-hypervisor-pic into main