WIP: Builds escape the per-slot CARGO_TARGET_DIR into an unbounded shared debug dir the janitor's 7-day rule can never prune #615

Draft

toasterson wants to merge 3 commits from claude/wi-01a00c91-builds-escape-the-per-slot-cargo-target into main

toasterson commented

2026-08-17 17:53:31 +00:00

Owner

Anima work item 01a00c91-57d7-7ef2-ad6b-85825744a332.

Found 2026-08-16 while diagnosing 160GB of runner-state volume growth on archibald (root fs hit 86%).

Symptom: anima_runner-1-state held 60G and anima_runner-opencode-state 73G in cargo-target/debug — a SHARED, un-slotted target dir. Per WI 019fcd81's design, every build is supposed to run with CARGO_TARGET_DIR=/var/lib/anima-runner/cargo-target/slot-{N} (size-capped at 5G, cleared on slot release — that part works; observed the janitor clear slot-0 at 5.6G).

Root cause (two halves):

The compose file sets a container-wide CARGO_TARGET_DIR=/var/lib/anima-runner/cargo-target with no slot suffix. Any process that inherits container env without a slot assignment — the pre-push test gate is the prime suspect — builds into the shared dir, and cargo appends /debug.
The janitor's shared-dir prune (prune_cargo_target, janitor.rs ~line 958) is age-based: 7 days. Measured: ZERO files older than 7 days, 6,909 files younger than 48h — the dir churns faster than a week, so the age rule never bounds it. The slot dirs got a size cap (WI 019fcd81); the shared dir got an age rule that cannot work under churn.

Fix shape:

Find every spawn path that builds without a slot override (pre-push gate first) and give it a slot dir or a dedicated capped dir.
Give the shared cargo-target root the same size-cap treatment as slots (or delete-on-sight for a debug/ subdir that should never exist), instead of / in addition to the 7-day age rule.
Consider a startup assertion/log when anything writes to the un-slotted path, so escapes are visible instead of silent.

Not the problem (verified, don't re-litigate): sccache is healthy — S3 round-trip proven (write + hit after flush delay), live production window on opencode showed 822 requests / 617 hits (~75%), every live rustc parented by sccache, CARGO_INCREMENTAL=0 present everywhere. The old WI-12-era 0% was fixed. Manual cleanup 2026-08-16 removed both shared debug dirs (~133G) and 4 dead solstice-ci job dirs (~20G); disk went 86% → ~66% expected.

Anima work item `01a00c91-57d7-7ef2-ad6b-85825744a332`. Found 2026-08-16 while diagnosing 160GB of runner-state volume growth on archibald (root fs hit 86%). **Symptom:** `anima_runner-1-state` held 60G and `anima_runner-opencode-state` 73G in `cargo-target/debug` — a SHARED, un-slotted target dir. Per WI 019fcd81's design, every build is supposed to run with `CARGO_TARGET_DIR=/var/lib/anima-runner/cargo-target/slot-{N}` (size-capped at 5G, cleared on slot release — that part works; observed the janitor clear slot-0 at 5.6G). **Root cause (two halves):** 1. The compose file sets a container-wide `CARGO_TARGET_DIR=/var/lib/anima-runner/cargo-target` with no slot suffix. Any process that inherits container env without a slot assignment — the pre-push test gate is the prime suspect — builds into the shared dir, and cargo appends `/debug`. 2. The janitor's shared-dir prune (`prune_cargo_target`, janitor.rs ~line 958) is age-based: 7 days. Measured: ZERO files older than 7 days, 6,909 files younger than 48h — the dir churns faster than a week, so the age rule never bounds it. The slot dirs got a size cap (WI 019fcd81); the shared dir got an age rule that cannot work under churn. **Fix shape:** - Find every spawn path that builds without a slot override (pre-push gate first) and give it a slot dir or a dedicated capped dir. - Give the shared `cargo-target` root the same size-cap treatment as slots (or delete-on-sight for a `debug/` subdir that should never exist), instead of / in addition to the 7-day age rule. - Consider a startup assertion/log when anything writes to the un-slotted path, so escapes are visible instead of silent. **Not the problem (verified, don't re-litigate):** sccache is healthy — S3 round-trip proven (write + hit after flush delay), live production window on opencode showed 822 requests / 617 hits (~75%), every live rustc parented by sccache, CARGO_INCREMENTAL=0 present everywhere. The old WI-12-era 0% was fixed. Manual cleanup 2026-08-16 removed both shared debug dirs (~133G) and 4 dead solstice-ci job dirs (~20G); disk went 86% → ~66% expected.

toasterson added 1 commit

2026-08-17 17:53:32 +00:00

Builds escape the per-slot CARGO_TARGET_DIR into an unbounded shared debug dir the janitor's 7-day rule can never prune b3837c2a5e

Anima implementation session.

toasterson added 1 commit

2026-08-17 19:14:03 +00:00

Builds escape the per-slot CARGO_TARGET_DIR into an unbounded shared debug dir the janitor's 7-day rule can never prune 16bea94469

Anima implementation session.

toasterson added 1 commit

2026-08-18 09:49:38 +00:00

Builds escape the per-slot CARGO_TARGET_DIR into an unbounded shared debug dir the janitor's 7-day rule can never prune 0e38c420ab

Anima implementation session.

This pull request is marked as a work in progress.

This branch is out-of-date with the base branch

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin claude/wi-01a00c91-builds-escape-the-per-slot-cargo-target:claude/wi-01a00c91-builds-escape-the-per-slot-cargo-target

git switch claude/wi-01a00c91-builds-escape-the-per-slot-cargo-target

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main

git merge --no-ff claude/wi-01a00c91-builds-escape-the-per-slot-cargo-target

git switch claude/wi-01a00c91-builds-escape-the-per-slot-cargo-target

git rebase main

git switch main

git merge --ff-only claude/wi-01a00c91-builds-escape-the-per-slot-cargo-target

git switch claude/wi-01a00c91-builds-escape-the-per-slot-cargo-target

git rebase main

git switch main

git merge --no-ff claude/wi-01a00c91-builds-escape-the-per-slot-cargo-target

git switch main

git merge --squash claude/wi-01a00c91-builds-escape-the-per-slot-cargo-target