The janitor's two sweeps both exclude the shared cargo target dir: it regrows ~75G/day and took archibald to 0 bytes, killing Forgejo, the registry and CI #496

Closed

toasterson wants to merge 0 commits from claude/wi-019faa43-the-janitor-s-two-sweeps-both-exclude-th into main

toasterson commented

2026-08-04 16:01:02 +00:00

Owner

Anima work item 019faa43-4e92-7800-b142-3e0ebc534fd4.

Measured 2026-07-28

archibald /  (/dev/md2)          436G   350G used   65G free   85%

anima_runner-opencode-state      40.6G
  cargo-target                    34.3G
  xdg                              5.6G   (opencode logs/state)
  worktrees                      723.5M   (4 session dirs)
anima_runner-1-state             33.1G
anima_dind-data                   2.7G

74 GB of shared cargo target dirs, and nothing ever removes any of it.

Why it is unbounded by construction

The target dir was deliberately moved out of per-session worktrees — the old layout held one target/ per session and reached 62 GB on runner-opencode alone (CARGO_INCREMENTAL=0 was set at the same time, because sccache cannot cache an incremental compilation). The compose file records the trade explicitly: one target dir per RUNNER, living on the state volume next to worktrees/, "so the janitor's boot sweep only deletes UUID-named session dirs, so this survives it by construction."

That survival is the point — it is what makes a new session's build warm instead of cold. It also means the directory is write-only: every branch, every dependency version, every abandoned experiment accretes and nothing collects it. Stale artifacts are not merely wasted bytes; cargo's own fingerprint dirs and old dep versions accumulate faster than the live set.

Why it matters now rather than eventually

archibald is the fleet's single builder and its disk is already the throughput ceiling in another dimension: /proc/pressure/io full avg10=58 while CPU pressure is ~0, because concurrent solstice CI qemu VMs saturate the RAID1. Less free space means worse allocation behaviour on a filesystem that is already the bottleneck. At the current rate the volumes will take the host past the point where a CI VM or an image build fails — and a full disk on this box takes out CI, the runner fleet and the image builds simultaneously.

There is also a correctness edge: a build that fails from ENOSPC looks like a job failure, and CI-red on a work item costs it a retry-budget strike (handle_job_failure). A disk problem would be charged to the code.

Ask

Bound the target dirs. cargo-sweep-style age pruning (--time 7) on a timer inside each runner, or a size ceiling that trims oldest-first. Prefer age: the warm-build benefit comes from the recent working set, and everything older than a week is almost certainly a merged branch.
Prune xdg too — 5.6 GB of opencode state on the opencode runner. It was moved onto the volume deliberately (a container recreate used to destroy the log being diagnosed), so it must survive a recreate, but it does not need to survive forever.
Make the janitor report what it reclaimed, at INFO, per sweep. Today its scope ("UUID-named session dirs only") is documented in a compose comment and nowhere in the logs, so the growth is invisible until df says 85%.
Alert before the wall. A free-space check on archibald that fails loudly at, say, 90% — the fleet currently discovers a full disk by failing a build.

Acceptance criteria

A runner's cargo-target stops growing monotonically: after a prune it is materially smaller, and a subsequent build still hits sccache/cargo cache for the current branch (warm-build benefit preserved — measure a rebuild before and after).
The janitor logs bytes reclaimed and dirs removed on every sweep.
Free space on archibald / recovers above 80% and a monitor exists that fires before it falls under 10% free.
A build failing from ENOSPC is not charged to the work item's retry budget (classify it the way WI-340's infra/job split classifies an infra failure).

project_archibald_disk_leaks covered the OLD per-session target/ layout; that leak was fixed by consolidating to one dir per runner. This is the consolidated dir having no retention of its own — a different problem with the same symptom.
reference_anima_throughput_ceiling — the IO saturation this compounds.
WI-340 / #377 — infra-vs-job CI failure classification, which acceptance criterion 4 extends.

Anima work item `019faa43-4e92-7800-b142-3e0ebc534fd4`. ## Measured 2026-07-28 ``` archibald / (/dev/md2) 436G 350G used 65G free 85% anima_runner-opencode-state 40.6G cargo-target 34.3G xdg 5.6G (opencode logs/state) worktrees 723.5M (4 session dirs) anima_runner-1-state 33.1G anima_dind-data 2.7G ``` **74 GB of shared cargo target dirs, and nothing ever removes any of it.** ## Why it is unbounded by construction The target dir was deliberately moved out of per-session worktrees — the old layout held one `target/` per session and reached 62 GB on runner-opencode alone (`CARGO_INCREMENTAL=0` was set at the same time, because sccache cannot cache an incremental compilation). The compose file records the trade explicitly: one target dir per RUNNER, living on the state volume next to `worktrees/`, "so the janitor's boot sweep only deletes UUID-named session dirs, so this survives it by construction." That survival is the point — it is what makes a new session's build warm instead of cold. It also means the directory is **write-only**: every branch, every dependency version, every abandoned experiment accretes and nothing collects it. Stale artifacts are not merely wasted bytes; cargo's own fingerprint dirs and old dep versions accumulate faster than the live set. ## Why it matters now rather than eventually archibald is the fleet's single builder and its disk is already the throughput ceiling in another dimension: `/proc/pressure/io full avg10=58` while CPU pressure is ~0, because concurrent solstice CI qemu VMs saturate the RAID1. Less free space means worse allocation behaviour on a filesystem that is already the bottleneck. At the current rate the volumes will take the host past the point where a CI VM or an image build fails — and a full disk on this box takes out CI, the runner fleet and the image builds simultaneously. There is also a correctness edge: a build that fails from ENOSPC looks like a job failure, and CI-red on a work item costs it a retry-budget strike (`handle_job_failure`). A disk problem would be charged to the code. ## Ask 1. **Bound the target dirs.** `cargo-sweep`-style age pruning (`--time 7`) on a timer inside each runner, or a size ceiling that trims oldest-first. Prefer age: the warm-build benefit comes from the *recent* working set, and everything older than a week is almost certainly a merged branch. 2. **Prune `xdg` too** — 5.6 GB of opencode state on the opencode runner. It was moved onto the volume deliberately (a container recreate used to destroy the log being diagnosed), so it must survive a recreate, but it does not need to survive forever. 3. **Make the janitor report what it reclaimed**, at INFO, per sweep. Today its scope ("UUID-named session dirs only") is documented in a compose comment and nowhere in the logs, so the growth is invisible until `df` says 85%. 4. **Alert before the wall.** A free-space check on archibald that fails loudly at, say, 90% — the fleet currently discovers a full disk by failing a build. ## Acceptance criteria - A runner's `cargo-target` stops growing monotonically: after a prune it is materially smaller, and a subsequent build still hits sccache/cargo cache for the current branch (warm-build benefit preserved — measure a rebuild before and after). - The janitor logs bytes reclaimed and dirs removed on every sweep. - Free space on archibald `/` recovers above 80% and a monitor exists that fires before it falls under 10% free. - A build failing from ENOSPC is not charged to the work item's retry budget (classify it the way WI-340's infra/job split classifies an infra failure). ## Related - [[project_archibald_disk_leaks]] covered the OLD per-session `target/` layout; that leak was fixed by consolidating to one dir per runner. This is the consolidated dir having no retention of its own — a different problem with the same symptom. - [[reference_anima_throughput_ceiling]] — the IO saturation this compounds. - WI-340 / #377 — infra-vs-job CI failure classification, which acceptance criterion 4 extends.

toasterson added 1 commit

2026-08-04 16:01:02 +00:00

The janitor's two sweeps both exclude the shared cargo target dir: it regrows ~75G/day and took archibald to 0 bytes, killing Forgejo, the registry and CI 607f47a584

Anima implementation session.

toasterson added 1 commit

2026-08-04 17:08:24 +00:00

The janitor's two sweeps both exclude the shared cargo target dir: it regrows ~75G/day and took archibald to 0 bytes, killing Forgejo, the registry and CI 107dc4ec03

Anima implementation session.

toasterson added 1 commit

2026-08-04 18:29:39 +00:00

The janitor's two sweeps both exclude the shared cargo target dir: it regrows ~75G/day and took archibald to 0 bytes, killing Forgejo, the registry and CI 9147b89729

Anima implementation session.

toasterson added 1 commit

2026-08-04 19:07:46 +00:00

fix(janitor): box recursive async function to avoid infinitely sized future

CI / conflict-check (push) Successful in 8m26s

Details

CI / docker (push) Successful in 29m3s

Details

49062928ff

The prune_cargo_target_recursive function was causing a compile error
(E0733) because Rust cannot determine the size of a recursive async
function at compile time. Fixed by:

1. Converting the async fn to a regular fn that returns Pin<Box<dyn Future>>
2. Adding Send bound to make the future Send-safe for tokio::spawn
3. Adding explicit lifetime annotations for all borrowed parameters

Also fixed missing newline at end of ci_failure_classification_test.rs.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

toasterson changed title from ~~WIP: The janitor's two sweeps both exclude the shared cargo target dir: it regrows ~75G/day and took archibald to 0 bytes, killing Forgejo, the registry and CI~~ to The janitor's two sweeps both exclude the shared cargo target dir: it regrows ~75G/day and took archibald to 0 bytes, killing Forgejo, the registry and CI

2026-08-04 20:16:34 +00:00

toasterson referenced this pull request from a commit

2026-08-04 20:52:51 +00:00

Merge pull request 'The janitor's two sweeps both exclude the shared cargo-target volumes (#496)' from claude/wi-019faa43-the-janitor-s-two-sweeps-both-exclude-th into main

toasterson referenced this pull request from a commit

2026-08-05 17:48:33 +00:00

fix(janitor): reconcile duplicate dir_size from the two disk PRs

toasterson closed this pull request

2026-08-06 16:54:29 +00:00