Job reaper trusts a bare PID from qemu.pid, so PID reuse strands job dirs forever (21.6 GB leaked on archibald, root fs to 94%) #16

Open
toasterson wants to merge 3 commits from anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5 into main
Owner

Proposed changes for work item 01a000fe-b127-72d3-84f4-1ee34d58c5a5.

Anima tracks the implementation, review and validation for this work item.

<!-- anima-work-item:01a000fe-b127-72d3-84f4-1ee34d58c5a5 --> Proposed changes for work item `01a000fe-b127-72d3-84f4-1ee34d58c5a5`. Anima tracks the implementation, review and validation for this work item.
Work item: 01a000fe-b127-72d3-84f4-1ee34d58c5a5
Phase: a0f4dd54-9f6a-5709-a704-ac9662f21328 (working)
Attempt: 1
Base: 11014ef315

Add startup, periodic and low-water sweeps with exact QEMU pidfile argument matching, exclusive work-directory ownership and active-job leases. Keep uncertain identities for inspection. Defer AMQP admission below a configurable available-space floor.

Validation: cargo test -p orchestrator --locked (38 passed); rustfmt 1.94.1 check; git diff --check.
Log entry iteration errors with the storage root and metadata errors with the job path, retaining uncertain entries while reclaiming later orphans. Preserve fatal root/procfs checks and existing identity and lease protection.

Add deterministic fault-injection regressions for iterator and metadata failures, plus a missing-root guard. Validation: cargo test -p orchestrator job_storage --locked (11 passed); git diff --check passed. rustfmt is unavailable in the supplied toolchain.

Work item: 01a000fe-b127-72d3-84f4-1ee34d58c5a5
Phase: a0f4dd54-9f6a-5709-a704-ac9662f21328
Attempt: 1
Assignment base: 157c2428db
Keep u64 widening for Unix targets with narrower statvfs fields while allowing the redundant casts on 64-bit Linux. This addresses the two unnecessary_cast CI errors without changing the accepted reaper or admission behavior.
This pull request can be merged automatically.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5:anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5
git switch anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5
git switch anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5
git rebase main
git switch main
git merge --ff-only anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5
git switch anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5
git rebase main
git switch main
git merge --no-ff anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5
git switch main
git merge --squash anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5
git switch main
git merge --ff-only anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5
git switch main
git merge anima/01a000fe-b127-72d3-84f4-1ee34d58c5a5
git push origin main
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
CloudNebulaProject/solstice-ci!16
No description provided.