toasterson/Anima
Watch
0
0
Fork
You've already forked Anima
 0
Code Issues 1 Pull requests 87 Projects Releases Packages Wiki Activity Actions

WIP: Build the runner image in CI from Dockerfile.runner so a fleet roll is a docker pull, not a hand build on archibald #611

Draft
toasterson wants to merge 4 commits from claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker into main
pull from: claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker
merge into: toasterson:main
Conversation 0 Commits 4 Files changed 2 +162 -10
toasterson commented 2026-08-17 02:20:31 +00:00
Owner
Copy link

Anima work item 01a004d3-b8ad-7191-b028-1a7afa985383.

Why

A fleet roll currently requires an operator on archibald. That is the only remaining reason — everything else in the 2026-08-14/15 roll (image bump, migration, Flux reconcile, drain, resume) was driven remotely. The runner image is built by hand there because no CI job produces it.

This is option 1 from WI-558. That work item took option 2 (stop publishing the broken image) plus deploy/build/build-and-push-runner.sh (8346121c), which fixed correctness — the script builds the right file and refuses to publish an entrypoint-less image — but did not remove the operator.

What exists now

deploy/build/build-and-push-runner.sh, run on archibald:

  • builds -f Dockerfile.runner
  • guards: refuses to tag or push if .Config.Entrypoint is empty
  • pushes <sha8> and latest to the same digest
  • prints the compose line to recreate the fleet

A CI job should do exactly this, on main, so anima-runner:latest tracks main without anyone logging in.

The cost, honestly

Dockerfile.runner is not the slim server image. It is rust:bookworm plus node 20, the ACP executor CLIs, sccache, a docker CLI and tini — two stages, the first a full cargo build of the runner binary. Building it on every main push would add real minutes and real registry bytes.

Mitigations worth considering, in rough order of value:

  1. Only build when it can have changed. Path-filter on crates/anima-runner/**, crates/anima-core/**, Cargo.lock, Dockerfile.runner, and the deploy/runner assets it COPYs. The 2026-08-15 roll is the case in point: 98630ff5..8346121c touched zero files under crates/, so the image was byte-identical and rebuilding it would have been pure waste.
  2. Reuse the existing build cache. The runner binary is the same cargo build the test lane already does; a shared sccache/registry cache mount should make the common case cheap.
  3. Tag-triggered only. Build it in release.yml on v* tags rather than every main push, accepting that latest then tracks releases rather than main.

Pick deliberately — (1)+(2) keeps latest == main, (3) is cheaper but changes what latest means, and that meaning is load-bearing (see below).

Constraints that must survive

  • Keep the entrypoint guard. It is the one-line check that would have caught WI-558 at the first push instead of months later during a roll. Whatever runs in CI must fail the build, not publish, on an empty entrypoint.
  • latest and <sha8> must move together, to the same digest. The staleness was the hazard: anima-runner:latest sat months old with a different entrypoint (no tini) than anything the fleet ran, so a roll that pulled it would have come up subtly wrong rather than loudly broken.
  • --short=8. Every publisher pins the sha length; a 7-vs-8 mismatch produces per-arch tags that cannot be paired.
  • amd64 only is correct today. Runners run on archibald and nowhere else — unlike anima-server, which schedules onto a mixed-arch cluster with no nodeSelector and therefore must be a stitched multi-arch manifest. If an arm64 runner host ever lands, build the arm64 half there and stitch; do not leave latest amd64-only.
  • Do not resurrect --target anima-runner. That target in the main Dockerfile is FROM scratch + one COPY, a binary-export stage. Publishing it is what produced 23 dead tags.

Done when

A push to main (or a tag, per the choice above) publishes anima-runner:<sha8> and :latest from Dockerfile.runner, with the entrypoint guard enforced in CI, and a fleet roll on archibald is:

docker compose -p anima -f docker-compose.runner.yml pull runner-1 runner-opencode
docker compose -p anima -f docker-compose.runner.yml up -d --force-recreate --no-deps runner-1 runner-opencode

with no local build step. deploy/build/build-and-push-runner.sh should stay as the manual fallback and keep working.

Anima work item `01a004d3-b8ad-7191-b028-1a7afa985383`. ## Why A fleet roll currently requires an operator on archibald. That is the *only* remaining reason — everything else in the 2026-08-14/15 roll (image bump, migration, Flux reconcile, drain, resume) was driven remotely. The runner image is built by hand there because no CI job produces it. This is option 1 from WI-558. That work item took option 2 (stop publishing the broken image) plus `deploy/build/build-and-push-runner.sh` (`8346121c`), which fixed *correctness* — the script builds the right file and refuses to publish an entrypoint-less image — but did not remove the operator. ## What exists now `deploy/build/build-and-push-runner.sh`, run on archibald: - builds `-f Dockerfile.runner` - guards: refuses to tag or push if `.Config.Entrypoint` is empty - pushes `<sha8>` and `latest` to the same digest - prints the compose line to recreate the fleet A CI job should do exactly this, on main, so `anima-runner:latest` tracks main without anyone logging in. ## The cost, honestly `Dockerfile.runner` is not the slim server image. It is `rust:bookworm` plus node 20, the ACP executor CLIs, sccache, a docker CLI and tini — two stages, the first a full `cargo build` of the runner binary. Building it on every main push would add real minutes and real registry bytes. Mitigations worth considering, in rough order of value: 1. **Only build when it can have changed.** Path-filter on `crates/anima-runner/**`, `crates/anima-core/**`, `Cargo.lock`, `Dockerfile.runner`, and the deploy/runner assets it COPYs. The 2026-08-15 roll is the case in point: `98630ff5..8346121c` touched zero files under `crates/`, so the image was byte-identical and rebuilding it would have been pure waste. 2. **Reuse the existing build cache.** The runner binary is the same `cargo build` the test lane already does; a shared sccache/registry cache mount should make the common case cheap. 3. **Tag-triggered only.** Build it in `release.yml` on `v*` tags rather than every main push, accepting that `latest` then tracks releases rather than main. Pick deliberately — (1)+(2) keeps `latest` == main, (3) is cheaper but changes what `latest` means, and that meaning is load-bearing (see below). ## Constraints that must survive - **Keep the entrypoint guard.** It is the one-line check that would have caught WI-558 at the first push instead of months later during a roll. Whatever runs in CI must fail the build, not publish, on an empty entrypoint. - **`latest` and `<sha8>` must move together, to the same digest.** The staleness was the hazard: `anima-runner:latest` sat months old with a *different* entrypoint (no tini) than anything the fleet ran, so a roll that pulled it would have come up subtly wrong rather than loudly broken. - **`--short=8`.** Every publisher pins the sha length; a 7-vs-8 mismatch produces per-arch tags that cannot be paired. - **amd64 only is correct today.** Runners run on archibald and nowhere else — unlike `anima-server`, which schedules onto a mixed-arch cluster with no nodeSelector and therefore must be a stitched multi-arch manifest. If an arm64 runner host ever lands, build the arm64 half there and stitch; do not leave `latest` amd64-only. - **Do not resurrect `--target anima-runner`.** That target in the main `Dockerfile` is `FROM scratch` + one COPY, a binary-export stage. Publishing it is what produced 23 dead tags. ## Done when A push to main (or a tag, per the choice above) publishes `anima-runner:<sha8>` and `:latest` from `Dockerfile.runner`, with the entrypoint guard enforced in CI, and a fleet roll on archibald is: ```sh docker compose -p anima -f docker-compose.runner.yml pull runner-1 runner-opencode docker compose -p anima -f docker-compose.runner.yml up -d --force-recreate --no-deps runner-1 runner-opencode ``` with no local build step. `deploy/build/build-and-push-runner.sh` should stay as the manual fallback and keep working.
Anima implementation session.
Anima implementation session.
Build the runner image in CI from Dockerfile.runner so a fleet roll is a docker pull, not a hand build on archibald
Some checks failed
CI / docker (pull_request) Has been skipped
Details
CI / runner (pull_request) Has been skipped
Details
CI / conflict-check (pull_request) Has been cancelled
Details
CI / test (pull_request) Has been cancelled
Details
8ab55f8f33 
Anima implementation session.
Build the runner image in CI from Dockerfile.runner so a fleet roll is a docker pull, not a hand build on archibald
Some checks failed
CI / docker (pull_request) Has been skipped
Details
CI / runner (pull_request) Has been skipped
Details
CI / conflict-check (pull_request) Failing after 2h3m41s
Details
CI / test (pull_request) Failing after 2h3m42s
Details
576b794e6b 
Anima implementation session.
Some checks failed
CI / docker (pull_request) Has been skipped
Details
CI / runner (pull_request) Has been skipped
Details
CI / conflict-check (pull_request) Failing after 2h3m41s
Details
CI / test (pull_request) Failing after 2h3m42s
Details
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker:claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker
git switch claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker
git switch claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker
git rebase main
git switch main
git merge --ff-only claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker
git switch claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker
git rebase main
git switch main
git merge --no-ff claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker
git switch main
git merge --squash claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker
git switch main
git merge --ff-only claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker
git switch main
git merge claude/wi-01a004d3-build-the-runner-image-in-ci-from-docker
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
Compat/Breaking
Breaking change that won't be backward compatible

  
Kind/Bug
Something is not working

  
Kind/Documentation
Documentation changes

  
Kind/Enhancement
Improve existing functionality

  
Kind/Feature
New functionality

  
Kind/Security
This is security issue

  
Kind/Testing
Issue or pull request related to testing

  
Priority
Critical
 The priority is critical

  
Priority
High
 The priority is high

  
Priority
Low
 The priority is low

  
Priority
Medium
 The priority is medium

  
Reviewed
Confirmed
 Issue has been confirmed

  
Reviewed
Duplicate
 This issue or pull request already exists

  
Reviewed
Invalid
 Invalid issue

  
Reviewed
Won't Fix
 This issue won't be fixed

  
Status
Abandoned
 Somebody has started to work on this but abandoned work

  
Status
Blocked
 Something is blocking this issue or pull request

  
Status
Need More Info
 Feedback is required to reproduce issue or to continue work

No labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/Anima!611
No description provided.