toasterson/Anima
Watch
0
0
Fork
You've already forked Anima
 0
Code Issues 1 Pull requests 87 Projects Releases Packages Wiki Activity Actions

Fleet unwedge: Opus 5 on the Claude lane, opencode un-stalled, agent commits no longer dropped #362

Closed
toasterson wants to merge 0 commits from claude/claude-lane-opus-5 into main
pull from: claude/claude-lane-opus-5
merge into: toasterson:main
Conversation 1 Commits - Files changed -
toasterson commented 2026-07-25 09:49:27 +00:00
Owner
Copy link

Six fixes from the 2026-07-25 fleet review. Nothing had merged to main since 2026-07-23 21:22 and all three lanes were failing silently.

1. The model override never reached the claude-code CLI

The spawn path exported the Assign's model as OPENCODE_MODEL for every executor. opencode reads that var; the claude-code CLI reads ANTHROPIC_MODEL — so a claude-code Assign carrying a model (the review-model registry's pick for reviewer sessions) was dropped and the CLI ran its own default. RunInput now carries the canonical executor name and the spawn path picks the env var from it. runner-1 also gets a lane default ANTHROPIC_MODEL=claude-opus-5.

2. EMFILE killed sessions silently — and stalled the whole opencode lane

claude-code-acp watches every file it touches, one worktree per slot, against Docker's 1024-fd default: EMFILE: too many open files, watch …/.claude out of session/new. Runner containers now get nofile 65536. (Host fs.inotify.max_user_instances 128 → 1024 on archibald, in /etc/sysctl.d/99-anima-runner.conf — not part of this diff.)

This was also the opencode lane's stall. Isolated probes exonerated everything else: bare ACP turn 11s, with the Anima MCP server 12s, against a real 7.6 GB worktree 8.4s, with a 49k-token prompt 9.2s. After the limits went up, a real dispatch ran 65 loop steps with real tool calls and exited cleanly in 9 minutes instead of dying at the 3600s cap.

3. An agent that commits its own work was having it dropped

open_or_update_pr decided there was nothing to push purely from git status being clean — but the forcing prompt asks the agent to commit, and opencode does, leaving a clean tree with its commit on the branch. The branch was never pushed and the turn's work died with the worktree. The very first good opencode turn hit this: it committed WI-B: executor token scope stripping as 684ad46, one commit ahead of origin/main, and the runner logged "no changes to push this turn". A clean tree is now checked against the base.

4. A work item created through MCP without an explicit phase was born unroutable

handle_create_work_item passed current_phase straight through, so omitting the documented-optional argument wrote NULL into the column the scheduler and every board bucket filter on. NULL is not a point on the phase axis. gRPC CreateWorkItem has defaulted to draft since ADR 0024; the MCP path now matches. WI 019f9698 was stranded this way and has been healed.

5. opencode's log survives a container recreate

It lived under the default XDG_DATA_HOME inside the container's writable layer — so recreating the container to apply a fix destroyed the evidence for the thing being fixed. Exactly that happened mid-diagnosis. Now on the state volume. (The richer surface is the WI thread in Anima, which carries the full ACP record — 64 tool calls for that session; this log only matters for a stall that never reaches ACP at all, which is precisely the failure being chased here.)

6. Token rotation that proves itself

deploy/runner/rotate-claude-token.sh takes the token on stdin (never argv, never shell history), backs up and rewrites the compose env, recreates runner-1, then drives a real turn through the container. A bad token fails at rotation time instead of 401-ing every dispatch for a day.

Verification: cargo test -p anima-runner — 38 passed, 0 failed, incl. the new model_override_reaches_each_cli_under_its_own_env_var. cargo clippy -p anima-runner --all-targets clean. cargo check -p anima-server clean; the server suite needs a Postgres container per test and was not run.

Deployed: the runner image was rebuilt and both lanes recreated on archibald from this branch. The server-side change (4) is not deployed — it needs the k8s roll-out.

🤖 Generated with Claude Code

Six fixes from the 2026-07-25 fleet review. Nothing had merged to `main` since 2026-07-23 21:22 and all three lanes were failing silently. ### 1. The model override never reached the claude-code CLI The spawn path exported the Assign's model as `OPENCODE_MODEL` for **every** executor. opencode reads that var; the claude-code CLI reads `ANTHROPIC_MODEL` — so a claude-code Assign carrying a model (the review-model registry's pick for reviewer sessions) was dropped and the CLI ran its own default. `RunInput` now carries the canonical executor name and the spawn path picks the env var from it. `runner-1` also gets a lane default `ANTHROPIC_MODEL=claude-opus-5`. ### 2. EMFILE killed sessions silently — and stalled the whole opencode lane claude-code-acp watches every file it touches, one worktree per slot, against Docker's 1024-fd default: `EMFILE: too many open files, watch …/.claude` out of `session/new`. Runner containers now get `nofile 65536`. (Host `fs.inotify.max_user_instances` 128 → 1024 on archibald, in `/etc/sysctl.d/99-anima-runner.conf` — not part of this diff.) This was also the opencode lane's stall. Isolated probes exonerated everything else: bare ACP turn 11s, with the Anima MCP server 12s, against a real 7.6 GB worktree 8.4s, with a 49k-token prompt 9.2s. After the limits went up, a real dispatch ran **65 loop steps** with real tool calls and exited cleanly in 9 minutes instead of dying at the 3600s cap. ### 3. An agent that commits its own work was having it dropped `open_or_update_pr` decided there was nothing to push purely from `git status` being clean — but the forcing prompt asks the agent to commit, and opencode does, leaving a clean tree with its commit on the branch. The branch was never pushed and the turn's work died with the worktree. The very first good opencode turn hit this: it committed `WI-B: executor token scope stripping` as `684ad46`, one commit ahead of `origin/main`, and the runner logged "no changes to push this turn". A clean tree is now checked against the base. ### 4. A work item created through MCP without an explicit phase was born unroutable `handle_create_work_item` passed `current_phase` straight through, so omitting the documented-optional argument wrote NULL into the column the scheduler and every board bucket filter on. NULL is not a point on the phase axis. gRPC `CreateWorkItem` has defaulted to `draft` since ADR 0024; the MCP path now matches. WI `019f9698` was stranded this way and has been healed. ### 5. opencode's log survives a container recreate It lived under the default `XDG_DATA_HOME` inside the container's writable layer — so recreating the container to apply a fix destroyed the evidence for the thing being fixed. Exactly that happened mid-diagnosis. Now on the state volume. (The richer surface is the WI thread in Anima, which carries the full ACP record — 64 tool calls for that session; this log only matters for a stall that never reaches ACP at all, which is precisely the failure being chased here.) ### 6. Token rotation that proves itself `deploy/runner/rotate-claude-token.sh` takes the token on stdin (never argv, never shell history), backs up and rewrites the compose env, recreates `runner-1`, then drives a real turn through the container. A bad token fails at rotation time instead of 401-ing every dispatch for a day. **Verification:** `cargo test -p anima-runner` — 38 passed, 0 failed, incl. the new `model_override_reaches_each_cli_under_its_own_env_var`. `cargo clippy -p anima-runner --all-targets` clean. `cargo check -p anima-server` clean; the server suite needs a Postgres container per test and was not run. **Deployed:** the runner image was rebuilt and both lanes recreated on archibald from this branch. The server-side change (4) is **not** deployed — it needs the k8s roll-out. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
fix(runner): the model override never reached the claude-code CLI
Some checks failed
CI / conflict-check (push) Successful in 31s
Details
CI / docker (push) Has been cancelled
Details
d847a031da
 
The spawn path exported the Assign's model as OPENCODE_MODEL for every
executor. opencode reads that var; the claude-code CLI reads
ANTHROPIC_MODEL — so a claude-code Assign carrying a model (the
review-model registry's pick for reviewer sessions) silently ran the
CLI's own default instead.

RunInput now carries the canonical executor name and the spawn path
picks the env var from it. runner-1 also gets a lane default of
claude-opus-5, so implementer/chat sessions — which carry no override —
run Opus 5 rather than whatever the CLI defaults to.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
claude-code-acp watches every file it touches in the worktree, one
worktree per slot, against Docker's 1024-fd default. On 2026-07-25 it
threw `EMFILE: too many open files, watch …/.claude` on session/new and
the turn failed with no failure recorded anywhere.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(mcp): a work item created without an explicit phase was born unroutable
All checks were successful
CI / conflict-check (push) Successful in 15m51s
Details
CI / docker (push) Successful in 26m41s
Details
7ad650d06d
 
The MCP create_work_item handler passed the caller's `current_phase`
straight through, so omitting the documented-optional argument wrote
NULL into the column the scheduler and every board bucket filter on.
NULL is not a point on the phase axis: the item matched no phase
predicate, so nothing could dispatch it and nothing could display it.
The gRPC CreateWorkItem has defaulted to `draft` since ADR 0024; this
brings the MCP path in line, and honours the prefactor invariant stated
two comments below it — every WI carries a valid phase from birth.

Planning sessions are still forced to `draft` regardless of what they ask
for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
toasterson changed title from fix(runner): the model override never reached the claude-code CLI; Claude lane defaults to Opus 5 to Fleet unwedge: Claude lane on Opus 5, runner fd limit, MCP-created WIs get a phase 2026-07-25 09:58:00 +00:00 
opencode's log lives under XDG_DATA_HOME, which defaulted into the
container's writable layer. That log is the only place a stalled turn is
visible — the runner reports nothing but the 3600s cap — so recreating
the container to apply a fix destroys the evidence for the thing being
fixed. Exactly that happened while diagnosing the stalled lane on
2026-07-25.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The claude-code lane's OAuth token expires roughly daily and carries no
refresh token in a reused login. When it lapses every turn 401s and the
runner reports "no changes to push this turn" — indistinguishable, to the
server, from a turn that genuinely had nothing to push. On 2026-07-25
that ran for over a day: ~20 dead turns, zero merges, every dashboard
reading "working".

rotate-claude-token.sh takes the token on stdin (never argv, never shell
history), backs up and rewrites the compose env, recreates runner-1, and
then drives a real one-turn probe through the container. A token that
does not work fails there, loudly, with the backup path in the message.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix(runner): an agent that commits its own work was having it dropped
All checks were successful
CI / conflict-check (push) Successful in 34s
Details
CI / docker (push) Successful in 10m31s
Details
08e69a7497
 
open_or_update_pr decided there was nothing to push purely from
`git status` being clean. But the forcing prompt asks the agent to
commit, and opencode does: it leaves a clean tree with its commit on the
branch. commit_all then returns None and the runner logs "no changes to
push this turn" — the branch is never pushed, no PR is opened, and the
whole turn's work dies with the worktree.

Observed 2026-07-25 on the opencode lane: a 65-step turn committed
"WI-B: executor token scope stripping" as 684ad46, one commit ahead of
origin/main with a clean tree, and the runner reported no changes.

A clean tree is now checked against the base: only a HEAD level with
origin/<base_branch> counts as an idle turn.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
toasterson changed title from Fleet unwedge: Claude lane on Opus 5, runner fd limit, MCP-created WIs get a phase to Fleet unwedge: Opus 5 on the Claude lane, opencode un-stalled, agent commits no longer dropped 2026-07-25 10:38:29 +00:00
toasterson commented 2026-07-28 12:03:10 +00:00
Author
Owner
Copy link

Closing: this branch is fully contained in main — git log origin/main..HEAD is empty. The fleet-unwedge content (including the TurnDone.stop_reason handling) landed through PR #366 and the lane-config changes that followed. Nothing left to merge.

Closing: this branch is fully contained in `main` — `git log origin/main..HEAD` is empty. The fleet-unwedge content (including the `TurnDone.stop_reason` handling) landed through PR #366 and the lane-config changes that followed. Nothing left to merge.
toasterson closed this pull request 2026-07-28 12:03:21 +00:00
All checks were successful
CI / conflict-check (push) Successful in 34s
Details
CI / docker (push) Successful in 10m31s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
Compat/Breaking
Breaking change that won't be backward compatible

  
Kind/Bug
Something is not working

  
Kind/Documentation
Documentation changes

  
Kind/Enhancement
Improve existing functionality

  
Kind/Feature
New functionality

  
Kind/Security
This is security issue

  
Kind/Testing
Issue or pull request related to testing

  
Priority
Critical
 The priority is critical

  
Priority
High
 The priority is high

  
Priority
Low
 The priority is low

  
Priority
Medium
 The priority is medium

  
Reviewed
Confirmed
 Issue has been confirmed

  
Reviewed
Duplicate
 This issue or pull request already exists

  
Reviewed
Invalid
 Invalid issue

  
Reviewed
Won't Fix
 This issue won't be fixed

  
Status
Abandoned
 Somebody has started to work on this but abandoned work

  
Status
Blocked
 Something is blocking this issue or pull request

  
Status
Need More Info
 Feedback is required to reproduce issue or to continue work

No labels
Compat/Breaking
Kind/Bug
Kind/Documentation
Kind/Enhancement
Kind/Feature
Kind/Security
Kind/Testing
Priority
Critical
Priority
High
Priority
Low
Priority
Medium
Reviewed
Confirmed
Reviewed
Duplicate
Reviewed
Invalid
Reviewed
Won't Fix
Status
Abandoned
Status
Blocked
Status
Need More Info
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/Anima!362
No description provided.