Panic in the turn completion path: tokio oneshot "called after complete" fires once per timed-out turn #282

Merged

toasterson merged 2 commits from claude/wi-019fb9d6-panic-in-the-turn-completion-path-tokio into main 2026-08-01 20:32:40 +00:00

Conversation 0 Commits 2 Files changed 1 +177 -6

toasterson commented 2026-07-31 23:56:30 +00:00

Owner

Copy link

Anima work item 019fb9d6-de2a-71b1-9911-782b1677c076.

Symptom

2026-07-31T19:25:56.016607Z thread 'tokio-runtime-worker' (921) panicked at
  /usr/local/cargo/registry/.../tokio-1.49.0/src/sync/oneshot.rs:1289:13:
called after complete

Observed on akh-thoth, image 0fc0416b257d (built from 53c5b86), during the
first tecton turn after #279 was deployed. One occurrence, mid-turn. The
process survived — the panicking worker died, the daemon kept running and the
learning daemon kept ticking either side of it.

Why it is almost certainly #279

oneshot.rs:1289 called after complete is a oneshot used twice: send (or
a completion poll) after the channel has already resolved. #279 introduced
exactly that shape — an explicit completion signal from the turn worker plus a
recv_timeout select in the drain loop, replacing the previous "wait for every
ChannelSink sender clone to drop". Before that merge this panic did not appear
in the daemon log.

Candidate paths to audit:

• the new completion signal vs. the pre-existing done_tx/done_rx oneshot
  around the turn worker (src/acp/mod.rs, the tokio::spawn that awaits
  handle and relays the join result)
• the interaction between a cancelled turn and the completion signal: this
  turn was killed by the runner's 3600s budget cap, so cancellation and normal
  completion may both fire.

Context — the turn it happened in

Session 019fb977-d7ff-70f3-b89f-d15f18566c65, workspace tecton, WI-259.
Started 18:37:46, killed 19:38:00 by the 3600s budget cap
({"outcome":"failed","stop_reason":"cancelled"}). The panic landed at 19:25:56,
about 48 minutes in — not aligned to either the turn start or the cap, so it
is not simply "cancellation raced completion at the deadline".

Note the idle watchdog did not fire in this turn (grep -c "chat turn stalled" = 0), which is #279 working as intended. This item is only about the
panic.

Required

1. Make the completion signal idempotent — a second signal, or a signal after
   cancellation, must be a no-op rather than a panic. Prefer a type that cannot
   be misused twice over a debug_assert.
2. Cover the cancelled-turn path in a test: cancel a turn mid-flight and assert
   the drain loop terminates without panicking.
3. A panicking tokio-runtime-worker should be visible as more than a bare stderr
   line — it currently carries no session id, no workspace, no context. Whoever
   fixes this should leave enough breadcrumb that the next one is greppable.

Acceptance

• No called after complete panic across a run that includes at least one
  cancelled turn and one normally-completed turn.
• A test exercising cancel-then-complete on the same turn.

Anima work item `019fb9d6-de2a-71b1-9911-782b1677c076`. ## Symptom ``` 2026-07-31T19:25:56.016607Z thread 'tokio-runtime-worker' (921) panicked at /usr/local/cargo/registry/.../tokio-1.49.0/src/sync/oneshot.rs:1289:13: called after complete ``` Observed on `akh-thoth`, image `0fc0416b257d` (built from `53c5b86`), during the first tecton turn after **#279** was deployed. One occurrence, mid-turn. The process survived — the panicking worker died, the daemon kept running and the learning daemon kept ticking either side of it. ## Why it is almost certainly #279 `oneshot.rs:1289` `called after complete` is a **oneshot used twice**: `send` (or a completion poll) after the channel has already resolved. #279 introduced exactly that shape — an explicit completion signal from the turn worker plus a `recv_timeout` select in the drain loop, replacing the previous "wait for every `ChannelSink` sender clone to drop". Before that merge this panic did not appear in the daemon log. Candidate paths to audit: - the new completion signal vs. the pre-existing `done_tx`/`done_rx` oneshot around the turn worker (`src/acp/mod.rs`, the `tokio::spawn` that awaits `handle` and relays the join result) - the interaction between a **cancelled** turn and the completion signal: this turn was killed by the runner's 3600s budget cap, so cancellation and normal completion may both fire. ## Context — the turn it happened in Session `019fb977-d7ff-70f3-b89f-d15f18566c65`, workspace `tecton`, WI-259. Started 18:37:46, killed 19:38:00 by the 3600s budget cap (`{"outcome":"failed","stop_reason":"cancelled"}`). The panic landed at 19:25:56, about 48 minutes in — **not** aligned to either the turn start or the cap, so it is not simply "cancellation raced completion at the deadline". Note the idle watchdog did **not** fire in this turn (`grep -c "chat turn stalled"` = 0), which is #279 working as intended. This item is only about the panic. ## Required 1. Make the completion signal idempotent — a second signal, or a signal after cancellation, must be a no-op rather than a panic. Prefer a type that cannot be misused twice over a `debug_assert`. 2. Cover the cancelled-turn path in a test: cancel a turn mid-flight and assert the drain loop terminates without panicking. 3. A panicking `tokio-runtime-worker` should be visible as more than a bare stderr line — it currently carries no session id, no workspace, no context. Whoever fixes this should leave enough breadcrumb that the next one is greppable. ## Acceptance - No `called after complete` panic across a run that includes at least one cancelled turn and one normally-completed turn. - A test exercising cancel-then-complete on the same turn.

toasterson added 1 commit 2026-07-31 23:56:30 +00:00

Panic in the turn completion path: tokio oneshot "called after complete" fires once per timed-out turn ... ab0403440e

Anima implementation session.

toasterson force-pushed claude/wi-019fb9d6-panic-in-the-turn-completion-path-tokio from ab0403440e to 408ff35b4e 2026-08-01 16:43:59 +00:00 Compare

toasterson changed title from WIP: Panic in the turn completion path: tokio oneshot "called after complete" fires once per timed-out turn to Panic in the turn completion path: tokio oneshot "called after complete" fires once per timed-out turn 2026-08-01 17:25:07 +00:00

toasterson merged commit 61e140a7fd into main 2026-08-01 20:32:40 +00:00

toasterson referenced this pull request from a commit 2026-08-01 20:32:44 +00:00

Merge pull request #282: Do not await done_rx after try_recv already consumed it

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!282

No description provided.