toasterson/akh-medu
Watch
1
0
Fork
You've already forked akh-medu
 0
Code Issues 10 Pull requests 22 Projects Releases 2 Packages Wiki Activity Actions 5

WIP: Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible #277

Closed
toasterson wants to merge 3 commits from claude/wi-019fa44f-sessions-on-one-akh-still-serialise-on-t into main
pull from: claude/wi-019fa44f-sessions-on-one-akh-still-serialise-on-t
merge into: toasterson:main
Conversation 1 Commits 3 Files changed 15 +822 -264
toasterson commented 2026-07-29 17:53:53 +00:00
Owner
Copy link

Anima work item 019fa44f-da1f-78e1-813d-1897f0ce5fd4.

Why this exists separately

akh-medu WI-221 ("Extract per-session SessionContext from ChatProcessor", merged 2026-07-27 as PR #237) removed the first of two serialisation points. It was scoped, correctly, to ChatProcessor — that is what its description asked for and that is what it delivered: kg_cache, trajectory and eliza moved into a per-session SessionContext, the RefCell-inside-Mutex anti-pattern removed, and the clone-modify-write per turn eliminated.

The second lock was never in scope. An operator believed they had extended WI-221 to cover it, but the channel used (description_revisions with author_id = 'operator') is filtered out before it reaches any agent — see the correction document anima-prompt-inventory-correction-20260727. Neither the implementer nor the reviewer ever saw the extension. This work item states it properly.

The remaining serialisation point

On origin/main:

src/bin/akhomed.rs:45     agents: Arc<RwLock<HashMap<String, Arc<Mutex<Agent>>>>>
src/bin/akhomed.rs:1826   shared_agent: Arc<Mutex<Agent>>
src/agent/daemon.rs:170   agent: Arc<Mutex<Agent>>   <-- background learning daemon

Every ACP turn takes the per-workspace Arc<Mutex<Agent>>, and the background learning daemon holds the same lock on its own schedule. So two sessions on one akh workspace cannot both make progress, regardless of WI-221.

Evidence this is real, not theoretical

2026-07-27 09:13:25 — three work items dispatched to tecton in the same second:

WI-24  implementation   2 events
WI-33  implementation  10 events   ← the one that got the lock
WI-37  implementation   4 events

WI-37's entire session:

09:13:25  session.started
09:13:26  acp.session_capabilities
09:18:26  "error[acp]: chat turn stalled: no progress for 300s"
09:18:26  acp.turn_completed {"stop_reason": "end_turn"}

No acp.agent_thought at all — the model never ran. Exactly 300s of nothing between handshake and watchdog. That is lock contention, not slow inference.

A second observation the same day is sharper: when two sessions ran concurrently (WI-37 and WI-33, 12:43-13:25), they did not cleanly serialise with one winning. Both crawled — WI-37 produced 27 events in 41 minutes and pushed nothing at all. Contention degrades both sessions rather than queueing them.

Required change

  1. Determine what the Agent mutex actually protects during a turn. Separate genuinely-shared workspace state (the engine, the KG) from per-turn state that belongs alongside SessionContext.
  2. Make the shared remainder safe for concurrent access. Do not simply widen the critical section, and do not swap Mutex for RwLock without checking that turns are read-mostly — measure rather than assume.
  3. The background learning daemon must not be able to block a live ACP turn for an unbounded time. Either it takes the lock in short bounded slices, or it yields to session work.

Acceptance

  • Two ACP sessions on the SAME workspace both make progress concurrently. Start two prompt turns and assert interleaved activity from both. A test that runs them sequentially and passes proves nothing — that is today's behaviour and it is what this item exists to change.
  • A turn's progress is not blocked by a background learning tick; assert a turn proceeds while the daemon is active.
  • Per-session state does not bleed: two concurrent sessions must not observe each other's kg_cache / trajectory / eliza. (WI-221 established this; keep it.)
  • After this lands, the akh gateway can honestly advertise per-workspace concurrency > 1 — which is the precondition for Anima WI-370.

Why it matters now

tecton is being run at WIP 1 by hand because of this. The target architecture (2026-07-27) puts throughput management in the runners and expects executors to advertise real capacity; an akh that can only ever run one turn at a time caps that at 1 per workspace. This is the blocker between the current fleet and per-akh parallelism.

Anima work item `019fa44f-da1f-78e1-813d-1897f0ce5fd4`. ## Why this exists separately akh-medu WI-221 ("Extract per-session SessionContext from ChatProcessor", merged 2026-07-27 as PR #237) removed the **first** of two serialisation points. It was scoped, correctly, to `ChatProcessor` — that is what its description asked for and that is what it delivered: `kg_cache`, `trajectory` and `eliza` moved into a per-session `SessionContext`, the `RefCell`-inside-`Mutex` anti-pattern removed, and the clone-modify-write per turn eliminated. The **second** lock was never in scope. An operator believed they had extended WI-221 to cover it, but the channel used (`description_revisions` with `author_id = 'operator'`) is filtered out before it reaches any agent — see the correction document `anima-prompt-inventory-correction-20260727`. Neither the implementer nor the reviewer ever saw the extension. This work item states it properly. ## The remaining serialisation point On `origin/main`: src/bin/akhomed.rs:45 agents: Arc<RwLock<HashMap<String, Arc<Mutex<Agent>>>>> src/bin/akhomed.rs:1826 shared_agent: Arc<Mutex<Agent>> src/agent/daemon.rs:170 agent: Arc<Mutex<Agent>> <-- background learning daemon Every ACP turn takes the per-workspace `Arc<Mutex<Agent>>`, and the background learning daemon holds the **same** lock on its own schedule. So two sessions on one akh workspace cannot both make progress, regardless of WI-221. ## Evidence this is real, not theoretical 2026-07-27 09:13:25 — three work items dispatched to tecton in the same second: WI-24 implementation 2 events WI-33 implementation 10 events ← the one that got the lock WI-37 implementation 4 events WI-37's entire session: 09:13:25 session.started 09:13:26 acp.session_capabilities 09:18:26 "error[acp]: chat turn stalled: no progress for 300s" 09:18:26 acp.turn_completed {"stop_reason": "end_turn"} No `acp.agent_thought` at all — the model never ran. Exactly 300s of nothing between handshake and watchdog. That is lock contention, not slow inference. A second observation the same day is sharper: when two sessions ran concurrently (WI-37 and WI-33, 12:43-13:25), they did **not** cleanly serialise with one winning. Both crawled — WI-37 produced 27 events in 41 minutes and pushed nothing at all. Contention degrades both sessions rather than queueing them. ## Required change 1. Determine what the `Agent` mutex actually protects during a turn. Separate genuinely-shared workspace state (the engine, the KG) from per-turn state that belongs alongside `SessionContext`. 2. Make the shared remainder safe for concurrent access. Do not simply widen the critical section, and do not swap `Mutex` for `RwLock` without checking that turns are read-mostly — measure rather than assume. 3. The background learning daemon must not be able to block a live ACP turn for an unbounded time. Either it takes the lock in short bounded slices, or it yields to session work. ## Acceptance - **Two ACP sessions on the SAME workspace both make progress concurrently.** Start two prompt turns and assert interleaved activity from both. A test that runs them sequentially and passes proves nothing — that is today's behaviour and it is what this item exists to change. - A turn's progress is not blocked by a background learning tick; assert a turn proceeds while the daemon is active. - Per-session state does not bleed: two concurrent sessions must not observe each other's `kg_cache` / `trajectory` / `eliza`. (WI-221 established this; keep it.) - After this lands, the akh gateway can honestly advertise per-workspace concurrency > 1 — which is the precondition for Anima WI-370. ## Why it matters now tecton is being run at WIP 1 by hand because of this. The target architecture (2026-07-27) puts throughput management in the runners and expects executors to advertise real capacity; an akh that can only ever run one turn at a time caps that at 1 per workspace. This is the blocker between the current fleet and per-akh parallelism.
Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible
Some checks failed
CI / check-seshat (push) Successful in 16m59s
Details
CI / check-seshat (pull_request) Successful in 17m32s
Details
CI / docker-seshd (pull_request) Successful in 19m54s
Details
CI / publish-chart (push) Successful in 19m58s
Details
CI / publish-chart (pull_request) Failing after 22m12s
Details
CI / docker-seshd (push) Successful in 28m27s
Details
d8bfe7e684 
Anima implementation session.
toasterson force-pushed claude/wi-019fa44f-sessions-on-one-akh-still-serialise-on-t from d8bfe7e684
Some checks failed
CI / check-seshat (push) Successful in 16m59s
Details
CI / check-seshat (pull_request) Successful in 17m32s
Details
CI / docker-seshd (pull_request) Successful in 19m54s
Details
CI / publish-chart (push) Successful in 19m58s
Details
CI / publish-chart (pull_request) Failing after 22m12s
Details
CI / docker-seshd (push) Successful in 28m27s
Details
to 597fdb2c8f
All checks were successful
CI / publish-chart (push) Successful in 10m8s
Details
CI / publish-chart (pull_request) Successful in 10m17s
Details
CI / check-seshat (pull_request) Successful in 11m4s
Details
CI / check-seshat (push) Successful in 11m7s
Details
CI / docker-seshd (push) Successful in 17m3s
Details
CI / docker-seshd (pull_request) Successful in 17m7s
Details
2026-07-29 18:50:29 +00:00 Compare
Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible
Some checks failed
CI / publish-chart (pull_request) Successful in 12m21s
Details
CI / check-seshat (pull_request) Successful in 13m56s
Details
CI / docker-seshd (pull_request) Successful in 18m6s
Details
CI / check-seshat (push) Failing after 22m19s
Details
CI / publish-chart (push) Successful in 9m49s
Details
CI / docker-seshd (push) Successful in 23m0s
Details
a5a249e519 
Anima implementation session.
toasterson commented 2026-08-14 10:38:13 +00:00
Author
Owner
Copy link

Closing. This branch carried a CHANGES-REQUESTED verdict (2026-07-31) for a stale base, pollution, and an unjustified 15-file blast radius, and it now has 7 conflicts against main. The follow-up rescue branch claude/wi-019fa44f-rebased-onto-main (594efd1) is itself 52 commits behind and conflicts in 5 files — src/bin/akhomed.rs alone has taken 9 commits since, including the ADR-049 advisory-lock work that restructured ServerState and the lock acquire/release paths.

The conflict is semantic, not textual: main has independently changed the lock-adjacent code this branch exists to restructure. A third rebase would be a third attempt to resolve a design question inside a merge tool. WI-259 is routed to working for a fresh implementation on current main, with the design conclusions worth keeping written up in the work item thread.

Closing. This branch carried a CHANGES-REQUESTED verdict (2026-07-31) for a stale base, pollution, and an unjustified 15-file blast radius, and it now has 7 conflicts against main. The follow-up rescue branch `claude/wi-019fa44f-rebased-onto-main` (`594efd1`) is itself 52 commits behind and conflicts in 5 files — `src/bin/akhomed.rs` alone has taken 9 commits since, including the ADR-049 advisory-lock work that restructured `ServerState` and the lock acquire/release paths. The conflict is semantic, not textual: main has independently changed the lock-adjacent code this branch exists to restructure. A third rebase would be a third attempt to resolve a design question inside a merge tool. WI-259 is routed to `working` for a fresh implementation on current main, with the design conclusions worth keeping written up in the work item thread.
toasterson closed this pull request 2026-08-14 10:38:17 +00:00
Some checks failed
CI / publish-chart (pull_request) Successful in 12m21s
Details
CI / check-seshat (pull_request) Successful in 13m56s
Details
CI / docker-seshd (pull_request) Successful in 18m6s
Details
CI / check-seshat (push) Failing after 22m19s
Details
CI / publish-chart (push) Successful in 9m49s
Details
CI / docker-seshd (push) Successful in 23m0s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
toasterson/akh-medu!277
No description provided.