Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible #318

Merged

toasterson merged 19 commits from claude/wi-019fa44f-sessions-on-one-akh-still-serialise-on-t into main 2026-08-18 15:02:50 +00:00

Conversation 0 Commits 19 Files changed 12 +681 -296

toasterson commented 2026-08-15 21:41:55 +00:00

Owner

Copy link

Anima work item 019fa44f-da1f-78e1-813d-1897f0ce5fd4.

Why this exists separately

akh-medu WI-221 ("Extract per-session SessionContext from ChatProcessor", merged 2026-07-27 as PR #237) removed the first of two serialisation points. It was scoped, correctly, to ChatProcessor — that is what its description asked for and that is what it delivered: kg_cache, trajectory and eliza moved into a per-session SessionContext, the RefCell-inside-Mutex anti-pattern removed, and the clone-modify-write per turn eliminated.

The second lock was never in scope. An operator believed they had extended WI-221 to cover it, but the channel used (description_revisions with author_id = 'operator') is filtered out before it reaches any agent — see the correction document anima-prompt-inventory-correction-20260727. Neither the implementer nor the reviewer ever saw the extension. This work item states it properly.

The remaining serialisation point

On origin/main:

src/bin/akhomed.rs:45     agents: Arc<RwLock<HashMap<String, Arc<Mutex<Agent>>>>>
src/bin/akhomed.rs:1826   shared_agent: Arc<Mutex<Agent>>
src/agent/daemon.rs:170   agent: Arc<Mutex<Agent>>   <-- background learning daemon

Every ACP turn takes the per-workspace Arc<Mutex<Agent>>, and the background learning daemon holds the same lock on its own schedule. So two sessions on one akh workspace cannot both make progress, regardless of WI-221.

Evidence this is real, not theoretical

2026-07-27 09:13:25 — three work items dispatched to tecton in the same second:

WI-24  implementation   2 events
WI-33  implementation  10 events   ← the one that got the lock
WI-37  implementation   4 events

WI-37's entire session:

09:13:25  session.started
09:13:26  acp.session_capabilities
09:18:26  "error[acp]: chat turn stalled: no progress for 300s"
09:18:26  acp.turn_completed {"stop_reason": "end_turn"}

No acp.agent_thought at all — the model never ran. Exactly 300s of nothing between handshake and watchdog. That is lock contention, not slow inference.

A second observation the same day is sharper: when two sessions ran concurrently (WI-37 and WI-33, 12:43-13:25), they did not cleanly serialise with one winning. Both crawled — WI-37 produced 27 events in 41 minutes and pushed nothing at all. Contention degrades both sessions rather than queueing them.

Required change

1. Determine what the Agent mutex actually protects during a turn. Separate genuinely-shared workspace state (the engine, the KG) from per-turn state that belongs alongside SessionContext.
2. Make the shared remainder safe for concurrent access. Do not simply widen the critical section, and do not swap Mutex for RwLock without checking that turns are read-mostly — measure rather than assume.
3. The background learning daemon must not be able to block a live ACP turn for an unbounded time. Either it takes the lock in short bounded slices, or it yields to session work.

Acceptance

• Two ACP sessions on the SAME workspace both make progress concurrently. Start two prompt turns and assert interleaved activity from both. A test that runs them sequentially and passes proves nothing — that is today's behaviour and it is what this item exists to change.
• A turn's progress is not blocked by a background learning tick; assert a turn proceeds while the daemon is active.
• Per-session state does not bleed: two concurrent sessions must not observe each other's kg_cache / trajectory / eliza. (WI-221 established this; keep it.)
• After this lands, the akh gateway can honestly advertise per-workspace concurrency > 1 — which is the precondition for Anima WI-370.

Why it matters now

tecton is being run at WIP 1 by hand because of this. The target architecture (2026-07-27) puts throughput management in the runners and expects executors to advertise real capacity; an akh that can only ever run one turn at a time caps that at 1 per workspace. This is the blocker between the current fleet and per-akh parallelism.

---

REVIEW — CHANGES REQUESTED (2026-07-31, PR #277 @ stale)

Rebase-and-reimplement on current main. The branch cannot land as it stands.

• mergeable: false, last touched 2026-07-29 19:48 — two days stale.
• It predates akh-medu #276 (302fb347) and #279 (0f9f1261), both now on
  main, which rewrote src/acp/mod.rs and src/agent/agentic_chat.rs — two of
  the files this branch also edits. The conflict is structural, not textual.
• The diff carries pollution that is not this work item:
  WI-222-IMPLEMENTATION-SUMMARY.md (+163) and
  docs/ai/plans/2026-07-29-concurrent-acp-sessions.md (+194) at the repo root /
  docs tree. Neither belongs in a change whose scope is the per-workspace Agent
  mutex. Drop them.
• Blast radius is 15 files including ooda.rs, oxifed.rs, email/mod.rs and
  trigger.rs. Justify each file that is not on the lock path, or leave it alone.

What has changed underneath this item

The idle watchdog no longer kills a turn during tool-call generation: #279
added a client-side 10s heartbeat delivered through a recv_timeout select
(not from inside the blocking read loop), plus AKH_SOCKET_READ_TIMEOUT_SECS
(default 60s) separating socket death from application stall. Deployed as image
0fc0416b257d on 2026-07-31.

That matters here: the previous eleven attempts at this work item were killed
mid-generation by that bug, which is why the branch looks half-finished. Start
from current main and the turn should now survive long edits.

The acceptance criteria in the original brief stand unchanged — in particular
two ACP sessions on the SAME workspace must both make progress concurrently;
a sequential test proves nothing.

Verdict entered by hand on the primary DB — the operator PAT lacks
workitem.submit_review.

Anima work item `019fa44f-da1f-78e1-813d-1897f0ce5fd4`. ## Why this exists separately akh-medu WI-221 ("Extract per-session SessionContext from ChatProcessor", merged 2026-07-27 as PR #237) removed the **first** of two serialisation points. It was scoped, correctly, to `ChatProcessor` — that is what its description asked for and that is what it delivered: `kg_cache`, `trajectory` and `eliza` moved into a per-session `SessionContext`, the `RefCell`-inside-`Mutex` anti-pattern removed, and the clone-modify-write per turn eliminated. The **second** lock was never in scope. An operator believed they had extended WI-221 to cover it, but the channel used (`description_revisions` with `author_id = 'operator'`) is filtered out before it reaches any agent — see the correction document `anima-prompt-inventory-correction-20260727`. Neither the implementer nor the reviewer ever saw the extension. This work item states it properly. ## The remaining serialisation point On `origin/main`: src/bin/akhomed.rs:45 agents: Arc<RwLock<HashMap<String, Arc<Mutex<Agent>>>>> src/bin/akhomed.rs:1826 shared_agent: Arc<Mutex<Agent>> src/agent/daemon.rs:170 agent: Arc<Mutex<Agent>> <-- background learning daemon Every ACP turn takes the per-workspace `Arc<Mutex<Agent>>`, and the background learning daemon holds the **same** lock on its own schedule. So two sessions on one akh workspace cannot both make progress, regardless of WI-221. ## Evidence this is real, not theoretical 2026-07-27 09:13:25 — three work items dispatched to tecton in the same second: WI-24 implementation 2 events WI-33 implementation 10 events ← the one that got the lock WI-37 implementation 4 events WI-37's entire session: 09:13:25 session.started 09:13:26 acp.session_capabilities 09:18:26 "error[acp]: chat turn stalled: no progress for 300s" 09:18:26 acp.turn_completed {"stop_reason": "end_turn"} No `acp.agent_thought` at all — the model never ran. Exactly 300s of nothing between handshake and watchdog. That is lock contention, not slow inference. A second observation the same day is sharper: when two sessions ran concurrently (WI-37 and WI-33, 12:43-13:25), they did **not** cleanly serialise with one winning. Both crawled — WI-37 produced 27 events in 41 minutes and pushed nothing at all. Contention degrades both sessions rather than queueing them. ## Required change 1. Determine what the `Agent` mutex actually protects during a turn. Separate genuinely-shared workspace state (the engine, the KG) from per-turn state that belongs alongside `SessionContext`. 2. Make the shared remainder safe for concurrent access. Do not simply widen the critical section, and do not swap `Mutex` for `RwLock` without checking that turns are read-mostly — measure rather than assume. 3. The background learning daemon must not be able to block a live ACP turn for an unbounded time. Either it takes the lock in short bounded slices, or it yields to session work. ## Acceptance - **Two ACP sessions on the SAME workspace both make progress concurrently.** Start two prompt turns and assert interleaved activity from both. A test that runs them sequentially and passes proves nothing — that is today's behaviour and it is what this item exists to change. - A turn's progress is not blocked by a background learning tick; assert a turn proceeds while the daemon is active. - Per-session state does not bleed: two concurrent sessions must not observe each other's `kg_cache` / `trajectory` / `eliza`. (WI-221 established this; keep it.) - After this lands, the akh gateway can honestly advertise per-workspace concurrency > 1 — which is the precondition for Anima WI-370. ## Why it matters now tecton is being run at WIP 1 by hand because of this. The target architecture (2026-07-27) puts throughput management in the runners and expects executors to advertise real capacity; an akh that can only ever run one turn at a time caps that at 1 per workspace. This is the blocker between the current fleet and per-akh parallelism. --- # REVIEW — CHANGES REQUESTED (2026-07-31, PR #277 @ stale) **Rebase-and-reimplement on current `main`. The branch cannot land as it stands.** - `mergeable: false`, last touched **2026-07-29 19:48** — two days stale. - It predates akh-medu **#276** (`302fb347`) and **#279** (`0f9f1261`), both now on `main`, which rewrote `src/acp/mod.rs` and `src/agent/agentic_chat.rs` — two of the files this branch also edits. The conflict is structural, not textual. - The diff carries **pollution that is not this work item**: `WI-222-IMPLEMENTATION-SUMMARY.md` (+163) and `docs/ai/plans/2026-07-29-concurrent-acp-sessions.md` (+194) at the repo root / docs tree. Neither belongs in a change whose scope is the per-workspace `Agent` mutex. Drop them. - Blast radius is 15 files including `ooda.rs`, `oxifed.rs`, `email/mod.rs` and `trigger.rs`. Justify each file that is not on the lock path, or leave it alone. ## What has changed underneath this item The idle watchdog no longer kills a turn during tool-call generation: `#279` added a client-side 10s heartbeat delivered through a `recv_timeout` select (not from inside the blocking read loop), plus `AKH_SOCKET_READ_TIMEOUT_SECS` (default 60s) separating socket death from application stall. Deployed as image `0fc0416b257d` on 2026-07-31. That matters here: the previous eleven attempts at this work item were killed mid-generation by that bug, which is why the branch looks half-finished. Start from current `main` and the turn should now survive long edits. The acceptance criteria in the original brief stand unchanged — in particular **two ACP sessions on the SAME workspace must both make progress concurrently**; a sequential test proves nothing. *Verdict entered by hand on the primary DB — the operator PAT lacks `workitem.submit_review`.*

toasterson added 1 commit 2026-08-15 21:41:55 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... d704d2fd47

Anima implementation session.

toasterson added 1 commit 2026-08-15 22:16:34 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 69d59f7cbe

Anima implementation session.

toasterson added 1 commit 2026-08-16 01:17:00 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 54be9eaa2f

Anima implementation session.

toasterson added 1 commit 2026-08-16 01:51:36 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... d5c8580254

Anima implementation session.

toasterson added 1 commit 2026-08-16 04:09:21 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... f3c9e31ee9

Anima implementation session.

toasterson added 1 commit 2026-08-16 04:44:00 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 08c2c5f5ae

Anima implementation session.

toasterson added 1 commit 2026-08-16 12:38:11 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 421383d637

Anima implementation session.

toasterson added 1 commit 2026-08-16 14:23:56 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 8015607659

Anima implementation session.

toasterson force-pushed claude/wi-019fa44f-sessions-on-one-akh-still-serialise-on-t from 8015607659 to fcc77d19ef 2026-08-16 17:27:00 +00:00 Compare

toasterson added 1 commit 2026-08-17 16:27:44 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... bf97187d3b

Anima implementation session.

toasterson added 1 commit 2026-08-17 23:17:53 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 9d56f0c19b

Anima implementation session.

toasterson added 1 commit 2026-08-18 00:28:13 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 038e2c2bf1

Anima implementation session.

toasterson added 1 commit 2026-08-18 01:02:49 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 3da0bfa1ec

Anima implementation session.

toasterson added 1 commit 2026-08-18 01:37:29 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 767317ca96

Anima implementation session.

toasterson added 1 commit 2026-08-18 02:44:17 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 9e4385ca7d

Anima implementation session.

toasterson added 1 commit 2026-08-18 03:56:26 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 7e638f1d76

Anima implementation session.

toasterson added 1 commit 2026-08-18 06:24:26 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... 9eb9c338e5

Anima implementation session.

toasterson added 1 commit 2026-08-18 09:26:46 +00:00

Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible ... bc0d7378b3

Anima implementation session.

toasterson force-pushed claude/wi-019fa44f-sessions-on-one-akh-still-serialise-on-t from bc0d7378b3 to 5e1e4703cf 2026-08-18 14:16:14 +00:00 Compare

toasterson added 1 commit 2026-08-18 14:59:54 +00:00

WI-259: agentic path no longer holds the ChatProcessor mutex — two-session concurrency test ... 086cadc598

run_turn's spawn_blocking closure acquired processor.lock() unconditionally
right after agent.read(), even though the agentic path (run_agentic_turn)
never dereferences it — only the symbolic fallback does after upgrading to
agent.write(). That serialized every agentic-path turn on a per-workspace
mutex nobody in that path used, defeating the ADR-027 RwLock migration this
work item is about. Drop the unused acquisition (and its matching drop());
the fallback's own processor_g2 acquisition is unaffected.

Adds two_session_prompt_turns_on_one_workspace_progress_concurrently: two
real session/prompt turns over separate Unix-socket connections against the
SAME shared workspace Agent (via serve_unix_loop), driven concurrently with
tokio::join!. Both turns' model calls are forced through a fake ChatBackend
that blocks on a two-party Barrier, so the test can only pass if both turns'
run_agentic_turn calls are genuinely in flight at once — sequential
execution deadlocks and fails via the outer timeout instead of hanging. This
full transport-level test was attempted before the fix and reliably
deadlocked (documented at the old comment site); it now passes. The lock-only
two_threads_hold_agent_read_lock_concurrently test stays as a narrower
complement.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

toasterson changed title from WIP: Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible to Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible 2026-08-18 15:02:49 +00:00

toasterson merged commit 12073d1426 into main 2026-08-18 15:02:50 +00:00

toasterson referenced this pull request from a commit 2026-08-18 15:02:51 +00:00

Merge pull request 'Sessions on one akh still serialise on the per-workspace Agent mutex, so parallel ACP sessions remain impossible' (#318) from claude/wi-019fa44f-sessions-on-one-akh-still-serialise-on-t into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!318

No description provided.