CI: build the akh-medu crate, and stop running every job twice #296

Merged

toasterson merged 4 commits from claude/wi-019fdd21-ci-never-builds-the-akh-medu-crate-it-on into main 2026-08-08 13:40:04 +00:00

Conversation 0 Commits 4 Files changed 2 +132 -13

toasterson commented 2026-08-07 18:40:19 +00:00

Owner

Copy link

Anima work item 019fdd21-7c12-7500-84ef-5f7278c2af15.

The problem

.forgejo/workflows/ci.yml runs exactly one build command: cargo check -p seshat. seshat is a sibling crate. The root akh-medu crate — where src/bin/akhomed.rs, src/acp/, src/agent/, src/message.rs and essentially all agent work lives — is never compiled by CI at all, and no tests are ever run.

The workflow's own comment documents this as a deliberate OOM workaround (the full build exhausted the runner), so this is a known trade-off that was never revisited. It has now become the single most expensive defect in the pipeline.

Why this is critical

Every akh implementer session treats "CI green" as its correctness signal, and the ci_green completion condition gates the exit bar. Both are lying. Verified on 2026-08-07 across one review wave:

• PR #294 (296ae33f) — CI reported green on the push run. cargo check --features server locally fails with two hard errors (E0277 and E0283) in the file the PR adds. The work item's thread shows roughly 15 sessions each declaring "Implementation Verification Complete" after reading the file, never once building it.
• PR #295 (bf8377aa) — Forgejo runs #7580 and #7581 both report success. Locally the crate fails with 7 errors (E0027 ×4, E0063 ×3) because a new struct field was added without updating its match and construction sites.
• PR #291 (74010c26) — merged, and genuinely fine, but its round-1 blocked revision 63a6df26 was equally green while the entire feature was dead, unregistered code. CI could not tell the difference.

An agent that cannot compile its own work, and whose CI will not compile it either, has no feedback loop. That is the root cause behind several multi-round session churns on this board, not agent incompetence.

What to build

Restore a real compile signal for the root crate without reintroducing the OOM:

1. Add a CI step that builds the root crate — at minimum cargo check --features server, ideally cargo clippy --features server -- -D warnings.
2. Mitigate the memory pressure that caused the original workaround rather than dodging it. Options, roughly in order of preference:
   • split the build into separate jobs/steps per crate or feature set so no single step holds the peak;
   • constrain parallelism and codegen memory (CARGO_BUILD_JOBS, -C codegen-units, CARGO_PROFILE_DEV_DEBUG=0 — debug info is a large share of peak RSS for this workspace);
   • lean on sccache, which is already the global rustc wrapper on the build hosts, so dependency compilation is cached across runs;
   • if the runner is simply too small, move this job to the archibald builder, which already builds the akhomed image.
3. Add a test run (cargo test --features server) if it fits the memory budget. Note that on developer machines the --features server --lib link step has repeatedly hit ENOSPC because the binary is enormous — budget disk on the runner accordingly, and prefer --features postgres for lib tests if that reproduces the same coverage more cheaply.
4. Make the ci_green completion condition depend on the step that actually builds the root crate, so a green exit bar means something.

Acceptance

• A pull request that fails cargo check --features server is reported red by CI. Verify this directly by pushing one of the two known-bad SHAs above (296ae33f or bf8377aa) to a scratch branch and confirming CI fails on it.
• A clean pull request is still green, and total CI wall-clock stays within roughly the current envelope (about 25 minutes).
• No OOM on the runner across at least three consecutive runs.
• The workflow comment explaining the seshat-only restriction is removed or rewritten to describe whatever the new arrangement is.

Anima work item `019fdd21-7c12-7500-84ef-5f7278c2af15`. ## The problem `.forgejo/workflows/ci.yml` runs exactly one build command: `cargo check -p seshat`. `seshat` is a sibling crate. The root `akh-medu` crate — where `src/bin/akhomed.rs`, `src/acp/`, `src/agent/`, `src/message.rs` and essentially all agent work lives — is **never compiled by CI at all**, and no tests are ever run. The workflow's own comment documents this as a deliberate OOM workaround (the full build exhausted the runner), so this is a known trade-off that was never revisited. It has now become the single most expensive defect in the pipeline. ## Why this is critical Every akh implementer session treats "CI green" as its correctness signal, and the `ci_green` completion condition gates the exit bar. Both are lying. Verified on 2026-08-07 across one review wave: - **PR #294** (`296ae33f`) — CI reported green on the push run. `cargo check --features server` locally fails with two hard errors (E0277 and E0283) in the file the PR adds. The work item's thread shows roughly 15 sessions each declaring "Implementation Verification Complete" after *reading* the file, never once building it. - **PR #295** (`bf8377aa`) — Forgejo runs #7580 and #7581 both report success. Locally the crate fails with 7 errors (E0027 ×4, E0063 ×3) because a new struct field was added without updating its match and construction sites. - **PR #291** (`74010c26`) — merged, and genuinely fine, but its *round-1* blocked revision `63a6df26` was equally green while the entire feature was dead, unregistered code. CI could not tell the difference. An agent that cannot compile its own work, and whose CI will not compile it either, has no feedback loop. That is the root cause behind several multi-round session churns on this board, not agent incompetence. ## What to build Restore a real compile signal for the root crate without reintroducing the OOM: 1. Add a CI step that builds the root crate — at minimum `cargo check --features server`, ideally `cargo clippy --features server -- -D warnings`. 2. Mitigate the memory pressure that caused the original workaround rather than dodging it. Options, roughly in order of preference: - split the build into separate jobs/steps per crate or feature set so no single step holds the peak; - constrain parallelism and codegen memory (`CARGO_BUILD_JOBS`, `-C codegen-units`, `CARGO_PROFILE_DEV_DEBUG=0` — debug info is a large share of peak RSS for this workspace); - lean on `sccache`, which is already the global rustc wrapper on the build hosts, so dependency compilation is cached across runs; - if the runner is simply too small, move this job to the archibald builder, which already builds the akhomed image. 3. Add a test run (`cargo test --features server`) if it fits the memory budget. Note that on developer machines the `--features server --lib` **link** step has repeatedly hit ENOSPC because the binary is enormous — budget disk on the runner accordingly, and prefer `--features postgres` for lib tests if that reproduces the same coverage more cheaply. 4. Make the `ci_green` completion condition depend on the step that actually builds the root crate, so a green exit bar means something. ## Acceptance - A pull request that fails `cargo check --features server` is reported red by CI. Verify this directly by pushing one of the two known-bad SHAs above (`296ae33f` or `bf8377aa`) to a scratch branch and confirming CI fails on it. - A clean pull request is still green, and total CI wall-clock stays within roughly the current envelope (about 25 minutes). - No OOM on the runner across at least three consecutive runs. - The workflow comment explaining the seshat-only restriction is removed or rewritten to describe whatever the new arrangement is.

toasterson added 1 commit 2026-08-07 18:40:25 +00:00

CI never builds the akh-medu crate — it only runs cargo check -p seshat, so "CI green" is not evidence the code compiles ... 45fd70b15b

Anima implementation session.

toasterson added 1 commit 2026-08-07 19:51:29 +00:00

CI never builds the akh-medu crate — it only runs cargo check -p seshat, so "CI green" is not evidence the code compiles ... 0b643a8530

Anima implementation session.

toasterson added 1 commit 2026-08-08 11:16:53 +00:00

Make root CI signal reflect compilation 96b255d3b5

toasterson added 1 commit 2026-08-08 13:28:51 +00:00

ci: stop running every job twice (push + pull_request on the same commit) ... 1762449b72

The widened CI from this branch works: on 96b255d, check-akh-medu (push)
succeeded in 38m33s and test-akh-medu (push) succeeded in 38m32s on 0b643a8.
The akh-medu crate does check and its lib tests do pass on the Solstice
runner — the original 'the workspace OOMs the runner' premise held for a
*release* build, not for cargo check, which does no codegen and no linking.

What was failing was the duplicate. 'on: [push, pull_request]' fires both
events for every push to a PR branch, so each commit started two identical
copies of every job — ten concurrent runs per commit with five jobs, all
contending for one 16-core host that also runs Forgejo, the akh daemons and
the Anima runners. Same commit, same job, two outcomes:

  check-akh-medu (push)          success   38m33s
  check-akh-medu (pull_request)  failure  1h3m42s

Durations climb monotonically with contention across the branch:
18m -> 20m -> 32m -> 38m -> 49m -> 1h3m -> 1h18m. Those timeouts are what
feeds Anima's CI-red loop, which parks a work item after three strikes.

Each event now belongs to exactly one lane: feature branches run under
pull_request (Anima opens a PR per work item and ci_green binds to the PR
head); main/develop run under push, which is the lane that publishes the
seshd image and the Helm chart. No behaviour is lost and no job stops
covering what it covered.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

toasterson changed title from WIP: CI never builds the akh-medu crate — it only runs cargo check -p seshat, so "CI green" is not evidence the code compiles to CI: build the akh-medu crate, and stop running every job twice 2026-08-08 13:39:20 +00:00

toasterson merged commit d00d5af118 into main 2026-08-08 13:40:04 +00:00

toasterson referenced this pull request from a commit 2026-08-08 13:40:06 +00:00

Merge pull request 'CI: build the akh-medu crate, and stop running every job twice' (#296) from claude/wi-019fdd21-ci-never-builds-the-akh-medu-crate-it-on into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

toasterson/akh-medu!296

No description provided.