toasterson/Anima

Watch

Fork 0

fix(ci): unstick the native Solstice lane — protoc too old for proto3 optional #369

Merged

toasterson merged 1 commit from claude/fix-native-ci-protoc into main

2026-07-25 18:37:46 +00:00

toasterson commented

2026-07-25 14:24:54 +00:00

Owner

The native CI lane has been red for two weeks and nobody saw it

runs_on=rust — the native Solstice dispatch path, the one the machine exit bar consumes — last went green on 2026-07-11 20:56. Since 2026-07-20 it has run 36 jobs, 0 successes. Each burns ~40 minutes of a CI VM before failing.

Root cause

.solstice/job.sh installed protobuf-compiler from jammy, which is protoc 3.12. protoc rejects proto3 optional fields before 3.15 unless --experimental_allow_proto3_optional is passed:

protoc failed: anima/v1/thread.proto: This file contains proto3 optional
fields, but --experimental_allow_proto3_optional was not set.

Commit 5c1dd37 (2026-07-11) added optional string project_id = 1; to thread.proto. The native lane's last green run was the same day. Exact correlation.

The Forgejo Actions lane never broke, because its ubuntu-22.04 image carries a newer protoc. So the outage was invisible on the path everyone watches, and the green Actions badge kept saying CI was fine.

The fix

Install a pinned protoc release into /usr/local (binary plus the well-known .proto include tree) when the detected protoc is older than 3.15, and export PROTOC so prost-build cannot fall back to the distro binary at /usr/bin/protoc.

Passing --experimental_allow_proto3_optional would also work, but it means threading a flag through every prost-build call site and keeping it there forever; a current protoc is the smaller standing cost.

The version probe treats missing or malformed protoc --version output as "install" rather than "skip", so it degrades toward doing the work. Verified against 3.12.4 → install, 27.3 → skip, empty → install, malformed → install.

How this was found

Not from the CI badge — from job_logs in solstice_prod directly. jobs.failure_summary for these runs contains only job script failed: exit_code=101, with no log tail, so the failure carries no diagnostic content anywhere it is surfaced. And solstice-mcp list_runs(owner, repo) cannot see these jobs at all because native-dispatch rows leave repo_owner/repo_name NULL.

Both of those are filed separately against the new solstice-ci Anima project:

Native rows write NULL attribution and a mislabelled source — 019f998b-be04-7f70-8138-27774a142d85
Jobs table never reconciles against live VMs (96 running rows vs 5 real VMs) — 019f998b-9cea-7bf3-87c4-c6a6ad24f0ef

Verification status

Not yet proven end to end. This change only takes effect on a native rust dispatch, which is triggered by an agent branch push, not by opening this PR — the Actions lane that will run on this PR is the one that was already passing. The fix should be confirmed by watching the first native job after merge, not by this PR going green.

🤖 Generated with Claude Code

## The native CI lane has been red for two weeks and nobody saw it `runs_on=rust` — the native Solstice dispatch path, the one the machine exit bar consumes — last went green on **2026-07-11 20:56**. Since 2026-07-20 it has run **36 jobs, 0 successes**. Each burns ~40 minutes of a CI VM before failing. ## Root cause `.solstice/job.sh` installed `protobuf-compiler` from jammy, which is protoc **3.12**. protoc rejects proto3 `optional` fields before 3.15 unless `--experimental_allow_proto3_optional` is passed: ``` protoc failed: anima/v1/thread.proto: This file contains proto3 optional fields, but --experimental_allow_proto3_optional was not set. ``` Commit `5c1dd37` (2026-07-11) added `optional string project_id = 1;` to `thread.proto`. The native lane's last green run was the same day. Exact correlation. The Forgejo Actions lane never broke, because its `ubuntu-22.04` image carries a newer protoc. So the outage was invisible on the path everyone watches, and the green Actions badge kept saying CI was fine. ## The fix Install a pinned protoc release into `/usr/local` (binary plus the well-known `.proto` include tree) when the detected protoc is older than 3.15, and export `PROTOC` so `prost-build` cannot fall back to the distro binary at `/usr/bin/protoc`. Passing `--experimental_allow_proto3_optional` would also work, but it means threading a flag through every `prost-build` call site and keeping it there forever; a current protoc is the smaller standing cost. The version probe treats missing or malformed `protoc --version` output as "install" rather than "skip", so it degrades toward doing the work. Verified against `3.12.4` → install, `27.3` → skip, empty → install, malformed → install. ## How this was found Not from the CI badge — from `job_logs` in `solstice_prod` directly. `jobs.failure_summary` for these runs contains only `job script failed: exit_code=101`, with no log tail, so the failure carries no diagnostic content anywhere it is surfaced. And `solstice-mcp list_runs(owner, repo)` cannot see these jobs at all because native-dispatch rows leave `repo_owner`/`repo_name` NULL. Both of those are filed separately against the new `solstice-ci` Anima project: - Native rows write NULL attribution and a mislabelled `source` — `019f998b-be04-7f70-8138-27774a142d85` - Jobs table never reconciles against live VMs (96 `running` rows vs 5 real VMs) — `019f998b-9cea-7bf3-87c4-c6a6ad24f0ef` ## Verification status **Not yet proven end to end.** This change only takes effect on a native `rust` dispatch, which is triggered by an agent branch push, not by opening this PR — the Actions lane that will run on this PR is the one that was already passing. The fix should be confirmed by watching the first native job after merge, not by this PR going green. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

toasterson added 1 commit

2026-07-25 14:24:54 +00:00

fix(ci): install a protoc that accepts proto3 optional on the native lane 8d371aabfa

Anima's native Solstice CI (`runs_on=rust`) has been red since 2026-07-11
and had 36 consecutive failures with zero successes after 2026-07-20.

Root cause: `.solstice/job.sh` installed protoc from jammy, which is 3.12.
protoc rejects proto3 `optional` fields before 3.15 unless
--experimental_allow_proto3_optional is passed:

  protoc failed: anima/v1/thread.proto: This file contains proto3 optional
  fields, but --experimental_allow_proto3_optional was not set.

Commit 5c1dd37 (2026-07-11) added `optional string project_id = 1` to
thread.proto. The native lane's last green run was 2026-07-11 20:56 — the
same day. The Forgejo Actions lane never broke because its ubuntu-22.04
image carries a newer protoc, so the outage was invisible on the path
everyone was watching.

Install a pinned protoc release into /usr/local (binary plus the
well-known .proto include tree) when the detected protoc is older than
3.15, and export PROTOC so prost-build cannot fall back to the distro
binary. The version probe treats missing/malformed output as "install",
so it degrades toward doing the work rather than skipping it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

toasterson commented

2026-07-25 14:44:11 +00:00

Author

Owner

Verification update — mechanism proven, install path still pending

The PR description said this was unproven. Part of it no longer is.

The protoc version theory is now confirmed directly, not inferred from the error message:

$ protoc --version
libprotoc 35.1

$ protoc -I proto --descriptor_set_out=/dev/null proto/anima/v1/thread.proto
$ echo $?
0

A modern protoc compiles thread.proto — including optional string project_id = 1; at line 74 — cleanly, with no --experimental_allow_proto3_optional. That is the exact file and the exact field the CI failure names, so the diagnosis holds: the only difference between red and green here is the protoc version.

The download path is also verified:

https://github.com/protocolbuffers/protobuf/releases/download/v27.3/protoc-27.3-linux-x86_64.zip → HTTP 200
Archive layout is what the script assumes: bin/protoc, plus include/google/protobuf/field_mask.proto and include/google/protobuf/timestamp.proto — the two well-known imports named in the build warnings. Unzipping to /usr/local therefore lands the binary on PATH ahead of /usr/bin/protoc and the includes where protoc looks for them.

What remains unproven is narrower than before: whether the install actually executes correctly inside the CI VM — sudo/unzip availability, PATH ordering in the runner's non-login shell, and prost-build honouring the exported PROTOC. Those only exercise on a native rust dispatch, which this PR does not trigger. Confirm on the first native job after merge.

Note that unzip was added to the apt line in this change; without it the install step would fail rather than silently skip, which is the intended direction.

## Verification update — mechanism proven, install path still pending The PR description said this was unproven. Part of it no longer is. **The protoc version theory is now confirmed directly**, not inferred from the error message: ``` $ protoc --version libprotoc 35.1 $ protoc -I proto --descriptor_set_out=/dev/null proto/anima/v1/thread.proto $ echo $? 0 ``` A modern protoc compiles `thread.proto` — including `optional string project_id = 1;` at line 74 — cleanly, with no `--experimental_allow_proto3_optional`. That is the exact file and the exact field the CI failure names, so the diagnosis holds: the only difference between red and green here is the protoc version. **The download path is also verified:** - `https://github.com/protocolbuffers/protobuf/releases/download/v27.3/protoc-27.3-linux-x86_64.zip` → HTTP 200 - Archive layout is what the script assumes: `bin/protoc`, plus `include/google/protobuf/field_mask.proto` and `include/google/protobuf/timestamp.proto` — the two well-known imports named in the build warnings. Unzipping to `/usr/local` therefore lands the binary on PATH ahead of `/usr/bin/protoc` and the includes where protoc looks for them. **What remains unproven** is narrower than before: whether the install actually executes correctly inside the CI VM — sudo/unzip availability, PATH ordering in the runner's non-login shell, and `prost-build` honouring the exported `PROTOC`. Those only exercise on a native `rust` dispatch, which this PR does not trigger. Confirm on the first native job after merge. Note that `unzip` was added to the apt line in this change; without it the install step would fail rather than silently skip, which is the intended direction.

toasterson commented

2026-07-25 14:50:33 +00:00

Author

Owner

Impact update: this bug is not just red CI, it is actively destroying work items

Measured on solstice_prod, last 24 hours:

failed_anima_rust_24h  = 24
of_which_protoc        = 23

23 of 24 native CI failures for this repo in the last day are this exact protoc error. That is not background noise — it is the near-totality of the lane.

The chain

protoc 3.12 fails every native build with exit_code=101.
anima_server::solstice reads that as a genuine CI-red and re-dispatches the work item with corrective context — telling an implementer to fix a build failure that has nothing to do with their change.
The CI-red counter reaches cap=3.
auto-parked: CI-red failed dispatch cap reached — the work item leaves the dispatch queue.

What it has cost

Seven work items auto-parked in the last 24 hours, five of them Anima:

parked	work item
11:06	WI-B: Executor token scope stripping
11:13	WI-C: Runner interceptor + checkpoint
12:19	WI-A: Backend escalation core
13:22	Reopen work items when their orphaned sessions…
14:40	WI-D: Frontend permission card

WI-A through WI-D are the entire ACP escalation slice. All four parked today, none for a reason connected to their own code. Every one of them also burned agent sessions first (one ran 2200s before failing), chasing a phantom.

The park is silent by design — it looks like a work item that legitimately gave up, and it is indistinguishable from one.

Follow-up after merge

Merging this stops the bleeding but does not undo it. The parked items need reopening once a native run goes green — they were parked on false evidence and their failed_dispatch_count should be reset, not merely decremented. Recommend confirming one green native job first, then unparking as a batch.

Worth noting how this compounds with the two lane-visibility gaps already filed: the Actions lane stayed green throughout, so from every dashboard this looked like a healthy repo steadily parking work for its own reasons.

## Impact update: this bug is not just red CI, it is actively destroying work items Measured on `solstice_prod`, last 24 hours: ``` failed_anima_rust_24h = 24 of_which_protoc = 23 ``` **23 of 24** native CI failures for this repo in the last day are this exact protoc error. That is not background noise — it is the near-totality of the lane. ### The chain 1. protoc 3.12 fails every native build with `exit_code=101`. 2. `anima_server::solstice` reads that as a genuine CI-red and re-dispatches the work item with corrective context — telling an implementer to fix a build failure that has nothing to do with their change. 3. The CI-red counter reaches `cap=3`. 4. `auto-parked: CI-red failed dispatch cap reached` — the work item leaves the dispatch queue. ### What it has cost Seven work items auto-parked in the last 24 hours, five of them Anima: | parked | work item | |---|---| | 11:06 | WI-B: Executor token scope stripping | | 11:13 | WI-C: Runner interceptor + checkpoint | | 12:19 | WI-A: Backend escalation core | | 13:22 | Reopen work items when their orphaned sessions… | | 14:40 | WI-D: Frontend permission card | WI-A through WI-D are **the entire ACP escalation slice**. All four parked today, none for a reason connected to their own code. Every one of them also burned agent sessions first (one ran 2200s before failing), chasing a phantom. The park is silent by design — it looks like a work item that legitimately gave up, and it is indistinguishable from one. ### Follow-up after merge Merging this stops the bleeding but does not undo it. The parked items need reopening once a native run goes green — they were parked on false evidence and their `failed_dispatch_count` should be reset, not merely decremented. Recommend confirming one green native job first, then unparking as a batch. Worth noting how this compounds with the two lane-visibility gaps already filed: the Actions lane stayed green throughout, so from every dashboard this looked like a healthy repo steadily parking work for its own reasons.

toasterson merged commit 17ffe87d41 into main

2026-07-25 18:37:46 +00:00

toasterson deleted branch claude/fix-native-ci-protoc

2026-07-25 18:37:48 +00:00

toasterson referenced this pull request from a commit

2026-07-25 18:37:48 +00:00

Merge pull request #369 'fix(ci): unstick the native Solstice lane — protoc too old for proto3 optional' from claude/fix-native-ci-protoc into main

toasterson referenced this pull request from a commit

2026-07-25 18:55:35 +00:00

fix(lint): add the two missing clippy allows blocking native CI

toasterson referenced this pull request

2026-07-25 18:55:55 +00:00