Native CI falls back to a job script that need not exist, then fails 100% silently (Akh-Medu is dark) #373

Merged

toasterson merged 2 commits from claude/wi-019f99ca-native-ci-falls-back-to-a-job-script-tha into main

2026-07-28 10:26:48 +00:00

toasterson commented

2026-07-25 19:42:19 +00:00

Owner

Anima work item 019f99ca-c3df-7f42-ba63-a2261fd41f1b.

When a project has default_ci_steps unset, native dispatch falls back to running .solstice/job.sh from the repo. Nothing checks that the file exists. If it does not, every native CI job fails immediately with job script not found at <workdir>/.solstice/job.sh.

Akh-Medu is in exactly this state

projects row for Akh-Medu has default_runs_on = NULL and default_ci_steps = NULL, and the repo has no .solstice/ directory at all. Measured on solstice_prod for the last 24 hours:

dispatch path	succeeded	failed
Forgejo Actions (`repo_owner` populated)	141	12
native (NULL attribution)	0	8

The Actions lane is at 92% success. The native lane — the one the machine exit bar consumes — is at 100% failure, and has never produced a single success. From every dashboard Akh-Medu looks like a healthy project, because the green lane is the visible one.

This is the second confirmed instance of the same shape. Anima's native lane was 100% red for two weeks on a protoc bug (PR #369) with an equally green Actions badge. Two projects, two totally dead native lanes, both invisible. Akh-Medu is the more expensive one: it carries 43 open work items with tecton working them continuously.

Why this is a code defect, not just misconfiguration

Configuring Akh-Medu fixes one project. The defect is that the system:

silently selects a fallback script without verifying it exists,
reports its absence as an ordinary job failure (exit_code=1) indistinguishable from a failing test,
feeds that into the CI-red corrective loop, which re-prompts an implementer to fix a build that never ran, and
eventually auto-parks the work item.

A project whose CI cannot possibly run should say so at configuration time, not once per dispatch, forever.

Required

Validate at write time. Reject or warn when a project's CI configuration cannot resolve — no default_ci_steps and no .solstice/job.sh on the default branch means CI can never run. Surface it on the project rather than discovering it per job.
Classify a missing job script as a configuration failure, not a job failure. It must not increment failed_dispatch_count and must not re-prompt an implementer. This is the same classification work as 019f9698; the missing-script case belongs in its non-code bucket.
Surface per-lane health. A lane with zero successes ever is a broken lane and must be visible as one incident, not as N independently failing work items.

Separately: decide Akh-Medu's CI configuration

This is a judgement call for the owner, not part of the code fix. Note the orchestrator image map records that Akh-Medu's release build OOMs at 4 GB and wants >=25 GB disk, and its Actions workflow targets the illumos label — so the native lane's default_runs_on should probably match that rather than defaulting to something smaller. Do not set this blind.

Acceptance criteria

Creating or updating a project with unresolvable CI configuration produces an immediate, legible error or warning.
A native job that fails because the job script is absent does not park the work item and does not re-prompt an implementer.
Querying "is this project's CI able to run?" has an answer that does not require reading solstice_prod.
A lane with zero successes over a meaningful window raises one alert.

019f9698 — CI-red corrective loop must distinguish infra/toolchain failures from job failures (critical).
019f9980 — pin CI verdicts to the commit under test; surface native vs Actions lane health separately.
019f99ac — create_project leaves projects undispatchable and silent about it. Same family: configuration that is absent rather than wrong, failing invisibly.
PR #369 — the Anima instance of a 100%-red native lane behind a green Actions badge.

Anima work item `019f99ca-c3df-7f42-ba63-a2261fd41f1b`. When a project has `default_ci_steps` unset, native dispatch falls back to running `.solstice/job.sh` from the repo. Nothing checks that the file exists. If it does not, every native CI job fails immediately with `job script not found at <workdir>/.solstice/job.sh`. ## Akh-Medu is in exactly this state `projects` row for Akh-Medu has `default_runs_on = NULL` and `default_ci_steps = NULL`, and the repo has no `.solstice/` directory at all. Measured on `solstice_prod` for the last 24 hours: | dispatch path | succeeded | failed | |---|---|---| | Forgejo Actions (`repo_owner` populated) | 141 | 12 | | native (NULL attribution) | **0** | **8** | The Actions lane is at 92% success. The native lane — the one the machine exit bar consumes — is at **100% failure, and has never produced a single success**. From every dashboard Akh-Medu looks like a healthy project, because the green lane is the visible one. This is the second confirmed instance of the same shape. Anima's native lane was 100% red for two weeks on a protoc bug (PR #369) with an equally green Actions badge. Two projects, two totally dead native lanes, both invisible. Akh-Medu is the more expensive one: it carries 43 open work items with tecton working them continuously. ## Why this is a code defect, not just misconfiguration Configuring Akh-Medu fixes one project. The defect is that the system: - silently selects a fallback script without verifying it exists, - reports its absence as an ordinary job failure (`exit_code=1`) indistinguishable from a failing test, - feeds that into the CI-red corrective loop, which re-prompts an implementer to fix a build that never ran, and - eventually auto-parks the work item. A project whose CI cannot possibly run should say so at configuration time, not once per dispatch, forever. ## Required 1. **Validate at write time.** Reject or warn when a project's CI configuration cannot resolve — no `default_ci_steps` and no `.solstice/job.sh` on the default branch means CI can never run. Surface it on the project rather than discovering it per job. 2. **Classify a missing job script as a configuration failure**, not a job failure. It must not increment `failed_dispatch_count` and must not re-prompt an implementer. This is the same classification work as `019f9698`; the missing-script case belongs in its non-code bucket. 3. **Surface per-lane health.** A lane with zero successes ever is a broken lane and must be visible as one incident, not as N independently failing work items. ## Separately: decide Akh-Medu's CI configuration This is a judgement call for the owner, not part of the code fix. Note the orchestrator image map records that Akh-Medu's release build OOMs at 4 GB and wants >=25 GB disk, and its Actions workflow targets the `illumos` label — so the native lane's `default_runs_on` should probably match that rather than defaulting to something smaller. Do not set this blind. ## Acceptance criteria - Creating or updating a project with unresolvable CI configuration produces an immediate, legible error or warning. - A native job that fails because the job script is absent does not park the work item and does not re-prompt an implementer. - Querying "is this project's CI able to run?" has an answer that does not require reading `solstice_prod`. - A lane with zero successes over a meaningful window raises one alert. ## Related - `019f9698` — CI-red corrective loop must distinguish infra/toolchain failures from job failures (critical). - `019f9980` — pin CI verdicts to the commit under test; surface native vs Actions lane health separately. - `019f99ac` — `create_project` leaves projects undispatchable and silent about it. Same family: configuration that is absent rather than wrong, failing invisibly. - PR #369 — the Anima instance of a 100%-red native lane behind a green Actions badge.

toasterson added 1 commit

2026-07-25 19:42:20 +00:00

Native CI falls back to a job script that need not exist, then fails 100% silently (Akh-Medu is dark) 62446111a3

Anima implementation session.

toasterson force-pushed claude/wi-019f99ca-native-ci-falls-back-to-a-job-script-tha from 62446111a3 to ab008a16e6

2026-07-25 20:21:04 +00:00

Compare

toasterson force-pushed claude/wi-019f99ca-native-ci-falls-back-to-a-job-script-tha from ab008a16e6 to aeed02d29d

2026-07-26 05:46:25 +00:00

Compare

toasterson force-pushed claude/wi-019f99ca-native-ci-falls-back-to-a-job-script-tha from aeed02d29d to 940f3c8eba

2026-07-28 08:09:10 +00:00

Compare

toasterson added 1 commit

2026-07-28 08:31:46 +00:00

Native CI falls back to a job script that need not exist, then fails 100% silently (Akh-Medu is dark) 2856527918

Anima implementation session.

toasterson changed title from ~~WIP: Native CI falls back to a job script that need not exist, then fails 100% silently (Akh-Medu is dark)~~ to Native CI falls back to a job script that need not exist, then fails 100% silently (Akh-Medu is dark)

2026-07-28 08:56:02 +00:00

toasterson force-pushed claude/wi-019f99ca-native-ci-falls-back-to-a-job-script-tha from 2856527918 to 868022e832

2026-07-28 10:26:16 +00:00

Compare

toasterson merged commit bf360cfc0c into main

2026-07-28 10:26:48 +00:00

toasterson referenced this pull request from a commit

2026-07-28 10:26:50 +00:00

Merge pull request 'Native CI falls back to a job script that need not exist, then fails 100% silently (Akh-Medu is dark)' (#373) from claude/wi-019f99ca-native-ci-falls-back-to-a-job-script-tha into main