Build the solstice rust CI image from a refraction-forger recipe, with a writable RUSTUP_HOME and clippy/rustfmt baked in #1

Merged

toasterson merged 2 commits from claude/wi-019faa04-build-the-solstice-rust-ci-image-from-a into main 2026-08-09 09:57:48 +00:00

Conversation 0 Commits 2 Files changed 3 +194

toasterson commented 2026-07-29 00:03:40 +00:00

Owner

Copy link

Anima work item 019faa04-868e-7f60-ae26-d7eab7114a3a.

Why now

The image behind solstice's runs_on: rust lane cannot install a pinned Rust toolchain. Every job that ships a rust-toolchain.toml dies here — reproduced twice on 2026-07-28, byte-identical, from the job's own log (job_logs, request 1a063681-97bc-4bbc-beee-ae6645af6ac6):

=== toolchain (pinned by rust-toolchain.toml) ===
/opt/rust/cargo/bin/cargo
info: syncing channel updates for 1.96.0-x86_64-unknown-linux-gnu
error: could not create temp root /opt/rust/rustup/tmp: Permission denied (os error 13)
2026-07-28T18:27:39Z ERROR workflow failed exit_code=1

RUSTUP_HOME is /opt/rust/rustup, owned by root, while the job runs as sol (workdir /home/sol/work). rustup resolves the pin, tries to download, cannot create its temp dir, and exits. This is not specific to 1.96 — no pinned toolchain can install, so the lane silently lints against whatever rustc the image happens to carry.

That blocks Anima WI-352 ("CI lints against an unpinned toolchain"), which has now been dispatched, diagnosed, rebased and parked at its retry cap three times against this one error.

What the live image actually is

/var/lib/solstice/images/solstice-rust.qcow2 — 5.4 GB, dated 2026-06-03, owned by solstice, hand-built and not reproducible from any recipe in git. Meanwhile refraction-forger already carries images/ubuntu-rust-ci.kdl, which is not what is running: it installs jammy's distro rustc/cargo and creates a user ci, whereas the live image has /opt/rust/{cargo,rustup} and a user sol. So the recipe and the running artifact have diverged completely.

Till's direction (2026-07-28): fix this via refraction-forger rather than by patching the live qcow2.

Ask

1. Make the recipe match, then supersede the artifact. Bring images/ubuntu-rust-ci.kdl (or a new solstice-rust.kdl) up to what the lane actually needs, and publish it as the image solstice boots for runs_on: rust. The point is that the CI image becomes reproducible from git — a hand-built qcow2 from June is why nobody could see what was wrong with it.
2. Fix the ownership. Either chown -R sol:sol /opt/rust (or the CI user the recipe creates) at build time, or set RUSTUP_HOME/CARGO_HOME under the job user's home. Keep CARGO_HOME pointing at a shared, pre-populated registry cache so jobs do not re-download crates every run — only RUSTUP_HOME needs to be writable.
3. Bake the components. Install the toolchain the repos pin, with rustfmt and clippy as components, so a pinned job resolves offline and does not depend on network at job time. This is the substance of Anima WI-227 ("Bake clippy + rustfmt into the solstice-rust runner image"), which is human-parked on the Anima side; this item is where the work actually lives.
4. Keep sccache working. The existing image bakes sccache and the lane's S3 cache config (tools.toml) assumes it. Do not regress that — a cache miss degrades silently into full compiles.

Acceptance criteria

• A job whose repo carries rust-toolchain.toml pinning 1.96.0 prints rustc 1.96.0 from the === toolchain === section and proceeds.
• cargo fmt --check and cargo clippy both run without the job installing anything at runtime.
• The image is produced by refraction-forger from a recipe in git, and rebuilding it twice yields the same toolchain versions.
• sccache still reports cache hits against the shared S3 bucket from inside the guest.
• Anima WI-352's CI goes green unmodified.

Verification hook

The fastest end-to-end check is Anima WI-352's branch (claude/wi-019f9aa3-ci-lints-against-an-unpinned-toolchain-d): it pins 1.96.0 and makes both fmt and clippy blocking. If that branch goes green, this is done.

Evidence

• Failing run: solstice jobs.request_id = 1a063681-97bc-4bbc-beee-ae6645af6ac6, and an identical earlier one at e34b42b0-a065-40d1-aa50-44896c077aca. Both readable via job_logs (note: repo_owner/repo_name are NULL on native-lane rows, so the MCP cannot find them by owner/repo — query job_logs by request_id directly).
• Live artifact: archibald:/var/lib/solstice/images/solstice-rust.qcow2.
• Existing recipe: refraction-forger/images/ubuntu-rust-ci.kdl.

Anima work item `019faa04-868e-7f60-ae26-d7eab7114a3a`. ## Why now The image behind solstice's `runs_on: rust` lane cannot install a pinned Rust toolchain. Every job that ships a `rust-toolchain.toml` dies here — reproduced twice on 2026-07-28, byte-identical, from the job's own log (`job_logs`, request `1a063681-97bc-4bbc-beee-ae6645af6ac6`): ``` === toolchain (pinned by rust-toolchain.toml) === /opt/rust/cargo/bin/cargo info: syncing channel updates for 1.96.0-x86_64-unknown-linux-gnu error: could not create temp root /opt/rust/rustup/tmp: Permission denied (os error 13) 2026-07-28T18:27:39Z ERROR workflow failed exit_code=1 ``` `RUSTUP_HOME` is `/opt/rust/rustup`, owned by root, while the job runs as `sol` (workdir `/home/sol/work`). rustup resolves the pin, tries to download, cannot create its temp dir, and exits. This is not specific to 1.96 — **no** pinned toolchain can install, so the lane silently lints against whatever rustc the image happens to carry. That blocks Anima WI-352 ("CI lints against an unpinned toolchain"), which has now been dispatched, diagnosed, rebased and parked at its retry cap three times against this one error. ## What the live image actually is `/var/lib/solstice/images/solstice-rust.qcow2` — 5.4 GB, dated **2026-06-03**, owned by `solstice`, hand-built and not reproducible from any recipe in git. Meanwhile `refraction-forger` already carries `images/ubuntu-rust-ci.kdl`, which is *not* what is running: it installs jammy's distro `rustc`/`cargo` and creates a user `ci`, whereas the live image has `/opt/rust/{cargo,rustup}` and a user `sol`. So the recipe and the running artifact have diverged completely. Till's direction (2026-07-28): fix this via refraction-forger rather than by patching the live qcow2. ## Ask 1. **Make the recipe match, then supersede the artifact.** Bring `images/ubuntu-rust-ci.kdl` (or a new `solstice-rust.kdl`) up to what the lane actually needs, and publish it as the image solstice boots for `runs_on: rust`. The point is that the CI image becomes reproducible from git — a hand-built qcow2 from June is why nobody could see what was wrong with it. 2. **Fix the ownership.** Either `chown -R sol:sol /opt/rust` (or the CI user the recipe creates) at build time, or set `RUSTUP_HOME`/`CARGO_HOME` under the job user's home. Keep `CARGO_HOME` pointing at a shared, pre-populated registry cache so jobs do not re-download crates every run — only `RUSTUP_HOME` needs to be writable. 3. **Bake the components.** Install the toolchain the repos pin, with `rustfmt` and `clippy` as components, so a pinned job resolves offline and does not depend on network at job time. This is the substance of Anima WI-227 ("Bake clippy + rustfmt into the solstice-rust runner image"), which is human-parked on the Anima side; this item is where the work actually lives. 4. **Keep sccache working.** The existing image bakes sccache and the lane's S3 cache config (`tools.toml`) assumes it. Do not regress that — a cache miss degrades silently into full compiles. ## Acceptance criteria - A job whose repo carries `rust-toolchain.toml` pinning 1.96.0 prints `rustc 1.96.0` from the `=== toolchain ===` section and proceeds. - `cargo fmt --check` and `cargo clippy` both run without the job installing anything at runtime. - The image is produced by `refraction-forger` from a recipe in git, and rebuilding it twice yields the same toolchain versions. - sccache still reports cache hits against the shared S3 bucket from inside the guest. - Anima WI-352's CI goes green unmodified. ## Verification hook The fastest end-to-end check is Anima WI-352's branch (`claude/wi-019f9aa3-ci-lints-against-an-unpinned-toolchain-d`): it pins 1.96.0 and makes both `fmt` and `clippy` blocking. If that branch goes green, this is done. ## Evidence - Failing run: solstice `jobs.request_id = 1a063681-97bc-4bbc-beee-ae6645af6ac6`, and an identical earlier one at `e34b42b0-a065-40d1-aa50-44896c077aca`. Both readable via `job_logs` (note: `repo_owner`/`repo_name` are NULL on native-lane rows, so the MCP cannot find them by owner/repo — query `job_logs` by `request_id` directly). - Live artifact: `archibald:/var/lib/solstice/images/solstice-rust.qcow2`. - Existing recipe: `refraction-forger/images/ubuntu-rust-ci.kdl`.

toasterson added 1 commit 2026-07-29 00:03:40 +00:00

images: add solstice-rust CI recipe (writable RUSTUP_HOME, baked clippy/rustfmt) ... 2812a8abf3

Add images/solstice-rust.kdl, a reproducible-from-git recipe for the
qcow2 behind solstice's runs_on: rust lane, superseding the hand-built
2026-06-03 solstice-rust.qcow2 that nobody could inspect.

The live image fails every job that ships a rust-toolchain.toml:

  info: syncing channel updates for 1.96.0-x86_64-unknown-linux-gnu
  error: could not create temp root /opt/rust/rustup/tmp: Permission denied (os error 13)

because RUSTUP_HOME (/opt/rust/rustup) is root-owned while the job runs as
sol. The existing images/ubuntu-rust-ci.kdl diverged from the running
artifact entirely (jammy distro rustc, user ci vs. rustup + user sol).

The new recipe matches what the lane actually needs and fixes the root
cause:

- Rust toolchain system-wide under /opt/rust via rustup, with the pinned
  1.96.0 toolchain plus stable installed and rustfmt + clippy + rust-src
  baked in, so a pinned job resolves offline (no network at job time).
- /opt/rust/rustup chowned to sol:sol (only RUSTUP_HOME is writable;
  CARGO_HOME stays root-owned as a shared, pre-populated registry cache)
  so rustup can create its temp dir when resolving a pin.
- sccache v0.8.2 (pinned, not latest) installed and wired via
  RUSTC_WRAPPER=sccache in both /etc/profile.d and /etc/environment so the
  lane's S3 cache keeps hitting -- a miss degrades silently into full
  compiles.
- cargo registry cache pre-warmed via cargo fetch so jobs do not
  re-download crates every run.
- Toolchain + sccache versions are pinned, so two builds of the recipe
  yield the same versions (an explicit acceptance criterion).

Rebuild with: forger build --spec images/solstice-rust.kdl

Validation: forger validate + inspect pass; spec-parser unit tests pass;
forger release binary still builds clean. Recipe ships no Rust code
changes.

toasterson changed title from WIP: Build the solstice rust CI image from a refraction-forger recipe, with a writable RUSTUP_HOME and clippy/rustfmt baked in to Build the solstice rust CI image from a refraction-forger recipe, with a writable RUSTUP_HOME and clippy/rustfmt baked in 2026-07-29 00:35:17 +00:00

toasterson added 1 commit 2026-07-29 00:40:39 +00:00

images/solstice-rust: fix rustup --component syntax (comma-separated, not spaces) ... ff96eefaba

rustup's --component flag is comma-separated, not space-separated: the
previous form

  rustup toolchain install 1.96.0 --profile minimal --component rustfmt clippy rust-src

was parsed as `--component rustfmt` plus two positional toolchain names
`clippy` and `rust-src`, failing with

  error: invalid value 'clippy' for '[TOOLCHAIN]...': invalid toolchain name: 'clippy'

before any toolchain was installed. The bake would therefore ship an image
with the 1.96.0 toolchain installed (from the initial rustup-init) but
*without* rustfmt/clippy/rust-src, directly violating the acceptance
criterion that cargo fmt --check and cargo clippy run without the job
installing anything at runtime.

Switch to the documented comma form (--component rustfmt,clippy,rust-src)
in both the rustup-init invocation and the two rustup toolchain install
calls.

Also tighten the 'warm cargo registry cache' step:
- rename to 'warm cargo registry index' and fix the comment, which claimed
  it pre-downloaded crates (it cannot -- an empty warmup crate has no deps);
  what cargo fetch actually warms is the crates.io *index* under
  CARGO_HOME, sparing the first job the one-time index fetch.
- redirect cargo new's banner to /dev/null so build logs stay clean.

No Rust code changes. forger validate + inspect pass; spec-parser tests
pass (25/25).

toasterson merged commit 1e84a806b1 into main 2026-08-09 09:57:48 +00:00

toasterson referenced this pull request from a commit 2026-08-09 09:57:50 +00:00

Build the solstice `rust` CI image from a refraction-forger recipe, with a writable RUSTUP_HOME and clippy/rustfmt baked in (#1)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

CloudNebulaProject/refraction-forger!1

No description provided.