CloudNebulaProject/solstice-ci
SHA256
Watch
1
0
Fork
You've already forked solstice-ci
 0
Code Issues Pull requests 1 Projects Releases Packages Wiki Activity Actions

WIP: Image push reads the wrong filename: 27-minute build succeeds, push fails on a hardcoded path #1

Draft
toasterson wants to merge 2 commits from claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m into main
pull from: claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m
merge into: CloudNebulaProject:main
Conversation 0 Commits 2 Files changed 31 +1032 -791
toasterson commented 2026-07-26 05:53:38 +00:00
Owner
Copy link

Anima work item 019f99d8-ffc9-7bf2-a6c0-3c2c0b54a4dd.

.forgejo/workflows/images.yml builds the runner image successfully and then fails to push it, because the build and push steps disagree about the output filename.

From the last run (2026-07-13, request in solstice_prod):

$ qemu-img convert -f raw -O qcow2 \
    target/forger-output/solstice-ubuntu-22.04.raw \
    target/forger-output/solstice-ubuntu-22.04.qcow2
✔ Target 'solstice-ubuntu-22.04' built successfully (1640720ms)

Build complete. Output: target/forger-output
total 4.4G
-rw-r--r-- 1 root root 4.4G Jul 13 15:31 solstice-ubuntu-22.04.qcow2

⸬ Pushing QCOW2 artifact → code.aopc.cloud/cloudnebulaproject/solstice-ubuntu-22.04:latest
  Reading target/forger-output/qcow2.qcow2
Error: Failed to read QCOW2 file: target/forger-output/qcow2.qcow2
       No such file or directory (os error 2)

forger names the artifact after the target — solstice-ubuntu-22.04.qcow2. The push step is hardcoded to target/forger-output/qcow2.qcow2. The workflow's own ls -lah prints the correct filename immediately before the step that looks for the wrong one.

The build takes 27 minutes and completes. Every one of those runs is thrown away at the last step.

Consequences

  • The image pipeline has been broken since at least 2026-07-13, and no image has been published since. The last successful run in the job history is before that.
  • solstice-rust was never pushed at all, which is why the orchestrator image map still consumes it via file:///var/lib/solstice/images/solstice-rust.qcow2 with the comment "registry push pending". The reason was never the memory ceiling that comment implies — it is this.
  • Consequently the deployed solstice-rust.qcow2 is stuck at its 2026-06-03 hand-build and does not include commit 6651cfb's rustfmt component fix, forcing every consumer to guard tool availability at job time.

Fix

IMAGE_NAME is already an env var in the workflow. Use it:

-  --image target/forger-output/qcow2.qcow2
+  --image target/forger-output/${IMAGE_NAME}.qcow2

Do not merely hardcode the other name — deriving it from IMAGE_NAME is also the change that makes matrixing over multiple images straightforward, which is the next work item in this plan.

Add an assertion between build and push that the artifact exists, so a future naming change fails in one second with a clear message instead of after 27 minutes of build time.

Acceptance criteria

  • forger push finds the artifact the build produced.
  • solstice-ubuntu-22.04:latest is republished and the digest changes.
  • A missing artifact fails immediately with a message naming both the expected and the actual directory contents.
  • Re-running the workflow twice produces two successful pushes (no dependence on leftover state).

Related

  • 019f998c-af60-7832-907d-3d1422a26acd — make solstice-rust pipeline-built and registry-published. Blocked on this; the pipeline it would reuse cannot currently push anything.
  • 019f998c-8698-79d2-9183-e7ec5f8c3746 — streaming push to bound memory. Still worth doing, but it is not what is blocking publication today.
Anima work item `019f99d8-ffc9-7bf2-a6c0-3c2c0b54a4dd`. `.forgejo/workflows/images.yml` builds the runner image successfully and then fails to push it, because the build and push steps disagree about the output filename. From the last run (2026-07-13, request in `solstice_prod`): ``` $ qemu-img convert -f raw -O qcow2 \ target/forger-output/solstice-ubuntu-22.04.raw \ target/forger-output/solstice-ubuntu-22.04.qcow2 ✔ Target 'solstice-ubuntu-22.04' built successfully (1640720ms) Build complete. Output: target/forger-output total 4.4G -rw-r--r-- 1 root root 4.4G Jul 13 15:31 solstice-ubuntu-22.04.qcow2 ⸬ Pushing QCOW2 artifact → code.aopc.cloud/cloudnebulaproject/solstice-ubuntu-22.04:latest Reading target/forger-output/qcow2.qcow2 Error: Failed to read QCOW2 file: target/forger-output/qcow2.qcow2 No such file or directory (os error 2) ``` forger names the artifact after the target — `solstice-ubuntu-22.04.qcow2`. The push step is hardcoded to `target/forger-output/qcow2.qcow2`. The workflow's own `ls -lah` prints the correct filename immediately before the step that looks for the wrong one. The build takes **27 minutes** and completes. Every one of those runs is thrown away at the last step. ## Consequences - The image pipeline has been broken since at least 2026-07-13, and no image has been published since. The last successful run in the job history is before that. - `solstice-rust` was never pushed at all, which is why the orchestrator image map still consumes it via `file:///var/lib/solstice/images/solstice-rust.qcow2` with the comment "registry push pending". The reason was never the memory ceiling that comment implies — it is this. - Consequently the deployed `solstice-rust.qcow2` is stuck at its 2026-06-03 hand-build and does not include commit 6651cfb's rustfmt component fix, forcing every consumer to guard tool availability at job time. ## Fix `IMAGE_NAME` is already an env var in the workflow. Use it: ``` - --image target/forger-output/qcow2.qcow2 + --image target/forger-output/${IMAGE_NAME}.qcow2 ``` Do not merely hardcode the other name — deriving it from `IMAGE_NAME` is also the change that makes matrixing over multiple images straightforward, which is the next work item in this plan. Add an assertion between build and push that the artifact exists, so a future naming change fails in one second with a clear message instead of after 27 minutes of build time. ## Acceptance criteria - `forger push` finds the artifact the build produced. - `solstice-ubuntu-22.04:latest` is republished and the digest changes. - A missing artifact fails immediately with a message naming both the expected and the actual directory contents. - Re-running the workflow twice produces two successful pushes (no dependence on leftover state). ## Related - `019f998c-af60-7832-907d-3d1422a26acd` — make solstice-rust pipeline-built and registry-published. Blocked on this; the pipeline it would reuse cannot currently push anything. - `019f998c-8698-79d2-9183-e7ec5f8c3746` — streaming push to bound memory. Still worth doing, but it is not what is blocking publication today.
Anima implementation session.
toasterson force-pushed claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m from e016e513eb to 715e417574 2026-07-26 06:43:48 +00:00 Compare
toasterson force-pushed claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m from 715e417574 to ca7df59eb6 2026-07-26 07:26:28 +00:00 Compare 
The runner-integration build.rs imports google/protobuf/{struct,timestamp}.proto
and probes /usr/include/google/protobuf/timestamp.proto to add /usr/include as
an include dir for tonic_prost_build. On Debian/Ubuntu, protobuf-compiler ships
only the protoc binary — the well-known .proto files live in the separate
libprotobuf-dev package. Without it, the build script panics with
"google/protobuf/struct.proto: File not found" and cargo exits 101 with
"warning: build failed, waiting for other jobs to finish..." — the exact CI
failure blocking the image-push fix.

Add libprotobuf-dev (apt) / protobuf-devel (dnf/yum/zypper) to setup-linux.sh.
apk (Alpine) and pacman (Arch) already bundle the well-known protos with their
protoc package, so they are left as-is.

Also add a sanity check in main() that the well-known protos exist after the
package install, failing loudly in seconds with the remediation package named
instead of after ~27 min of build-script panic.
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m:claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m
git switch claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch main
git merge --no-ff claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m
git switch claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m
git rebase main
git switch main
git merge --ff-only claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m
git switch claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m
git rebase main
git switch main
git merge --no-ff claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m
git switch main
git merge --squash claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m
git switch main
git merge --ff-only claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m
git switch main
git merge claude/wi-019f99d8-image-push-reads-the-wrong-filename-27-m
git push origin main
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
CloudNebulaProject/solstice-ci!1
No description provided.