solstice-ci/docs/ai/2025-11-01-vm-build-scripts.md

### VM build scripts and cross-built runner serving

Summary
- Added mise tasks to cross-build the workflow runner and run builds inside both Linux and Illumos VMs.
- Enhanced orchestrator cloud-init to support multiple runner URLs (SOLSTICE_RUNNER_URLS) and auto-pick by OS.

New tasks
- build:runner-cross — cross builds workflow-runner for:
  - x86_64-unknown-linux-gnu (release)
  - x86_64-unknown-illumos (release)
- run:runner-serve-multi — serves both binaries via simple HTTP servers:
  - http://HOST:8090/solstice-runner-linux
  - http://HOST:8091/solstice-runner-illumos
- ci:vm-build — end-to-end:
  - Brings up RabbitMQ
  - Cross-builds the runner
  - Serves both binaries and exports SOLSTICE_RUNNER_URLS
  - Starts the orchestrator with ORCH_CONTACT_ADDR set so VMs can stream logs back
  - Enqueues two jobs for this repo/commit: ubuntu-22.04 and illumos-latest
  - Tails logs briefly and cleans up

Guest job script
- Added .solstice/job.sh (executed by the runner in the VM) to:
  - Best-effort install basic toolchain (curl/git/protoc/Rust) depending on OS
  - Build the workflow-runner crate (release preferred)
  - Avoids full workspace build to sidestep known sea-orm-cli issues during development

Usage
- End-to-end run across both VMs:
  - mise run ci:vm-build
- Individual steps:
  - mise run build:runner-cross
  - mise run run:runner-serve-multi
  - mise run run:orchestrator (in another terminal)
  - SOL_RUNS_ON=ubuntu-22.04 mise run run:forge-enqueue
  - SOL_RUNS_ON=illumos-latest mise run run:forge-enqueue

Notes
- Ensure the example orchestrator image map is configured and images are accessible. Ubuntu entry is provided; illumos points to OpenIndiana Hipster cloud image.
- The runner uses system git and /bin/sh in the VM; ensure they exist in the base images.
- Known: a transitive sea-orm-cli build failure can break full workspace builds; tasks avoid building that crate for the fast loop.
Add support for multi-OS VM builds with cross-built runners and improved local development tooling This commit introduces: - Flexible runner URL configuration via `SOLSTICE_RUNNER_URL(S)` for cloud-init. - Automated detection of OS-specific runner binaries during VM boot. - Tasks for cross-building, serving, and orchestrating Solstice runners. - End-to-end VM build flows for Linux and Illumos environments. - Enhanced orchestration with multi-runner HTTP serving and log streaming. 2025-11-01 14:31:48 +01:00			`### VM build scripts and cross-built runner serving`

			`Summary`
			`- Added mise tasks to cross-build the workflow runner and run builds inside both Linux and Illumos VMs.`
			`- Enhanced orchestrator cloud-init to support multiple runner URLs (SOLSTICE_RUNNER_URLS) and auto-pick by OS.`

			`New tasks`
			`- build:runner-cross — cross builds workflow-runner for:`
			`- x86_64-unknown-linux-gnu (release)`
			`- x86_64-unknown-illumos (release)`
			`- run:runner-serve-multi — serves both binaries via simple HTTP servers:`
			`- http://HOST:8090/solstice-runner-linux`
			`- http://HOST:8091/solstice-runner-illumos`
			`- ci:vm-build — end-to-end:`
			`- Brings up RabbitMQ`
			`- Cross-builds the runner`
			`- Serves both binaries and exports SOLSTICE_RUNNER_URLS`
			`- Starts the orchestrator with ORCH_CONTACT_ADDR set so VMs can stream logs back`
			`- Enqueues two jobs for this repo/commit: ubuntu-22.04 and illumos-latest`
			`- Tails logs briefly and cleans up`

			`Guest job script`
			`- Added .solstice/job.sh (executed by the runner in the VM) to:`
			`- Best-effort install basic toolchain (curl/git/protoc/Rust) depending on OS`
			`- Build the workflow-runner crate (release preferred)`
			`- Avoids full workspace build to sidestep known sea-orm-cli issues during development`

			`Usage`
			`- End-to-end run across both VMs:`
			`- mise run ci:vm-build`
			`- Individual steps:`
			`- mise run build:runner-cross`
			`- mise run run:runner-serve-multi`
			`- mise run run:orchestrator (in another terminal)`
			`- SOL_RUNS_ON=ubuntu-22.04 mise run run:forge-enqueue`
			`- SOL_RUNS_ON=illumos-latest mise run run:forge-enqueue`

			`Notes`
			`- Ensure the example orchestrator image map is configured and images are accessible. Ubuntu entry is provided; illumos points to OpenIndiana Hipster cloud image.`
			`- The runner uses system git and /bin/sh in the VM; ensure they exist in the base images.`
			`- Known: a transitive sea-orm-cli build failure can break full workspace builds; tasks avoid building that crate for the fast loop.`