diff --git a/.idea/solstice-ci.iml b/.idea/solstice-ci.iml index c1ffe81..bdfeb09 100644 --- a/.idea/solstice-ci.iml +++ b/.idea/solstice-ci.iml @@ -10,6 +10,7 @@ + diff --git a/docs/ai/2025-10-25-orchestrator-scheduling-and-libvirt.md b/docs/ai/2025-10-25-orchestrator-scheduling-and-libvirt.md index 14d31b5..9214e55 100644 --- a/docs/ai/2025-10-25-orchestrator-scheduling-and-libvirt.md +++ b/docs/ai/2025-10-25-orchestrator-scheduling-and-libvirt.md @@ -1,32 +1,34 @@ -### Solstice CI — Orchestrator Scheduling, Image Map Config, and Libvirt/Zones Backends (MVP) +### Solstice CI — Orchestrator Scheduling, Image Map Config, and Libvirt Lifecycle (MVP) -This document summarizes the initial implementation of Orchestrator scheduling, a YAML-based image map configuration, cloud image preparation, and a hypervisor abstraction with Linux/KVM (libvirt) and illumos zones scaffolding. +This document reflects the current implementation of the Orchestrator: scheduling/capacity, a YAML-based image map, cloud image preparation, and a hypervisor abstraction with a working Linux/KVM (libvirt) backend and an illumos zones scaffold. -#### What’s included (MVP) +#### What’s included (current status) - Scheduler and capacity - Global max concurrency (`MAX_CONCURRENCY`, default 2) with backpressure by aligning AMQP `prefetch` to concurrency. - Optional per-label capacity via `CAPACITY_MAP` (e.g., `illumos-latest=2,ubuntu-22.04=4`). - Ack-on-accept: AMQP message is acked after basic validation and enqueue to scheduler; errors during provisioning are handled internally. -- YAML image map configuration +- YAML image map configuration (backend-agnostic images) - Loaded at startup from `--config` / `ORCH_CONFIG`; defaults to `examples/orchestrator-image-map.yaml`. - - Keys: `default_label`, `aliases`, optional `sizes` presets, and `images` map with backend (`zones` or `libvirt`), `source` URL, `local_path`, `decompress` (`zstd` or none), `nocloud` (bool), and per-image default resources. + - Keys: `default_label`, `aliases`, optional `sizes` presets, and `images` map with `source` URL, `local_path`, `decompress` (`zstd` or none), `nocloud` (bool), and per-image default resources. - Default mapping provided: - `default_label: illumos-latest` - Alias: `illumos-latest → openindiana-hipster` - - `openindiana-hipster` image points to current OI cloud image: `https://dlc.openindiana.org/isos/hipster/20250402/OI-hipster-cloudimage.img.zstd`, marked `nocloud: true` and `backend: zones`. - - Size presets (not yet consumed directly by jobs): `small` (1 CPU, 1 GiB), `medium` (2 CPU, 2 GiB), `large` (4 CPU, 4 GiB). + - `openindiana-hipster` image points to current OI cloud image: `https://dlc.openindiana.org/isos/hipster/20250402/OI-hipster-cloudimage.img.zstd` and is marked `nocloud: true`. + - Size presets (operator convenience): `small` (1 CPU, 1 GiB), `medium` (2 CPU, 2 GiB), `large` (4 CPU, 4 GiB). - Image preparation (downloader) - On startup, the orchestrator ensures each configured image exists at `local_path`. - If missing, downloads from `source` and optionally decompresses with Zstd into the target path. - Hypervisor abstraction - `Hypervisor` trait and `RouterHypervisor` dispatcher. - Backends: - - `libvirt` (Linux/KVM): skeleton that connects to libvirt in `prepare`; domain XML/overlay/NoCloud seed wiring to follow. - - `zones` (illumos/bhyve): stub scaffold (not yet functional); will integrate with `zone` crate + ZFS clones in a follow-up. + - `libvirt` (Linux/KVM): IMPLEMENTED — creates qcow2 overlays (qemu-img), generates domain XML with virtio devices, builds/attaches NoCloud seed ISO (mkisofs/genisoimage), defines and starts the domain, shuts down via ACPI with timeout and forces destroy if needed. Ensures the libvirt network (`default` by default) is active and autostarted. + - `zones` (illumos/bhyve): scaffold (not yet functional); will integrate with `zone` crate + ZFS clones in a follow-up. - `NoopHypervisor` for development on hosts without privileges. - Orchestrator MQ wiring - Consumes `JobRequest` messages and builds `VmSpec` from resolved label and image defaults. - - Injects minimal cloud-init user-data content (NoCloud) into the spec for future seeding. + - Injects minimal cloud-init user-data content (NoCloud) into the spec for seeding. +- Graceful shutdown + - On SIGINT/SIGTERM the consumer is stopped, the scheduler is allowed to drain, and active VMs are asked to shutdown gracefully before being destroyed. #### Configuration (CLI/env) - `--config`, `ORCH_CONFIG` — path to YAML image map (default `examples/orchestrator-image-map.yaml`). @@ -34,6 +36,7 @@ This document summarizes the initial implementation of Orchestrator scheduling, - `--capacity-map`, `CAPACITY_MAP` — per-label capacity (e.g., `illumos-latest=2,ubuntu-22.04=4`). - AMQP: `AMQP_URL`, `AMQP_EXCHANGE`, `AMQP_QUEUE`, `AMQP_ROUTING_KEY`, `AMQP_PREFETCH` (defaulted to `MAX_CONCURRENCY`). - Libvirt (Linux): `LIBVIRT_URI` (default `qemu:///system`), `LIBVIRT_NETWORK` (default `default`). +- Requirements for libvirt lifecycle on Linux: `libvirtd` running, `qemu-img`, and `mkisofs` (or `genisoimage`) available on PATH. #### Local usage (dev) 1. Ensure RabbitMQ is running (docker-compose service `rabbitmq`). @@ -44,19 +47,14 @@ This document summarizes the initial implementation of Orchestrator scheduling, --max-concurrency 2 ``` On first run, the OI cloud image will be downloaded and decompressed to the configured `local_path`. -3. In another terminal, enqueue a job (Forge Integration webhook or CLI `enqueue`). The orchestrator will resolve `runs_on` (or default label) and schedule a VM using the configured backend. - -Note: The current libvirt/zones backends are partial; actual VM boot is a follow-up. The scheduler and config wiring are complete and ready for backend integration. +3. In another terminal, enqueue a job (Forge Integration webhook or CLI `enqueue`). On Linux with libvirt enabled, the orchestrator will resolve `runs_on` (or default label), prepare an overlay and seed ISO, define `job-` and start it. #### What’s next (planned) -- Libvirt backend: - - Create qcow2 overlays, generate domain XML (virtio devices), attach NoCloud ISO seed, define, start, shutdown, and destroy. - - Ensure libvirt default network is active at startup if necessary. - Illumos zones backend: - Integrate `oxidecomputer/zone` and ZFS clone workflow; set bhyve attributes (`vcpus`, `ram`, `bootdisk`), networking, and SMF. - Lifecycle and runner coordination: - gRPC Orchestrator↔Runner for logs/status, job completion handling, and cleanup. - Persistence and recovery: - - Store job/VM state in Postgres; graceful recovery on restart. + - Store job/VM state in Postgres; reconcile on restart. - Tests and docs: - - Unit tests for config parsing and scheduler; feature-gated libvirt smoke test; expand docs. + - Unit tests for config parsing, scheduler capacity accounting, and cloud-init seed creation; an opt-in libvirt smoke test; expand docs.