This commit adds:
- A unified configuration system (`AppConfig`) that aggregates KDL files and environment variables with precedence handling.
- Example KDL configuration files for the orchestrator and forge-integration modules.
- Updates to orchestrator and forge-integration to load and apply configurations from `AppConfig`.
- Improved AMQP and database configuration with overlays from CLI, environment, or KDL.
- Deprecated `TODO.txt` as it's now represented in the configuration examples.
This commit introduces:
- Automatic detection of the orchestrator contact address when not explicitly provided.
- Platform-specific logic for determining reachable IPs, including libvirt network parsing (Linux) and external IP detection.
- Updates to GRPC address processing to handle both specific and unspecified hosts.
- Additional utility functions for parsing and detecting IPs in libvirt configurations.
This commit includes:
- Adjusted runner logs from `info` to `debug` for reduced deployment log verbosity while retaining visibility in CI.
- Added functionality to serve runner binaries directly from the orchestrator via HTTP.
- Introduced new `RUNNER_DIR` configuration to specify the binary directory, with default paths and URL composition.
- Updated HTTP routing to include runner file serving with validation and logging.
- Improved AMQP body logging with a utility for better error debugging.
- Updated task scripts for runner cross-building and serving, consolidating configurations and removing redundant files.
This commit introduces:
- A utility function to parse repository owner and name from URLs, supporting HTTPS, SSH, and Git formats.
- Enhancements to job messages and results with optional `repo_owner` and `repo_name` fields for downstream integrations.
- Updated orchestrator and forge-integration workflows to leverage parsed repository details for status updates and accurate routing.
This commit introduces:
- A heuristic to mark jobs as failed if VMs stop quickly without generating logs.
- Improved configuration for runner URLs, including auto-detection of host IPs and default multi-OS runner URLs.
- Updates to the orchestrator's HTTP routing for consistency.
- New task scripts for Forge integration and updates to environment defaults for local development.
This commit introduces:
- Log persistence feature with a new `job_logs` table and related APIs for recording and retrieving job logs.
- An HTTP server for serving log endpoints and job results.
- Updates to the CI pipeline to enable persistence by default and ensure PostgreSQL readiness.
- Docker Compose updates with a Postgres service and MinIO integration for object storage.
- Packaging scripts for Arch Linux, including systemd service units for deployment.
This commit introduces the following updates:
- Adds an environment variable (`SOLSTICE_ALLOW_INSECURE`) to enable insecure TLS as a fallback for curl.
- Improves CA certificate handling and automatic installation on SunOS using IPS or pkgin.
- Extends fallback logic for repository fetching to cover scenarios with missing CA bundles.
- Updates Solstice job script dependencies to include `cmake`.
This commit improves the runner's functionality by adding:
- Detailed log streaming with request ID, stdout, and stderr line counts.
- Fallback mechanisms for repository fetch using HTTP archive when git commands fail.
- Enhanced error reporting for missing job scripts and reading errors.
- Updates to ensure compatibility with SunOS environments and non-interactive shells.
This commit replaces `openindiana-hipster` with `omnios-bloody` as the default illumos image in the orchestrator. It adds detailed configuration for the new image, including source URL, local path, and resource defaults, while retaining `openindiana-hipster` as a reference. Corresponding test cases and YAML updates are included.
This commit enhances image handling in the orchestrator by adding support for `file://` sources. It introduces logic to handle both local file copying and decompression options, complementing the existing `http(s)://` download functionality.
This commit introduces:
- VM suspend support for timeout scenarios, allowing investigation of frozen states.
- Enhanced orchestrator persistence initialization with skip option for faster startup.
- Improvements to orchestrator logging, job state tracking, and VM runtime monitoring.
- Updates to CI tasks for capturing job request IDs and tracking completion statuses.
- Extended hypervisor capabilities, including libvirt console logging configuration.
This commit introduces `bindgen-cli` installation and sets up `BINDGEN_EXTRA_CLANG_ARGS` passthrough in `Cross.toml`. It also updates the pre-build process with necessary packages (`clang`, `libclang-dev`), alternative compilers, and adds `aws-lc-rs` with `bindgen` support to `workflow-runner` dependencies.
This commit updates multiple dependencies, including:
- `axum` upgraded to 0.8 for HTTP and webhook functionality.
- `tonic` upgraded to 0.14 for gRPC support.
- `prost` upgraded to 0.14 for protobuf processing.
- Addition of `tonic-prost` and `tonic-prost-build` for updated gRPC build configurations.
Relevant Cargo.toml entries and `build.rs` are adjusted to reflect these updates.
This commit updates parsing logic by simplifying `.and_then(|e| e.value().as_string())` calls to `.and_then(|v| v.as_string())`. Additionally, it upgrades several crate dependencies, including `thiserror`, `sea-orm`, `lapin`, `virt`, and `kdl`, to their latest compatible versions for improved functionality and stability.
This commit introduces:
- Flexible runner URL configuration via `SOLSTICE_RUNNER_URL(S)` for cloud-init.
- Automated detection of OS-specific runner binaries during VM boot.
- Tasks for cross-building, serving, and orchestrating Solstice runners.
- End-to-end VM build flows for Linux and Illumos environments.
- Enhanced orchestration with multi-runner HTTP serving and log streaming.
This commit introduces gRPC-based log streaming between the VM runner (`solstice-runner`) and orchestrator. Key updates include:
- Implemented gRPC server in the orchestrator for receiving and processing runner logs.
- Added log streaming and job result reporting in the `solstice-runner` client.
- Defined `runner.proto` with messages (`LogItem`, `JobEnd`) and the `Runner` service.
- Updated orchestrator to accept gRPC settings and start the server.
- Modified cloud-init user data to include gRPC endpoint and request ID for runners.
- Enhanced message queue logic to handle job results via `publish_job_result`.
- Configured `Cross.toml` for cross-compilation of the runner.
This commit replaces inline workflow preparation logic with a dedicated `solstice-bootstrap.sh` script, simplifying workspace setup, job execution, and shutdown processes. The change ensures cleaner orchestration and improves maintainability by centralizing the bootstrapping logic.
This commit enhances the `Scheduler` to monitor VM states for completion, enabling more accurate termination detection. It introduces periodic polling combined with shutdown signals to halt operations gracefully. Additionally, VM lifecycle management in the hypervisor is updated with `state` retrieval for precise status assessments. The VM domain configuration now includes serial console support.
This commit updates the `Scheduler` to support cooperative shutdown using `Notify`, allowing graceful termination of tasks and cleanup of placeholder VMs. Additionally, the AMQP consumer is enhanced with an explicit shutdown mechanism, ensuring proper resource cleanup, including closing channels and connections.
This commit introduces the ability to configure placeholder VM run time via an environment variable (`VM_PLACEHOLDER_RUN_SECS`) and updates the `Scheduler` to accept this duration. Additionally, it implements a graceful shutdown mechanism for the orchestrator, allowing cooperative shutdown of consumers and cleanup of resources.
This commit improves the hypervisor by:
- Adding support for detecting base image formats using `qemu-img info`.
- Dynamically setting the base image format for overlay creation.
- Automatically converting non-raw images to raw format for bhyve compatibility.
- Updating `Cargo.toml` to include `serde_json` for JSON parsing.
- Modifying default working directory logic for `ZonesHypervisor`.
This commit replaces the `libvirt` crate with the `virt` crate for managing the libvirt backend on Linux. Key changes include:
- Updated `Cargo.toml` dependencies and feature configuration.
- Refactored hypervisor implementation to align with `virt` crate API.
- Improved error handling and lifecycle management for VMs and networks.
This commit introduces a persistence layer to the Orchestrator, enabling it to optionally connect to a Postgres database for recording job and VM states. It includes:
- SeaORM integration with support for migrations from the migration crate.
- `Persist` module with methods for job and VM state upserts.
- No-op fallback when persistence is disabled or unavailable.
- Documentation updates and test coverage for persistence functionality.
