- Introduce `slugify_step_name` to generate URL-friendly step name slugs.
- Attach per-step categories to NDJSON logs for better traceability.
- Update stdout and stderr logging with step-specific categories.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Introduce new `/jobs` endpoint for listing jobs grouped by `(repo_url, commit_sha)`, ordered by update timestamp.
- Add models `JobGroup`, `JobSummary`, and `JobLinks` to structure grouped job details.
- Implement grouping logic using `BTreeMap` for structured output.
- Extend router with the new endpoint and integrate ORM-backed query for fetching job data.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Add detection and usage of GNU `tar` for platforms where BSD `tar` is incompatible with required options.
- Refactor `job.sh` to delegate all environment setup to newly introduced per-OS setup scripts.
- Add initial support for workflow setups via `workflow.kdl`, running pre-defined setup scripts before executing workflow steps.
- Integrate step-wise execution and logging for workflows, with structured NDJSON output for detailed traceability.
- Increment orchestrator version to 0.1.16.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Change the default SSH username from 'ubuntu' to 'sol' for consistency with Solstice CI environment.
- Modify cloud-init user configuration to align with the new default, adding enhanced permissions and settings for 'sol' user.
- Increment orchestrator version to 0.1.15.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Add parsing and execution support for `.solstice/workflow.kdl` with job-specific configurations, including `runs_on`, `script path`, and `workflow_job_id`.
- Enable job grouping via `group_id` for cohesive workflow processing.
- Update orchestrator to pass workflow-specific parameters to `cloud-init` for finer control over execution.
- Refactor enqueue logic to handle multiple jobs per workflow with fallback to single job when no workflow is defined.
- Enhance dependencies for workflow parsing by integrating `base64`, `regex`, and `uuid`.
- Increment orchestrator version to 0.1.14 for release.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Enhance runner logging with status-specific messages and structured JSON fields for better traceability.
- Add SHA256 object format detection and initialization for Git repos when applicable.
- Improve shell script execution by adding verbose mode and safe commands handling.
- Extend package installation scripts to support Clang and related tooling across multiple environments.
- Increment orchestrator version for release.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Enhance runner tool check diagnostics with more descriptive output and JSON fields for better observability.
- Replace raw SQL queries in `logs-service` with ORM-based logic for portable and backend-agnostic log categorization.
- Add error category aggregation and structured summary reporting in logs-service.
- Improve environment variable fallback mechanics for runner workdir selection.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Add `logs-service` crate as a separate microservice to handle job log storage, retrieval, and categorization.
- Update orchestrator to redirect log endpoints to the new service with optional permanent redirects using `LOGS_BASE_URL`.
- Enhance log persistence by introducing structured fields such as category, level, and error flags.
- Implement migration to add new columns and indexes for job logs.
- Add ANSI escape sequence stripping and structured logging for cleaner log storage.
- Improve SSH log handling with interleaved stdout/stderr processing and pty request support.
- Revise Docker files and compose setup to include logs-service, with support for PostgreSQL and secure connections.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Refactor runner upload logic to use temporary files and atomic renaming for safer updates.
- Improve file permission handling during temporary file creation.
- Increment orchestrator version to 0.1.11.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Add validation for `RUNNER_LINUX_PATH` and `RUNNER_ILLUMOS_PATH` with detailed warnings and diagnostics for misconfigurations.
- Log fallback to default paths and warn if binaries are missing.
- Increment orchestrator version to 0.1.10.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Introduce `libvirt_uri` and `libvirt_network` in configuration structs, replacing reliance on environment variables.
- Update all `virsh`-related logic to use explicit parameters for libvirt connection and network settings.
- Align codebase with new guidelines rejecting runtime environment variable mutations.
- Document breaking changes in `.junie/guidelines.md`.
- Increment orchestrator version to 0.1.9.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Update IP selection logic to prefer the latest lease based on epoch timestamp.
- Remove redundant IP discovery logic in `net-dhcp-leases`.
- Increment orchestrator version to 0.1.8 for release.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Add default `LIBVIRT_URI`, `HOME`, and `XDG_CACHE_HOME` environment variable handling for `virsh` commands.
- Ensure writable cache directories for the service user in packaging scripts.
- Update systemd service to include libvirt-related environment defaults.
- Bump orchestrator version to 0.1.7.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Introduce `boot_wait_secs` configuration to delay IP discovery/SSH after VM startup.
- Capture console logs when no SSH logs are available for better debugging during failures.
- Add a utility function to snapshot and persist console logs into job logs.
- Update CLI and environment variable support for the `boot_wait_secs` parameter.
- Bump orchestrator version to 0.1.5.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Introduce `results_queue` and `results_routing_key` to MQ configuration.
- Update message publishing and queue declaration logic to leverage new fields.
- Increment orchestrator version to 0.1.4.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Add `--features libvirt` to orchestrator's Debian package build process.
- Update orchestrator version to 0.1.2 in `Cargo.toml`.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Introduce Debian package build script using `cargo-deb` for orchestrator releases.
- Add systemd unit file and post-installation script for automatic service setup.
- Update `compose.yml` with host-only port bindings for Postgres and RabbitMQ.
- Introduce NGINX-based log proxy for orchestrator logs with Traefik support.
- Bump orchestrator version to 0.1.1 and update related Cargo metadata for packaging.
- Add example environment file for orchestrator configuration.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Implement SSH execution retries with exponential backoff and timeout handling.
- Replace `virsh domifaddr` with a multi-strategy IP discovery approach.
- Introduce `OrchestratorError` for consistent, structured error reporting.
- Improve runner deployment and SSH session utilities for readability and reliability.
- Add dependencies: `thiserror`, `anyhow` for streamlined error handling.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Implement retries for SSH-based job execution with configurable timeouts.
- Introduce `OrchestratorError` for consistent error handling across modules.
- Replace `virsh domifaddr` based guest IP discovery with a robust, multi-strategy approach.
- Refactor runner deployment and SSH-related utility functions for clarity.
- Add `thiserror` and `anyhow` dependencies for error management.
- Update persistence layer with improved error handling for database operations.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Introduce fields in `JobContext` for per-job SSH configuration, including user, key paths, and PEM contents.
- Update the scheduler to support SSH-based execution of jobs, including VM lifecycle management and SSH session handling.
- Add utility functions for SSH execution, guest IP discovery, and runner deployment.
- Remove the unused `/runners/{name}` HTTP endpoint and its associated logic.
- Simplify router creation by refactoring out disabled runner directory handling.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Introduce options for specifying public runner base URLs (`SOLSTICE_RUNNER_BASE_URL`) and orchestrator contact addresses (`ORCH_CONTACT_ADDR`).
- Update `.env.sample` and `compose.yml` with new configuration fields for external log streaming and runner binary serving.
- Refactor runner URL handling and generation logic for improved flexibility.
- Enhance `cloud-init` templates with updated runner URL environment variables (`RUNNER_SINGLE` and `RUNNER_URLS`).
- Add unit tests for runner URL generation to verify various input cases.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Change data source name and JDBC URL for production environment.
- Add new data source mapping file for SQL console.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Extend `.env.sample` with `RUNNER_DIR_HOST` for serving workflow runner binaries.
- Update `compose.yml` with `RUNNER_DIR` and corresponding volume mount.
- Add instructions for runner binary setup and serving in `README.md`.
- Enhance `mise.toml` with new tooling dependencies for building runners.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Extend `.env.sample` with libvirt configuration, Forgejo secrets, and image mapping defaults.
- Update `compose.yml` to enable libvirt integration, including required mounts, devices, and environment variables.
- Add Forgejo webhook configuration and commit status reporting with optional HMAC validation.
- Enhance the orchestrator container with libvirt dependencies and optional features for VM management.
- Document host preparation for libvirt/KVM and image directories in the README.
- Set default fallback values for Traefik ACME CA server.
Signed-off-by: Till Wegmueller <toasterson@gmail.com>
- Document rootless Podman port binding limitations and workarounds in README.
- Update `.env.sample` with notes and default high ports for rootless runs.
- Adjust `compose.yml` for network configuration and privileged port handling.
- Introduce fixes for Traefik DNS timeouts using explicit public resolvers and network tweaks.
- Switch MinIO and MinIO setup to use the latest images for better compatibility.
This commit introduces:
- A production-ready Podman Compose stack using Traefik as a reverse proxy with Let's Encrypt integration.
- Per-environment logical separation for Postgres, RabbitMQ, and MinIO services.
- New deployment utilities, including a `.env.sample` template, `compose.yml`, and setup scripts for MinIO and Postgres.
- Updates to `github-integration` HTTP server with basic webhook handling using `axum` and configurable paths.
- Adjustments to packaging tasks for better tarball generation via `git archive`.
- Expanded dependencies for `PKGBUILD` to support SQLite and PostgreSQL libraries.
- Containerfiles for orchestrator and integration services to enable Rust multi-stage builds without sccache.
This enables simplified and secure CI deployments with automatic routing, TLS, and volume persistence.
This commit adds:
- A unified configuration system (`AppConfig`) that aggregates KDL files and environment variables with precedence handling.
- Example KDL configuration files for the orchestrator and forge-integration modules.
- Updates to orchestrator and forge-integration to load and apply configurations from `AppConfig`.
- Improved AMQP and database configuration with overlays from CLI, environment, or KDL.
- Deprecated `TODO.txt` as it's now represented in the configuration examples.
This commit introduces:
- Automatic detection of the orchestrator contact address when not explicitly provided.
- Platform-specific logic for determining reachable IPs, including libvirt network parsing (Linux) and external IP detection.
- Updates to GRPC address processing to handle both specific and unspecified hosts.
- Additional utility functions for parsing and detecting IPs in libvirt configurations.
This commit includes:
- Adjusted runner logs from `info` to `debug` for reduced deployment log verbosity while retaining visibility in CI.
- Added functionality to serve runner binaries directly from the orchestrator via HTTP.
- Introduced new `RUNNER_DIR` configuration to specify the binary directory, with default paths and URL composition.
- Updated HTTP routing to include runner file serving with validation and logging.
- Improved AMQP body logging with a utility for better error debugging.
- Updated task scripts for runner cross-building and serving, consolidating configurations and removing redundant files.
This commit introduces:
- A utility function to parse repository owner and name from URLs, supporting HTTPS, SSH, and Git formats.
- Enhancements to job messages and results with optional `repo_owner` and `repo_name` fields for downstream integrations.
- Updated orchestrator and forge-integration workflows to leverage parsed repository details for status updates and accurate routing.
This commit introduces:
- A heuristic to mark jobs as failed if VMs stop quickly without generating logs.
- Improved configuration for runner URLs, including auto-detection of host IPs and default multi-OS runner URLs.
- Updates to the orchestrator's HTTP routing for consistency.
- New task scripts for Forge integration and updates to environment defaults for local development.
This commit introduces:
- Log persistence feature with a new `job_logs` table and related APIs for recording and retrieving job logs.
- An HTTP server for serving log endpoints and job results.
- Updates to the CI pipeline to enable persistence by default and ensure PostgreSQL readiness.
- Docker Compose updates with a Postgres service and MinIO integration for object storage.
- Packaging scripts for Arch Linux, including systemd service units for deployment.
This commit introduces the following updates:
- Adds an environment variable (`SOLSTICE_ALLOW_INSECURE`) to enable insecure TLS as a fallback for curl.
- Improves CA certificate handling and automatic installation on SunOS using IPS or pkgin.
- Extends fallback logic for repository fetching to cover scenarios with missing CA bundles.
- Updates Solstice job script dependencies to include `cmake`.
This commit improves the runner's functionality by adding:
- Detailed log streaming with request ID, stdout, and stderr line counts.
- Fallback mechanisms for repository fetch using HTTP archive when git commands fail.
- Enhanced error reporting for missing job scripts and reading errors.
- Updates to ensure compatibility with SunOS environments and non-interactive shells.
This commit replaces `openindiana-hipster` with `omnios-bloody` as the default illumos image in the orchestrator. It adds detailed configuration for the new image, including source URL, local path, and resource defaults, while retaining `openindiana-hipster` as a reference. Corresponding test cases and YAML updates are included.
This commit enhances image handling in the orchestrator by adding support for `file://` sources. It introduces logic to handle both local file copying and decompression options, complementing the existing `http(s)://` download functionality.
This commit introduces:
- VM suspend support for timeout scenarios, allowing investigation of frozen states.
- Enhanced orchestrator persistence initialization with skip option for faster startup.
- Improvements to orchestrator logging, job state tracking, and VM runtime monitoring.
- Updates to CI tasks for capturing job request IDs and tracking completion statuses.
- Extended hypervisor capabilities, including libvirt console logging configuration.
This commit introduces `bindgen-cli` installation and sets up `BINDGEN_EXTRA_CLANG_ARGS` passthrough in `Cross.toml`. It also updates the pre-build process with necessary packages (`clang`, `libclang-dev`), alternative compilers, and adds `aws-lc-rs` with `bindgen` support to `workflow-runner` dependencies.
This commit updates multiple dependencies, including:
- `axum` upgraded to 0.8 for HTTP and webhook functionality.
- `tonic` upgraded to 0.14 for gRPC support.
- `prost` upgraded to 0.14 for protobuf processing.
- Addition of `tonic-prost` and `tonic-prost-build` for updated gRPC build configurations.
Relevant Cargo.toml entries and `build.rs` are adjusted to reflect these updates.
This commit updates the Solstice job script to install the Rust toolchain (`developer/rustc`) via the IPS package manager on Solaris-based systems (SunOS). It also adjusts the `ensure_rust` function to prioritize system package installation before falling back to `rustup`.
