mirror of
https://codeberg.org/Toasterson/solstice-ci.git
synced 2026-04-10 13:20:41 +00:00
Add GitHub App support, AMQP integration, and webhook enhancements
- Extend GitHub webhook handler with signature validation, push, and pull request event handling. - Add GitHub App authentication via JWT and installation token retrieval. - Parse `.solstice/workflow.kdl` for job queuing with `runs_on`, `script`, and job grouping support. - Integrate AMQP consumer for orchestrator results and structured job enqueueing. - Add S3-compatible storage configuration for log uploads. - Refactor CLI options and internal state for improved configuration management. - Enhance dependencies for signature, JSON, and AMQP handling. - Document GitHub integration Signed-off-by: Till Wegmueller <toasterson@gmail.com>
This commit is contained in:
Till Wegmueller
parent
b53ccfb4e2
commit
a1592cd6c9
3 changed files with 1505 additions and 13 deletions
|
|
@ -8,5 +8,25 @@ common = { path = "../common" }
|
|||
clap = { version = "4", features = ["derive", "env"] }
|
||||
miette = { version = "7", features = ["fancy"] }
|
||||
tracing = "0.1"
|
||||
tokio = { version = "1", features = ["rt-multi-thread", "macros", "signal"] }
|
||||
tokio = { version = "1", features = ["rt-multi-thread", "macros", "signal", "fs", "io-util", "time"] }
|
||||
# HTTP + Webhooks
|
||||
axum = { version = "0.8", features = ["macros"] }
|
||||
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls-native-roots"] }
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = "1"
|
||||
# Signature verification
|
||||
hmac = "0.12"
|
||||
sha2 = "0.10"
|
||||
hex = "0.4"
|
||||
# GitHub App auth
|
||||
jsonwebtoken = "9"
|
||||
time = { version = "0.3", features = ["formatting"] }
|
||||
# AMQP consumer for results
|
||||
lapin = { version = "2" }
|
||||
futures-util = "0.3"
|
||||
# S3/Garage upload
|
||||
aws-config = { version = "1", default-features = false, features = ["behavior-version-latest", "rt-tokio"] }
|
||||
aws-sdk-s3 = { version = "1", default-features = false, features = ["rt-tokio", "rustls"] }
|
||||
# Workflow parsing helpers
|
||||
base64 = "0.22"
|
||||
uuid = { version = "1", features = ["v4"] }
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
170
docs/ai/2025-10-25-github-webhooks-to-jobrequest.md
Normal file
170
docs/ai/2025-10-25-github-webhooks-to-jobrequest.md
Normal file
|
|
@ -0,0 +1,170 @@
|
|||
### GitHub Webhooks → JobRequest Mapping (Integration Layer)
|
||||
|
||||
This document explains how the GitHub Integration service maps GitHub webhooks to internal `JobRequest` messages, publishes them to RabbitMQ, and reports status back via the GitHub Checks API.
|
||||
|
||||
---
|
||||
|
||||
### Overview
|
||||
- Service: `crates/github-integration`
|
||||
- Endpoint: `POST /webhooks/github` (configurable via `WEBHOOK_PATH`)
|
||||
- Auth: HMAC-SHA256 validation using `GITHUB_WEBHOOK_SECRET` (or `WEBHOOK_SECRET` fallback)
|
||||
- Events handled: `push`, `pull_request` (`opened`, `synchronize`, `reopened`)
|
||||
- Output: `JobRequest` (JSON) published to exchange `solstice.jobs` with routing key `jobrequest.v1`
|
||||
- Status reporting: GitHub Checks API (via GitHub App)
|
||||
|
||||
---
|
||||
|
||||
### Headers and Security
|
||||
- Event type header: `X-GitHub-Event`
|
||||
- Delivery ID header (optional, for logs): `X-GitHub-Delivery`
|
||||
- Signature header: `X-Hub-Signature-256`
|
||||
- Value is `sha256=<hex>` where `<hex>` is `HMAC_SHA256(secret, raw_request_body)`.
|
||||
- If `GITHUB_WEBHOOK_SECRET` is set, the service requires a valid signature and returns `401` on mismatch/missing header.
|
||||
- If unset, the service accepts requests (dev mode) and logs a warning.
|
||||
|
||||
Signature example (shell):
|
||||
```bash
|
||||
SECRET=your_shared_secret
|
||||
BODY='{"after":"deadbeef", "repository":{}}'
|
||||
SIG=$(printf %s "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -binary | xxd -p -c 256)
|
||||
# Send:
|
||||
curl -sS -X POST http://127.0.0.1:8082/webhooks/github \
|
||||
-H "Content-Type: application/json" \
|
||||
-H "X-GitHub-Event: push" \
|
||||
-H "X-Hub-Signature-256: sha256=$SIG" \
|
||||
--data "$BODY"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Payload Mapping → JobRequest
|
||||
|
||||
We only deserialize the minimal fields required to construct a `JobRequest`. Unused fields are ignored.
|
||||
|
||||
- Push event (`X-GitHub-Event: push`):
|
||||
- `repo_url` <- `repository.clone_url` (fallback `repository.ssh_url`)
|
||||
- `commit_sha` <- `after`
|
||||
- Ignore branch deletions where `after` is all zeros
|
||||
- `source` = `github`
|
||||
|
||||
Minimal push payload shape:
|
||||
```json
|
||||
{
|
||||
"after": "0123456789abcdef0123456789abcdef01234567",
|
||||
"repository": {
|
||||
"clone_url": "https://github.com/org/repo.git",
|
||||
"ssh_url": "git@github.com:org/repo.git"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- Pull request event (`X-GitHub-Event: pull_request`):
|
||||
- Only actions: `opened`, `synchronize`, `reopened` (others → 204 No Content)
|
||||
- `repo_url` <- `pull_request.head.repo.clone_url` (fallback `ssh_url`)
|
||||
- `commit_sha` <- `pull_request.head.sha`
|
||||
- `source` = `github`
|
||||
|
||||
Minimal PR payload shape:
|
||||
```json
|
||||
{
|
||||
"action": "synchronize",
|
||||
"pull_request": {
|
||||
"head": {
|
||||
"sha": "89abcdef0123456789abcdef0123456789abcd",
|
||||
"repo": {
|
||||
"clone_url": "https://github.com/org/repo.git",
|
||||
"ssh_url": "git@github.com:org/repo.git"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`JobRequest` fields set now:
|
||||
- `schema_version = "jobrequest.v1"`
|
||||
- `request_id = Uuid::v4()`
|
||||
- `source = github`
|
||||
- `repo_url` as above
|
||||
- `commit_sha` as above
|
||||
- `workflow_path = null` (may be inferred later)
|
||||
- `workflow_job_id = null`
|
||||
- `runs_on = null` (future enhancement to infer)
|
||||
- `submitted_at = now(UTC)`
|
||||
|
||||
---
|
||||
|
||||
### Workflow Expansion (.solstice/workflow.kdl)
|
||||
If the GitHub App credentials are configured and the repo includes `.solstice/workflow.kdl`, the integration will:
|
||||
- Fetch the KDL file at the exact commit SHA (via the Contents API).
|
||||
- Parse job blocks and enqueue one `JobRequest` per job.
|
||||
- Set `workflow_path` to `.solstice/workflow.kdl` and `workflow_job_id` to the job ID.
|
||||
- Use `runs_on` from the workflow job if present, otherwise infer from labels or defaults.
|
||||
|
||||
If the workflow is absent or cannot be parsed, a single job is enqueued.
|
||||
|
||||
---
|
||||
|
||||
### Checks API Status Updates
|
||||
- On webhook enqueue, the integration creates a Check Run for each job (status `queued`).
|
||||
- The Check Run `external_id` is the `request_id`, enabling later lookup without persistent storage.
|
||||
- When a `JobResult` arrives from MQ, the integration locates the matching Check Run and marks it `completed` with `success` or `failure`.
|
||||
- If no matching Check Run is found, it creates a completed Check Run so the commit still shows the result.
|
||||
|
||||
---
|
||||
|
||||
### AMQP Publication
|
||||
- Exchange: `solstice.jobs` (durable, direct)
|
||||
- Routing key: `jobrequest.v1`
|
||||
- Queue: `solstice.jobs.v1` (declared by both publisher and consumer)
|
||||
- DLX/DLQ: `solstice.dlx` / `solstice.jobs.v1.dlq`
|
||||
- Publisher confirms enabled; messages are persistent (`delivery_mode = 2`).
|
||||
|
||||
Env/CLI (defaults):
|
||||
- `AMQP_URL=amqp://127.0.0.1:5672/%2f`
|
||||
- `AMQP_EXCHANGE=solstice.jobs`
|
||||
- `AMQP_ROUTING_KEY=jobrequest.v1`
|
||||
- `AMQP_QUEUE=solstice.jobs.v1`
|
||||
- `AMQP_DLX=solstice.dlx`
|
||||
- `AMQP_DLQ=solstice.jobs.v1.dlq`
|
||||
|
||||
---
|
||||
|
||||
### Configuration
|
||||
- HTTP address: `HTTP_ADDR` (default `0.0.0.0:8082`)
|
||||
- Webhook path: `WEBHOOK_PATH` (default `/webhooks/github`)
|
||||
- Shared secret: `GITHUB_WEBHOOK_SECRET` (required in prod)
|
||||
- GitHub API base: `GITHUB_API_BASE` (default `https://api.github.com`)
|
||||
- GitHub App ID: `GITHUB_APP_ID`
|
||||
- GitHub App key: `GITHUB_APP_KEY_PATH` or `GITHUB_APP_KEY`
|
||||
- Check name: `GITHUB_CHECK_NAME` (default `Solstice CI`)
|
||||
- Logs base URL: `LOGS_BASE_URL` (preferred) or `ORCH_HTTP_BASE` (deprecated)
|
||||
- S3 upload: `S3_ENDPOINT`, `S3_BUCKET`
|
||||
- Runs-on overrides: `RUNS_ON_DEFAULT`, `RUNS_ON_MAP` (owner/repo=label)
|
||||
|
||||
Example run:
|
||||
```bash
|
||||
export GITHUB_WEBHOOK_SECRET=devsecret
|
||||
export GITHUB_APP_ID=123456
|
||||
export GITHUB_APP_KEY_PATH=/path/to/app.pem
|
||||
cargo run -p github-integration -- --http-addr 0.0.0.0:8082 --webhook-path /webhooks/github
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### GitHub App Setup (Minimum)
|
||||
1. Create a GitHub App with webhook URL `http://<your-host>:8082/webhooks/github` and set the webhook secret.
|
||||
2. Grant permissions:
|
||||
- Checks: Read & write
|
||||
- Contents: Read
|
||||
- Metadata: Read
|
||||
3. Subscribe to events:
|
||||
- Push
|
||||
- Pull request
|
||||
4. Install the App on the target repositories.
|
||||
|
||||
---
|
||||
|
||||
### Notes & Next Steps
|
||||
- Consider adding idempotency keyed by `X-GitHub-Delivery` to avoid duplicate enqueues.
|
||||
- Consider supporting `check_suite` events for GitHub-native UI flows.
|
||||
- Add optional allowlists/branch filters to reduce noise.
|
||||
Loading…
Add table
Reference in a new issue