solstice-ci/docs/ai/2025-10-25-github-webhooks-to-jobrequest.md

GitHub Webhooks → JobRequest Mapping (Integration Layer)

This document explains how the GitHub Integration service maps GitHub webhooks to internal JobRequest messages, publishes them to RabbitMQ, and reports status back via the GitHub Checks API.

---

Overview

• Service: crates/github-integration
• Endpoint: POST /webhooks/github (configurable via WEBHOOK_PATH)
• Auth: HMAC-SHA256 validation using GITHUB_WEBHOOK_SECRET (or WEBHOOK_SECRET fallback)
• Events handled: push, pull_request (opened, synchronize, reopened)
• Output: JobRequest (JSON) published to exchange solstice.jobs with routing key jobrequest.v1
• Status reporting: GitHub Checks API (via GitHub App)

---

Headers and Security

• Event type header: X-GitHub-Event
• Delivery ID header (optional, for logs): X-GitHub-Delivery
• Signature header: X-Hub-Signature-256
  • Value is sha256=<hex> where <hex> is HMAC_SHA256(secret, raw_request_body).
  • If GITHUB_WEBHOOK_SECRET is set, the service requires a valid signature and returns 401 on mismatch/missing header.
  • If unset, the service accepts requests (dev mode) and logs a warning.

Signature example (shell):

SECRET=your_shared_secret
BODY='{"after":"deadbeef", "repository":{}}'
SIG=$(printf %s "$BODY" | openssl dgst -sha256 -hmac "$SECRET" -binary | xxd -p -c 256)
# Send:
curl -sS -X POST http://127.0.0.1:8082/webhooks/github \
  -H "Content-Type: application/json" \
  -H "X-GitHub-Event: push" \
  -H "X-Hub-Signature-256: sha256=$SIG" \
  --data "$BODY"

---

Payload Mapping → JobRequest

We only deserialize the minimal fields required to construct a JobRequest. Unused fields are ignored.

• Push event (X-GitHub-Event: push):
  • repo_url <- repository.clone_url (fallback repository.ssh_url)
  • commit_sha <- after
  • Ignore branch deletions where after is all zeros
  • source = github

Minimal push payload shape:

{
  "after": "0123456789abcdef0123456789abcdef01234567",
  "repository": {
    "clone_url": "https://github.com/org/repo.git",
    "ssh_url":   "git@github.com:org/repo.git"
  }
}

• Pull request event (X-GitHub-Event: pull_request):
  • Only actions: opened, synchronize, reopened (others → 204 No Content)
  • repo_url <- pull_request.head.repo.clone_url (fallback ssh_url)
  • commit_sha <- pull_request.head.sha
  • source = github

Minimal PR payload shape:

{
  "action": "synchronize",
  "pull_request": {
    "head": {
      "sha": "89abcdef0123456789abcdef0123456789abcd",
      "repo": {
        "clone_url": "https://github.com/org/repo.git",
        "ssh_url":   "git@github.com:org/repo.git"
      }
    }
  }
}

JobRequest fields set now:

• schema_version = "jobrequest.v1"
• request_id = Uuid::v4()
• source = github
• repo_url as above
• commit_sha as above
• workflow_path = null (may be inferred later)
• workflow_job_id = null
• runs_on = null (future enhancement to infer)
• submitted_at = now(UTC)

---

Workflow Expansion (.solstice/workflow.kdl)

If the GitHub App credentials are configured and the repo includes .solstice/workflow.kdl, the integration will:

• Fetch the KDL file at the exact commit SHA (via the Contents API).
• Parse job blocks and enqueue one JobRequest per job.
• Set workflow_path to .solstice/workflow.kdl and workflow_job_id to the job ID.
• Use runs_on from the workflow job if present, otherwise infer from labels or defaults.

If the workflow is absent or cannot be parsed, a single job is enqueued.

---

Checks API Status Updates

• On webhook enqueue, the integration creates a Check Run for each job (status queued).
• The Check Run external_id is the request_id, enabling later lookup without persistent storage.
• When a JobResult arrives from MQ, the integration locates the matching Check Run and marks it completed with success or failure.
• If no matching Check Run is found, it creates a completed Check Run so the commit still shows the result.

---

AMQP Publication

• Exchange: solstice.jobs (durable, direct)
• Routing key: jobrequest.v1
• Queue: solstice.jobs.v1 (declared by both publisher and consumer)
• DLX/DLQ: solstice.dlx / solstice.jobs.v1.dlq
• Publisher confirms enabled; messages are persistent (delivery_mode = 2).

Env/CLI (defaults):

• AMQP_URL=amqp://127.0.0.1:5672/%2f
• AMQP_EXCHANGE=solstice.jobs
• AMQP_ROUTING_KEY=jobrequest.v1
• AMQP_QUEUE=solstice.jobs.v1
• AMQP_DLX=solstice.dlx
• AMQP_DLQ=solstice.jobs.v1.dlq

---

Configuration

• HTTP address: HTTP_ADDR (default 0.0.0.0:8082)
• Webhook path: WEBHOOK_PATH (default /webhooks/github)
• Shared secret: GITHUB_WEBHOOK_SECRET (required in prod)
• GitHub API base: GITHUB_API_BASE (default https://api.github.com)
• GitHub App ID: GITHUB_APP_ID
• GitHub App key: GITHUB_APP_KEY_PATH or GITHUB_APP_KEY
• Check name: GITHUB_CHECK_NAME (default Solstice CI)
• Logs base URL: LOGS_BASE_URL (preferred) or ORCH_HTTP_BASE (deprecated)
• S3 upload: S3_ENDPOINT, S3_BUCKET
• Runs-on overrides: RUNS_ON_DEFAULT, RUNS_ON_MAP (owner/repo=label)

Example run:

export GITHUB_WEBHOOK_SECRET=devsecret
export GITHUB_APP_ID=123456
export GITHUB_APP_KEY_PATH=/path/to/app.pem
cargo run -p github-integration -- --http-addr 0.0.0.0:8082 --webhook-path /webhooks/github

---

GitHub App Setup (Minimum)

1. Create a GitHub App with webhook URL http://<your-host>:8082/webhooks/github and set the webhook secret.
2. Grant permissions:
   • Checks: Read & write
   • Contents: Read
   • Metadata: Read
3. Subscribe to events:
   • Push
   • Pull request
4. Install the App on the target repositories.

---

Notes & Next Steps

• Consider adding idempotency keyed by X-GitHub-Delivery to avoid duplicate enqueues.
• Consider supporting check_suite events for GitHub-native UI flows.
• Add optional allowlists/branch filters to reduce noise.