alexandre.pires
8724360fb2
- Updated AGENTS.md to include new directories for execution plans and implemented feature reports. - Modified Ansible host variables for dev-02 and gpu-01 to change API endpoints and add new configurations for Scaleway. - Adjusted firewall rules in gpu-01 to allow HTTPS traffic instead of the previous Ollama port. - Added OAuth2 proxy configurations to gpu-01 for enhanced security. - Removed deprecated open-webui module from Terraform app configurations. - Updated Terraform configurations to reflect changes in local variables and removed unused secrets. - Created a new sectool.json file for CloudNS integration with Bitwarden. - Enhanced SELinux configurations for nginx to allow connections to any port and added oauth2-proxy port.
234 lines
9.4 KiB
Markdown
234 lines
9.4 KiB
Markdown
# Gitea → MicroK8s Webhook Orchestrator → Scaleway Containers (CI Runner System)
|
|
|
|
A lightweight CI orchestration system where a self-hosted Gitea instance triggers a Go-based webhook service running inside MicroK8s. The service decides whether to run jobs locally or launch ephemeral CI runners in Scaleway Serverless Containers.
|
|
|
|
---
|
|
|
|
## 1. High-Level Architecture
|
|
|
|
```text
|
|
On-Premises (MicroK8s Cluster)
|
|
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ Gitea │
|
|
│ │
|
|
│ Repo push / PR event │
|
|
└───────────────┬──────────────────────────────────────────────┘
|
|
│ Webhook (internal cluster network)
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ ci-orchestrator (Go service) │
|
|
│ │
|
|
│ - Validates webhook signature (HMAC) │
|
|
│ - Parses event payload │
|
|
│ - Applies routing rules │
|
|
│ - Decides runner target (local vs cloud) │
|
|
└───────────────┬──────────────────────────────────────────────┘
|
|
│ HTTPS API call (authenticated)
|
|
▼
|
|
──────────────────────── Internet ───────────────────────────────
|
|
│
|
|
▼
|
|
┌──────────────────────────────────────────────────────────────┐
|
|
│ Scaleway Serverless Containers API │
|
|
│ │
|
|
│ - Starts ephemeral container │
|
|
│ - Runs Gitea Runner │
|
|
│ - Executes CI job │
|
|
│ - Shuts down after completion │
|
|
└──────────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Current State
|
|
|
|
| Component | Location | Status |
|
|
|---|---|---|
|
|
| **Gitea** | `prod-01.alexpires.me` (K8s) | Deployed via Terraform module, TLS, ingress, S3 storage, MariaDB |
|
|
| **MicroK8s** | `prod-01`, `dev-01` | nginx ingress, cert-manager, multiple apps running |
|
|
| **Scaleway** | `fr-par` | Registry + S3 buckets + IAM keys exist, no Serverless Containers yet |
|
|
| **Gitea Runners** | `ci-01.lab.alexpires.me` | Windows Podman + native runners already deployed |
|
|
| **DNS** | CoreDNS in `dev-01` | `ci-01 → 10.19.4.207` already resolved |
|
|
|
|
---
|
|
|
|
## 3. Design Decisions
|
|
|
|
1. **Orchestrator cluster:** `prod-01` for low latency to Gitea
|
|
2. **Routing:** Cloud runners only trigger when `cloud` label is present
|
|
3. **Runner image:** `gitea/act_runner` (official)
|
|
4. **Webhook registration:** Terraform-managed via `go-gitea/gitea` provider
|
|
|
|
---
|
|
|
|
## 4. Implementation Plan
|
|
|
|
### Phase 1: Scaleway IaC (`terraform/iac/scaleway/`)
|
|
|
|
**New Terraform resources:**
|
|
- `scaleway_container_namespace` — CI runners namespace
|
|
- `scaleway_container` — the `gitea/act_runner` container (ephemeral, privacy=private)
|
|
- `scaleway_iam_policy` — `ContainersInvoke` permission for the CI orchestrator app
|
|
- `scaleway_container_domain` (optional) — custom domain if desired
|
|
|
|
**IAM:**
|
|
- New app `ci_runner_app` with policy granting `ContainersInvoke`
|
|
- API key for the Go service to invoke containers
|
|
|
|
**Existing reuse:**
|
|
- `data.terraform_remote_state.scaleway` in `prod-01` for outputs
|
|
- Existing `scaleway_registry_namespace` for pushing the runner image
|
|
|
|
### Phase 2: Go Service (`ci-orchestrator/`)
|
|
|
|
```
|
|
ci-orchestrator/
|
|
├── go.mod
|
|
├── main.go # Entry: HTTP server
|
|
├── Dockerfile # Multi-stage, distroless
|
|
├── .dockerignore
|
|
└── internal/
|
|
├── webhook/
|
|
│ ├── gitea.go # Parse Gitea push/PR payloads
|
|
│ └── hmac.go # HMAC-SHA256 validation
|
|
├── router/
|
|
│ ├── router.go # Check if job has "cloud" label
|
|
│ └── rules.go # Config-based routing rules
|
|
└── runner/
|
|
└── scaleway.go # Invoke SCW Serverless Container
|
|
```
|
|
|
|
**Routing logic:**
|
|
- Parse webhook → extract repo, branch, commit, labels (from PR body or commit message convention)
|
|
- If `cloud` label → invoke Scaleway container
|
|
- If no label → ignore (existing runners on `ci-01` handle it)
|
|
|
|
**Endpoints:**
|
|
- `POST /webhook/gitea` — main webhook receiver
|
|
- `GET /health` — health check
|
|
|
|
### Phase 3: Runner Container Image
|
|
|
|
```dockerfile
|
|
FROM gitea/act_runner:latest
|
|
ENV GITEA_RUNNER_IGNORE_RUNNING_JOBS_IN_SCRIPT=true
|
|
# Entrypoint configured via Terraform env vars
|
|
```
|
|
|
|
Push to: `rg.fr-par.scw.cloud/<ci-namespace>/act-runner:<tag>`
|
|
|
|
### Phase 4: Terraform K8s Deployment (`terraform/modules/apps/ci-orchestrator/`)
|
|
|
|
Deployed in `prod-01` namespace:
|
|
|
|
| Resource | Purpose |
|
|
|---|---|
|
|
| `kubernetes_namespace` | `ci-orchestrator` |
|
|
| `kubernetes_deployment` | Go service (hardened security context like other modules) |
|
|
| `kubernetes_service` | ClusterIP, port 8080 |
|
|
| `kubernetes_ingress_v1` | Ingress via nginx, path `/webhook/gitea` |
|
|
| `kubernetes_config_map` | Routing rules config |
|
|
| `kubernetes_secret` | HMAC key, Scaleway API credentials, Gitea URL |
|
|
|
|
### Phase 5: Gitea Webhook Registration
|
|
|
|
```hcl
|
|
provider "gitea" {
|
|
address = "https://code.alexpires.me"
|
|
username = "<user>"
|
|
token = "<token>"
|
|
}
|
|
|
|
resource "gitea_repository_webhook" "ci_orchestrator" {
|
|
username = "<org-or-user>"
|
|
name = "<repo-name>"
|
|
type = "gitea"
|
|
url = "https://code.alexpires.me/webhook/gitea"
|
|
secret = var.webhook_secret
|
|
content_type = "json"
|
|
branch_filter = "*"
|
|
events = ["push", "pull_request"]
|
|
active = true
|
|
}
|
|
```
|
|
|
|
Can be per-repo or a single wildcard. The Go service routes based on labels.
|
|
|
|
### Phase 6: CI Flow
|
|
|
|
```
|
|
push/PR to Gitea
|
|
│
|
|
▼
|
|
Webhook → ci-orchestrator (prod-01)
|
|
│
|
|
├─ no cloud label → ignore (local runners handle it)
|
|
│
|
|
└─ cloud label → invoke Scaleway container
|
|
│
|
|
▼
|
|
gitea/act_runner starts
|
|
Registers with Gitea
|
|
Receives and executes job
|
|
Exits → container shuts down (cost: per-second)
|
|
```
|
|
|
|
---
|
|
|
|
## 5. Provider Compatibility
|
|
|
|
### `go-gitea/gitea` (v0.7.0)
|
|
- `gitea_repository_webhook` resource — creates and manages webhooks
|
|
- Supports: `url`, `secret`, `content_type`, `events`, `branch_filter`, `active`, `type`
|
|
|
|
### `scaleway/scaleway` (v2.77.1)
|
|
- `scaleway_container` — Serverless Container resource
|
|
- `privacy` can be set to `private` (requires IAM auth)
|
|
- `min_scale = 0`, `max_scale = 1` for ephemeral execution
|
|
- Supports `command`, `args`, `environment_variables`, `secret_environment_variables`
|
|
- `scaleway_container_namespace` — namespace for containers
|
|
- `scaleway_iam_policy` — IAM policies for container invocation
|
|
|
|
---
|
|
|
|
## 7. Knowns (Updated)
|
|
|
|
### Gitea
|
|
- URL: `https://code.alexpires.me`
|
|
- Admin user: `alexandre.pires`
|
|
- API token: `${GITEA_ADMIN_TOKEN}` (via sectool)
|
|
- Provider config: `gitea_credentials` in `terraform/iac/scaleway/` or `prod-01/`
|
|
|
|
### Webhook registration
|
|
- Target: all existing repos + new ones
|
|
- Method: one-time Ansible script to register webhooks on all repos, then run once per new repo
|
|
- Webhook target URL: `https://code.alexpires.me/webhook/gitea` (points to Go service ingress)
|
|
|
|
### Runner resources
|
|
- CPU: 2 vCPU
|
|
- Memory: 4Gi
|
|
- Image: `gitea/act_runner:latest` (or pinned version)
|
|
|
|
### Repository
|
|
- Moved to Gitea: `https://code.alexpires.me/alexpires.me/a13labs.infra`
|
|
- Origin remote updated to `ssh://git@code.alexpires.me:22/alexpires.me/a13labs.infra.git`
|
|
- All branches and tags pushed
|
|
|
|
### CI repo workflow
|
|
- `ci-01` runner builds Terraform images using Gitea Actions
|
|
- Images pushed to Scaleway Registry
|
|
- Terraform state managed via S3 on Scaleway
|
|
|
|
---
|
|
|
|
## 8. Open Decisions (to be addressed during implementation)
|
|
|
|
1. **Routing labels** — How will the Go service detect "cloud" intent? Options:
|
|
- PR body labels (`[cloud]` in title/body)
|
|
- Commit message convention (`[cloud]` prefix)
|
|
- Branch naming convention
|
|
- Separate "cloud" branch filter per repo
|
|
2. **Go service ingress path** — Current plan: `/webhook/gitea`. Need to ensure it doesn't conflict with Gitea's own routes.
|
|
3. **Scaling strategy** — `max_scale=1` per invocation, but multiple concurrent jobs? May need pool or queue.
|