Enhance infrastructure and application configurations
- Updated AGENTS.md to include new directories for execution plans and implemented feature reports. - Modified Ansible host variables for dev-02 and gpu-01 to change API endpoints and add new configurations for Scaleway. - Adjusted firewall rules in gpu-01 to allow HTTPS traffic instead of the previous Ollama port. - Added OAuth2 proxy configurations to gpu-01 for enhanced security. - Removed deprecated open-webui module from Terraform app configurations. - Updated Terraform configurations to reflect changes in local variables and removed unused secrets. - Created a new sectool.json file for CloudNS integration with Bitwarden. - Enhanced SELinux configurations for nginx to allow connections to any port and added oauth2-proxy port.
This commit is contained in:
@@ -0,0 +1,233 @@
|
||||
# Gitea → MicroK8s Webhook Orchestrator → Scaleway Containers (CI Runner System)
|
||||
|
||||
A lightweight CI orchestration system where a self-hosted Gitea instance triggers a Go-based webhook service running inside MicroK8s. The service decides whether to run jobs locally or launch ephemeral CI runners in Scaleway Serverless Containers.
|
||||
|
||||
---
|
||||
|
||||
## 1. High-Level Architecture
|
||||
|
||||
```text
|
||||
On-Premises (MicroK8s Cluster)
|
||||
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Gitea │
|
||||
│ │
|
||||
│ Repo push / PR event │
|
||||
└───────────────┬──────────────────────────────────────────────┘
|
||||
│ Webhook (internal cluster network)
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ ci-orchestrator (Go service) │
|
||||
│ │
|
||||
│ - Validates webhook signature (HMAC) │
|
||||
│ - Parses event payload │
|
||||
│ - Applies routing rules │
|
||||
│ - Decides runner target (local vs cloud) │
|
||||
└───────────────┬──────────────────────────────────────────────┘
|
||||
│ HTTPS API call (authenticated)
|
||||
▼
|
||||
──────────────────────── Internet ───────────────────────────────
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Scaleway Serverless Containers API │
|
||||
│ │
|
||||
│ - Starts ephemeral container │
|
||||
│ - Runs Gitea Runner │
|
||||
│ - Executes CI job │
|
||||
│ - Shuts down after completion │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Current State
|
||||
|
||||
| Component | Location | Status |
|
||||
|---|---|---|
|
||||
| **Gitea** | `prod-01.alexpires.me` (K8s) | Deployed via Terraform module, TLS, ingress, S3 storage, MariaDB |
|
||||
| **MicroK8s** | `prod-01`, `dev-01` | nginx ingress, cert-manager, multiple apps running |
|
||||
| **Scaleway** | `fr-par` | Registry + S3 buckets + IAM keys exist, no Serverless Containers yet |
|
||||
| **Gitea Runners** | `ci-01.lab.alexpires.me` | Windows Podman + native runners already deployed |
|
||||
| **DNS** | CoreDNS in `dev-01` | `ci-01 → 10.19.4.207` already resolved |
|
||||
|
||||
---
|
||||
|
||||
## 3. Design Decisions
|
||||
|
||||
1. **Orchestrator cluster:** `prod-01` for low latency to Gitea
|
||||
2. **Routing:** Cloud runners only trigger when `cloud` label is present
|
||||
3. **Runner image:** `gitea/act_runner` (official)
|
||||
4. **Webhook registration:** Terraform-managed via `go-gitea/gitea` provider
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation Plan
|
||||
|
||||
### Phase 1: Scaleway IaC (`terraform/iac/scaleway/`)
|
||||
|
||||
**New Terraform resources:**
|
||||
- `scaleway_container_namespace` — CI runners namespace
|
||||
- `scaleway_container` — the `gitea/act_runner` container (ephemeral, privacy=private)
|
||||
- `scaleway_iam_policy` — `ContainersInvoke` permission for the CI orchestrator app
|
||||
- `scaleway_container_domain` (optional) — custom domain if desired
|
||||
|
||||
**IAM:**
|
||||
- New app `ci_runner_app` with policy granting `ContainersInvoke`
|
||||
- API key for the Go service to invoke containers
|
||||
|
||||
**Existing reuse:**
|
||||
- `data.terraform_remote_state.scaleway` in `prod-01` for outputs
|
||||
- Existing `scaleway_registry_namespace` for pushing the runner image
|
||||
|
||||
### Phase 2: Go Service (`ci-orchestrator/`)
|
||||
|
||||
```
|
||||
ci-orchestrator/
|
||||
├── go.mod
|
||||
├── main.go # Entry: HTTP server
|
||||
├── Dockerfile # Multi-stage, distroless
|
||||
├── .dockerignore
|
||||
└── internal/
|
||||
├── webhook/
|
||||
│ ├── gitea.go # Parse Gitea push/PR payloads
|
||||
│ └── hmac.go # HMAC-SHA256 validation
|
||||
├── router/
|
||||
│ ├── router.go # Check if job has "cloud" label
|
||||
│ └── rules.go # Config-based routing rules
|
||||
└── runner/
|
||||
└── scaleway.go # Invoke SCW Serverless Container
|
||||
```
|
||||
|
||||
**Routing logic:**
|
||||
- Parse webhook → extract repo, branch, commit, labels (from PR body or commit message convention)
|
||||
- If `cloud` label → invoke Scaleway container
|
||||
- If no label → ignore (existing runners on `ci-01` handle it)
|
||||
|
||||
**Endpoints:**
|
||||
- `POST /webhook/gitea` — main webhook receiver
|
||||
- `GET /health` — health check
|
||||
|
||||
### Phase 3: Runner Container Image
|
||||
|
||||
```dockerfile
|
||||
FROM gitea/act_runner:latest
|
||||
ENV GITEA_RUNNER_IGNORE_RUNNING_JOBS_IN_SCRIPT=true
|
||||
# Entrypoint configured via Terraform env vars
|
||||
```
|
||||
|
||||
Push to: `rg.fr-par.scw.cloud/<ci-namespace>/act-runner:<tag>`
|
||||
|
||||
### Phase 4: Terraform K8s Deployment (`terraform/modules/apps/ci-orchestrator/`)
|
||||
|
||||
Deployed in `prod-01` namespace:
|
||||
|
||||
| Resource | Purpose |
|
||||
|---|---|
|
||||
| `kubernetes_namespace` | `ci-orchestrator` |
|
||||
| `kubernetes_deployment` | Go service (hardened security context like other modules) |
|
||||
| `kubernetes_service` | ClusterIP, port 8080 |
|
||||
| `kubernetes_ingress_v1` | Ingress via nginx, path `/webhook/gitea` |
|
||||
| `kubernetes_config_map` | Routing rules config |
|
||||
| `kubernetes_secret` | HMAC key, Scaleway API credentials, Gitea URL |
|
||||
|
||||
### Phase 5: Gitea Webhook Registration
|
||||
|
||||
```hcl
|
||||
provider "gitea" {
|
||||
address = "https://code.alexpires.me"
|
||||
username = "<user>"
|
||||
token = "<token>"
|
||||
}
|
||||
|
||||
resource "gitea_repository_webhook" "ci_orchestrator" {
|
||||
username = "<org-or-user>"
|
||||
name = "<repo-name>"
|
||||
type = "gitea"
|
||||
url = "https://code.alexpires.me/webhook/gitea"
|
||||
secret = var.webhook_secret
|
||||
content_type = "json"
|
||||
branch_filter = "*"
|
||||
events = ["push", "pull_request"]
|
||||
active = true
|
||||
}
|
||||
```
|
||||
|
||||
Can be per-repo or a single wildcard. The Go service routes based on labels.
|
||||
|
||||
### Phase 6: CI Flow
|
||||
|
||||
```
|
||||
push/PR to Gitea
|
||||
│
|
||||
▼
|
||||
Webhook → ci-orchestrator (prod-01)
|
||||
│
|
||||
├─ no cloud label → ignore (local runners handle it)
|
||||
│
|
||||
└─ cloud label → invoke Scaleway container
|
||||
│
|
||||
▼
|
||||
gitea/act_runner starts
|
||||
Registers with Gitea
|
||||
Receives and executes job
|
||||
Exits → container shuts down (cost: per-second)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Provider Compatibility
|
||||
|
||||
### `go-gitea/gitea` (v0.7.0)
|
||||
- `gitea_repository_webhook` resource — creates and manages webhooks
|
||||
- Supports: `url`, `secret`, `content_type`, `events`, `branch_filter`, `active`, `type`
|
||||
|
||||
### `scaleway/scaleway` (v2.77.1)
|
||||
- `scaleway_container` — Serverless Container resource
|
||||
- `privacy` can be set to `private` (requires IAM auth)
|
||||
- `min_scale = 0`, `max_scale = 1` for ephemeral execution
|
||||
- Supports `command`, `args`, `environment_variables`, `secret_environment_variables`
|
||||
- `scaleway_container_namespace` — namespace for containers
|
||||
- `scaleway_iam_policy` — IAM policies for container invocation
|
||||
|
||||
---
|
||||
|
||||
## 7. Knowns (Updated)
|
||||
|
||||
### Gitea
|
||||
- URL: `https://code.alexpires.me`
|
||||
- Admin user: `alexandre.pires`
|
||||
- API token: `${GITEA_ADMIN_TOKEN}` (via sectool)
|
||||
- Provider config: `gitea_credentials` in `terraform/iac/scaleway/` or `prod-01/`
|
||||
|
||||
### Webhook registration
|
||||
- Target: all existing repos + new ones
|
||||
- Method: one-time Ansible script to register webhooks on all repos, then run once per new repo
|
||||
- Webhook target URL: `https://code.alexpires.me/webhook/gitea` (points to Go service ingress)
|
||||
|
||||
### Runner resources
|
||||
- CPU: 2 vCPU
|
||||
- Memory: 4Gi
|
||||
- Image: `gitea/act_runner:latest` (or pinned version)
|
||||
|
||||
### Repository
|
||||
- Moved to Gitea: `https://code.alexpires.me/alexpires.me/a13labs.infra`
|
||||
- Origin remote updated to `ssh://git@code.alexpires.me:22/alexpires.me/a13labs.infra.git`
|
||||
- All branches and tags pushed
|
||||
|
||||
### CI repo workflow
|
||||
- `ci-01` runner builds Terraform images using Gitea Actions
|
||||
- Images pushed to Scaleway Registry
|
||||
- Terraform state managed via S3 on Scaleway
|
||||
|
||||
---
|
||||
|
||||
## 8. Open Decisions (to be addressed during implementation)
|
||||
|
||||
1. **Routing labels** — How will the Go service detect "cloud" intent? Options:
|
||||
- PR body labels (`[cloud]` in title/body)
|
||||
- Commit message convention (`[cloud]` prefix)
|
||||
- Branch naming convention
|
||||
- Separate "cloud" branch filter per repo
|
||||
2. **Go service ingress path** — Current plan: `/webhook/gitea`. Need to ensure it doesn't conflict with Gitea's own routes.
|
||||
3. **Scaling strategy** — `max_scale=1` per invocation, but multiple concurrent jobs? May need pool or queue.
|
||||
Reference in New Issue
Block a user