diff --git a/.opencode/plans/ci-orchestrator.md b/.opencode/plans/ci-orchestrator.md new file mode 100644 index 0000000..91ca637 --- /dev/null +++ b/.opencode/plans/ci-orchestrator.md @@ -0,0 +1,233 @@ +# Gitea → MicroK8s Webhook Orchestrator → Scaleway Containers (CI Runner System) + +A lightweight CI orchestration system where a self-hosted Gitea instance triggers a Go-based webhook service running inside MicroK8s. The service decides whether to run jobs locally or launch ephemeral CI runners in Scaleway Serverless Containers. + +--- + +## 1. High-Level Architecture + +```text + On-Premises (MicroK8s Cluster) + +┌──────────────────────────────────────────────────────────────┐ +│ Gitea │ +│ │ +│ Repo push / PR event │ +└───────────────┬──────────────────────────────────────────────┘ + │ Webhook (internal cluster network) + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ ci-orchestrator (Go service) │ +│ │ +│ - Validates webhook signature (HMAC) │ +│ - Parses event payload │ +│ - Applies routing rules │ +│ - Decides runner target (local vs cloud) │ +└───────────────┬──────────────────────────────────────────────┘ + │ HTTPS API call (authenticated) + ▼ +──────────────────────── Internet ─────────────────────────────── + │ + ▼ +┌──────────────────────────────────────────────────────────────┐ +│ Scaleway Serverless Containers API │ +│ │ +│ - Starts ephemeral container │ +│ - Runs Gitea Runner │ +│ - Executes CI job │ +│ - Shuts down after completion │ +└──────────────────────────────────────────────────────────────┘ +``` + +--- + +## 2. Current State + +| Component | Location | Status | +|---|---|---| +| **Gitea** | `prod-01.alexpires.me` (K8s) | Deployed via Terraform module, TLS, ingress, S3 storage, MariaDB | +| **MicroK8s** | `prod-01`, `dev-01` | nginx ingress, cert-manager, multiple apps running | +| **Scaleway** | `fr-par` | Registry + S3 buckets + IAM keys exist, no Serverless Containers yet | +| **Gitea Runners** | `ci-01.lab.alexpires.me` | Windows Podman + native runners already deployed | +| **DNS** | CoreDNS in `dev-01` | `ci-01 → 10.19.4.207` already resolved | + +--- + +## 3. Design Decisions + +1. **Orchestrator cluster:** `prod-01` for low latency to Gitea +2. **Routing:** Cloud runners only trigger when `cloud` label is present +3. **Runner image:** `gitea/act_runner` (official) +4. **Webhook registration:** Terraform-managed via `go-gitea/gitea` provider + +--- + +## 4. Implementation Plan + +### Phase 1: Scaleway IaC (`terraform/iac/scaleway/`) + +**New Terraform resources:** +- `scaleway_container_namespace` — CI runners namespace +- `scaleway_container` — the `gitea/act_runner` container (ephemeral, privacy=private) +- `scaleway_iam_policy` — `ContainersInvoke` permission for the CI orchestrator app +- `scaleway_container_domain` (optional) — custom domain if desired + +**IAM:** +- New app `ci_runner_app` with policy granting `ContainersInvoke` +- API key for the Go service to invoke containers + +**Existing reuse:** +- `data.terraform_remote_state.scaleway` in `prod-01` for outputs +- Existing `scaleway_registry_namespace` for pushing the runner image + +### Phase 2: Go Service (`ci-orchestrator/`) + +``` +ci-orchestrator/ +├── go.mod +├── main.go # Entry: HTTP server +├── Dockerfile # Multi-stage, distroless +├── .dockerignore +└── internal/ + ├── webhook/ + │ ├── gitea.go # Parse Gitea push/PR payloads + │ └── hmac.go # HMAC-SHA256 validation + ├── router/ + │ ├── router.go # Check if job has "cloud" label + │ └── rules.go # Config-based routing rules + └── runner/ + └── scaleway.go # Invoke SCW Serverless Container +``` + +**Routing logic:** +- Parse webhook → extract repo, branch, commit, labels (from PR body or commit message convention) +- If `cloud` label → invoke Scaleway container +- If no label → ignore (existing runners on `ci-01` handle it) + +**Endpoints:** +- `POST /webhook/gitea` — main webhook receiver +- `GET /health` — health check + +### Phase 3: Runner Container Image + +```dockerfile +FROM gitea/act_runner:latest +ENV GITEA_RUNNER_IGNORE_RUNNING_JOBS_IN_SCRIPT=true +# Entrypoint configured via Terraform env vars +``` + +Push to: `rg.fr-par.scw.cloud//act-runner:` + +### Phase 4: Terraform K8s Deployment (`terraform/modules/apps/ci-orchestrator/`) + +Deployed in `prod-01` namespace: + +| Resource | Purpose | +|---|---| +| `kubernetes_namespace` | `ci-orchestrator` | +| `kubernetes_deployment` | Go service (hardened security context like other modules) | +| `kubernetes_service` | ClusterIP, port 8080 | +| `kubernetes_ingress_v1` | Ingress via nginx, path `/webhook/gitea` | +| `kubernetes_config_map` | Routing rules config | +| `kubernetes_secret` | HMAC key, Scaleway API credentials, Gitea URL | + +### Phase 5: Gitea Webhook Registration + +```hcl +provider "gitea" { + address = "https://code.alexpires.me" + username = "" + token = "" +} + +resource "gitea_repository_webhook" "ci_orchestrator" { + username = "" + name = "" + type = "gitea" + url = "https://code.alexpires.me/webhook/gitea" + secret = var.webhook_secret + content_type = "json" + branch_filter = "*" + events = ["push", "pull_request"] + active = true +} +``` + +Can be per-repo or a single wildcard. The Go service routes based on labels. + +### Phase 6: CI Flow + +``` +push/PR to Gitea + │ + ▼ +Webhook → ci-orchestrator (prod-01) + │ + ├─ no cloud label → ignore (local runners handle it) + │ + └─ cloud label → invoke Scaleway container + │ + ▼ + gitea/act_runner starts + Registers with Gitea + Receives and executes job + Exits → container shuts down (cost: per-second) +``` + +--- + +## 5. Provider Compatibility + +### `go-gitea/gitea` (v0.7.0) +- `gitea_repository_webhook` resource — creates and manages webhooks +- Supports: `url`, `secret`, `content_type`, `events`, `branch_filter`, `active`, `type` + +### `scaleway/scaleway` (v2.77.1) +- `scaleway_container` — Serverless Container resource + - `privacy` can be set to `private` (requires IAM auth) + - `min_scale = 0`, `max_scale = 1` for ephemeral execution + - Supports `command`, `args`, `environment_variables`, `secret_environment_variables` +- `scaleway_container_namespace` — namespace for containers +- `scaleway_iam_policy` — IAM policies for container invocation + +--- + +## 7. Knowns (Updated) + +### Gitea +- URL: `https://code.alexpires.me` +- Admin user: `alexandre.pires` +- API token: `${GITEA_ADMIN_TOKEN}` (via sectool) +- Provider config: `gitea_credentials` in `terraform/iac/scaleway/` or `prod-01/` + +### Webhook registration +- Target: all existing repos + new ones +- Method: one-time Ansible script to register webhooks on all repos, then run once per new repo +- Webhook target URL: `https://code.alexpires.me/webhook/gitea` (points to Go service ingress) + +### Runner resources +- CPU: 2 vCPU +- Memory: 4Gi +- Image: `gitea/act_runner:latest` (or pinned version) + +### Repository +- Moved to Gitea: `https://code.alexpires.me/alexpires.me/a13labs.infra` +- Origin remote updated to `ssh://git@code.alexpires.me:22/alexpires.me/a13labs.infra.git` +- All branches and tags pushed + +### CI repo workflow +- `ci-01` runner builds Terraform images using Gitea Actions +- Images pushed to Scaleway Registry +- Terraform state managed via S3 on Scaleway + +--- + +## 8. Open Decisions (to be addressed during implementation) + +1. **Routing labels** — How will the Go service detect "cloud" intent? Options: + - PR body labels (`[cloud]` in title/body) + - Commit message convention (`[cloud]` prefix) + - Branch naming convention + - Separate "cloud" branch filter per repo +2. **Go service ingress path** — Current plan: `/webhook/gitea`. Need to ensure it doesn't conflict with Gitea's own routes. +3. **Scaling strategy** — `max_scale=1` per invocation, but multiple concurrent jobs? May need pool or queue. diff --git a/.opencode/plans/gateway-api-migration.md b/.opencode/plans/gateway-api-migration.md new file mode 100644 index 0000000..d6dffc8 --- /dev/null +++ b/.opencode/plans/gateway-api-migration.md @@ -0,0 +1,321 @@ +# Gateway API Migration & Cloudflare Integration + +## Overview + +Migrate prod-01 and dev-01 clusters from the deprecated nginx-ingress controller (retired March 2026) to NGINX Gateway Fabric via the Gateway API, improve cross-cluster WireGuard connectivity, and add Cloudflare Tunnel for edge protection. + +## Current Architecture + +``` +Internet → Contabo Public IP → OPNsense → prod-01 + ↓ + nginx-ingress plugin (MicroK8s) + ↓ + MetalLB (10.2.0.40-49) + ↓ + ┌───────────────┼─────────────────┐ + ↓ ↓ ↓ + [proxy pod] [app pods] [TCP streams] + (nginx + WG) (gitea, m3uproxy, etc) (hostPort) + ↓ + WireGuard → aristotle (Contabo VPN) + ↓ + dev-01 backends (open-webui, nextcloud) +``` + +**Key facts:** +- 14 `kubernetes_ingress_v1` resources across both clusters +- All use `ingress.class: public` + nginx-specific annotations +- Proxy module on prod-01 creates a double-proxy (Ingress → proxy pod → backends) +- TCP/UDP streams use nginx with hostPorts +- DNS: ClouDNS, no Cloudflare involvement +- cert-manager with DNS-01 (dev-01) and HTTP-01 (prod-01) for Let's Encrypt +- **Both clusters are single-node** — no need for MetalLB, NodePort is sufficient + +## Decisions Made + +| Decision | Choice | Rationale | +|---|---|---| +| Gateway API implementation | **NGINX Gateway Fabric** | Keeps nginx data plane, minimal behavioral changes, conformant v1.4.1 | +| Cross-cluster connectivity | **Improve WireGuard** | Keep through aristotle, allowed_ips to dev-01 only (10.19.4.136/32), remove gpu-01 | +| Ollama removal | **Remove** | gpu-01 runs llamacpp (not ollama), prod-01 ollama proxy is broken, dev-01 DNS CNAME stale | +| Cloudflare scope | **Tunnels only** | Get DDoS+WAF, keep DNS with ClouDNS, least disruptive | +| Proxy module | **All through proxy** | Gateway routes everything to proxy Service, simpler Terraform | +| Website module | **Direct Gateway route** | Keep separate pods, no proxy overhead for static sites | +| TLS termination | **At Cloudflare edge** | Simpler cert management, Cloudflare provides origin cert | + +## Target Architecture + +``` +Internet → Cloudflare Tunnel → Cloudflare Edge (TLS termination) + ↓ + cloudflared pods (DaemonSet) + ↓ + Gateway (NGINX Gateway Fabric, NodePort Service) + HTTP listener :80 (from cloudflared) + ↓ + ┌───────────────┼─────────────────┐ + ↓ ↓ ↓ + [proxy Service] [app Services] [TCP/UDP Routes] + ↓ ↓ ↓ + [proxy pod + WG] [direct pods] [direct pods] + (nginx) (website, etc.) (rustdesk, etc.) + ↓ + Cross-cluster → dev-01 backends + (open-webui, nextcloud) +``` + +--- + +## Phase 1: Gateway API Migration + +### 1A. Install NGINX Gateway Fabric (Ansible) + +**New file: `ansible/roles/microk8s/tasks/gateway-api.yml`** + +``` +- Install `snap` package manager (if needed) +- Install helm via snap +- Add nginx-stable helm repo +- Install Gateway API CRDs (standard channel + experimental for TCPRoute/UDPRoute) +- Install NGINX Gateway Fabric via Helm (experimental features enabled) +- Create GatewayClass named `public` +- Create Gateway Service with type = NodePort (single-node, no MetalLB needed) +``` + +**Modify: `ansible/roles/microk8s/tasks/main.yml`** +- Include `gateway-api.yml` AFTER existing setup (runs alongside nginx-ingress initially) + +**Modify: `ansible/roles/microk8s/defaults/main.yml`** +- Add variables: `gateway_fabric_version`, `gateway_fabric_experimental_enabled` +- Remove MetalLB-related vars (`microk8s_lb_provider`, `microk8s_lb_ips`, `microk8s_metallb_version`) + +### 1B. Create Gateway Module (Terraform) + +**New module: `terraform/modules/utils/gateway/`** +- `main.tf`: Gateway resource with HTTP (:80) listener, `ServiceType: NodePort` +- `variables.tf`: TLS secret names, listener config +- `outputs.tf`: Gateway service name, NodePort port numbers + +**Deploy in both clusters** (`prod-01/apps.tf`, `dev-01/apps.tf`) + +### 1C. Migrate Terraform Modules (Ingress → Gateway API Resources) + +#### Proxy module (`terraform/modules/utils/proxy/main.tf`) + +- **Remove**: `kubernetes_ingress_v1.ingress` resource (lines 160-203) +- **Remove**: `host_port` from stream container ports in Deployment +- **Remove**: ollama proxy service definition from prod-01 config.tf +- **Add**: HTTPRoute for cross-cluster services (open-webui, nextcloud) +- **Add**: TCPRoute resources for RustDesk (21115, 21116, 21117 TCP) and Prometheus (9090) +- **Add**: UDPRoute resource for RustDesk (21116 UDP) +- The proxy pod's nginx stream block can be removed (Gateway handles L4 routing) +- **Keep**: WireGuard sidecar, auth sidecar, internal nginx reverse proxy config + +#### Website module (`terraform/modules/apps/nginx/main.tf`) + +- **Remove**: `kubernetes_ingress_v1.website_ingress` (lines 103-145) +- **Add**: HTTPRoute per website with backendRef → website Service +- Gateway handles TLS termination, website pods receive plain HTTP +- Website pods can drop internal TLS (simplification) + +#### Gitea module (`terraform/modules/apps/gitea/ingress.tf`) + +- **Remove**: `kubernetes_ingress_v1.gitea_ingress` +- **Add**: HTTPRoute with backendRef → Gitea Service +- Use `BackendTLSPolicy` if Gitea serves HTTPS internally + +#### M3UProxy module (`terraform/modules/apps/m3uproxy/main.tf`) + +- **Remove**: `kubernetes_ingress_v1.m3uproxy_ingress_http` (lines 241-274) +- **Add**: HTTPRoute with backendRef → m3uproxy Service + +#### dev-01 modules + +| Module | Current | Change | +|---|---|---| +| `open-webui/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute | +| `grafana/grafana.tf` | kubernetes_ingress_v1 | HTTPRoute | +| `mstream/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute | +| `searxng/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute | +| `vaultwarden/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute | +| `terraform-mcp/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute | +| `nextcloud/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute | +| `proxy/main.tf` | kubernetes_ingress_v1 | HTTPRoute (opencode, vscode) | + +#### Annotation → Gateway API Mapping + +| Current nginx annotation | Gateway API equivalent | +|---|---| +| `nginx.org/ssl-redirect: true` | HTTPRoute with `RequestRedirect` filter | +| `nginx.org/redirect-to-https: true` | HTTPRoute on HTTP listener with redirect | +| `nginx.ingress.kubernetes.io/backend-protocol: HTTPS` | `BackendTLSPolicy` resource | +| `nginx.ingress.kubernetes.io/proxy-body-size` | `UpstreamSettings` policy (NGF custom) | +| `nginx.ingress.kubernetes.io/proxy-request-buffering` | `ProxySettings` policy (NGF custom) | +| `nginx.ingress.kubernetes.io/proxy-buffer-size` | `UpstreamSettings` policy | +| `cert-manager.io/cluster-issuer` | cert-manager annotation on Gateway | + +### 1D. cert-manager Integration + +Use **approach B**: Keep existing Certificate resources, reference secrets in Gateway TLS config. + +- Each module's `kubernetes_manifest` Certificate resource stays unchanged +- Gateway HTTPS listener references cert secrets via `certificateRefs` +- This avoids alpha cert-manager Gateway integration + +### 1E. Migration Execution Order + +1. Apply new Ansible role (installs Gateway Fabric alongside nginx-ingress) +2. Deploy Gateway resource to both clusters +3. Migrate modules on **dev-01 first** (lower risk): + - Deploy HTTPRoutes for each module, one at a time + - Verify each service works + - Remove old Ingress resources +4. Migrate modules on **prod-01**: + - Same process, one module at a time +5. Disable nginx-ingress plugin on both clusters +6. Clean up IngressClass, old LoadBalancer Service + +--- + +## Phase 2: Cross-Cluster Connectivity + +### 2A. Improve WireGuard + +**Modify: `terraform/apps/prod-01/config.tf`** (WireGuard config): +- Change `allowed_ips` from `[10.19.4.106/32, 10.19.4.136/32]` → `["10.19.4.136/32"]` (dev-01 only) +- Remove gpu-01 (10.19.4.106) - ollama proxy being removed, no need to reach gpu-01 +- `persistent_keepalive` already set to 25 + +**Modify: `terraform/modules/utils/proxy/resources/wireguard/wireguard.conf.tpl`**: +- Template `PersistentKeepalive` from config variable +- Template `AllowedIPs` from config list + +**Also clean up**: +- Remove `"ollama"` entry from `proxy_services` in prod-01 config.tf +- Remove `ollama` CNAME from dev-01 DNS zones in dev-01 config.tf + +**Modify: `terraform/modules/utils/proxy/main.tf`** (WireGuard container): +- Improve liveness probe: `wg show` instead of `wg show | grep -q transfer` +- Add readiness probe +- Add resource limits (memory cap) + +### 2B. DNS Resolution + +No changes needed. The proxy's per-location resolver in `default.conf.tpl` already handles DNS correctly for cross-cluster backends. + +--- + +## Phase 3: Cloudflare Tunnels + +### 3A. Cloudflare Setup + +- Create/verify Cloudflare account with `alexpires.me` domain +- Keep DNS with ClouDNS (Cloudflare acts as reverse proxy only) +- Create Cloudflare Tunnel for prod-01 +- Configure route rules: `https://*.alexpires.me` → Gateway Service +- Set up origin rules (TLS terminates at Cloudflare edge → HTTP to cloudflared) + +### 3B. Deploy cloudflared + +**New module: `terraform/modules/utils/cloudflare-tunnel/`** +- DaemonSet of `cloudflare/cloudflared` pods +- ConfigMap with tunnel credentials and route definitions +- Each public FQDN mapped to `http://.:80` + +### 3C. Gateway Configuration Changes + +- Gateway uses HTTP (:80) NodePort listener for Cloudflare Tunnel traffic +- Cloudflare Tunnel connects to `http://:` +- No HTTPS listener needed since all traffic comes through Cloudflare Tunnel + +### 3D. Firewall Changes + +- Close UFW ports 80/443 (cloudflared uses ephemeral outbound ports) +- Keep SSH + WireGuard ports open + +### 3E. Security Benefits + +With Cloudflare in front: +- No open ports on prod-01 (80, 443, etc. closed) +- DDoS protection (automatic with Cloudflare) +- WAF rules (OWASP Top 10, custom rules) +- Rate limiting on public endpoints +- Bot management + +--- + +## Phase 4: Cleanup + +- Remove nginx-ingress plugin from MicroK8s (`microk8s.disable ingress`) +- Remove unused IngressClass resources +- Remove existing MetalLB (delete metallb-system namespace, IPAddressPool, L2Advertisement) +- Delete orphaned `tasks/loadbalancer.yml` and `tasks/loadbalancer/metallb.yaml` +- Remove MetalLB vars from `defaults/main.yml` +- Update documentation + +--- + +## File Change Summary + +### New Files +| File | Purpose | +|---|---| +| `ansible/roles/microk8s/tasks/gateway-api.yml` | NGINX Gateway Fabric install | +| `terraform/modules/utils/gateway/main.tf` | Gateway + ReferenceGrant resources | +| `terraform/modules/utils/gateway/variables.tf` | Gateway configuration | +| `terraform/modules/utils/gateway/outputs.tf` | Gateway outputs | +| `terraform/modules/utils/cloudflare-tunnel/main.tf` | cloudflared DaemonSet | +| `terraform/modules/utils/cloudflare-tunnel/variables.tf` | Tunnel config | + +### Deleted Files +| File | Reason | +|---|---| +| `ansible/roles/microk8s/tasks/loadbalancer.yml` | Not wired into main.yml, was for nginx-ingress LB | +| `ansible/roles/microk8s/tasks/loadbalancer/metallb.yaml` | MetalLB no longer needed (single-node, NodePort) | + +### Modified Files +| File | Change | +|---|---| +| `ansible/roles/microk8s/tasks/main.yml` | Include gateway-api.yml | +| `ansible/roles/microk8s/defaults/main.yml` | Add Gateway Fabric defaults, remove MetalLB vars | +| `terraform/modules/utils/proxy/main.tf` | Ingress → HTTPRoute + TCPRoute + UDPRoute | +| `terraform/modules/utils/proxy/variables.tf` | Update for Gateway API | +| `terraform/modules/utils/proxy/resources/wireguard/wireguard.conf.tpl` | Add PersistentKeepalive | +| `terraform/modules/apps/nginx/main.tf` | Ingress → HTTPRoute | +| `terraform/modules/apps/gitea/ingress.tf` | Ingress → HTTPRoute | +| `terraform/modules/apps/m3uproxy/main.tf` | Ingress → HTTPRoute | +| `terraform/modules/apps/open-webui/ingress.tf` | Ingress → HTTPRoute | +| `terraform/modules/apps/grafana/grafana.tf` | Ingress → HTTPRoute | +| `terraform/modules/apps/mstream/ingress.tf` | Ingress → HTTPRoute | +| `terraform/modules/apps/searxng/ingress.tf` | Ingress → HTTPRoute | +| `terraform/modules/apps/vaultwarden/ingress.tf` | Ingress → HTTPRoute | +| `terraform/modules/apps/terraform-mcp/ingress.tf` | Ingress → HTTPRoute | +| `terraform/modules/apps/nextcloud/ingress.tf` | Ingress → HTTPRoute | +| `terraform/modules/apps/ssh_locker_web/` | Ingress → HTTPRoute | +| `terraform/apps/prod-01/apps.tf` | Add gateway module | +| `terraform/apps/prod-01/config.tf` | WireGuard (remove gpu-01), remove ollama proxy | +| `terraform/apps/dev-01/config.tf` | Remove ollama DNS CNAME | +| `terraform/apps/dev-01/apps.tf` | Add gateway module | + +--- + +## Risk Assessment + +| Risk | Mitigation | +|---|---| +| TCPRoute/UDPRoute experimental in NGF | Start with HTTPRoute only, keep proxy streams temporarily | +| Cloudflare Tunnel adds latency | Monitor, acceptable for reverse proxy traffic | +| cert-manager Gateway integration alpha | Use approach B (separate Certificate resources) | +| Downtime during migration | Run both controllers in parallel, switch via DNS TTL | +| Removing gpu-01 from WireGuard allowed_ips | Verify no active proxy depends on gpu-01 reachability | +| Removing existing MetalLB on prod-01 | Do after Cloudflare Tunnel is fully operational | + +## Timeline Estimate + +| Phase | Duration | Dependencies | +|---|---|---| +| Phase 1 (Gateway API) | 2-3 weeks | None | +| Phase 2 (WireGuard) | 1-2 weeks | Can run parallel with Phase 1 | +| Phase 3 (Cloudflare) | 1-2 weeks | After Phase 1 | +| Phase 4 (Cleanup) | 1 week | After Phase 3 | diff --git a/.opencode/plans/gitea-actions-migration.md b/.opencode/plans/gitea-actions-migration.md new file mode 100644 index 0000000..9e8bbf3 --- /dev/null +++ b/.opencode/plans/gitea-actions-migration.md @@ -0,0 +1,170 @@ +# Gitea Actions Migration Plan — Validate-Only Phase + +## Overview + +Migrate validate-only workflows from `.github/workflows/` (all `.disabled`) to `.gitea/workflows/` on the `alexpires.me/a13labs.infra` repo. Runs on `cicd-base` runner label. Terraform removed entirely — OpenTofu only. + +## Status: IMPLEMENTED + +All files created, secrets configured as placeholders. Ready for push and testing. + +## Scope + +### Workflows In Scope (6) + +| Workflow | Trigger | Path Filter | What It Does | +|---|---|---|---| +| `ansible-validate-on-pr` | PR (opened/synchronize/reopened) → `main` | `ansible/**` | ansible-lint + syntax-check | +| `aws-validate-on-pr` | PR → `main` | `terraform/iac/aws/**` | tofu init, validate, fmt -check, plan | +| `cloudns-validate-on-pr` | PR → `main` | `terraform/iac/cloudns/**` | tofu init, validate, fmt -check, plan | +| `contabo-validate-on-pr` | PR → `main` | `terraform/iac/contabo/**` | tofu init, validate, fmt -check, plan | +| `scaleway-validate-on-pr` | PR → `main` | `terraform/iac/scaleway/**` | tofu init, validate, fmt -check, plan | +| `k8s-apps-validate` | PR → `main` | `terraform/apps/**` | ansible+k8s env + tofu validate dev-01 & prod-01 | + +### Workflows Deferred (8) + +All apply workflows, `deploy-websites`, `actions-validate`, `ansible-validate-playbook-on-pr` — out of scope. + +## Directory Layout + +``` +.gitea/ +├── actions/ +│ ├── setup-sectool/ +│ │ └── action.yml +│ ├── setup-env/ +│ │ └── action.yml +│ ├── setup-ansible-env/ +│ │ └── action.yml +│ ├── setup-k8s-env/ +│ │ └── action.yml +│ ├── setup-opentofu-env/ +│ │ └── action.yml +│ └── setup-scaleway-registry/ +│ └── action.yml +└── workflows/ + ├── ansible-validate-on-pr.yml + ├── aws-validate-on-pr.yml + ├── build-and-push-images.yml + ├── cloudns-validate-on-pr.yml + ├── contabo-validate-on-pr.yml + ├── k8s-apps-validate.yml + ├── scaleway-validate-on-pr.yml + └── weekly-image-rebuild.yml +``` + +## Action Migration Details + +### `setup-sectool` (New) + +Replaces `uses: a13labs/setup-sectool@v1` (external GitHub action). Local composite action that downloads the sectool binary from `a13labs/sectool` GitHub releases. + +- Fetches latest release from `https://api.github.com/repos/a13labs/sectool/releases/latest` +- Downloads Linux amd64 binary (tar.gz) +- Installs to `/usr/local/bin/sectool` +- Runs `sectool version` to verify + +### `setup-env` (Converted) + +Changes from `.github/actions/setup-env/action.yml`: +- `outputs.changed_files` → uses `$GITEA_OUTPUT` instead of `$GITHUB_OUTPUT` +- `git diff --name-only ${{ github.event.before }} ${{ github.sha }}` → `${{ gitea.event.before }} ${{ gitea.sha }}` +- `uses: a13labs/setup-sectool@v1` → `uses: ./.gitea/actions/setup-sectool` + +### `setup-ansible-env` (Converted) + +Changes from `.github/actions/setup-ansible-env/action.yml`: +- `outputs.ssh-auth-sock` and `outputs.hosts` → use `$GITEA_OUTPUT` +- No `github.event` references — minimal conversion needed + +### `setup-opentofu-env` (Unchanged) + +No changes needed. `opentofu/setup-opentofu@v1` stays as-is (Gitea pulls from GitHub). + +### `setup-k8s-env` (Unchanged) + +Carried over as-is. No GitHub-specific variables in this action. + +### `setup-terraform-env` (Not Migrated) + +Dropped entirely. Terraform is no longer used in this project. + +### `setup-scaleway-registry` (New) + +Local composite action for image build workflows. Creates `~/.docker/config.json` with registry auth and configures buildah storage/network settings for rootless operation. + +- Inputs: `registry-endpoint`, `registry-username`, `registry-password` +- Used by: `build-and-push-images.yml`, `weekly-image-rebuild.yml` + +## New Workflows (Image Build & Push) + +### `build-and-push-images.yml` (New) + +Builds and pushes runner profile Docker images to Scaleway registry. + +| Trigger | Tags | Scanning | +|---|---|---| +| Push to main (dockerfiles changed) | `{sha}` + `latest` | Yes (Trivy, warn only) | +| Push tag (`v*`) | `{tag}` + `{sha}` + `latest` | Yes (Trivy, warn only) | +| Pull request | `pr-{number}` | No | + +Jobs: `check-paths` → `auth` → `build-base` → `build-images` (matrix: 6 images) + +### `weekly-image-rebuild.yml` (New) + +Scheduled weekly (Monday 6AM). Rebuilds all images against updated upstream bases, tags only `:latest`, full Trivy scanning. + +See `plans/image-build-push.md` for full details. + +## Path Corrections + +GitHub workflows referenced stale paths from a previous repo layout: + +| Source (GitHub) | Actual | Notes | +|---|---|---| +| `terraform/aws-iac/` | `terraform/iac/aws/` | Restructured | +| `terraform/cloudns-iac/` | `terraform/iac/cloudns/` | Restructured | +| `terraform/contabo-iac/` | `terraform/iac/contabo/` | Restructured | +| `terraform/scaleway-iac/` | `terraform/iac/scaleway/` | Restructured | +| `terraform/k8sapps/` | `terraform/apps/dev-01/` + `terraform/apps/prod-01/` | Renamed + split | + +## Gitea Syntax Conversions + +| GitHub | Gitea | +|---|---| +| `$GITHUB_OUTPUT` | `$GITEA_OUTPUT` | +| `${{ github.event.before }}` | `${{ gitea.event.before }}` | +| `${{ github.sha }}` | `${{ gitea.sha }}` | +| `runs-on: ubuntu-latest` | `runs-on: cicd-base` | +| `uses: ./.github/actions/...` | `uses: ./.gitea/actions/...` | + +## Repo Secrets + +| Secret | Used By | Purpose | Status | +|---|---|---|---| +| `BW_PROJECT_ID` | All workflows | Bitwarden project ID | **Configured** | +| `BW_ORGANIZATION_ID` | All workflows | Bitwarden org ID | **Configured** | +| `BW_ACCESS_TOKEN` | All workflows | Bitwarden API token | **Configured** | +| `ANSIBLE_USER` | Ansible + K8s workflows | SSH user for remote hosts | **Configured** | +| `SCALEWAY_REGISTRY_ENDPOINT` | Image build workflows | Scaleway registry endpoint | **Needs setup** | +| `SCALEWAY_CI_ACCESS_KEY` | Image build workflows | CI IAM API access key | **Needs setup** | +| `SCALEWAY_CI_SECRET_KEY` | Image build workflows | CI IAM API secret key | **Needs setup** | + +## Phase B: Scaleway IaC Changes + +IaC changes in `terraform/iac/scaleway/` add CI IAM application and outputs. See `plans/image-build-push.md` for details. + +- `iam.tf` — added `ci_app` IAM application, policy, and API key +- `outputs.tf` — added `registry_endpoint`, `ci_app_api_access_key`, `ci_app_api_secret_key` +- `sectool.env` — no changes needed (CI app resources are hardcoded) + +## Next Steps + +1. **Push** all changes to `alexpires.me/a13labs.infra` on Gitea (note: `registry.tf` changed `is_public` from `false` to `true` on `a13labs` namespace) +2. **SSH to ci-01** — rebuild `cicd-base` image locally, update Gitea runner container — DONE +3. **Add Bitwarden secrets** — `SCALEWAY_CI_ACCESS_KEY` and `SCALEWAY_CI_SECRET_KEY` (populated after IaC apply) +4. **Apply Scaleway IaC** — `sectool exec tofu apply` in `terraform/iac/scaleway/` +5. **Extract IaC outputs** — `registry_endpoint`, `ci_app_api_access_key`, `ci_app_api_secret_key` +6. **Configure Gitea secrets** — `SCALEWAY_REGISTRY_ENDPOINT`, `SCALEWAY_CI_ACCESS_KEY`, `SCALEWAY_CI_SECRET_KEY` +7. **Test validate workflows** — PR touching `ansible/` or `terraform/iac/` paths +8. **Test build workflow** — PR touching `ansible/files/dockerfiles/linux-runners/` diff --git a/.opencode/plans/image-build-push.md b/.opencode/plans/image-build-push.md new file mode 100644 index 0000000..0734493 --- /dev/null +++ b/.opencode/plans/image-build-push.md @@ -0,0 +1,155 @@ +# Image Build & Push to Scaleway Registry + +## Goal +Build runner profile container images from `ansible/files/dockerfiles/linux-runners/`, scan for vulnerabilities, and push to the Scaleway container registry (`a13labs` namespace). + +## Decisions + +| Decision | Choice | Rationale | +|---|---|---| +| Registry namespace | `a13labs` (existing) | Clean project branding | +| Tagging strategy | Git tag + SHA + latest | Full traceability and history | +| Initial scope | Runner profiles only (7 → 4) | Consolidated image-build, infra-tools, security-scan into cicd-base | +| Trivy scan severity | Warn only | Review manually, never block builds | +| Update consumers | Not this iteration | Build+push pipeline first, consumers later | +| Path filtering | In-workflow check | Gitea paths filter unreliable with tags | +| PR scanning | Disabled | Keep PR builds fast | + +## Image Inventory & Dependencies + +``` +docker.io/gitea/runner:latest + ↓ + gitea-runner-base (build stage 1, pushed first) + ↓ + ┌─────────────────────────────┐ + │ cicd-base, cpp-build, go-build │ (build stage 2, all parallel) + └─────────────────────────────┘ +``` + +| Image | Dockerfile | Base | +|---|---|---| +| gitea-runner-base | `Dockerfile.gitea-runner-base` | `docker.io/gitea/runner:latest` | +| cicd-base | `Dockerfile.cicd-base` | `local/gitea-runner-base:latest` | +| cpp-build | `Dockerfile.cpp-build` | `local/gitea-runner-base:latest` | +| go-build | `Dockerfile.go-build` | `local/gitea-runner-base:latest` | + +> **Note:** `image-build`, `infra-tools`, and `security-scan` runners were decommissioned and their tooling consolidated into `cicd-base`. All workflows now run on `cicd-base` or the language-specific runners (`cpp-build`, `go-build`). + +## Registry Naming + +Images pushed as `fr-par-2.registry.scaleway.com/a13labs/{image-name}:{tag}` + +## Phases + +### Phase A: Prepare CI Runner (DONE) + +**Step 1 — Add build tools to `Dockerfile.cicd-base`** + +DONE. Added the following packages to the existing cicd-base image: +- Build packages: `buildah`, `fuse-overlayfs`, `podman`, `shadow-uidmap`, `skopeo`, `slirp4netns`, `iptables` (from `Dockerfile.image-build`) +- Storage config: `storage.conf` (vfs driver) and `containers.conf` (from `Dockerfile.image-build`) +- Trivy binary download (from `Dockerfile.security-scan`, v0.72.0) + +**Step 2 — Register updated cicd-base on Gitea runner** (DONE) + +cicd-base image rebuilt on `ci-01` and Gitea runner container registration updated. + +### Phase B: Scaleway IaC Changes (DONE - code changes, manual apply needed) + +**Step 3 — Add CI IAM app** (`terraform/iac/scaleway/iam.tf`) - DONE +- New `scaleway_iam_application` "ci_app" with `RegistryFullAccess` permission set +- New `scaleway_iam_policy` "registry_full_access" granting ci_app registry access +- New `scaleway_iam_api_key` "ci_app_api" for ci_app credentials + +**Step 4 — Add registry outputs** (`terraform/iac/scaleway/outputs.tf`) - DONE +- `registry_endpoint` — from `scaleway_registry_namespace.a13labs.endpoint` +- `ci_app_api_access_key`, `ci_app_api_secret_key` — sensitive + +**Step 5 — No `sectool.env` changes needed** - DONE +The CI app resources are hardcoded, no TF_VAR mappings required. + +**Step 6 — Apply and extract secrets** (MANUAL) +- Add `SCALEWAY_CI_ACCESS_KEY` and `SCALEWAY_CI_SECRET_KEY` to Bitwarden +- Run `sectool exec tofu apply` in `terraform/iac/scaleway/` +- Add `SCALEWAY_REGISTRY_ENDPOINT`, `SCALEWAY_CI_ACCESS_KEY`, `SCALEWAY_CI_SECRET_KEY` as Gitea repo secrets + +### Phase C: Gitea Actions & Workflow (DONE) + +**Step 7 — Create `.gitea/actions/setup-scaleway-registry/action.yml`** - DONE +Composite action that: +- Creates `~/.docker/config.json` with Scaleway registry auth +- Configures buildah storage and network settings for rootless operation +- Inputs: `registry-endpoint`, `registry-username`, `registry-password` + +**Step 8 — Create `.gitea/workflows/build-and-push-images.yml`** - DONE + +Jobs: +- `check-paths`: Determines if dockerfiles changed (handles push, PR, tag events) +- `auth`: Sets up registry authentication (gates on path check) +- `build-base`: Builds and pushes `gitea-runner-base` +- `build-images`: Matrix build of 6 dependent images + +Tagging strategy: +- Push to main: `{sha}` + `latest` +- Push tag (v*): `{tag}` + `{sha}` + `latest` +- PR: `pr-{number}` (no scanning) + +**Step 9 — Create `.gitea/workflows/weekly-image-rebuild.yml`** - DONE +Scheduled weekly (`cron: "0 6 * * 1"` — Monday 6AM): +- Same build jobs but only tag `:latest` +- Uses `--pull-always` to force rebuild against updated upstream base images +- Full Trivy scan (warn only, artifacts) + +## Build Order & Dependency Resolution + +Each dependent Dockerfile says `FROM local/gitea-runner-base:latest`. The workflow replaces `local/gitea-runner-base` with the full registry URL in a temp copy before building, so buildah pulls the freshly pushed base image. + +## Manual Setup Checklist + +Before the workflows can run successfully, complete these steps: + +- [ ] Push code changes to repo (note: `registry.tf` changed `is_public` from `false` to `true` on `a13labs` namespace) +- [x] SSH to `ci-01`, rebuild `cicd-base` image locally, update Gitea runner container — DONE +- [ ] Add `SCALEWAY_CI_ACCESS_KEY` and `SCALEWAY_CI_SECRET_KEY` to Bitwarden (leave blank initially, they'll be populated after IaC apply) +- [ ] Apply Scaleway IaC: `sectool exec tofu apply` in `terraform/iac/scaleway/` +- [ ] From IaC outputs, extract `registry_endpoint`, `ci_app_api_access_key`, `ci_app_api_secret_key` +- [ ] Add as Gitea repo secrets: `SCALEWAY_REGISTRY_ENDPOINT`, `SCALEWAY_CI_ACCESS_KEY`, `SCALEWAY_CI_SECRET_KEY` +- [ ] Push a test commit touching `ansible/files/dockerfiles/linux-runners/` to trigger workflow +- [ ] Verify workflow runs and images appear in Scaleway registry + +## Gitea Secrets Required + +| Secret | Source | Used by | +|---|---|---| +| `SCALEWAY_REGISTRY_ENDPOINT` | IaC output `registry_endpoint` | build-and-push, weekly-rebuild | +| `SCALEWAY_CI_ACCESS_KEY` | IaC output `ci_app_api_access_key` | build-and-push, weekly-rebuild | +| `SCALEWAY_CI_SECRET_KEY` | IaC output `ci_app_api_secret_key` | build-and-push, weekly-rebuild | + +## File Summary + +| Action | File | Status | +|---|---|---| +| Modify | `ansible/files/dockerfiles/linux-runners/Dockerfile.cicd-base` | DONE | +| Modify | `terraform/iac/scaleway/iam.tf` | DONE | +| Modify | `terraform/iac/scaleway/outputs.tf` | DONE | +| Modify | `terraform/iac/scaleway/sectool.env` | DONE | +| Create | `.gitea/actions/setup-scaleway-registry/action.yml` | DONE | +| Create | `.gitea/workflows/build-and-push-images.yml` | DONE | +| Create | `.gitea/workflows/weekly-image-rebuild.yml` | DONE | + +## Post-Iteration (NOT now) + +- Update runner Dockerfiles to use registry base URLs +- Update Ansible playbooks to `pull` from registry instead of `build` +- Add PR close job to clean up `pr-{number}` tags from registry +- Expand to AI service Dockerfiles (comfyui, ollama, llamacpp variants) + +## Consolidation + +`image-build`, `infra-tools`, and `security-scan` runner profiles have been decommissioned. Their tooling was already absorbed into `cicd-base`: +- **image-build** tools (buildah, podman, storage config) → merged into `Dockerfile.cicd-base` +- **security-scan** tools (trivy) → merged into `Dockerfile.cicd-base` +- **infra-tools** tools (ansible, pip packages) → installed at workflow runtime via `requirements/ansible.txt` + +Unused Dockerfiles removed: `Dockerfile.image-build`, `Dockerfile.infra-tools`, `Dockerfile.security-scan` diff --git a/.opencode/plans/llama_server_template_command.md b/.opencode/plans/llama_server_template_command.md new file mode 100644 index 0000000..96e1c91 --- /dev/null +++ b/.opencode/plans/llama_server_template_command.md @@ -0,0 +1,271 @@ +# Plan: Refactor llama_server role — template-based container command + +## Current State + +The `llama_server` role deploys `llama.cpp` server via Podman (EL) or NSSB (Windows) in router mode. + +**Existing structure:** + +| Component | Location | +|---|---| +| `defaults/main.yml` | User/group, image, tag, port, bind, models list, `argv_extra`, exporter config, Windows config | +| `tasks/el/main.yml` | User/group creation → podman image pull → **container created inline** with hardcoded flags → systemd wrapper → HF model sync → exporter | +| `tasks/windows/main.yml` | Binary download → NSSM install with same hardcoded flags → HF sync → firewall → exporter | +| `templates/podman.service.j2` | Thin systemd wrapper: `podman start/stop llamacpp` | +| `templates/llama_exporter.service.j2` | Exporter systemd service | +| `vars/main.yml` | Internal non-overridable vars | +| `files/scripts/` | `llama_hf_sync.py` (Linux + Windows), `llama_exporter.py`, tests | + +**Current container command** (`tasks/el/main.yml:142-152`): + +```yaml +command: >- + {{ + [ + '--metrics', + '--models-preset', + '/models/.router/models.ini', + '--models-max', + (llama_server_models_max | string) + ] + + (llama_server_argv_extra | default([])) + }} +``` + +**Current Windows command** (`tasks/windows/main.yml:220-238`): + +```yaml +argv: >- + {{ + [ + 'nssm.exe', + 'install', + 'llama-server', + (llama_server_windows_install_dir ~ '\\' ~ llama_server_windows_bin_name), + '--port', + (llama_server_port | default(11434) | string), + '--host', + (llama_server_host | default('127.0.0.1') | string), + '--metrics', + '--models-preset', + (__llama_preset_file | string), + '--models-max', + (llama_server_models_max | string) + ] + + (llama_server_argv_extra | default([])) + }} +``` + +**Problem:** Both commands are hardcoded inline. The only extensibility is `llama_server_argv_extra` (append-only list), which has no validation, no conditional logic, and no self-documentation. + +## Goals + +- Replace hardcoded command construction with Jinja2 templates for both EL and Windows paths. +- Add role variables for all major llama.cpp server flags with sensible defaults and conditional rendering. +- Maintain backward compatibility: `llama_server_argv_extra` still works as a suffix. +- Enable future configuration of MCP servers, SSL/TLS, API keys, chat templates, and performance tuning without manual playbook edits. + +## Changes Required + +### 1. Add new role variables to `defaults/main.yml` + +Add a new section after the existing defaults: + +```yaml +# --- Server configuration --- +llama_server_bind: "0.0.0.0" +llama_server_host: "127.0.0.1" +llama_server_port: 11434 + +# SSL/TLS +llama_server_ssl_enabled: false +llama_server_ssl_cert: "" +llama_server_ssl_key: "" + +# Authentication +llama_server_api_key: "" + +# Chat +llama_server_chat_template: "" + +# Performance +llama_server_ctx_size: 4096 +llama_server_n_gpu_layers: 35 +llama_server_batch_size: 512 +llama_server_ubatch_size: 512 +llama_server_threads: 0 +llama_server_threads_batch: 0 + +# Features +llama_server_embeddings: false +llama_server_cors: false +llama_server_log_timestamps: false +llama_server_log_level: "warn" +llama_server_log_print: false +llama_server_slot_save_context: false + +# Web UI +llama_server_webui: "" +llama_server_webui_author: "" + +# MCP +llama_server_mcp_servers: [] + +# KV cache quantization +llama_server_cache_type_q4: false + +# Speculative decoding +llama_server_speculative_model: "" +llama_server_draft: 0 + +# RoPE scaling +llama_server_rope_freq_scale: 0.0 + +# GPU offload +llama_server_tensor_split: 0 +llama_server_split_mode: "layer" +llama_server_main_gpu: 0 +``` + +### 2. Create EL container command template + +**File:** `templates/llama_server_command_el.j2` + +```jinja2 +{{ + [ + '--metrics' + ] + + (llama_server_models_preset_global_path | default('/models/.router/models.ini') | ternary(['--models-preset', llama_server_models_preset_global_path], [])) + + (llama_server_models_max | default(1) | string | ternary(['--models-max', llama_server_models_max | string], [])) + + (llama_server_host | default('127.0.0.1') | ternary(['--host', llama_server_host], [])) + + (llama_server_ssl_enabled | default(false) | bool | ternary( + ['--ssl', '--ssl-cert', llama_server_ssl_cert, '--ssl-key', llama_server_ssl_key] if llama_server_ssl_cert and llama_server_ssl_key else [], [])) + + (llama_server_api_key | default('') | ternary(['--api-key', llama_server_api_key], [])) + + (llama_server_chat_template | default('') | ternary(['--chat-template', llama_server_chat_template], [])) + + (llama_server_ctx_size | default(4096) | ternary(['--ctx-size', llama_server_ctx_size | string], [])) + + (llama_server_n_gpu_layers | default(35) | ternary(['--n-gpu-layers', llama_server_n_gpu_layers | string], [])) + + (llama_server_batch_size | default(512) | ternary(['--batch-size', llama_server_batch_size | string], [])) + + (llama_server_ubatch_size | default(512) | ternary(['--ubatch-size', llama_server_ubatch_size | string], [])) + + (llama_server_threads | default(0) | ternary(['--threads', llama_server_threads | string], [])) + + (llama_server_threads_batch | default(0) | ternary(['--threads-batch', llama_server_threads_batch | string], [])) + + (llama_server_embeddings | default(false) | bool | ternary(['--embedding'], [])) + + (llama_server_cors | default(false) | bool | ternary(['--cors'], [])) + + (llama_server_log_timestamps | default(false) | bool | ternary(['--log-timestamps'], [])) + + (llama_server_log_level | default('warn') | ternary(['--log-level', llama_server_log_level], [])) + + (llama_server_log_print | default(false) | bool | ternary(['--log-print'], [])) + + (llama_server_slot_save_context | default(false) | bool | ternary(['--slot-save-context'], [])) + + (llama_server_webui | default('') | ternary(['--webui-name', llama_server_webui], [])) + + (llama_server_webui_author | default('') | ternary(['--webui-author', llama_server_webui_author], [])) + + (llama_server_mcp_servers | default([]) | length | ternary(['--mcp-servers', (llama_server_mcp_servers | to_json)], [])) + + (llama_server_cache_type_q4 | default(false) | bool | ternary(['--cache-type-q4'], [])) + + (llama_server_speculative_model | default('') | ternary(['--speculative-model', llama_server_speculative_model], [])) + + (llama_server_draft | default(0) | ternary(['--draft', llama_server_draft | string], [])) + + (llama_server_rope_freq_scale | default(0.0) | ternary(['--rope-freq-scale', llama_server_rope_freq_scale | string], [])) + + (llama_server_tensor_split | default(0) | ternary(['--tensor-split', llama_server_tensor_split | string], [])) + + (llama_server_split_mode | default('layer') | ternary(['--split-mode', llama_server_split_mode], [])) + + (llama_server_main_gpu | default(0) | ternary(['--main-gpu', llama_server_main_gpu | string], [])) + + (llama_server_argv_extra | default([])) +}} +``` + +**Note:** The above uses a Jinja2 expression that produces a JSON array string. The `ternary()` calls handle conditional inclusion. The `llama_server_mcp_servers` serializes to JSON for the `--mcp-servers` flag. + +### 3. Create Windows service command template + +**File:** `templates/llama_server_command_windows.j2` + +Same logic as EL but output as a flat list suitable for NSSM `nssm.exe install` argv (each flag and value as separate list elements). + +### 4. Refactor `tasks/el/main.yml` + +Replace the hardcoded command block (lines 142-152) in the `podman_container` task: + +```yaml +- name: Set fact llama_server_el_command + ansible.builtin.set_fact: + llama_server_el_command: "{{ lookup('template', 'templates/llama_server_command_el.j2') }}" + +- name: Create pods + containers.podman.podman_container: + name: "llamacpp" + image: "{{ llama_server_image }}:{{ llama_server_tag }}" + state: started + device: "{{ llama_server_devices | default(omit) }}" + env: "{{ llama_server_env | default(omit) }}" + ports: "{{ llama_server_bind }}:{{ llama_server_port }}:8080/tcp" + volumes: + - "{{ llama_server_home }}/models:/models:Z" + - "{{ llama_server_home }}/.cache:/app/.cache:Z" + command: "{{ llama_server_el_command }}" +``` + +### 5. Refactor `tasks/windows/main.yml` + +Replace the hardcoded `nssm.exe install` argv block (lines 220-238): + +```yaml +- name: Set fact llama_server_windows_argv + ansible.builtin.set_fact: + llama_server_windows_argv: "{{ lookup('template', 'templates/llama_server_command_windows.j2') }}" + +- name: Install llama-server service via NSSM + ansible.windows.win_command: + argv: "{{ llama_server_windows_argv }}" + args: + chdir: "{{ llama_server_windows_install_dir }}" + register: __llama_nssm_install + changed_when: __llama_nssm_install.rc == 0 +``` + +### 6. Create MCP servers preset file template (optional, for future) + +**File:** `templates/mcp_servers.json.j2` + +```json +{{ llama_server_mcp_servers | to_nice_json }} +``` + +Used when MCP servers need to be written to a file and passed via `--mcp-servers` flag pointing to a path. + +### 7. Add tasks to write SSL/TLS and API key files (conditional) + +When `llama_server_ssl_cert` or `llama_server_api_key` are set, ensure the referenced files exist on the remote host. This may require: +- A task to create the API key file if provided as a variable (not file path) +- A task to validate SSL cert/key paths exist + +## Files to modify + +| File | Change | +|------|--------| +| `ansible/roles/llama_server/defaults/main.yml` | Add new role variables section | +| `ansible/roles/llama_server/templates/llama_server_command_el.j2` | **New** — EL container command template | +| `ansible/roles/llama_server/templates/llama_server_command_windows.j2` | **New** — Windows NSSM command template | +| `ansible/roles/llama_server/templates/mcp_servers.json.j2` | **New** — MCP servers JSON preset (optional) | +| `ansible/roles/llama_server/tasks/el/main.yml` | Replace hardcoded command with `lookup('template')` + `set_fact` | +| `ansible/roles/llama_server/tasks/windows/main.yml` | Replace hardcoded nssm argv with `lookup('template')` + `set_fact` | +| `ansible/roles/llama_server/tasks/el/llama_exporter.yml` | No change | +| `ansible/roles/llama_server/files/scripts/` | No change | + +## Rollback plan + +- The `llama_server_argv_extra` variable continues to work as a suffix in both templates. +- If template rendering fails, revert `tasks/el/main.yml` and `tasks/windows/main.yml` to the hardcoded command blocks. +- Delete the new template files. +- No data loss — templates only affect command construction, not state. + +## Testing + +1. **Baseline test:** Run role with existing defaults — should produce identical command to current hardcoded version. +2. **Variable test:** Set individual variables (e.g., `llama_server_ctx_size: 8192`, `llama_server_embeddings: true`) and verify they appear in the rendered command. +3. **Conditional test:** Set `llama_server_ssl_enabled: true` with cert/key paths — verify `--ssl`, `--ssl-cert`, `--ssl-key` flags are included. +4. **MCP test:** Set `llama_server_mcp_servers` to a list of dicts — verify JSON serialization in `--mcp-servers` flag. +5. **Backward compat test:** Use `llama_server_argv_extra` alongside new variables — verify both are present. +6. **Windows test:** Same scenarios on Windows target with NSSM. + +## Open questions + +1. **Should `--models-preset` and `--models-max` remain in the template or stay hardcoded?** They're core to router mode and unlikely to change. Keeping them in the template makes the command self-contained but adds complexity. +2. **For `mcp_servers`** — does llama.cpp accept the flag as `--mcp-servers '{"name":"..."}'` (JSON string) or `--mcp-servers /path/to/config.json` (file path)? This affects whether we need a separate file template. +3. **Should defaults be opinionated** (e.g., `llama_server_ctx_size: 4096`) or minimal (only required flags)? Opinionated defaults reduce host var boilerplate but may not suit all use cases. +4. **Should we add assertions** for mutually exclusive flags (e.g., `--split-mode` values)? diff --git a/.opencode/plans/llamacpp-nginx-proxy.md b/.opencode/plans/llamacpp-nginx-proxy.md new file mode 100644 index 0000000..247dce2 --- /dev/null +++ b/.opencode/plans/llamacpp-nginx-proxy.md @@ -0,0 +1,234 @@ +# Plan: Replace OpenWebUI with llama.cpp behind nginx proxy + +## Current State + +- **llama.cpp** runs on `gpu-01.lab.alexpires.me` via Podman on port `11434` (bound to `127.0.0.1` with API key) +- **nginx proxy** role runs on `dev-02.lab.alexpires.me` and `gpu-01.lab.alexpires.me` +- **OpenWebUI** has been decommissioned (removed from K8s, DNS, inventory) +- **DNS** for `lab.alexpires.me` is managed via Terraform (`terraform/apps/dev-01/config.tf` zones) +- **oauth2-proxy** template uses OIDC provider (`code.alexpires.me`), `set_xauthrequest = true` + +## Goals + +- Deploy llama.cpp behind nginx proxy at `llm.lab.alexpires.me` ✅ +- API paths (`/api/*`) accessible without OAuth2 (token auth enforced via `--api-key`) ✅ +- All other paths (chat UI, etc.) protected with OAuth2 via oauth2-proxy ✅ +- Issue SSL certificate for `llm.lab.alexpires.me` via CloudNS DNS ✅ +- OpenWebUI decommissioned ✅ + +## Important Constraint + +> **Resolved.** Phase 2 completed: llama.cpp now bound to `127.0.0.1` with API key. OpenWebUI decommissioned. + +## Changes Required + +> All changes have been implemented. See execution order below for completed steps. + +### 1. Add DNS record for llm.lab.alexpires.me ✅ + +**File:** `terraform/apps/dev-01/config.tf` + +Add A record to the `lab.alexpires.me` zone: +```hcl +{ + name = "llm" + type = "A" + content = "10.19.4.106" +} +``` + +Run `tofu plan/apply` on `terraform/apps/dev-01/` before Phase 1 so DNS resolves. + +### 2. Add gpu-01 to `[nginx]` inventory group ✅ + +### 3. Update nginx-server.conf.j2 — per-site `no_auth` conditional ✅ + +### 4. Configure gpu-01 host vars for nginx proxy ✅ + +### 5. Add firewall rules for HTTPS ✅ + +### 6. Update llama_server role — support configurable bind address ✅ + +### 7. Two-phase transition ✅ + +**Phase 1 — Deploy nginx proxy (llama.cpp still on 0.0.0.0, no API key)** + +- Keep `llama_server_bind: "0.0.0.0"` (current behavior, default) +- Do NOT set API key +- nginx proxies to `127.0.0.1:11434` (container is bound to `0.0.0.0:11434`, reachable from localhost) +- OpenWebUI continues to work (still connects to `gpu-01:11434` directly) +- Test: `llm.lab.alexpires.me/` requires OAuth2 login, `llm.lab.alexpires.me/api/*` is open + +**Phase 2 — Switch to localhost binding + enable API key (after validation)** + +- Change `llama_server_bind: "127.0.0.1"` in host vars +- Add `--api-key` to `llama_server_argv_extra` in host vars: + ```yaml + llama_server_argv_extra: + - "--api-key" + - "{{ lookup('env', 'LLAMA_SERVER_API_KEY') }}" + # ... existing args ... + ``` +- OpenWebUI will break (can no longer reach llama.cpp on `gpu-01:11434`) — decommission OpenWebUI +- Verify all consumers migrate to `llm.lab.alexpires.me/api/*` with `Authorization: Bearer ` +- Remove old firewall rules for port 11434 (no longer externally reachable) + +## Execution Order + +### Phase 1 — Deploy nginx proxy (llama.cpp unchanged, OpenWebUI still works) + +> **Status: COMPLETE** — All steps executed and verified. + +1. **Update Terraform DNS** — add `llm` A record in `terraform/apps/dev-01/config.tf`, run `tofu apply` ✅ +2. **Update inventory** — add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` group ✅ +3. **Update nginx-server.conf.j2** — add `no_auth` per-site conditional (template stays generic) ✅ +4. **Update llama_server role** — add `llama_server_bind` variable (default `0.0.0.0`) ✅ +5. **Update host vars** — add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules ✅ +6. **Run nginx proxy playbook** — installs nginx + oauth2-proxy + certbot + deploys config ✅ +7. **Fix SELinux** — add `nginx_connect_any` boolean + port 4180 to `http_port_t` (manually applied, added to role for persistence) ✅ +8. **Verify chat UI** — `llm.lab.alexpires.me/` returns oauth2-proxy login page (200) ✅ +9. **Verify API** — `llm.lab.alexpires.me/api/v1/models` returns 200 with models list ✅ +10. **Verify OpenWebUI still works** — llama.cpp still bound to `0.0.0.0:11434` ✅ + +1. **Update Terraform DNS** — add `llm` A record in `terraform/apps/dev-01/config.tf`, run `tofu apply` +2. **Update inventory** — add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` group +3. **Update nginx-server.conf.j2** — add `no_auth` per-site conditional (template stays generic) +4. **Update llama_server role** — add `llama_server_bind` variable (default `0.0.0.0`) +5. **Update host vars** — add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules +6. **Run nginx proxy playbook** — installs nginx + oauth2-proxy + certbot + deploys config (all in one run) +7. **Verify chat UI** — test `llm.lab.alexpires.me/` with OAuth2 login +8. **Verify API** — test `llm.lab.alexpires.me/api/*` (no auth yet, same as current direct access) +9. **Verify OpenWebUI still works** — confirm dev-01 still connects to gpu-01:11434 + +### Phase 2 — Switch to localhost + API key (after validation, decommission OpenWebUI) + +> **Status: COMPLETE** — All steps executed and verified. + +10. **Update host vars** — change `llama_server_bind: "127.0.0.1"`, add `--api-key` to `llama_server_argv_extra` ✅ +11. **Update firewall** — remove port 11434 rules from `fw_allowed_ports` ✅ +12. **Run llama_server playbook** — restart container with new binding + API key ✅ +13. **Run hardening playbook** — `playbook_hardening_cis.yml --limit gpu-01.lab.alexpires.me -t ufw` to apply firewall changes ✅ +14. **Verify API** — `llm.lab.alexpires.me/api/v1/chat/completions` returns 401 without key, 200 with key ✅ +15. **Decommission OpenWebUI** — removed from `terraform/apps/dev-01/apps.tf` ✅ +16. **Remove OpenWebUI DNS records** — removed `open-webui` CNAME from `terraform/apps/dev-01/config.tf` zones ✅ +17. **Remove OpenWebUI locals** — removed `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` from config.tf ✅ +18. **Remove `webui_secret_key` variable** — removed from `terraform/apps/dev-01/secrets.tf` ✅ +19. **Remove `[openwebui]` group** — cleaned up `inventory/hosts` ✅ +20. **Run `tofu apply`** on `terraform/apps/dev-01/` — destroyed 8 OpenWebUI K8s resources (namespace, deployment, service, ingress, secret, service accounts) ✅ + +### Phase 2 — Switch to localhost + API key (after validation, decommission OpenWebUI) + +10. **Update host vars** — change `llama_server_bind: "127.0.0.1"`, add `--api-key` to `llama_server_argv_extra` +11. **Update firewall** — remove port 11434 rules from `fw_allowed_ports` +12. **Run llama_server playbook** — restart container with new binding + API key +13. **Run hardening playbook** — `playbook_hardening_cis.yml --limit gpu-01.lab.alexpires.me -t ufw` to apply firewall changes +14. **Verify API** — test `llm.lab.alexpires.me/api/*` requires API key +15. **Decommission OpenWebUI** — remove Terraform module from `terraform/apps/dev-01/apps.tf` and `config.tf` +16. **Remove OpenWebUI DNS records** — clean up `open-webui` CNAME in `terraform/apps/dev-01/config.tf` zones +17. **Remove `[openwebui]` group** — clean up `inventory/hosts` +18. **Run `tofu apply`** on `terraform/apps/dev-01/` — destroys OpenWebUI K8s resources + +## Playbooks to run + +### Phase 1 + +> **COMPLETE** — All steps executed and verified. + +```sh +cd ansible +sectool exec ansible-playbook -u $ANSIBLE_USER playbook_nginx_proxy.yml # nginx + oauth2 + certbot (all-in-one) +``` + +### Phase 2 + +> **COMPLETE** — All steps executed and verified. + +```sh +cd ansible +sectool exec ansible-playbook -u $ANSIBLE_USER playbook_llama_server.yml # container binding + API key +sectool exec ansible-playbook -u $ANSIBLE_USER playbook_hardening_cis.yml -t ufw # remove port 11434 rules +``` + +Then in terraform: +```sh +cd terraform/apps/dev-01 +sectool exec tofu plan # review OpenWebUI destruction +sectool exec tofu apply --auto-approve # apply +``` + +## Files to modify + +| File | Phase | Change | +|------|-------|--------| +| `terraform/apps/dev-01/config.tf` | 1 | Add `llm` A record to zones | +| `inventory/hosts` | 1 | Add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` | +| `ansible/roles/nginx_proxy/templates/nginx-server.conf.j2` | 1 | Add `no_auth` per-site conditional (3 blocks) | +| `ansible/roles/llama_server/defaults/main.yml` | 1 | Add `llama_server_bind` variable (default `0.0.0.0`) | +| `ansible/roles/llama_server/tasks/el/main.yml` | 1 | Use `llama_server_bind` in podman port mapping | +| `ansible/roles/nginx_proxy/tasks/nginx.yml` | 1 | Add SELinux tasks (`nginx_connect_any` boolean + oauth2-proxy port to `http_port_t`) | +| `ansible/host_vars/gpu-01.lab.alexpires.me.yml` | 1 | Add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules | +| `ansible/host_vars/gpu-01.lab.alexpires.me.yml` | 2 | Change `llama_server_bind` to `127.0.0.1`, add `--api-key` to `llama_server_argv_extra`, remove port 11434 fw rules ✅ | +| `terraform/apps/dev-01/apps.tf` | 2 | Remove `module "open-webui"` block ✅ | +| `terraform/apps/dev-01/config.tf` | 2 | Remove `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` locals; remove `open-webui` DNS record from zones ✅ | +| `terraform/apps/dev-01/secrets.tf` | 2 | Remove `webui_secret_key` variable ✅ | +| `inventory/hosts` | 2 | Remove `[openwebui]` group ✅ | + +## Rollback plan + +### Phase 1 rollback (nginx proxy broken) + +- Remove gpu-01 from `[nginx]` group in `inventory/hosts` +- Re-run hardening playbook on gpu-01 to clean up nginx/oauth2-proxy packages +- Revert `nginx-server.conf.j2` template changes +- llama.cpp continues running on `0.0.0.0:11434` — no impact +- OpenWebUI continues to work + +## Issues Encountered & Resolved + +### 1. nginx `copy` module missing `remote_src: true` +**File:** `ansible/roles/nginx_proxy/tasks/oauth2_proxy.yml:16` +The `copy` module was trying to copy from the controller instead of the remote host. Added `remote_src: true`. + +### 2. SELinux blocking nginx → oauth2-proxy / llama.cpp connections +**Symptom:** `connect() to 127.0.0.1:4180 failed (13: Permission denied)` in nginx error log +**Fix:** +- Set SELinux boolean `nginx_connect_any` to `true` (persistent) +- Added port 4180 to `http_port_t` via `semanage port` +- Added both as Ansible tasks in `nginx.yml` for persistence + +### 3. Template `no_auth` attribute error +**Symptom:** `object of type 'dict' has no attribute 'no_auth'` +**Fix:** Changed `{% if not item.no_auth %}` to `{% if not (item.no_auth | default(false) | bool) %}` to handle missing attribute on dicts. + +### 4. Playbook failed on model query after container restart (Phase 2) +**Symptom:** `urlopen error [Errno 111] Connection refused` when querying `http://127.0.0.1:11434/models?reload=1` +**Cause:** The playbook's "Query llama router model list" task ran before the container was fully started. The container did start correctly with `127.0.0.1:11434` binding and API key active. +**Impact:** Non-blocking — the container was running correctly; only the model list query task failed. + +## Known Warnings (non-blocking) + +- `types_hash_bucket_size: 64` — nginx warning, non-blocking +- oauth2-proxy PKCE warning (`--code-challenge-method`) — non-blocking, S256 is auto-negotiated + +## Rollback plan + +### Phase 1 rollback (nginx proxy broken) + +- Revert `llama_server_bind` to `0.0.0.0` in host vars +- Remove `--api-key` from `llama_server_argv_extra` +- Re-add port 11434 firewall rules +- Re-run `playbook_llama_server.yml` + `playbook_hardening_cis.yml -t ufw` — container restarts with old settings +- OpenWebUI still works (llama.cpp is back on `0.0.0.0` without API key) +- Fix issues and re-attempt Phase 2 + +### Phase 2 rollback (after OpenWebUI decommission) + +- Restore `llama_server_bind: "0.0.0.0"` in host vars +- Remove `--api-key` from `llama_server_argv_extra` +- Re-add port 11434 firewall rules +- Restore `module "open-webui"` in `terraform/apps/dev-01/apps.tf` +- Restore `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` locals in `config.tf` +- Restore `open-webui` DNS CNAME record in `config.tf` zones +- Restore `webui_secret_key` variable in `secrets.tf` +- Re-add `[openwebui]` group in `inventory/hosts` +- Run `tofu apply` to recreate OpenWebUI K8s resources diff --git a/.opencode/plans/tls-oauth2-sso-dev-02.md b/.opencode/plans/tls-oauth2-sso-dev-02.md new file mode 100644 index 0000000..8b74de9 --- /dev/null +++ b/.opencode/plans/tls-oauth2-sso-dev-02.md @@ -0,0 +1,605 @@ +# TLS + OAuth2 SSO for dev-02 + +## Goal + +Add HTTPS, TLS termination, and OAuth2/OIDC SSO to dev-02 services (code-server + opencode), +exposed directly on the internal network for VPN/office access. + +## Architecture + +``` + ┌──────────────────────────────────────────┐ + │ dev-02.lab.alexpires.me │ + │ (Fedora bare metal) │ + │ │ + │ ┌──────┐ auth_request ┌───────────┐ │ + │ │nginx │─────────────────►│oauth2- │ │ + │ │:443 │◄──────────────────│proxy:4180 │ │ + │ │TLS │ │(session) │ │ + │ │SSO │ │ │ │ + │ └──┬───┘ └─────┬─────┘ │ + │ │ proxy_pass (trusts auth) │ │ + │ │ │OIDC │ + │ ┌──▼──────┐ ┌──────▼────┐ │ + │ │ide:3000 │ │Gitea IdP │ │ + │ │(127.0.0.1) │code. │ │ + │ │code- │ │alexpires. │ │ + │ │server │ │me (pub) │ │ + │ └────────┘ └───────────┘ │ + │ ┌────────┐ │ + │ │agent:4096│ │ + │ │(127.0.0.1) │ + │ │opencode │ │ + │ └────────┘ │ + └──────────────────────────────────────────┘ + │ │ + HTTPS 443 │ │ OIDC + (internal) │ │ + ▼ ▼ + ┌──────────────────────────────────────────┐ + │ Browser (VPN/office) │ code.alexpires.me (public) + │ ide.lab.alexpires.me:443 ──────────► Gitea IdP + │ agent.lab.alexpires.me:443 ──────────► + └──────────────────────────────────────────┘ +``` + +## SSO Flow + +1. User visits `https://ide.lab.alexpires.me:443` (direct, requires VPN/office) +2. nginx detects no valid session cookie → `auth_request` to oauth2-proxy +3. oauth2-proxy returns 401 → nginx redirects browser to Gitea login +4. Browser → `code.alexpires.me` (Gitea, public, always reachable) +5. User logs in → Gitea redirects back to `https://ide.lab.alexpires.me/oauth2/callback` +6. oauth2-proxy exchanges code → access_token → creates session cookie +7. Cookie scoped to `*.lab.alexpires.me` → SSO across lab subdomains +8. Subsequent requests: nginx trusts cookie → proxies to backend (no re-login) + +**Access constraint:** SSO only works when the user can reach `lab.alexpires.me` (VPN or office network). +This is inherent to the architecture — the OAuth2 callback URL (`ide.lab.alexpires.me/oauth2/callback`) +must be reachable by the browser after Gitea redirects back. All `lab.alexpires.me` services +follow this same constraint. + +## Certificate Strategy + +- **SAN cert** for `ide.lab.alexpires.me` and `agent.lab.alexpires.me` +- Obtained via certbot DNS-01 challenge (cloudns plugin) +- Auto-renewal via cron with `--deploy-hook "systemctl reload nginx"` +- No port 80/443 exposure needed (DNS-01 challenge doesn't need HTTP) +- Certbot stores certs in `/etc/letsencrypt/live/` (symlinked from `/etc/letsencrypt/archive/`) +- nginx references: + - `ssl_certificate /etc/letsencrypt/live/ide.lab.alexpires.me/fullchain.pem;` + - `ssl_certificate_key /etc/letsencrypt/live/ide.lab.alexpires.me/privkey.pem;` + +## Service Binding + +| Service | Before | After | +|---------|--------|-------| +| code-server | `0.0.0.0:3000` | `127.0.0.1:3000` | +| opencode | `0.0.0.0:4096` | `127.0.0.1:4096` | +| oauth2-proxy | N/A | `127.0.0.1:4180` | +| nginx | N/A | `0.0.0.0:443` (only public port) | + +## Pre-requisites + +Before applying, register an OAuth2 app in Gitea: +- **Name:** `lab-services` +- **Redirect URI:** `https://ide.lab.alexpires.me/oauth2/callback` +- **Scopes:** `openid`, `profile`, `email` +- **Result:** `client_id` + `client_secret` → stored in `sectool` / env vars + +Generate a cookie secret: +```sh +openssl rand -base64 32 +``` +Store via `sectool` as `OAUTH2_PROXY_COOKIE_SECRET`, `OAUTH2_PROXY_CLIENT_ID`, `OAUTH2_PROXY_CLIENT_SECRET`. + +## Changes Required + +### 1. New Role: `nginx_proxy` + +``` +ansible/roles/nginx_proxy/ +├── defaults/main.yml # certbot, nginx, oauth2 config, upstreams +├── tasks/ +│ ├── main.yml # Entry point (includes certbot, nginx, oauth2_proxy) +│ ├── certbot.yml # Install certbot, DNS credentials, obtain cert, cron renewal +│ ├── nginx.yml # Install nginx, deploy config, systemd, SELinux context +│ └── oauth2_proxy.yml # Download binary, config, systemd +├── templates/ +│ ├── nginx.conf.j2 # Main nginx config (worker processes, logging, etc.) +│ ├── nginx-server.conf.j2 # Server blocks (SAN domains, TLS, auth_request, proxy) +│ └── oauth2-proxy.cfg.j2 # OAuth2/OIDC config for Gitea +└── files/ + └── .gitkeep +``` + +#### `defaults/main.yml` + +```yaml +nginx_proxy_certbot_enabled: true +nginx_proxy_certbot_email: "c.alexandre.pires@alexpires.me" +nginx_proxy_certbot_domains: + - "ide.lab.alexpires.me" + - "agent.lab.alexpires.me" +nginx_proxy_certbot_cron_minute: "0" +nginx_proxy_certbot_cron_hour: "3" +nginx_proxy_certbot_credential_file: "/etc/letsencrypt/cloudns.ini" +nginx_proxy_certbot_credential_mode: "0600" +nginx_proxy_certbot_nginx_reload: true + +nginx_proxy_nginx_enabled: true +nginx_proxy_nginx_bind: "0.0.0.0" +nginx_proxy_nginx_port: 443 +nginx_proxy_nginx_worker_connections: 1024 +nginx_proxy_nginx_log_format: "main" +nginx_proxy_nginx_access_log: "/var/log/nginx/access.log" +nginx_proxy_nginx_error_log: "/var/log/nginx/error.log warn" +nginx_proxy_nginx_ssl_protocols: "TLSv1.2 TLSv1.3" +nginx_proxy_nginx_ssl_ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384" +nginx_proxy_nginx_ssl_prefer_server_ciphers: "on" +nginx_proxy_nginx_hsts_max_age: "31536000" +nginx_proxy_nginx_hsts_include_subdomains: true + +nginx_proxy_oauth2_proxy_enabled: true +nginx_proxy_oauth2_proxy_bind: "127.0.0.1" +nginx_proxy_oauth2_proxy_port: 4180 +nginx_proxy_oauth2_proxy_version: "2.13.0" +nginx_proxy_oauth2_proxy_cookie_domain: ".lab.alexpires.me" +nginx_proxy_oauth2_proxy_cookie_name: "_alexpires_oauth2_proxy" +nginx_proxy_oauth2_proxy_cookie_secret: "{{ lookup('env', 'OAUTH2_PROXY_COOKIE_SECRET', default='') }}" +nginx_proxy_oauth2_proxy_client_id: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_ID', default='') }}" +nginx_proxy_oauth2_proxy_client_secret: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_SECRET', default='') }}" +nginx_proxy_oauth2_proxy_redirect_url: "https://ide.lab.alexpires.me/oauth2/callback" +nginx_proxy_oauth2_proxy_provider: "gitea" +nginx_proxy_oauth2_proxy_provider_url: "https://code.alexpires.me" +nginx_proxy_oauth2_proxy_scopes: "openid profile email" +nginx_proxy_oauth2_proxy_email_domains: [] +nginx_proxy_oauth2_proxy_upstream: "static://200" +nginx_proxy_oauth2_proxy_skip_jwt: false +nginx_proxy_oauth2_proxy_set_basic_auth: false +nginx_proxy_oauth2_proxy_set_xauthrequest: true +nginx_proxy_oauth2_proxy_ssl_insecure_skip_verify: false +nginx_proxy_oauth2_proxy_login_button: "Sign in with Gitea" +nginx_proxy_oauth2_proxy_proxy_prefix: "/oauth2" +nginx_proxy_oauth2_proxy_silent_login: false +nginx_proxy_oauth2_proxy_skip_provider_whoami: false +nginx_proxy_oauth2_proxy_cookie_refresh: "0s" +nginx_proxy_oauth2_proxy_cookie_expire: "168h" +nginx_proxy_oauth2_proxy_session_storage_type: "memory" +nginx_proxy_oauth2_proxy_request_logging: true +nginx_proxy_oauth2_proxy_log_file: "/var/log/oauth2-proxy.log" +nginx_proxy_oauth2_proxy_log_level: "info" + +nginx_proxy_upstreams: + ide: + host: "127.0.0.1" + port: 3000 + agent: + host: "127.0.0.1" + port: 4096 + +nginx_proxy_cloudns_auth_id: "{{ lookup('env', 'CLOUDNS_AUTH_ID') }}" +nginx_proxy_cloudns_auth_password: "{{ lookup('env', 'CLOUDNS_PASSWORD') }}" +nginx_proxy_cloudns_nameserver: "109.201.133.111" +``` + +#### `tasks/certbot.yml` + +- Install certbot + certbot-dns-cloudns via pip +- Create CloudDNS credentials file at `/etc/letsencrypt/cloudns.ini`: + ```ini + dns_cloudns_auth_id = {{ nginx_proxy_cloudns_auth_id }} + dns_cloudns_auth_password = {{ nginx_proxy_cloudns_auth_password }} + ``` +- Permissions: `0600`, owner: `root:root` +- Obtain SAN cert: + ```sh + certbot certonly --dns-cloudns \ + -d ide.lab.alexpires.me \ + -d agent.lab.alexpires.me \ + --non-interactive \ + --agree-tos \ + -m {{ nginx_proxy_certbot_email }} + ``` +- Create cron job for renewal (daily at 3:00 AM): + ```sh + certbot renew --deploy-hook "systemctl reload nginx" + ``` +- Certs stored in `/etc/letsencrypt/live/ide.lab.alexpires.me/` + +#### `tasks/nginx.yml` + +- Install nginx package +- Render `nginx.conf` (global): + ```nginx + user nginx; + worker_processes auto; + error_log {{ nginx_proxy_nginx_error_log }}; + pid /run/nginx.pid; + + events { + worker_connections {{ nginx_proxy_nginx_worker_connections }}; + } + + http { + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status $body_bytes_sent "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for"'; + + access_log {{ nginx_proxy_nginx_access_log }} main; + + sendfile on; + tcp_nopush on; + tcp_nodelay on; + keepalive_timeout 65; + types_hash_max_size 2048; + + include /etc/nginx/mime.types; + default_type application/octet-stream; + + # WebSocket upgrade mapping + map $http_upgrade $connection_upgrade { + default upgrade; + '' close; + } + + include /etc/nginx/conf.d/*.conf; + } + ``` +- Render `nginx-server.conf` (server blocks): + ```nginx + # IDE (code-server) + server { + listen 443 ssl http2; + server_name ide.lab.alexpires.me; + + ssl_certificate /etc/letsencrypt/live/ide.lab.alexpires.me/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/ide.lab.alexpires.me/privkey.pem; + ssl_protocols {{ nginx_proxy_nginx_ssl_protocols }}; + ssl_ciphers {{ nginx_proxy_nginx_ssl_ciphers }}; + ssl_prefer_server_ciphers {{ nginx_proxy_nginx_ssl_prefer_server_ciphers }}; + + add_header Strict-Transport-Security "max-age={{ nginx_proxy_nginx_hsts_max_age }}; {{ nginx_proxy_nginx_hsts_include_subdomains ? 'includeSubDomains' : '' }}" always; + + # OAuth2 auth_request + location = /oauth2/ { + proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }}; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Scheme $scheme; + proxy_set_header X-Auth-Request-Redirect $request_uri; + } + + location = /oauth2/callback { + proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }}/oauth2/callback; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Scheme $scheme; + proxy_set_header X-Auth-Request-Redirect $request_uri; + } + + location /oauth2/ { + proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }}; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Scheme $scheme; + } + + location / { + auth_request /oauth2/; + auth_request_set $auth_status $upstream_status; + auth_request_set $auth_cookie $upstream_http_set_cookie; + + add_header Set-Cookie $auth_cookie; + + proxy_pass http://127.0.0.1:{{ nginx_proxy_upstreams.ide.port }}; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Port $server_port; + + # WebSocket support + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + + # Timeouts for code-server + proxy_read_timeout 3600s; + proxy_send_timeout 3600s; + } + } + + # Agent (opencode) + server { + listen 443 ssl http2; + server_name agent.lab.alexpires.me; + + ssl_certificate /etc/letsencrypt/live/ide.lab.alexpires.me/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/ide.lab.alexpires.me/privkey.pem; + ssl_protocols {{ nginx_proxy_nginx_ssl_protocols }}; + ssl_ciphers {{ nginx_proxy_nginx_ssl_ciphers }}; + ssl_prefer_server_ciphers {{ nginx_proxy_nginx_ssl_prefer_server_ciphers }}; + + add_header Strict-Transport-Security "max-age={{ nginx_proxy_nginx_hsts_max_age }}; {{ nginx_proxy_nginx_hsts_include_subdomains ? 'includeSubDomains' : '' }}" always; + + # OAuth2 auth_request + location = /oauth2/ { + proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }}; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Scheme $scheme; + proxy_set_header X-Auth-Request-Redirect $request_uri; + } + + location = /oauth2/callback { + proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }}/oauth2/callback; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Scheme $scheme; + proxy_set_header X-Auth-Request-Redirect $request_uri; + } + + location /oauth2/ { + proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }}; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Scheme $scheme; + } + + location / { + auth_request /oauth2/; + auth_request_set $auth_status $upstream_status; + auth_request_set $auth_cookie $upstream_http_set_cookie; + + add_header Set-Cookie $auth_cookie; + + proxy_pass http://127.0.0.1:{{ nginx_proxy_upstreams.agent.port }}; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header X-Forwarded-Host $host; + proxy_set_header X-Forwarded-Port $server_port; + + # WebSocket support + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + + # Timeouts for opencode + proxy_read_timeout 3600s; + proxy_send_timeout 3600s; + } + } + ``` +- Enable + start nginx service +- Set SELinux context: `semanage fcontext -a -t httpd_cert_t "/etc/letsencrypt(/.*)?"` + `restorecon -Rv /etc/letsencrypt` + +#### `tasks/oauth2_proxy.yml` + +- Download latest release from GitHub (Go binary for linux/amd64): + ```sh + wget -qO /tmp/oauth2-proxy.tar.gz "https://github.com/oauth2-proxy/oauth2-proxy/releases/download/v{{ nginx_proxy_oauth2_proxy_version }}/oauth2-proxy-v{{ nginx_proxy_oauth2_proxy_version }}.linux-amd64.tar.gz" + tar xzf /tmp/oauth2-proxy.tar.gz -C /tmp/ + mv /tmp/oauth2-proxy-v{{ nginx_proxy_oauth2_proxy_version }}.linux-amd64/oauth2-proxy /usr/local/bin/ + ``` +- Render config file at `/etc/oauth2-proxy/oauth2-proxy.cfg`: + ```ini + provider = "{{ nginx_proxy_oauth2_proxy_provider }}" + provider_url = "{{ nginx_proxy_oauth2_proxy_provider_url }}" + client_id = "{{ nginx_proxy_oauth2_proxy_client_id }}" + client_secret = "{{ nginx_proxy_oauth2_proxy_client_secret }}" + cookie_domain = "{{ nginx_proxy_oauth2_proxy_cookie_domain }}" + cookie_name = "{{ nginx_proxy_oauth2_proxy_cookie_name }}" + cookie_secret = "{{ nginx_proxy_oauth2_proxy_cookie_secret }}" + cookie_expire = "{{ nginx_proxy_oauth2_proxy_cookie_expire | default('168h') }}" + cookie_refresh = "{{ nginx_proxy_oauth2_proxy_cookie_refresh | default('0s') }}" + redirect_url = "{{ nginx_proxy_oauth2_proxy_redirect_url }}" + login_url = "{{ nginx_proxy_oauth2_proxy_provider_url }}/login/oauth/authorize" + redeem_url = "{{ nginx_proxy_oauth2_proxy_provider_url }}/login/oauth/access_token" + openid_connectissuerurl = "{{ nginx_proxy_oauth2_proxy_provider_url }}" + scopes = "{{ nginx_proxy_oauth2_proxy_scopes }}" + email_domains = {{ nginx_proxy_oauth2_proxy_email_domains | to_json }} + upstream = "{{ nginx_proxy_oauth2_proxy_upstream }}" + skip_jwt_bearer_tokens = {{ nginx_proxy_oauth2_proxy_skip_jwt | lower }} + set_basic_auth = {{ nginx_proxy_oauth2_proxy_set_basic_auth | lower }} + set_xauthrequest = {{ nginx_proxy_oauth2_proxy_set_xauthrequest | lower }} + ssl_insecure_skip_verify = {{ nginx_proxy_oauth2_proxy_ssl_insecure_skip_verify | lower }} + proxy_prefix = "{{ nginx_proxy_oauth2_proxy_proxy_prefix }}" + silent_login = {{ nginx_proxy_oauth2_proxy_silent_login | lower }} + skip_provider_whoami = {{ nginx_proxy_oauth2_proxy_skip_provider_whoami | lower }} + listen = "{{ nginx_proxy_oauth2_proxy_bind }}:{{ nginx_proxy_oauth2_proxy_port }}" + {% if nginx_proxy_oauth2_proxy_request_logging | default(true) %} + logfile_filename = "{{ nginx_proxy_oauth2_proxy_log_file | default('/var/log/oauth2-proxy.log') }}" + {% endif %} + {% if nginx_proxy_oauth2_proxy_log_level | default('info') %} + log_level = "{{ nginx_proxy_oauth2_proxy_log_level | default('info') }}" + {% endif %} + cookie_secure = false + {% if nginx_proxy_oauth2_proxy_login_button | default('Sign in with Gitea') %} + custom_login_message = "Sign in with your Gitea account" + {% endif %} + ``` +- Create systemd service at `/etc/systemd/system/oauth2-proxy.service`: + ```ini + [Unit] + Description=oauth2-proxy for {{ nginx_proxy_oauth2_proxy_cookie_domain }} + After=network.target + + [Service] + Type=simple + User=nginx + ExecStart=/usr/local/bin/oauth2-proxy --config /etc/oauth2-proxy/oauth2-proxy.cfg + Restart=always + RestartSec=5 + + [Install] + WantedBy=multi-user.target + ``` +- Enable + start service +- Set config file permissions: `0640`, owner `root:nginx` (contains `client_secret`) +- Create log directory with proper permissions: `mkdir -p /var/log && chown nginx:nginx /var/log/oauth2-proxy.log` + +### Notes + +- **opencode without credentials** — Verify opencode starts and functions correctly when `OPENCODE_SERVER_USERNAME`/`OPENCODE_SERVER_PASSWORD` are not set. If opencode requires them to start, we may need to keep them but they will be ignored since nginx/oauth2-proxy handles auth. + +### 2. Update `linux_dev_station` Role + +#### `defaults/main.yml` + +- Change `linux_dev_station_code_server_host` default to `127.0.0.1` +- Change `linux_dev_station_opencode_server_host` default to `127.0.0.1` + +#### `templates/code-server.service.j2` + +- Change `--bind-addr` to `127.0.0.1:3000` +- **Remove** `Environment="PASSWORD={{ linux_dev_station_code_server_password }}"` — SSO handles auth, no password needed + +#### `templates/opencode.service.j2` + +- Change `--hostname` to `127.0.0.1` +- **Remove** `Environment="OPENCODE_SERVER_USERNAME={{ linux_dev_station_opencode_server_username }}"` — SSO handles auth +- **Remove** `Environment="OPENCODE_SERVER_PASSWORD={{ linux_dev_station_opencode_server_password }}"` — SSO handles auth + +### 3. Update dev-01 DNS & Remove Proxy Config + +#### `terraform/apps/dev-01/config.tf` + +**Add DNS records** for `ide` and `agent` pointing to dev-02 (`10.19.4.210`): + +```hcl +{ + name = "ide" + type = "A" + content = "10.19.4.210" +}, +{ + name = "agent" + type = "A" + content = "10.19.4.210" +}, +``` + +**Remove** the `opencode` and `code-server` entries from `proxy_services` (no longer needed, dev-02 handles its own TLS). + +**Remove** the `opencode` and `vscode` CNAME records from the `zones` list: +```hcl +// Remove these: +{ + name = "opencode" + type = "CNAME" + content = "dev-01.${local.lab_domain}." +}, +{ + name = "vscode" + type = "CNAME" + content = "dev-01.${local.lab_domain}." +}, +``` + +### 4. Update Host Vars & Inventory + +#### `ansible/host_vars/dev-02.lab.alexpires.me.yml` + +Add: + +```yaml +# Certbot +nginx_proxy_certbot_enabled: true +nginx_proxy_certbot_email: "c.alexandre.pires@alexpires.me" +nginx_proxy_certbot_domains: + - "ide.lab.alexpires.me" + - "agent.lab.alexpires.me" +nginx_proxy_certbot_credential_file: "/etc/letsencrypt/cloudns.ini" +nginx_proxy_cloudns_auth_id: "{{ lookup('env', 'CLOUDNS_AUTH_ID') }}" +nginx_proxy_cloudns_auth_password: "{{ lookup('env', 'CLOUDNS_PASSWORD') }}" +nginx_proxy_cloudns_nameserver: "109.201.133.111" + +# OAuth2 / Gitea +nginx_proxy_oauth2_proxy_cookie_secret: "{{ lookup('env', 'OAUTH2_PROXY_COOKIE_SECRET') }}" +nginx_proxy_oauth2_proxy_client_id: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_ID') }}" +nginx_proxy_oauth2_proxy_client_secret: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_SECRET') }}" +``` + +#### `inventory/hosts` + +Add `dev-02.lab.alexpires.me` to `[certbot]` group: +```ini +[certbot] +dev-02.lab.alexpires.me +``` + +### 5. Gitea OAuth2 Application + +Register one OAuth2 app in Gitea: +- **Name:** `lab-services` +- **Redirect URI:** `https://ide.lab.alexpires.me/oauth2/callback` +- **Scopes:** `openid`, `profile`, `email` +- **Result:** `client_id` + `client_secret` → stored in `sectool` / env vars + +### 6. Firewall Changes (handled by existing firewall role) + +| Port | Before | After | +|------|--------|-------| +| 22 | `10.19.4.0/24` + VPN | Same | +| 3000 | `10.19.4.0/24` | **CLOSED** | +| 4096 | `10.19.4.0/24` | **CLOSED** | +| 443 | — | `10.19.4.0/24` + VPN | + +The `linux_dev_station` role already manages firewall rules. Update `fw_allowed_ports` in host_vars: +```yaml +fw_allowed_ports: + - { rule: "allow", port: "22", proto: "tcp", from: "10.19.4.0/24" } + - { rule: "allow", port: "22", proto: "tcp", from: "10.5.5.5/32" } + - { rule: "allow", port: "443", proto: "tcp", from: "10.19.4.0/24" } + - { rule: "allow", port: "443", proto: "tcp", from: "10.5.5.5/32" } +``` + +## SELinux Considerations + +- Fedora ships with SELinux enforcing by default +- nginx needs permission to read certs: `semanage fcontext -a -t httpd_cert_t "/etc/letsencrypt(/.*)?"` + `restorecon -Rv /etc/letsencrypt` +- oauth2-proxy runs as `nginx` user — ensure systemd service file sets `User=nginx` +- nginx needs outbound HTTP/HTTPS to Gitea and CloudDNS API — these are allowed by default for the `httpd_t` domain + +## Playbook Structure + +```yaml +# playbook_linux_dev_station.yml +- hosts: linux_dev + roles: + - linux_dev_station + +# playbook_nginx_proxy.yml +- hosts: nginx + roles: + - nginx_proxy +``` + +The `nginx_proxy` role is a standalone, generic role — not tied to `linux_dev_station`. + +## Apply Order + +1. Register Gitea OAuth2 app → store creds in `sectool` +2. Run `playbook_nginx_proxy.yml` on dev-02 (Ansible) +3. Update `fw_allowed_ports` in dev-02 host_vars (close 3000/4096, add 443) +4. Update dev-01 DNS + remove proxy config (Terraform) +5. Verify: `https://ide.lab.alexpires.me` → SSO via Gitea +6. Verify: `https://agent.lab.alexpires.me` → SSO via Gitea + +## Files to Create + +| File | Action | +|------|--------| +| `ansible/roles/nginx_proxy/` | **NEW** role | +| `ansible/playbook_nginx_proxy.yml` | **NEW** standalone playbook (`hosts: certbot`) | +| `ansible/roles/linux_dev_station/defaults/main.yml` | Update bind to `127.0.0.1` | +| `ansible/roles/linux_dev_station/templates/code-server.service.j2` | `--bind-addr 127.0.0.1:3000` | +| `ansible/roles/linux_dev_station/templates/opencode.service.j2` | `--hostname 127.0.0.1` | +| `ansible/host_vars/dev-02.lab.alexpires.me.yml` | Add certbot + oauth2 vars | +| `inventory/hosts` | Add dev-02 to `[certbot]` | +| `terraform/apps/dev-01/config.tf` | Add DNS records for ide/agent, remove proxy config, remove CNAMEs | +| `implemented/linux_dev_station.md` | Update report | diff --git a/AGENTS.md b/AGENTS.md index a7c0782..c058040 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -3,6 +3,8 @@ ## Repo Structure - **`ansible/`** — Playbooks, roles, inventory. Runs against bare-metal servers. +- **`.opencode/plans/`** — Plans under execution +- **`implemented/`** — Reports of implemented features - **`terraform/`** — IaC (OpenTofu for K8s app deployments + cloud provisioning). - **`inventory/hosts`** — Ansible inventory. All hosts and groups. - **`requirements/`** — pip requirements files: `ansible.txt`, `dev.txt`, `opentofu.txt`. @@ -26,7 +28,11 @@ export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES # if on a MAC ``` ### Running playbooks -```sh + +**Note:** ```sectool``` just need to be used when secrets must be injected to the environment, if the command does not requires secrets injection do not use ```sectool``` + +``` +sh cd ansible sectool exec ansible-playbook -u $ANSIBLE_USER playbook_.yml ``` @@ -64,11 +70,33 @@ Excluded paths: `.git/`, `.github/`, `tests/`. Warns on: `line-length`, `yaml[li - `terraform/iac/cloudns/` — DNS (cloudns) - `terraform/iac/scaleway/` — Scaleway infrastructure -### Secrets +### Running Terraform / OpenTofu + Each terraform directory has its own `sectool.env`. Run `sectool` to load secrets before `tofu init/plan/apply`: +To validate plane (mandatory before apply) ```sh -sectool env -d terraform/apps/dev-01/ -- tofu plan +sectool exec tofu plan ``` +Apply: +```sh +sectool exec tofu apply --auto-approve +``` + +## Secrets + +To list secrets: +```sh +sectool vault list +``` +To set a secret: +```sh +sectool vault set +``` +To set a secret: +```sh +sectool vault get +``` + ### Module structure - `terraform/modules/apps/` — application modules (Nextcloud, etc.) @@ -112,3 +140,8 @@ sectool exec cntb get instances --oauth2-clientid="$CONTABO_CLIENT_ID" ... - Ansible roles follow galaxy conventions: `defaults/`, `tasks/`, `meta/`, `files/`, `templates/` - `tasks/main.yml` is the entrypoint for every role - Host vars: `ansible/host_vars/.yml` + + +## Troubleshooting and planning + +- When troubleshooting and planning use deepwiki whenever possible to get documentation or information, deepwiki provides information about projects hosted on github to agents. diff --git a/ansible/host_vars/dev-02.lab.alexpires.me.yml b/ansible/host_vars/dev-02.lab.alexpires.me.yml index 4b17047..9edbca4 100644 --- a/ansible/host_vars/dev-02.lab.alexpires.me.yml +++ b/ansible/host_vars/dev-02.lab.alexpires.me.yml @@ -83,7 +83,7 @@ linux_dev_station_opencode_config: name: "alexpires.me" npm: "@ai-sdk/openai-compatible" options: - baseURL: "https://open-webui.lab.alexpires.me/api" + baseURL: "https://llm.lab.alexpires.me/api/v1" apiKey: "{{ lookup('env', 'OPEN_WEBUI_API_KEY', default='') }}" timeout: 3000000 chunkTimeout: 3000000 @@ -92,6 +92,12 @@ linux_dev_station_opencode_config: name: "qwen3-6-27b-mtp-q4-0" qwen3-6-35b-a3b-mtp-ud-q4-m: name: "qwen3-6-35b-a3b-mtp-ud-q4-m" + scaleway: + name: "scaleway" + npm: "@ai-sdk/openai-compatible" + options: + baseURL: "https://api.scaleway.ai/{{ lookup('env', 'SCW_DEFAULT_ORGANIZATION_ID', default='') }}/v1" + apiKey: "{{ lookup('env', 'SCW_INFERENCE_API_KEY', default='') }}" mcp: deepwiki: type: "remote" diff --git a/ansible/host_vars/gpu-01.lab.alexpires.me.yml b/ansible/host_vars/gpu-01.lab.alexpires.me.yml index 30ce3cd..7c6a0c8 100644 --- a/ansible/host_vars/gpu-01.lab.alexpires.me.yml +++ b/ansible/host_vars/gpu-01.lab.alexpires.me.yml @@ -39,8 +39,8 @@ ufw_enable: true fw_allowed_ports: - { rule: "allow", port: "22", proto: "tcp", from: "10.5.5.5/32" } # Wireguard VPN (Laptop) - { rule: "allow", port: "22", proto: "tcp", from: "10.19.4.0/24" } # local network - - { rule: "allow", port: "11434", proto: "tcp", from: "10.19.4.0/24" } # Ollama (local network) - - { rule: "allow", port: "11434", proto: "tcp", from: "10.5.5.5/32" } # Ollama (Laptop) + - { rule: "allow", port: "443", proto: "tcp", from: "10.19.4.0/24" } # HTTPS (nginx reverse proxy) + - { rule: "allow", port: "443", proto: "tcp", from: "10.5.5.5/32" } # HTTPS (nginx reverse proxy, VPN) - { rule: "allow", port: "8880", proto: "tcp", from: "10.19.4.0/24" } # Kokoro-FastAPI (local network) - { rule: "allow", port: "8880", proto: "tcp", from: "10.5.5.5/32" } # Kokoro-FastAPI (Laptop) - { rule: "allow", port: "8188", proto: "tcp", from: "10.19.4.0/24" } # ComfyUI (local network) @@ -112,6 +112,7 @@ llama_server_home: "/mnt/data/llamacpp" llama_server_user_pubkey: "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHHOPBR9p9kq5Cqzpe4cr3jHnweaYrHPXv5sXNzt+sCXP54uc5rWUBhxW2OQVvQzJ47dEVhEKi4WSC7LcuKS2G5AQDzWXNiasHvYIYQU3F/EknVCZnsiXYqXphYkJA6rJCNRnISZCIC1mocq6PB7J08ONdRFCvjfUBuVRT8BNGXNmQ/zQ==" llama_server_port: 11434 +llama_server_bind: "127.0.0.1" llama_server_devices: - "nvidia.com/gpu=all" - "/dev/kfd:/dev/kfd:rw" @@ -202,6 +203,8 @@ llama_server_models: llama_server_argv_extra: [ + "--api-key", + "{{ lookup('env', 'OPEN_WEBUI_API_KEY', default='') }}", "-t", 10, "-np", @@ -249,3 +252,33 @@ auto_suspend_cpu_idle_threshold: 90.0 # llama_server_preset_global: # ctx-size: 98304 # n-gpu-layers: all + +# nginx_proxy: TLS + OAuth2/OIDC SSO for llm.lab.alexpires.me +nginx_proxy_certbot_email: "c.alexandre.pires@alexpires.me" +nginx_proxy_oauth2_proxy_enabled: true +nginx_proxy_oauth2_proxy_provider: "oidc" +nginx_proxy_oauth2_proxy_provider_url: "https://code.alexpires.me" +nginx_proxy_oauth2_proxy_redirect_urls: + - "https://llm.lab.alexpires.me/oauth2/callback" +nginx_proxy_oauth2_proxy_email_domains: + - "*" +nginx_proxy_oauth2_proxy_cookie_secret: "{{ lookup('env', 'OAUTH2_PROXY_COOKIE_SECRET', default='') }}" +nginx_proxy_oauth2_proxy_client_id: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_ID', default='') }}" +nginx_proxy_oauth2_proxy_client_secret: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_SECRET', default='') }}" +nginx_proxy_sites: + - name: llm + server_name: "llm.lab.alexpires.me" + upstream: + host: "127.0.0.1" + port: 11434 + websocket: true + extra_locations: + - path: /api/ + config: + proxy_pass: "http://127.0.0.1:11434/" + proxy_buffering: "off" + proxy_read_timeout: "1800s" + proxy_send_timeout: "1800s" + proxy_http_version: "1.1" + proxy_set_header Upgrade: "$http_upgrade" + proxy_set_header Connection: "$connection_upgrade" diff --git a/ansible/roles/llama_server/defaults/main.yml b/ansible/roles/llama_server/defaults/main.yml index 51f1416..60c20b1 100644 --- a/ansible/roles/llama_server/defaults/main.yml +++ b/ansible/roles/llama_server/defaults/main.yml @@ -10,6 +10,7 @@ llama_server_extra_groups: "users,video" llama_server_devices: [] llama_server_env: {} llama_server_port: 11434 +llama_server_bind: "0.0.0.0" llama_server_preset_global: {} llama_server_host: "127.0.0.1" llama_server_argv_extra: [] diff --git a/ansible/roles/llama_server/tasks/el/main.yml b/ansible/roles/llama_server/tasks/el/main.yml index 22836d9..617dd77 100644 --- a/ansible/roles/llama_server/tasks/el/main.yml +++ b/ansible/roles/llama_server/tasks/el/main.yml @@ -135,7 +135,7 @@ state: started device: "{{ llama_server_devices | default(omit) }}" env: "{{ llama_server_env | default(omit) }}" - ports: "0.0.0.0:{{ llama_server_port }}:8080/tcp" + ports: "{{ llama_server_bind }}:{{ llama_server_port }}:8080/tcp" volumes: - "{{ llama_server_home }}/models:/models:Z" - "{{ llama_server_home }}/.cache:/app/.cache:Z" diff --git a/ansible/roles/nginx_proxy/tasks/nginx.yml b/ansible/roles/nginx_proxy/tasks/nginx.yml index c97d00a..58ebf7e 100644 --- a/ansible/roles/nginx_proxy/tasks/nginx.yml +++ b/ansible/roles/nginx_proxy/tasks/nginx.yml @@ -44,3 +44,18 @@ become: true ansible.builtin.command: restorecon -Rv /etc/letsencrypt changed_when: false + +- name: Set SELinux boolean to allow nginx to connect to any port + become: true + ansible.builtin.command: semanage boolean -m --on nginx_connect_any + failed_when: false + changed_when: false + when: ansible_selinux.status == "enabled" + +- name: Add oauth2-proxy port to SELinux http_port_t + become: true + ansible.builtin.command: >- + semanage port -a -t http_port_t -p tcp {{ nginx_proxy_oauth2_proxy_port }} + failed_when: false + changed_when: false + when: ansible_selinux.status == "enabled" diff --git a/ansible/roles/nginx_proxy/tasks/oauth2_proxy.yml b/ansible/roles/nginx_proxy/tasks/oauth2_proxy.yml index 461557c..d0d7f17 100644 --- a/ansible/roles/nginx_proxy/tasks/oauth2_proxy.yml +++ b/ansible/roles/nginx_proxy/tasks/oauth2_proxy.yml @@ -22,6 +22,7 @@ mode: "0755" owner: root group: root + remote_src: true - name: Create oauth2-proxy config directory become: true diff --git a/ansible/roles/nginx_proxy/templates/nginx-server.conf.j2 b/ansible/roles/nginx_proxy/templates/nginx-server.conf.j2 index 5d7cedf..6d90f90 100644 --- a/ansible/roles/nginx_proxy/templates/nginx-server.conf.j2 +++ b/ansible/roles/nginx_proxy/templates/nginx-server.conf.j2 @@ -13,6 +13,7 @@ server { add_header Strict-Transport-Security "max-age={{ nginx_proxy_nginx_hsts_max_age }}; {{ nginx_proxy_nginx_hsts_include_subdomains | ternary('includeSubDomains', '') }}" always; +{% if not (item.no_auth | default(false) | bool) %} # Redirect unauthenticated requests to OAuth2 sign-in error_page 401 = /oauth2/sign_in; @@ -52,13 +53,16 @@ server { proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Scheme $scheme; } +{% endif %} location / { +{% if not (item.no_auth | default(false) | bool) %} auth_request /oauth2/auth; auth_request_set $auth_status $upstream_status; auth_request_set $auth_cookie $upstream_http_set_cookie; add_header Set-Cookie $auth_cookie; +{% endif %} add_header Strict-Transport-Security "max-age={{ nginx_proxy_nginx_hsts_max_age }}; {{ nginx_proxy_nginx_hsts_include_subdomains | ternary('includeSubDomains', '') }}" always; proxy_pass http://{{ item.upstream.host }}:{{ item.upstream.port }}; diff --git a/ansible/sectool.env b/ansible/sectool.env index 3824541..169516a 100644 --- a/ansible/sectool.env +++ b/ansible/sectool.env @@ -24,3 +24,5 @@ OPEN_WEBUI_API_KEY=$OPEN_WEBUI_API_KEY OAUTH2_PROXY_COOKIE_SECRET=$OAUTH2_PROXY_COOKIE_SECRET OAUTH2_PROXY_CLIENT_ID=$OAUTH2_PROXY_CLIENT_ID OAUTH2_PROXY_CLIENT_SECRET=$OAUTH2_PROXY_CLIENT_SECRET +SCW_DEFAULT_ORGANIZATION_ID=$SCALEWAY_ORGANIZATION_ID +SCW_INFERENCE_API_KEY=$SCW_INFERENCE_API_KEY \ No newline at end of file diff --git a/inventory/hosts b/inventory/hosts index 49d52df..c091b87 100644 --- a/inventory/hosts +++ b/inventory/hosts @@ -27,6 +27,7 @@ gpu-01.lab.alexpires.me [nginx] dev-02.lab.alexpires.me +gpu-01.lab.alexpires.me [public] prod-01.alexpires.me @@ -92,9 +93,6 @@ gpu-01.lab.alexpires.me [comfyui] gpu-01.lab.alexpires.me -[openwebui] -dev-01.lab.alexpires.me - [mcp] dev-01.lab.alexpires.me diff --git a/terraform/apps/dev-01/apps.tf b/terraform/apps/dev-01/apps.tf index 12686a9..a6d0945 100644 --- a/terraform/apps/dev-01/apps.tf +++ b/terraform/apps/dev-01/apps.tf @@ -1,14 +1,3 @@ -module "open-webui" { - source = "../../modules/apps/open-webui" - - persistent_folder = local.persistent_folder - fqdn = local.open_webui_fqdn - llm_proxy = local.llm_proxy - secret_key = var.webui_secret_key - tag = "v0.9.2-slim" - log_level = local.open_webui_log_level -} - module "samba" { source = "../../modules/apps/samba" diff --git a/terraform/apps/dev-01/config.tf b/terraform/apps/dev-01/config.tf index 10ef9f0..cebcb42 100644 --- a/terraform/apps/dev-01/config.tf +++ b/terraform/apps/dev-01/config.tf @@ -6,10 +6,6 @@ locals { lab_domain = "lab.alexpires.me" internal_issuer = "internal-ca" - # open-webui - open_webui_fqdn = "open-webui.${local.lab_domain}" - open_webui_log_level = "DEBUG" - # samba samba_shares = [ { @@ -61,12 +57,7 @@ locals { type = "CNAME" content = "dev-01.${local.lab_domain}." }, - { - name = "open-webui" - type = "CNAME" - content = "dev-01.${local.lab_domain}." - }, - { +{ name = "smb" type = "CNAME" content = "dev-01.${local.lab_domain}." @@ -116,6 +107,11 @@ locals { type = "A" content = "10.19.4.210" }, + { + name = "llm" + type = "A" + content = "10.19.4.106" + }, { name = "ide" type = "A" @@ -180,14 +176,6 @@ locals { } } - llm_proxy = { - enabled = true - mac = "04:7c:16:d6:63:66" - ip = "10.19.4.106" - port = 11434 - scheme = "http" - } - llm_proxy_1 = { enabled = true mac = "52:54:00:CD:DD:2E" diff --git a/terraform/apps/dev-01/secrets.tf b/terraform/apps/dev-01/secrets.tf index 30f1abb..bd4ae3a 100644 --- a/terraform/apps/dev-01/secrets.tf +++ b/terraform/apps/dev-01/secrets.tf @@ -22,12 +22,6 @@ variable "telegram_bot_token" { sensitive = true } -variable "webui_secret_key" { - description = "The secret key for WEBUI" - type = string - sensitive = true -} - variable "searxng_secret_key" { description = "The secret key for SearXNG" type = string diff --git a/terraform/apps/prod-01/config.tf b/terraform/apps/prod-01/config.tf index 98b8e6b..fc99506 100644 --- a/terraform/apps/prod-01/config.tf +++ b/terraform/apps/prod-01/config.tf @@ -147,28 +147,6 @@ locals { } proxy_services = { - "open-webui" = { - http_port = 8080 - https_port = 8443 - tls = true - resolver = "10.19.4.136" - fqdn = ["ai.alexpires.me"] - read_timeout = "1800" - send_timeout = "1800" - upstream = "https://open-webui.lab.alexpires.me" - locations = [ - { - path = "/ws/socket.io/" - headers = { - "Upgrade" = "$http_upgrade" - "Connection" = "upgrade" - } - }, - { - path = "/" - }, - ] - }, "nextcloud" = { http_port = 8081 https_port = 8444 diff --git a/terraform/iac/cloudns/config.tf b/terraform/iac/cloudns/config.tf index c6daf0d..efbfd4e 100644 --- a/terraform/iac/cloudns/config.tf +++ b/terraform/iac/cloudns/config.tf @@ -170,16 +170,6 @@ locals { type = "TXT" value = "v=DKIM1; h=sha256; k=rsa; p=${var.alexpires_me_scaleway_dkim_value}" }, - { - name = "webdav" - type = "CNAME" - value = "${local.vps_1.name}.alexpires.me" - }, - { - name = "ai" - type = "CNAME" - value = "${local.vps_1.name}.alexpires.me" - }, ] } diff --git a/terraform/iac/cloudns/sectool.json b/terraform/iac/cloudns/sectool.json new file mode 100644 index 0000000..b50832d --- /dev/null +++ b/terraform/iac/cloudns/sectool.json @@ -0,0 +1,8 @@ +{ + "provider": "bitwarden", + "bitwarden": { + "api_url": "https://api.bitwarden.eu", + "identity_url": "https://identity.bitwarden.eu", + "project": "c6bd56a4-9fec-4388-b74a-b2a100c0ae07" + } +} \ No newline at end of file