Enhance infrastructure and application configurations
- Updated AGENTS.md to include new directories for execution plans and implemented feature reports. - Modified Ansible host variables for dev-02 and gpu-01 to change API endpoints and add new configurations for Scaleway. - Adjusted firewall rules in gpu-01 to allow HTTPS traffic instead of the previous Ollama port. - Added OAuth2 proxy configurations to gpu-01 for enhanced security. - Removed deprecated open-webui module from Terraform app configurations. - Updated Terraform configurations to reflect changes in local variables and removed unused secrets. - Created a new sectool.json file for CloudNS integration with Bitwarden. - Enhanced SELinux configurations for nginx to allow connections to any port and added oauth2-proxy port.
This commit is contained in:
@@ -0,0 +1,233 @@
|
||||
# Gitea → MicroK8s Webhook Orchestrator → Scaleway Containers (CI Runner System)
|
||||
|
||||
A lightweight CI orchestration system where a self-hosted Gitea instance triggers a Go-based webhook service running inside MicroK8s. The service decides whether to run jobs locally or launch ephemeral CI runners in Scaleway Serverless Containers.
|
||||
|
||||
---
|
||||
|
||||
## 1. High-Level Architecture
|
||||
|
||||
```text
|
||||
On-Premises (MicroK8s Cluster)
|
||||
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Gitea │
|
||||
│ │
|
||||
│ Repo push / PR event │
|
||||
└───────────────┬──────────────────────────────────────────────┘
|
||||
│ Webhook (internal cluster network)
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ ci-orchestrator (Go service) │
|
||||
│ │
|
||||
│ - Validates webhook signature (HMAC) │
|
||||
│ - Parses event payload │
|
||||
│ - Applies routing rules │
|
||||
│ - Decides runner target (local vs cloud) │
|
||||
└───────────────┬──────────────────────────────────────────────┘
|
||||
│ HTTPS API call (authenticated)
|
||||
▼
|
||||
──────────────────────── Internet ───────────────────────────────
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────┐
|
||||
│ Scaleway Serverless Containers API │
|
||||
│ │
|
||||
│ - Starts ephemeral container │
|
||||
│ - Runs Gitea Runner │
|
||||
│ - Executes CI job │
|
||||
│ - Shuts down after completion │
|
||||
└──────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Current State
|
||||
|
||||
| Component | Location | Status |
|
||||
|---|---|---|
|
||||
| **Gitea** | `prod-01.alexpires.me` (K8s) | Deployed via Terraform module, TLS, ingress, S3 storage, MariaDB |
|
||||
| **MicroK8s** | `prod-01`, `dev-01` | nginx ingress, cert-manager, multiple apps running |
|
||||
| **Scaleway** | `fr-par` | Registry + S3 buckets + IAM keys exist, no Serverless Containers yet |
|
||||
| **Gitea Runners** | `ci-01.lab.alexpires.me` | Windows Podman + native runners already deployed |
|
||||
| **DNS** | CoreDNS in `dev-01` | `ci-01 → 10.19.4.207` already resolved |
|
||||
|
||||
---
|
||||
|
||||
## 3. Design Decisions
|
||||
|
||||
1. **Orchestrator cluster:** `prod-01` for low latency to Gitea
|
||||
2. **Routing:** Cloud runners only trigger when `cloud` label is present
|
||||
3. **Runner image:** `gitea/act_runner` (official)
|
||||
4. **Webhook registration:** Terraform-managed via `go-gitea/gitea` provider
|
||||
|
||||
---
|
||||
|
||||
## 4. Implementation Plan
|
||||
|
||||
### Phase 1: Scaleway IaC (`terraform/iac/scaleway/`)
|
||||
|
||||
**New Terraform resources:**
|
||||
- `scaleway_container_namespace` — CI runners namespace
|
||||
- `scaleway_container` — the `gitea/act_runner` container (ephemeral, privacy=private)
|
||||
- `scaleway_iam_policy` — `ContainersInvoke` permission for the CI orchestrator app
|
||||
- `scaleway_container_domain` (optional) — custom domain if desired
|
||||
|
||||
**IAM:**
|
||||
- New app `ci_runner_app` with policy granting `ContainersInvoke`
|
||||
- API key for the Go service to invoke containers
|
||||
|
||||
**Existing reuse:**
|
||||
- `data.terraform_remote_state.scaleway` in `prod-01` for outputs
|
||||
- Existing `scaleway_registry_namespace` for pushing the runner image
|
||||
|
||||
### Phase 2: Go Service (`ci-orchestrator/`)
|
||||
|
||||
```
|
||||
ci-orchestrator/
|
||||
├── go.mod
|
||||
├── main.go # Entry: HTTP server
|
||||
├── Dockerfile # Multi-stage, distroless
|
||||
├── .dockerignore
|
||||
└── internal/
|
||||
├── webhook/
|
||||
│ ├── gitea.go # Parse Gitea push/PR payloads
|
||||
│ └── hmac.go # HMAC-SHA256 validation
|
||||
├── router/
|
||||
│ ├── router.go # Check if job has "cloud" label
|
||||
│ └── rules.go # Config-based routing rules
|
||||
└── runner/
|
||||
└── scaleway.go # Invoke SCW Serverless Container
|
||||
```
|
||||
|
||||
**Routing logic:**
|
||||
- Parse webhook → extract repo, branch, commit, labels (from PR body or commit message convention)
|
||||
- If `cloud` label → invoke Scaleway container
|
||||
- If no label → ignore (existing runners on `ci-01` handle it)
|
||||
|
||||
**Endpoints:**
|
||||
- `POST /webhook/gitea` — main webhook receiver
|
||||
- `GET /health` — health check
|
||||
|
||||
### Phase 3: Runner Container Image
|
||||
|
||||
```dockerfile
|
||||
FROM gitea/act_runner:latest
|
||||
ENV GITEA_RUNNER_IGNORE_RUNNING_JOBS_IN_SCRIPT=true
|
||||
# Entrypoint configured via Terraform env vars
|
||||
```
|
||||
|
||||
Push to: `rg.fr-par.scw.cloud/<ci-namespace>/act-runner:<tag>`
|
||||
|
||||
### Phase 4: Terraform K8s Deployment (`terraform/modules/apps/ci-orchestrator/`)
|
||||
|
||||
Deployed in `prod-01` namespace:
|
||||
|
||||
| Resource | Purpose |
|
||||
|---|---|
|
||||
| `kubernetes_namespace` | `ci-orchestrator` |
|
||||
| `kubernetes_deployment` | Go service (hardened security context like other modules) |
|
||||
| `kubernetes_service` | ClusterIP, port 8080 |
|
||||
| `kubernetes_ingress_v1` | Ingress via nginx, path `/webhook/gitea` |
|
||||
| `kubernetes_config_map` | Routing rules config |
|
||||
| `kubernetes_secret` | HMAC key, Scaleway API credentials, Gitea URL |
|
||||
|
||||
### Phase 5: Gitea Webhook Registration
|
||||
|
||||
```hcl
|
||||
provider "gitea" {
|
||||
address = "https://code.alexpires.me"
|
||||
username = "<user>"
|
||||
token = "<token>"
|
||||
}
|
||||
|
||||
resource "gitea_repository_webhook" "ci_orchestrator" {
|
||||
username = "<org-or-user>"
|
||||
name = "<repo-name>"
|
||||
type = "gitea"
|
||||
url = "https://code.alexpires.me/webhook/gitea"
|
||||
secret = var.webhook_secret
|
||||
content_type = "json"
|
||||
branch_filter = "*"
|
||||
events = ["push", "pull_request"]
|
||||
active = true
|
||||
}
|
||||
```
|
||||
|
||||
Can be per-repo or a single wildcard. The Go service routes based on labels.
|
||||
|
||||
### Phase 6: CI Flow
|
||||
|
||||
```
|
||||
push/PR to Gitea
|
||||
│
|
||||
▼
|
||||
Webhook → ci-orchestrator (prod-01)
|
||||
│
|
||||
├─ no cloud label → ignore (local runners handle it)
|
||||
│
|
||||
└─ cloud label → invoke Scaleway container
|
||||
│
|
||||
▼
|
||||
gitea/act_runner starts
|
||||
Registers with Gitea
|
||||
Receives and executes job
|
||||
Exits → container shuts down (cost: per-second)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Provider Compatibility
|
||||
|
||||
### `go-gitea/gitea` (v0.7.0)
|
||||
- `gitea_repository_webhook` resource — creates and manages webhooks
|
||||
- Supports: `url`, `secret`, `content_type`, `events`, `branch_filter`, `active`, `type`
|
||||
|
||||
### `scaleway/scaleway` (v2.77.1)
|
||||
- `scaleway_container` — Serverless Container resource
|
||||
- `privacy` can be set to `private` (requires IAM auth)
|
||||
- `min_scale = 0`, `max_scale = 1` for ephemeral execution
|
||||
- Supports `command`, `args`, `environment_variables`, `secret_environment_variables`
|
||||
- `scaleway_container_namespace` — namespace for containers
|
||||
- `scaleway_iam_policy` — IAM policies for container invocation
|
||||
|
||||
---
|
||||
|
||||
## 7. Knowns (Updated)
|
||||
|
||||
### Gitea
|
||||
- URL: `https://code.alexpires.me`
|
||||
- Admin user: `alexandre.pires`
|
||||
- API token: `${GITEA_ADMIN_TOKEN}` (via sectool)
|
||||
- Provider config: `gitea_credentials` in `terraform/iac/scaleway/` or `prod-01/`
|
||||
|
||||
### Webhook registration
|
||||
- Target: all existing repos + new ones
|
||||
- Method: one-time Ansible script to register webhooks on all repos, then run once per new repo
|
||||
- Webhook target URL: `https://code.alexpires.me/webhook/gitea` (points to Go service ingress)
|
||||
|
||||
### Runner resources
|
||||
- CPU: 2 vCPU
|
||||
- Memory: 4Gi
|
||||
- Image: `gitea/act_runner:latest` (or pinned version)
|
||||
|
||||
### Repository
|
||||
- Moved to Gitea: `https://code.alexpires.me/alexpires.me/a13labs.infra`
|
||||
- Origin remote updated to `ssh://git@code.alexpires.me:22/alexpires.me/a13labs.infra.git`
|
||||
- All branches and tags pushed
|
||||
|
||||
### CI repo workflow
|
||||
- `ci-01` runner builds Terraform images using Gitea Actions
|
||||
- Images pushed to Scaleway Registry
|
||||
- Terraform state managed via S3 on Scaleway
|
||||
|
||||
---
|
||||
|
||||
## 8. Open Decisions (to be addressed during implementation)
|
||||
|
||||
1. **Routing labels** — How will the Go service detect "cloud" intent? Options:
|
||||
- PR body labels (`[cloud]` in title/body)
|
||||
- Commit message convention (`[cloud]` prefix)
|
||||
- Branch naming convention
|
||||
- Separate "cloud" branch filter per repo
|
||||
2. **Go service ingress path** — Current plan: `/webhook/gitea`. Need to ensure it doesn't conflict with Gitea's own routes.
|
||||
3. **Scaling strategy** — `max_scale=1` per invocation, but multiple concurrent jobs? May need pool or queue.
|
||||
@@ -0,0 +1,321 @@
|
||||
# Gateway API Migration & Cloudflare Integration
|
||||
|
||||
## Overview
|
||||
|
||||
Migrate prod-01 and dev-01 clusters from the deprecated nginx-ingress controller (retired March 2026) to NGINX Gateway Fabric via the Gateway API, improve cross-cluster WireGuard connectivity, and add Cloudflare Tunnel for edge protection.
|
||||
|
||||
## Current Architecture
|
||||
|
||||
```
|
||||
Internet → Contabo Public IP → OPNsense → prod-01
|
||||
↓
|
||||
nginx-ingress plugin (MicroK8s)
|
||||
↓
|
||||
MetalLB (10.2.0.40-49)
|
||||
↓
|
||||
┌───────────────┼─────────────────┐
|
||||
↓ ↓ ↓
|
||||
[proxy pod] [app pods] [TCP streams]
|
||||
(nginx + WG) (gitea, m3uproxy, etc) (hostPort)
|
||||
↓
|
||||
WireGuard → aristotle (Contabo VPN)
|
||||
↓
|
||||
dev-01 backends (open-webui, nextcloud)
|
||||
```
|
||||
|
||||
**Key facts:**
|
||||
- 14 `kubernetes_ingress_v1` resources across both clusters
|
||||
- All use `ingress.class: public` + nginx-specific annotations
|
||||
- Proxy module on prod-01 creates a double-proxy (Ingress → proxy pod → backends)
|
||||
- TCP/UDP streams use nginx with hostPorts
|
||||
- DNS: ClouDNS, no Cloudflare involvement
|
||||
- cert-manager with DNS-01 (dev-01) and HTTP-01 (prod-01) for Let's Encrypt
|
||||
- **Both clusters are single-node** — no need for MetalLB, NodePort is sufficient
|
||||
|
||||
## Decisions Made
|
||||
|
||||
| Decision | Choice | Rationale |
|
||||
|---|---|---|
|
||||
| Gateway API implementation | **NGINX Gateway Fabric** | Keeps nginx data plane, minimal behavioral changes, conformant v1.4.1 |
|
||||
| Cross-cluster connectivity | **Improve WireGuard** | Keep through aristotle, allowed_ips to dev-01 only (10.19.4.136/32), remove gpu-01 |
|
||||
| Ollama removal | **Remove** | gpu-01 runs llamacpp (not ollama), prod-01 ollama proxy is broken, dev-01 DNS CNAME stale |
|
||||
| Cloudflare scope | **Tunnels only** | Get DDoS+WAF, keep DNS with ClouDNS, least disruptive |
|
||||
| Proxy module | **All through proxy** | Gateway routes everything to proxy Service, simpler Terraform |
|
||||
| Website module | **Direct Gateway route** | Keep separate pods, no proxy overhead for static sites |
|
||||
| TLS termination | **At Cloudflare edge** | Simpler cert management, Cloudflare provides origin cert |
|
||||
|
||||
## Target Architecture
|
||||
|
||||
```
|
||||
Internet → Cloudflare Tunnel → Cloudflare Edge (TLS termination)
|
||||
↓
|
||||
cloudflared pods (DaemonSet)
|
||||
↓
|
||||
Gateway (NGINX Gateway Fabric, NodePort Service)
|
||||
HTTP listener :80 (from cloudflared)
|
||||
↓
|
||||
┌───────────────┼─────────────────┐
|
||||
↓ ↓ ↓
|
||||
[proxy Service] [app Services] [TCP/UDP Routes]
|
||||
↓ ↓ ↓
|
||||
[proxy pod + WG] [direct pods] [direct pods]
|
||||
(nginx) (website, etc.) (rustdesk, etc.)
|
||||
↓
|
||||
Cross-cluster → dev-01 backends
|
||||
(open-webui, nextcloud)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: Gateway API Migration
|
||||
|
||||
### 1A. Install NGINX Gateway Fabric (Ansible)
|
||||
|
||||
**New file: `ansible/roles/microk8s/tasks/gateway-api.yml`**
|
||||
|
||||
```
|
||||
- Install `snap` package manager (if needed)
|
||||
- Install helm via snap
|
||||
- Add nginx-stable helm repo
|
||||
- Install Gateway API CRDs (standard channel + experimental for TCPRoute/UDPRoute)
|
||||
- Install NGINX Gateway Fabric via Helm (experimental features enabled)
|
||||
- Create GatewayClass named `public`
|
||||
- Create Gateway Service with type = NodePort (single-node, no MetalLB needed)
|
||||
```
|
||||
|
||||
**Modify: `ansible/roles/microk8s/tasks/main.yml`**
|
||||
- Include `gateway-api.yml` AFTER existing setup (runs alongside nginx-ingress initially)
|
||||
|
||||
**Modify: `ansible/roles/microk8s/defaults/main.yml`**
|
||||
- Add variables: `gateway_fabric_version`, `gateway_fabric_experimental_enabled`
|
||||
- Remove MetalLB-related vars (`microk8s_lb_provider`, `microk8s_lb_ips`, `microk8s_metallb_version`)
|
||||
|
||||
### 1B. Create Gateway Module (Terraform)
|
||||
|
||||
**New module: `terraform/modules/utils/gateway/`**
|
||||
- `main.tf`: Gateway resource with HTTP (:80) listener, `ServiceType: NodePort`
|
||||
- `variables.tf`: TLS secret names, listener config
|
||||
- `outputs.tf`: Gateway service name, NodePort port numbers
|
||||
|
||||
**Deploy in both clusters** (`prod-01/apps.tf`, `dev-01/apps.tf`)
|
||||
|
||||
### 1C. Migrate Terraform Modules (Ingress → Gateway API Resources)
|
||||
|
||||
#### Proxy module (`terraform/modules/utils/proxy/main.tf`)
|
||||
|
||||
- **Remove**: `kubernetes_ingress_v1.ingress` resource (lines 160-203)
|
||||
- **Remove**: `host_port` from stream container ports in Deployment
|
||||
- **Remove**: ollama proxy service definition from prod-01 config.tf
|
||||
- **Add**: HTTPRoute for cross-cluster services (open-webui, nextcloud)
|
||||
- **Add**: TCPRoute resources for RustDesk (21115, 21116, 21117 TCP) and Prometheus (9090)
|
||||
- **Add**: UDPRoute resource for RustDesk (21116 UDP)
|
||||
- The proxy pod's nginx stream block can be removed (Gateway handles L4 routing)
|
||||
- **Keep**: WireGuard sidecar, auth sidecar, internal nginx reverse proxy config
|
||||
|
||||
#### Website module (`terraform/modules/apps/nginx/main.tf`)
|
||||
|
||||
- **Remove**: `kubernetes_ingress_v1.website_ingress` (lines 103-145)
|
||||
- **Add**: HTTPRoute per website with backendRef → website Service
|
||||
- Gateway handles TLS termination, website pods receive plain HTTP
|
||||
- Website pods can drop internal TLS (simplification)
|
||||
|
||||
#### Gitea module (`terraform/modules/apps/gitea/ingress.tf`)
|
||||
|
||||
- **Remove**: `kubernetes_ingress_v1.gitea_ingress`
|
||||
- **Add**: HTTPRoute with backendRef → Gitea Service
|
||||
- Use `BackendTLSPolicy` if Gitea serves HTTPS internally
|
||||
|
||||
#### M3UProxy module (`terraform/modules/apps/m3uproxy/main.tf`)
|
||||
|
||||
- **Remove**: `kubernetes_ingress_v1.m3uproxy_ingress_http` (lines 241-274)
|
||||
- **Add**: HTTPRoute with backendRef → m3uproxy Service
|
||||
|
||||
#### dev-01 modules
|
||||
|
||||
| Module | Current | Change |
|
||||
|---|---|---|
|
||||
| `open-webui/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
|
||||
| `grafana/grafana.tf` | kubernetes_ingress_v1 | HTTPRoute |
|
||||
| `mstream/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
|
||||
| `searxng/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
|
||||
| `vaultwarden/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
|
||||
| `terraform-mcp/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
|
||||
| `nextcloud/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
|
||||
| `proxy/main.tf` | kubernetes_ingress_v1 | HTTPRoute (opencode, vscode) |
|
||||
|
||||
#### Annotation → Gateway API Mapping
|
||||
|
||||
| Current nginx annotation | Gateway API equivalent |
|
||||
|---|---|
|
||||
| `nginx.org/ssl-redirect: true` | HTTPRoute with `RequestRedirect` filter |
|
||||
| `nginx.org/redirect-to-https: true` | HTTPRoute on HTTP listener with redirect |
|
||||
| `nginx.ingress.kubernetes.io/backend-protocol: HTTPS` | `BackendTLSPolicy` resource |
|
||||
| `nginx.ingress.kubernetes.io/proxy-body-size` | `UpstreamSettings` policy (NGF custom) |
|
||||
| `nginx.ingress.kubernetes.io/proxy-request-buffering` | `ProxySettings` policy (NGF custom) |
|
||||
| `nginx.ingress.kubernetes.io/proxy-buffer-size` | `UpstreamSettings` policy |
|
||||
| `cert-manager.io/cluster-issuer` | cert-manager annotation on Gateway |
|
||||
|
||||
### 1D. cert-manager Integration
|
||||
|
||||
Use **approach B**: Keep existing Certificate resources, reference secrets in Gateway TLS config.
|
||||
|
||||
- Each module's `kubernetes_manifest` Certificate resource stays unchanged
|
||||
- Gateway HTTPS listener references cert secrets via `certificateRefs`
|
||||
- This avoids alpha cert-manager Gateway integration
|
||||
|
||||
### 1E. Migration Execution Order
|
||||
|
||||
1. Apply new Ansible role (installs Gateway Fabric alongside nginx-ingress)
|
||||
2. Deploy Gateway resource to both clusters
|
||||
3. Migrate modules on **dev-01 first** (lower risk):
|
||||
- Deploy HTTPRoutes for each module, one at a time
|
||||
- Verify each service works
|
||||
- Remove old Ingress resources
|
||||
4. Migrate modules on **prod-01**:
|
||||
- Same process, one module at a time
|
||||
5. Disable nginx-ingress plugin on both clusters
|
||||
6. Clean up IngressClass, old LoadBalancer Service
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Cross-Cluster Connectivity
|
||||
|
||||
### 2A. Improve WireGuard
|
||||
|
||||
**Modify: `terraform/apps/prod-01/config.tf`** (WireGuard config):
|
||||
- Change `allowed_ips` from `[10.19.4.106/32, 10.19.4.136/32]` → `["10.19.4.136/32"]` (dev-01 only)
|
||||
- Remove gpu-01 (10.19.4.106) - ollama proxy being removed, no need to reach gpu-01
|
||||
- `persistent_keepalive` already set to 25
|
||||
|
||||
**Modify: `terraform/modules/utils/proxy/resources/wireguard/wireguard.conf.tpl`**:
|
||||
- Template `PersistentKeepalive` from config variable
|
||||
- Template `AllowedIPs` from config list
|
||||
|
||||
**Also clean up**:
|
||||
- Remove `"ollama"` entry from `proxy_services` in prod-01 config.tf
|
||||
- Remove `ollama` CNAME from dev-01 DNS zones in dev-01 config.tf
|
||||
|
||||
**Modify: `terraform/modules/utils/proxy/main.tf`** (WireGuard container):
|
||||
- Improve liveness probe: `wg show` instead of `wg show | grep -q transfer`
|
||||
- Add readiness probe
|
||||
- Add resource limits (memory cap)
|
||||
|
||||
### 2B. DNS Resolution
|
||||
|
||||
No changes needed. The proxy's per-location resolver in `default.conf.tpl` already handles DNS correctly for cross-cluster backends.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Cloudflare Tunnels
|
||||
|
||||
### 3A. Cloudflare Setup
|
||||
|
||||
- Create/verify Cloudflare account with `alexpires.me` domain
|
||||
- Keep DNS with ClouDNS (Cloudflare acts as reverse proxy only)
|
||||
- Create Cloudflare Tunnel for prod-01
|
||||
- Configure route rules: `https://*.alexpires.me` → Gateway Service
|
||||
- Set up origin rules (TLS terminates at Cloudflare edge → HTTP to cloudflared)
|
||||
|
||||
### 3B. Deploy cloudflared
|
||||
|
||||
**New module: `terraform/modules/utils/cloudflare-tunnel/`**
|
||||
- DaemonSet of `cloudflare/cloudflared` pods
|
||||
- ConfigMap with tunnel credentials and route definitions
|
||||
- Each public FQDN mapped to `http://<gateway-service>.<namespace>:80`
|
||||
|
||||
### 3C. Gateway Configuration Changes
|
||||
|
||||
- Gateway uses HTTP (:80) NodePort listener for Cloudflare Tunnel traffic
|
||||
- Cloudflare Tunnel connects to `http://<node_ip>:<nodeport>`
|
||||
- No HTTPS listener needed since all traffic comes through Cloudflare Tunnel
|
||||
|
||||
### 3D. Firewall Changes
|
||||
|
||||
- Close UFW ports 80/443 (cloudflared uses ephemeral outbound ports)
|
||||
- Keep SSH + WireGuard ports open
|
||||
|
||||
### 3E. Security Benefits
|
||||
|
||||
With Cloudflare in front:
|
||||
- No open ports on prod-01 (80, 443, etc. closed)
|
||||
- DDoS protection (automatic with Cloudflare)
|
||||
- WAF rules (OWASP Top 10, custom rules)
|
||||
- Rate limiting on public endpoints
|
||||
- Bot management
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Cleanup
|
||||
|
||||
- Remove nginx-ingress plugin from MicroK8s (`microk8s.disable ingress`)
|
||||
- Remove unused IngressClass resources
|
||||
- Remove existing MetalLB (delete metallb-system namespace, IPAddressPool, L2Advertisement)
|
||||
- Delete orphaned `tasks/loadbalancer.yml` and `tasks/loadbalancer/metallb.yaml`
|
||||
- Remove MetalLB vars from `defaults/main.yml`
|
||||
- Update documentation
|
||||
|
||||
---
|
||||
|
||||
## File Change Summary
|
||||
|
||||
### New Files
|
||||
| File | Purpose |
|
||||
|---|---|
|
||||
| `ansible/roles/microk8s/tasks/gateway-api.yml` | NGINX Gateway Fabric install |
|
||||
| `terraform/modules/utils/gateway/main.tf` | Gateway + ReferenceGrant resources |
|
||||
| `terraform/modules/utils/gateway/variables.tf` | Gateway configuration |
|
||||
| `terraform/modules/utils/gateway/outputs.tf` | Gateway outputs |
|
||||
| `terraform/modules/utils/cloudflare-tunnel/main.tf` | cloudflared DaemonSet |
|
||||
| `terraform/modules/utils/cloudflare-tunnel/variables.tf` | Tunnel config |
|
||||
|
||||
### Deleted Files
|
||||
| File | Reason |
|
||||
|---|---|
|
||||
| `ansible/roles/microk8s/tasks/loadbalancer.yml` | Not wired into main.yml, was for nginx-ingress LB |
|
||||
| `ansible/roles/microk8s/tasks/loadbalancer/metallb.yaml` | MetalLB no longer needed (single-node, NodePort) |
|
||||
|
||||
### Modified Files
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `ansible/roles/microk8s/tasks/main.yml` | Include gateway-api.yml |
|
||||
| `ansible/roles/microk8s/defaults/main.yml` | Add Gateway Fabric defaults, remove MetalLB vars |
|
||||
| `terraform/modules/utils/proxy/main.tf` | Ingress → HTTPRoute + TCPRoute + UDPRoute |
|
||||
| `terraform/modules/utils/proxy/variables.tf` | Update for Gateway API |
|
||||
| `terraform/modules/utils/proxy/resources/wireguard/wireguard.conf.tpl` | Add PersistentKeepalive |
|
||||
| `terraform/modules/apps/nginx/main.tf` | Ingress → HTTPRoute |
|
||||
| `terraform/modules/apps/gitea/ingress.tf` | Ingress → HTTPRoute |
|
||||
| `terraform/modules/apps/m3uproxy/main.tf` | Ingress → HTTPRoute |
|
||||
| `terraform/modules/apps/open-webui/ingress.tf` | Ingress → HTTPRoute |
|
||||
| `terraform/modules/apps/grafana/grafana.tf` | Ingress → HTTPRoute |
|
||||
| `terraform/modules/apps/mstream/ingress.tf` | Ingress → HTTPRoute |
|
||||
| `terraform/modules/apps/searxng/ingress.tf` | Ingress → HTTPRoute |
|
||||
| `terraform/modules/apps/vaultwarden/ingress.tf` | Ingress → HTTPRoute |
|
||||
| `terraform/modules/apps/terraform-mcp/ingress.tf` | Ingress → HTTPRoute |
|
||||
| `terraform/modules/apps/nextcloud/ingress.tf` | Ingress → HTTPRoute |
|
||||
| `terraform/modules/apps/ssh_locker_web/` | Ingress → HTTPRoute |
|
||||
| `terraform/apps/prod-01/apps.tf` | Add gateway module |
|
||||
| `terraform/apps/prod-01/config.tf` | WireGuard (remove gpu-01), remove ollama proxy |
|
||||
| `terraform/apps/dev-01/config.tf` | Remove ollama DNS CNAME |
|
||||
| `terraform/apps/dev-01/apps.tf` | Add gateway module |
|
||||
|
||||
---
|
||||
|
||||
## Risk Assessment
|
||||
|
||||
| Risk | Mitigation |
|
||||
|---|---|
|
||||
| TCPRoute/UDPRoute experimental in NGF | Start with HTTPRoute only, keep proxy streams temporarily |
|
||||
| Cloudflare Tunnel adds latency | Monitor, acceptable for reverse proxy traffic |
|
||||
| cert-manager Gateway integration alpha | Use approach B (separate Certificate resources) |
|
||||
| Downtime during migration | Run both controllers in parallel, switch via DNS TTL |
|
||||
| Removing gpu-01 from WireGuard allowed_ips | Verify no active proxy depends on gpu-01 reachability |
|
||||
| Removing existing MetalLB on prod-01 | Do after Cloudflare Tunnel is fully operational |
|
||||
|
||||
## Timeline Estimate
|
||||
|
||||
| Phase | Duration | Dependencies |
|
||||
|---|---|---|
|
||||
| Phase 1 (Gateway API) | 2-3 weeks | None |
|
||||
| Phase 2 (WireGuard) | 1-2 weeks | Can run parallel with Phase 1 |
|
||||
| Phase 3 (Cloudflare) | 1-2 weeks | After Phase 1 |
|
||||
| Phase 4 (Cleanup) | 1 week | After Phase 3 |
|
||||
@@ -0,0 +1,170 @@
|
||||
# Gitea Actions Migration Plan — Validate-Only Phase
|
||||
|
||||
## Overview
|
||||
|
||||
Migrate validate-only workflows from `.github/workflows/` (all `.disabled`) to `.gitea/workflows/` on the `alexpires.me/a13labs.infra` repo. Runs on `cicd-base` runner label. Terraform removed entirely — OpenTofu only.
|
||||
|
||||
## Status: IMPLEMENTED
|
||||
|
||||
All files created, secrets configured as placeholders. Ready for push and testing.
|
||||
|
||||
## Scope
|
||||
|
||||
### Workflows In Scope (6)
|
||||
|
||||
| Workflow | Trigger | Path Filter | What It Does |
|
||||
|---|---|---|---|
|
||||
| `ansible-validate-on-pr` | PR (opened/synchronize/reopened) → `main` | `ansible/**` | ansible-lint + syntax-check |
|
||||
| `aws-validate-on-pr` | PR → `main` | `terraform/iac/aws/**` | tofu init, validate, fmt -check, plan |
|
||||
| `cloudns-validate-on-pr` | PR → `main` | `terraform/iac/cloudns/**` | tofu init, validate, fmt -check, plan |
|
||||
| `contabo-validate-on-pr` | PR → `main` | `terraform/iac/contabo/**` | tofu init, validate, fmt -check, plan |
|
||||
| `scaleway-validate-on-pr` | PR → `main` | `terraform/iac/scaleway/**` | tofu init, validate, fmt -check, plan |
|
||||
| `k8s-apps-validate` | PR → `main` | `terraform/apps/**` | ansible+k8s env + tofu validate dev-01 & prod-01 |
|
||||
|
||||
### Workflows Deferred (8)
|
||||
|
||||
All apply workflows, `deploy-websites`, `actions-validate`, `ansible-validate-playbook-on-pr` — out of scope.
|
||||
|
||||
## Directory Layout
|
||||
|
||||
```
|
||||
.gitea/
|
||||
├── actions/
|
||||
│ ├── setup-sectool/
|
||||
│ │ └── action.yml
|
||||
│ ├── setup-env/
|
||||
│ │ └── action.yml
|
||||
│ ├── setup-ansible-env/
|
||||
│ │ └── action.yml
|
||||
│ ├── setup-k8s-env/
|
||||
│ │ └── action.yml
|
||||
│ ├── setup-opentofu-env/
|
||||
│ │ └── action.yml
|
||||
│ └── setup-scaleway-registry/
|
||||
│ └── action.yml
|
||||
└── workflows/
|
||||
├── ansible-validate-on-pr.yml
|
||||
├── aws-validate-on-pr.yml
|
||||
├── build-and-push-images.yml
|
||||
├── cloudns-validate-on-pr.yml
|
||||
├── contabo-validate-on-pr.yml
|
||||
├── k8s-apps-validate.yml
|
||||
├── scaleway-validate-on-pr.yml
|
||||
└── weekly-image-rebuild.yml
|
||||
```
|
||||
|
||||
## Action Migration Details
|
||||
|
||||
### `setup-sectool` (New)
|
||||
|
||||
Replaces `uses: a13labs/setup-sectool@v1` (external GitHub action). Local composite action that downloads the sectool binary from `a13labs/sectool` GitHub releases.
|
||||
|
||||
- Fetches latest release from `https://api.github.com/repos/a13labs/sectool/releases/latest`
|
||||
- Downloads Linux amd64 binary (tar.gz)
|
||||
- Installs to `/usr/local/bin/sectool`
|
||||
- Runs `sectool version` to verify
|
||||
|
||||
### `setup-env` (Converted)
|
||||
|
||||
Changes from `.github/actions/setup-env/action.yml`:
|
||||
- `outputs.changed_files` → uses `$GITEA_OUTPUT` instead of `$GITHUB_OUTPUT`
|
||||
- `git diff --name-only ${{ github.event.before }} ${{ github.sha }}` → `${{ gitea.event.before }} ${{ gitea.sha }}`
|
||||
- `uses: a13labs/setup-sectool@v1` → `uses: ./.gitea/actions/setup-sectool`
|
||||
|
||||
### `setup-ansible-env` (Converted)
|
||||
|
||||
Changes from `.github/actions/setup-ansible-env/action.yml`:
|
||||
- `outputs.ssh-auth-sock` and `outputs.hosts` → use `$GITEA_OUTPUT`
|
||||
- No `github.event` references — minimal conversion needed
|
||||
|
||||
### `setup-opentofu-env` (Unchanged)
|
||||
|
||||
No changes needed. `opentofu/setup-opentofu@v1` stays as-is (Gitea pulls from GitHub).
|
||||
|
||||
### `setup-k8s-env` (Unchanged)
|
||||
|
||||
Carried over as-is. No GitHub-specific variables in this action.
|
||||
|
||||
### `setup-terraform-env` (Not Migrated)
|
||||
|
||||
Dropped entirely. Terraform is no longer used in this project.
|
||||
|
||||
### `setup-scaleway-registry` (New)
|
||||
|
||||
Local composite action for image build workflows. Creates `~/.docker/config.json` with registry auth and configures buildah storage/network settings for rootless operation.
|
||||
|
||||
- Inputs: `registry-endpoint`, `registry-username`, `registry-password`
|
||||
- Used by: `build-and-push-images.yml`, `weekly-image-rebuild.yml`
|
||||
|
||||
## New Workflows (Image Build & Push)
|
||||
|
||||
### `build-and-push-images.yml` (New)
|
||||
|
||||
Builds and pushes runner profile Docker images to Scaleway registry.
|
||||
|
||||
| Trigger | Tags | Scanning |
|
||||
|---|---|---|
|
||||
| Push to main (dockerfiles changed) | `{sha}` + `latest` | Yes (Trivy, warn only) |
|
||||
| Push tag (`v*`) | `{tag}` + `{sha}` + `latest` | Yes (Trivy, warn only) |
|
||||
| Pull request | `pr-{number}` | No |
|
||||
|
||||
Jobs: `check-paths` → `auth` → `build-base` → `build-images` (matrix: 6 images)
|
||||
|
||||
### `weekly-image-rebuild.yml` (New)
|
||||
|
||||
Scheduled weekly (Monday 6AM). Rebuilds all images against updated upstream bases, tags only `:latest`, full Trivy scanning.
|
||||
|
||||
See `plans/image-build-push.md` for full details.
|
||||
|
||||
## Path Corrections
|
||||
|
||||
GitHub workflows referenced stale paths from a previous repo layout:
|
||||
|
||||
| Source (GitHub) | Actual | Notes |
|
||||
|---|---|---|
|
||||
| `terraform/aws-iac/` | `terraform/iac/aws/` | Restructured |
|
||||
| `terraform/cloudns-iac/` | `terraform/iac/cloudns/` | Restructured |
|
||||
| `terraform/contabo-iac/` | `terraform/iac/contabo/` | Restructured |
|
||||
| `terraform/scaleway-iac/` | `terraform/iac/scaleway/` | Restructured |
|
||||
| `terraform/k8sapps/` | `terraform/apps/dev-01/` + `terraform/apps/prod-01/` | Renamed + split |
|
||||
|
||||
## Gitea Syntax Conversions
|
||||
|
||||
| GitHub | Gitea |
|
||||
|---|---|
|
||||
| `$GITHUB_OUTPUT` | `$GITEA_OUTPUT` |
|
||||
| `${{ github.event.before }}` | `${{ gitea.event.before }}` |
|
||||
| `${{ github.sha }}` | `${{ gitea.sha }}` |
|
||||
| `runs-on: ubuntu-latest` | `runs-on: cicd-base` |
|
||||
| `uses: ./.github/actions/...` | `uses: ./.gitea/actions/...` |
|
||||
|
||||
## Repo Secrets
|
||||
|
||||
| Secret | Used By | Purpose | Status |
|
||||
|---|---|---|---|
|
||||
| `BW_PROJECT_ID` | All workflows | Bitwarden project ID | **Configured** |
|
||||
| `BW_ORGANIZATION_ID` | All workflows | Bitwarden org ID | **Configured** |
|
||||
| `BW_ACCESS_TOKEN` | All workflows | Bitwarden API token | **Configured** |
|
||||
| `ANSIBLE_USER` | Ansible + K8s workflows | SSH user for remote hosts | **Configured** |
|
||||
| `SCALEWAY_REGISTRY_ENDPOINT` | Image build workflows | Scaleway registry endpoint | **Needs setup** |
|
||||
| `SCALEWAY_CI_ACCESS_KEY` | Image build workflows | CI IAM API access key | **Needs setup** |
|
||||
| `SCALEWAY_CI_SECRET_KEY` | Image build workflows | CI IAM API secret key | **Needs setup** |
|
||||
|
||||
## Phase B: Scaleway IaC Changes
|
||||
|
||||
IaC changes in `terraform/iac/scaleway/` add CI IAM application and outputs. See `plans/image-build-push.md` for details.
|
||||
|
||||
- `iam.tf` — added `ci_app` IAM application, policy, and API key
|
||||
- `outputs.tf` — added `registry_endpoint`, `ci_app_api_access_key`, `ci_app_api_secret_key`
|
||||
- `sectool.env` — no changes needed (CI app resources are hardcoded)
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. **Push** all changes to `alexpires.me/a13labs.infra` on Gitea (note: `registry.tf` changed `is_public` from `false` to `true` on `a13labs` namespace)
|
||||
2. **SSH to ci-01** — rebuild `cicd-base` image locally, update Gitea runner container — DONE
|
||||
3. **Add Bitwarden secrets** — `SCALEWAY_CI_ACCESS_KEY` and `SCALEWAY_CI_SECRET_KEY` (populated after IaC apply)
|
||||
4. **Apply Scaleway IaC** — `sectool exec tofu apply` in `terraform/iac/scaleway/`
|
||||
5. **Extract IaC outputs** — `registry_endpoint`, `ci_app_api_access_key`, `ci_app_api_secret_key`
|
||||
6. **Configure Gitea secrets** — `SCALEWAY_REGISTRY_ENDPOINT`, `SCALEWAY_CI_ACCESS_KEY`, `SCALEWAY_CI_SECRET_KEY`
|
||||
7. **Test validate workflows** — PR touching `ansible/` or `terraform/iac/` paths
|
||||
8. **Test build workflow** — PR touching `ansible/files/dockerfiles/linux-runners/`
|
||||
@@ -0,0 +1,155 @@
|
||||
# Image Build & Push to Scaleway Registry
|
||||
|
||||
## Goal
|
||||
Build runner profile container images from `ansible/files/dockerfiles/linux-runners/`, scan for vulnerabilities, and push to the Scaleway container registry (`a13labs` namespace).
|
||||
|
||||
## Decisions
|
||||
|
||||
| Decision | Choice | Rationale |
|
||||
|---|---|---|
|
||||
| Registry namespace | `a13labs` (existing) | Clean project branding |
|
||||
| Tagging strategy | Git tag + SHA + latest | Full traceability and history |
|
||||
| Initial scope | Runner profiles only (7 → 4) | Consolidated image-build, infra-tools, security-scan into cicd-base |
|
||||
| Trivy scan severity | Warn only | Review manually, never block builds |
|
||||
| Update consumers | Not this iteration | Build+push pipeline first, consumers later |
|
||||
| Path filtering | In-workflow check | Gitea paths filter unreliable with tags |
|
||||
| PR scanning | Disabled | Keep PR builds fast |
|
||||
|
||||
## Image Inventory & Dependencies
|
||||
|
||||
```
|
||||
docker.io/gitea/runner:latest
|
||||
↓
|
||||
gitea-runner-base (build stage 1, pushed first)
|
||||
↓
|
||||
┌─────────────────────────────┐
|
||||
│ cicd-base, cpp-build, go-build │ (build stage 2, all parallel)
|
||||
└─────────────────────────────┘
|
||||
```
|
||||
|
||||
| Image | Dockerfile | Base |
|
||||
|---|---|---|
|
||||
| gitea-runner-base | `Dockerfile.gitea-runner-base` | `docker.io/gitea/runner:latest` |
|
||||
| cicd-base | `Dockerfile.cicd-base` | `local/gitea-runner-base:latest` |
|
||||
| cpp-build | `Dockerfile.cpp-build` | `local/gitea-runner-base:latest` |
|
||||
| go-build | `Dockerfile.go-build` | `local/gitea-runner-base:latest` |
|
||||
|
||||
> **Note:** `image-build`, `infra-tools`, and `security-scan` runners were decommissioned and their tooling consolidated into `cicd-base`. All workflows now run on `cicd-base` or the language-specific runners (`cpp-build`, `go-build`).
|
||||
|
||||
## Registry Naming
|
||||
|
||||
Images pushed as `fr-par-2.registry.scaleway.com/a13labs/{image-name}:{tag}`
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase A: Prepare CI Runner (DONE)
|
||||
|
||||
**Step 1 — Add build tools to `Dockerfile.cicd-base`**
|
||||
|
||||
DONE. Added the following packages to the existing cicd-base image:
|
||||
- Build packages: `buildah`, `fuse-overlayfs`, `podman`, `shadow-uidmap`, `skopeo`, `slirp4netns`, `iptables` (from `Dockerfile.image-build`)
|
||||
- Storage config: `storage.conf` (vfs driver) and `containers.conf` (from `Dockerfile.image-build`)
|
||||
- Trivy binary download (from `Dockerfile.security-scan`, v0.72.0)
|
||||
|
||||
**Step 2 — Register updated cicd-base on Gitea runner** (DONE)
|
||||
|
||||
cicd-base image rebuilt on `ci-01` and Gitea runner container registration updated.
|
||||
|
||||
### Phase B: Scaleway IaC Changes (DONE - code changes, manual apply needed)
|
||||
|
||||
**Step 3 — Add CI IAM app** (`terraform/iac/scaleway/iam.tf`) - DONE
|
||||
- New `scaleway_iam_application` "ci_app" with `RegistryFullAccess` permission set
|
||||
- New `scaleway_iam_policy` "registry_full_access" granting ci_app registry access
|
||||
- New `scaleway_iam_api_key` "ci_app_api" for ci_app credentials
|
||||
|
||||
**Step 4 — Add registry outputs** (`terraform/iac/scaleway/outputs.tf`) - DONE
|
||||
- `registry_endpoint` — from `scaleway_registry_namespace.a13labs.endpoint`
|
||||
- `ci_app_api_access_key`, `ci_app_api_secret_key` — sensitive
|
||||
|
||||
**Step 5 — No `sectool.env` changes needed** - DONE
|
||||
The CI app resources are hardcoded, no TF_VAR mappings required.
|
||||
|
||||
**Step 6 — Apply and extract secrets** (MANUAL)
|
||||
- Add `SCALEWAY_CI_ACCESS_KEY` and `SCALEWAY_CI_SECRET_KEY` to Bitwarden
|
||||
- Run `sectool exec tofu apply` in `terraform/iac/scaleway/`
|
||||
- Add `SCALEWAY_REGISTRY_ENDPOINT`, `SCALEWAY_CI_ACCESS_KEY`, `SCALEWAY_CI_SECRET_KEY` as Gitea repo secrets
|
||||
|
||||
### Phase C: Gitea Actions & Workflow (DONE)
|
||||
|
||||
**Step 7 — Create `.gitea/actions/setup-scaleway-registry/action.yml`** - DONE
|
||||
Composite action that:
|
||||
- Creates `~/.docker/config.json` with Scaleway registry auth
|
||||
- Configures buildah storage and network settings for rootless operation
|
||||
- Inputs: `registry-endpoint`, `registry-username`, `registry-password`
|
||||
|
||||
**Step 8 — Create `.gitea/workflows/build-and-push-images.yml`** - DONE
|
||||
|
||||
Jobs:
|
||||
- `check-paths`: Determines if dockerfiles changed (handles push, PR, tag events)
|
||||
- `auth`: Sets up registry authentication (gates on path check)
|
||||
- `build-base`: Builds and pushes `gitea-runner-base`
|
||||
- `build-images`: Matrix build of 6 dependent images
|
||||
|
||||
Tagging strategy:
|
||||
- Push to main: `{sha}` + `latest`
|
||||
- Push tag (v*): `{tag}` + `{sha}` + `latest`
|
||||
- PR: `pr-{number}` (no scanning)
|
||||
|
||||
**Step 9 — Create `.gitea/workflows/weekly-image-rebuild.yml`** - DONE
|
||||
Scheduled weekly (`cron: "0 6 * * 1"` — Monday 6AM):
|
||||
- Same build jobs but only tag `:latest`
|
||||
- Uses `--pull-always` to force rebuild against updated upstream base images
|
||||
- Full Trivy scan (warn only, artifacts)
|
||||
|
||||
## Build Order & Dependency Resolution
|
||||
|
||||
Each dependent Dockerfile says `FROM local/gitea-runner-base:latest`. The workflow replaces `local/gitea-runner-base` with the full registry URL in a temp copy before building, so buildah pulls the freshly pushed base image.
|
||||
|
||||
## Manual Setup Checklist
|
||||
|
||||
Before the workflows can run successfully, complete these steps:
|
||||
|
||||
- [ ] Push code changes to repo (note: `registry.tf` changed `is_public` from `false` to `true` on `a13labs` namespace)
|
||||
- [x] SSH to `ci-01`, rebuild `cicd-base` image locally, update Gitea runner container — DONE
|
||||
- [ ] Add `SCALEWAY_CI_ACCESS_KEY` and `SCALEWAY_CI_SECRET_KEY` to Bitwarden (leave blank initially, they'll be populated after IaC apply)
|
||||
- [ ] Apply Scaleway IaC: `sectool exec tofu apply` in `terraform/iac/scaleway/`
|
||||
- [ ] From IaC outputs, extract `registry_endpoint`, `ci_app_api_access_key`, `ci_app_api_secret_key`
|
||||
- [ ] Add as Gitea repo secrets: `SCALEWAY_REGISTRY_ENDPOINT`, `SCALEWAY_CI_ACCESS_KEY`, `SCALEWAY_CI_SECRET_KEY`
|
||||
- [ ] Push a test commit touching `ansible/files/dockerfiles/linux-runners/` to trigger workflow
|
||||
- [ ] Verify workflow runs and images appear in Scaleway registry
|
||||
|
||||
## Gitea Secrets Required
|
||||
|
||||
| Secret | Source | Used by |
|
||||
|---|---|---|
|
||||
| `SCALEWAY_REGISTRY_ENDPOINT` | IaC output `registry_endpoint` | build-and-push, weekly-rebuild |
|
||||
| `SCALEWAY_CI_ACCESS_KEY` | IaC output `ci_app_api_access_key` | build-and-push, weekly-rebuild |
|
||||
| `SCALEWAY_CI_SECRET_KEY` | IaC output `ci_app_api_secret_key` | build-and-push, weekly-rebuild |
|
||||
|
||||
## File Summary
|
||||
|
||||
| Action | File | Status |
|
||||
|---|---|---|
|
||||
| Modify | `ansible/files/dockerfiles/linux-runners/Dockerfile.cicd-base` | DONE |
|
||||
| Modify | `terraform/iac/scaleway/iam.tf` | DONE |
|
||||
| Modify | `terraform/iac/scaleway/outputs.tf` | DONE |
|
||||
| Modify | `terraform/iac/scaleway/sectool.env` | DONE |
|
||||
| Create | `.gitea/actions/setup-scaleway-registry/action.yml` | DONE |
|
||||
| Create | `.gitea/workflows/build-and-push-images.yml` | DONE |
|
||||
| Create | `.gitea/workflows/weekly-image-rebuild.yml` | DONE |
|
||||
|
||||
## Post-Iteration (NOT now)
|
||||
|
||||
- Update runner Dockerfiles to use registry base URLs
|
||||
- Update Ansible playbooks to `pull` from registry instead of `build`
|
||||
- Add PR close job to clean up `pr-{number}` tags from registry
|
||||
- Expand to AI service Dockerfiles (comfyui, ollama, llamacpp variants)
|
||||
|
||||
## Consolidation
|
||||
|
||||
`image-build`, `infra-tools`, and `security-scan` runner profiles have been decommissioned. Their tooling was already absorbed into `cicd-base`:
|
||||
- **image-build** tools (buildah, podman, storage config) → merged into `Dockerfile.cicd-base`
|
||||
- **security-scan** tools (trivy) → merged into `Dockerfile.cicd-base`
|
||||
- **infra-tools** tools (ansible, pip packages) → installed at workflow runtime via `requirements/ansible.txt`
|
||||
|
||||
Unused Dockerfiles removed: `Dockerfile.image-build`, `Dockerfile.infra-tools`, `Dockerfile.security-scan`
|
||||
@@ -0,0 +1,271 @@
|
||||
# Plan: Refactor llama_server role — template-based container command
|
||||
|
||||
## Current State
|
||||
|
||||
The `llama_server` role deploys `llama.cpp` server via Podman (EL) or NSSB (Windows) in router mode.
|
||||
|
||||
**Existing structure:**
|
||||
|
||||
| Component | Location |
|
||||
|---|---|
|
||||
| `defaults/main.yml` | User/group, image, tag, port, bind, models list, `argv_extra`, exporter config, Windows config |
|
||||
| `tasks/el/main.yml` | User/group creation → podman image pull → **container created inline** with hardcoded flags → systemd wrapper → HF model sync → exporter |
|
||||
| `tasks/windows/main.yml` | Binary download → NSSM install with same hardcoded flags → HF sync → firewall → exporter |
|
||||
| `templates/podman.service.j2` | Thin systemd wrapper: `podman start/stop llamacpp` |
|
||||
| `templates/llama_exporter.service.j2` | Exporter systemd service |
|
||||
| `vars/main.yml` | Internal non-overridable vars |
|
||||
| `files/scripts/` | `llama_hf_sync.py` (Linux + Windows), `llama_exporter.py`, tests |
|
||||
|
||||
**Current container command** (`tasks/el/main.yml:142-152`):
|
||||
|
||||
```yaml
|
||||
command: >-
|
||||
{{
|
||||
[
|
||||
'--metrics',
|
||||
'--models-preset',
|
||||
'/models/.router/models.ini',
|
||||
'--models-max',
|
||||
(llama_server_models_max | string)
|
||||
]
|
||||
+ (llama_server_argv_extra | default([]))
|
||||
}}
|
||||
```
|
||||
|
||||
**Current Windows command** (`tasks/windows/main.yml:220-238`):
|
||||
|
||||
```yaml
|
||||
argv: >-
|
||||
{{
|
||||
[
|
||||
'nssm.exe',
|
||||
'install',
|
||||
'llama-server',
|
||||
(llama_server_windows_install_dir ~ '\\' ~ llama_server_windows_bin_name),
|
||||
'--port',
|
||||
(llama_server_port | default(11434) | string),
|
||||
'--host',
|
||||
(llama_server_host | default('127.0.0.1') | string),
|
||||
'--metrics',
|
||||
'--models-preset',
|
||||
(__llama_preset_file | string),
|
||||
'--models-max',
|
||||
(llama_server_models_max | string)
|
||||
]
|
||||
+ (llama_server_argv_extra | default([]))
|
||||
}}
|
||||
```
|
||||
|
||||
**Problem:** Both commands are hardcoded inline. The only extensibility is `llama_server_argv_extra` (append-only list), which has no validation, no conditional logic, and no self-documentation.
|
||||
|
||||
## Goals
|
||||
|
||||
- Replace hardcoded command construction with Jinja2 templates for both EL and Windows paths.
|
||||
- Add role variables for all major llama.cpp server flags with sensible defaults and conditional rendering.
|
||||
- Maintain backward compatibility: `llama_server_argv_extra` still works as a suffix.
|
||||
- Enable future configuration of MCP servers, SSL/TLS, API keys, chat templates, and performance tuning without manual playbook edits.
|
||||
|
||||
## Changes Required
|
||||
|
||||
### 1. Add new role variables to `defaults/main.yml`
|
||||
|
||||
Add a new section after the existing defaults:
|
||||
|
||||
```yaml
|
||||
# --- Server configuration ---
|
||||
llama_server_bind: "0.0.0.0"
|
||||
llama_server_host: "127.0.0.1"
|
||||
llama_server_port: 11434
|
||||
|
||||
# SSL/TLS
|
||||
llama_server_ssl_enabled: false
|
||||
llama_server_ssl_cert: ""
|
||||
llama_server_ssl_key: ""
|
||||
|
||||
# Authentication
|
||||
llama_server_api_key: ""
|
||||
|
||||
# Chat
|
||||
llama_server_chat_template: ""
|
||||
|
||||
# Performance
|
||||
llama_server_ctx_size: 4096
|
||||
llama_server_n_gpu_layers: 35
|
||||
llama_server_batch_size: 512
|
||||
llama_server_ubatch_size: 512
|
||||
llama_server_threads: 0
|
||||
llama_server_threads_batch: 0
|
||||
|
||||
# Features
|
||||
llama_server_embeddings: false
|
||||
llama_server_cors: false
|
||||
llama_server_log_timestamps: false
|
||||
llama_server_log_level: "warn"
|
||||
llama_server_log_print: false
|
||||
llama_server_slot_save_context: false
|
||||
|
||||
# Web UI
|
||||
llama_server_webui: ""
|
||||
llama_server_webui_author: ""
|
||||
|
||||
# MCP
|
||||
llama_server_mcp_servers: []
|
||||
|
||||
# KV cache quantization
|
||||
llama_server_cache_type_q4: false
|
||||
|
||||
# Speculative decoding
|
||||
llama_server_speculative_model: ""
|
||||
llama_server_draft: 0
|
||||
|
||||
# RoPE scaling
|
||||
llama_server_rope_freq_scale: 0.0
|
||||
|
||||
# GPU offload
|
||||
llama_server_tensor_split: 0
|
||||
llama_server_split_mode: "layer"
|
||||
llama_server_main_gpu: 0
|
||||
```
|
||||
|
||||
### 2. Create EL container command template
|
||||
|
||||
**File:** `templates/llama_server_command_el.j2`
|
||||
|
||||
```jinja2
|
||||
{{
|
||||
[
|
||||
'--metrics'
|
||||
]
|
||||
+ (llama_server_models_preset_global_path | default('/models/.router/models.ini') | ternary(['--models-preset', llama_server_models_preset_global_path], []))
|
||||
+ (llama_server_models_max | default(1) | string | ternary(['--models-max', llama_server_models_max | string], []))
|
||||
+ (llama_server_host | default('127.0.0.1') | ternary(['--host', llama_server_host], []))
|
||||
+ (llama_server_ssl_enabled | default(false) | bool | ternary(
|
||||
['--ssl', '--ssl-cert', llama_server_ssl_cert, '--ssl-key', llama_server_ssl_key] if llama_server_ssl_cert and llama_server_ssl_key else [], []))
|
||||
+ (llama_server_api_key | default('') | ternary(['--api-key', llama_server_api_key], []))
|
||||
+ (llama_server_chat_template | default('') | ternary(['--chat-template', llama_server_chat_template], []))
|
||||
+ (llama_server_ctx_size | default(4096) | ternary(['--ctx-size', llama_server_ctx_size | string], []))
|
||||
+ (llama_server_n_gpu_layers | default(35) | ternary(['--n-gpu-layers', llama_server_n_gpu_layers | string], []))
|
||||
+ (llama_server_batch_size | default(512) | ternary(['--batch-size', llama_server_batch_size | string], []))
|
||||
+ (llama_server_ubatch_size | default(512) | ternary(['--ubatch-size', llama_server_ubatch_size | string], []))
|
||||
+ (llama_server_threads | default(0) | ternary(['--threads', llama_server_threads | string], []))
|
||||
+ (llama_server_threads_batch | default(0) | ternary(['--threads-batch', llama_server_threads_batch | string], []))
|
||||
+ (llama_server_embeddings | default(false) | bool | ternary(['--embedding'], []))
|
||||
+ (llama_server_cors | default(false) | bool | ternary(['--cors'], []))
|
||||
+ (llama_server_log_timestamps | default(false) | bool | ternary(['--log-timestamps'], []))
|
||||
+ (llama_server_log_level | default('warn') | ternary(['--log-level', llama_server_log_level], []))
|
||||
+ (llama_server_log_print | default(false) | bool | ternary(['--log-print'], []))
|
||||
+ (llama_server_slot_save_context | default(false) | bool | ternary(['--slot-save-context'], []))
|
||||
+ (llama_server_webui | default('') | ternary(['--webui-name', llama_server_webui], []))
|
||||
+ (llama_server_webui_author | default('') | ternary(['--webui-author', llama_server_webui_author], []))
|
||||
+ (llama_server_mcp_servers | default([]) | length | ternary(['--mcp-servers', (llama_server_mcp_servers | to_json)], []))
|
||||
+ (llama_server_cache_type_q4 | default(false) | bool | ternary(['--cache-type-q4'], []))
|
||||
+ (llama_server_speculative_model | default('') | ternary(['--speculative-model', llama_server_speculative_model], []))
|
||||
+ (llama_server_draft | default(0) | ternary(['--draft', llama_server_draft | string], []))
|
||||
+ (llama_server_rope_freq_scale | default(0.0) | ternary(['--rope-freq-scale', llama_server_rope_freq_scale | string], []))
|
||||
+ (llama_server_tensor_split | default(0) | ternary(['--tensor-split', llama_server_tensor_split | string], []))
|
||||
+ (llama_server_split_mode | default('layer') | ternary(['--split-mode', llama_server_split_mode], []))
|
||||
+ (llama_server_main_gpu | default(0) | ternary(['--main-gpu', llama_server_main_gpu | string], []))
|
||||
+ (llama_server_argv_extra | default([]))
|
||||
}}
|
||||
```
|
||||
|
||||
**Note:** The above uses a Jinja2 expression that produces a JSON array string. The `ternary()` calls handle conditional inclusion. The `llama_server_mcp_servers` serializes to JSON for the `--mcp-servers` flag.
|
||||
|
||||
### 3. Create Windows service command template
|
||||
|
||||
**File:** `templates/llama_server_command_windows.j2`
|
||||
|
||||
Same logic as EL but output as a flat list suitable for NSSM `nssm.exe install` argv (each flag and value as separate list elements).
|
||||
|
||||
### 4. Refactor `tasks/el/main.yml`
|
||||
|
||||
Replace the hardcoded command block (lines 142-152) in the `podman_container` task:
|
||||
|
||||
```yaml
|
||||
- name: Set fact llama_server_el_command
|
||||
ansible.builtin.set_fact:
|
||||
llama_server_el_command: "{{ lookup('template', 'templates/llama_server_command_el.j2') }}"
|
||||
|
||||
- name: Create pods
|
||||
containers.podman.podman_container:
|
||||
name: "llamacpp"
|
||||
image: "{{ llama_server_image }}:{{ llama_server_tag }}"
|
||||
state: started
|
||||
device: "{{ llama_server_devices | default(omit) }}"
|
||||
env: "{{ llama_server_env | default(omit) }}"
|
||||
ports: "{{ llama_server_bind }}:{{ llama_server_port }}:8080/tcp"
|
||||
volumes:
|
||||
- "{{ llama_server_home }}/models:/models:Z"
|
||||
- "{{ llama_server_home }}/.cache:/app/.cache:Z"
|
||||
command: "{{ llama_server_el_command }}"
|
||||
```
|
||||
|
||||
### 5. Refactor `tasks/windows/main.yml`
|
||||
|
||||
Replace the hardcoded `nssm.exe install` argv block (lines 220-238):
|
||||
|
||||
```yaml
|
||||
- name: Set fact llama_server_windows_argv
|
||||
ansible.builtin.set_fact:
|
||||
llama_server_windows_argv: "{{ lookup('template', 'templates/llama_server_command_windows.j2') }}"
|
||||
|
||||
- name: Install llama-server service via NSSM
|
||||
ansible.windows.win_command:
|
||||
argv: "{{ llama_server_windows_argv }}"
|
||||
args:
|
||||
chdir: "{{ llama_server_windows_install_dir }}"
|
||||
register: __llama_nssm_install
|
||||
changed_when: __llama_nssm_install.rc == 0
|
||||
```
|
||||
|
||||
### 6. Create MCP servers preset file template (optional, for future)
|
||||
|
||||
**File:** `templates/mcp_servers.json.j2`
|
||||
|
||||
```json
|
||||
{{ llama_server_mcp_servers | to_nice_json }}
|
||||
```
|
||||
|
||||
Used when MCP servers need to be written to a file and passed via `--mcp-servers` flag pointing to a path.
|
||||
|
||||
### 7. Add tasks to write SSL/TLS and API key files (conditional)
|
||||
|
||||
When `llama_server_ssl_cert` or `llama_server_api_key` are set, ensure the referenced files exist on the remote host. This may require:
|
||||
- A task to create the API key file if provided as a variable (not file path)
|
||||
- A task to validate SSL cert/key paths exist
|
||||
|
||||
## Files to modify
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `ansible/roles/llama_server/defaults/main.yml` | Add new role variables section |
|
||||
| `ansible/roles/llama_server/templates/llama_server_command_el.j2` | **New** — EL container command template |
|
||||
| `ansible/roles/llama_server/templates/llama_server_command_windows.j2` | **New** — Windows NSSM command template |
|
||||
| `ansible/roles/llama_server/templates/mcp_servers.json.j2` | **New** — MCP servers JSON preset (optional) |
|
||||
| `ansible/roles/llama_server/tasks/el/main.yml` | Replace hardcoded command with `lookup('template')` + `set_fact` |
|
||||
| `ansible/roles/llama_server/tasks/windows/main.yml` | Replace hardcoded nssm argv with `lookup('template')` + `set_fact` |
|
||||
| `ansible/roles/llama_server/tasks/el/llama_exporter.yml` | No change |
|
||||
| `ansible/roles/llama_server/files/scripts/` | No change |
|
||||
|
||||
## Rollback plan
|
||||
|
||||
- The `llama_server_argv_extra` variable continues to work as a suffix in both templates.
|
||||
- If template rendering fails, revert `tasks/el/main.yml` and `tasks/windows/main.yml` to the hardcoded command blocks.
|
||||
- Delete the new template files.
|
||||
- No data loss — templates only affect command construction, not state.
|
||||
|
||||
## Testing
|
||||
|
||||
1. **Baseline test:** Run role with existing defaults — should produce identical command to current hardcoded version.
|
||||
2. **Variable test:** Set individual variables (e.g., `llama_server_ctx_size: 8192`, `llama_server_embeddings: true`) and verify they appear in the rendered command.
|
||||
3. **Conditional test:** Set `llama_server_ssl_enabled: true` with cert/key paths — verify `--ssl`, `--ssl-cert`, `--ssl-key` flags are included.
|
||||
4. **MCP test:** Set `llama_server_mcp_servers` to a list of dicts — verify JSON serialization in `--mcp-servers` flag.
|
||||
5. **Backward compat test:** Use `llama_server_argv_extra` alongside new variables — verify both are present.
|
||||
6. **Windows test:** Same scenarios on Windows target with NSSM.
|
||||
|
||||
## Open questions
|
||||
|
||||
1. **Should `--models-preset` and `--models-max` remain in the template or stay hardcoded?** They're core to router mode and unlikely to change. Keeping them in the template makes the command self-contained but adds complexity.
|
||||
2. **For `mcp_servers`** — does llama.cpp accept the flag as `--mcp-servers '{"name":"..."}'` (JSON string) or `--mcp-servers /path/to/config.json` (file path)? This affects whether we need a separate file template.
|
||||
3. **Should defaults be opinionated** (e.g., `llama_server_ctx_size: 4096`) or minimal (only required flags)? Opinionated defaults reduce host var boilerplate but may not suit all use cases.
|
||||
4. **Should we add assertions** for mutually exclusive flags (e.g., `--split-mode` values)?
|
||||
@@ -0,0 +1,234 @@
|
||||
# Plan: Replace OpenWebUI with llama.cpp behind nginx proxy
|
||||
|
||||
## Current State
|
||||
|
||||
- **llama.cpp** runs on `gpu-01.lab.alexpires.me` via Podman on port `11434` (bound to `127.0.0.1` with API key)
|
||||
- **nginx proxy** role runs on `dev-02.lab.alexpires.me` and `gpu-01.lab.alexpires.me`
|
||||
- **OpenWebUI** has been decommissioned (removed from K8s, DNS, inventory)
|
||||
- **DNS** for `lab.alexpires.me` is managed via Terraform (`terraform/apps/dev-01/config.tf` zones)
|
||||
- **oauth2-proxy** template uses OIDC provider (`code.alexpires.me`), `set_xauthrequest = true`
|
||||
|
||||
## Goals
|
||||
|
||||
- Deploy llama.cpp behind nginx proxy at `llm.lab.alexpires.me` ✅
|
||||
- API paths (`/api/*`) accessible without OAuth2 (token auth enforced via `--api-key`) ✅
|
||||
- All other paths (chat UI, etc.) protected with OAuth2 via oauth2-proxy ✅
|
||||
- Issue SSL certificate for `llm.lab.alexpires.me` via CloudNS DNS ✅
|
||||
- OpenWebUI decommissioned ✅
|
||||
|
||||
## Important Constraint
|
||||
|
||||
> **Resolved.** Phase 2 completed: llama.cpp now bound to `127.0.0.1` with API key. OpenWebUI decommissioned.
|
||||
|
||||
## Changes Required
|
||||
|
||||
> All changes have been implemented. See execution order below for completed steps.
|
||||
|
||||
### 1. Add DNS record for llm.lab.alexpires.me ✅
|
||||
|
||||
**File:** `terraform/apps/dev-01/config.tf`
|
||||
|
||||
Add A record to the `lab.alexpires.me` zone:
|
||||
```hcl
|
||||
{
|
||||
name = "llm"
|
||||
type = "A"
|
||||
content = "10.19.4.106"
|
||||
}
|
||||
```
|
||||
|
||||
Run `tofu plan/apply` on `terraform/apps/dev-01/` before Phase 1 so DNS resolves.
|
||||
|
||||
### 2. Add gpu-01 to `[nginx]` inventory group ✅
|
||||
|
||||
### 3. Update nginx-server.conf.j2 — per-site `no_auth` conditional ✅
|
||||
|
||||
### 4. Configure gpu-01 host vars for nginx proxy ✅
|
||||
|
||||
### 5. Add firewall rules for HTTPS ✅
|
||||
|
||||
### 6. Update llama_server role — support configurable bind address ✅
|
||||
|
||||
### 7. Two-phase transition ✅
|
||||
|
||||
**Phase 1 — Deploy nginx proxy (llama.cpp still on 0.0.0.0, no API key)**
|
||||
|
||||
- Keep `llama_server_bind: "0.0.0.0"` (current behavior, default)
|
||||
- Do NOT set API key
|
||||
- nginx proxies to `127.0.0.1:11434` (container is bound to `0.0.0.0:11434`, reachable from localhost)
|
||||
- OpenWebUI continues to work (still connects to `gpu-01:11434` directly)
|
||||
- Test: `llm.lab.alexpires.me/` requires OAuth2 login, `llm.lab.alexpires.me/api/*` is open
|
||||
|
||||
**Phase 2 — Switch to localhost binding + enable API key (after validation)**
|
||||
|
||||
- Change `llama_server_bind: "127.0.0.1"` in host vars
|
||||
- Add `--api-key` to `llama_server_argv_extra` in host vars:
|
||||
```yaml
|
||||
llama_server_argv_extra:
|
||||
- "--api-key"
|
||||
- "{{ lookup('env', 'LLAMA_SERVER_API_KEY') }}"
|
||||
# ... existing args ...
|
||||
```
|
||||
- OpenWebUI will break (can no longer reach llama.cpp on `gpu-01:11434`) — decommission OpenWebUI
|
||||
- Verify all consumers migrate to `llm.lab.alexpires.me/api/*` with `Authorization: Bearer <key>`
|
||||
- Remove old firewall rules for port 11434 (no longer externally reachable)
|
||||
|
||||
## Execution Order
|
||||
|
||||
### Phase 1 — Deploy nginx proxy (llama.cpp unchanged, OpenWebUI still works)
|
||||
|
||||
> **Status: COMPLETE** — All steps executed and verified.
|
||||
|
||||
1. **Update Terraform DNS** — add `llm` A record in `terraform/apps/dev-01/config.tf`, run `tofu apply` ✅
|
||||
2. **Update inventory** — add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` group ✅
|
||||
3. **Update nginx-server.conf.j2** — add `no_auth` per-site conditional (template stays generic) ✅
|
||||
4. **Update llama_server role** — add `llama_server_bind` variable (default `0.0.0.0`) ✅
|
||||
5. **Update host vars** — add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules ✅
|
||||
6. **Run nginx proxy playbook** — installs nginx + oauth2-proxy + certbot + deploys config ✅
|
||||
7. **Fix SELinux** — add `nginx_connect_any` boolean + port 4180 to `http_port_t` (manually applied, added to role for persistence) ✅
|
||||
8. **Verify chat UI** — `llm.lab.alexpires.me/` returns oauth2-proxy login page (200) ✅
|
||||
9. **Verify API** — `llm.lab.alexpires.me/api/v1/models` returns 200 with models list ✅
|
||||
10. **Verify OpenWebUI still works** — llama.cpp still bound to `0.0.0.0:11434` ✅
|
||||
|
||||
1. **Update Terraform DNS** — add `llm` A record in `terraform/apps/dev-01/config.tf`, run `tofu apply`
|
||||
2. **Update inventory** — add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` group
|
||||
3. **Update nginx-server.conf.j2** — add `no_auth` per-site conditional (template stays generic)
|
||||
4. **Update llama_server role** — add `llama_server_bind` variable (default `0.0.0.0`)
|
||||
5. **Update host vars** — add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules
|
||||
6. **Run nginx proxy playbook** — installs nginx + oauth2-proxy + certbot + deploys config (all in one run)
|
||||
7. **Verify chat UI** — test `llm.lab.alexpires.me/` with OAuth2 login
|
||||
8. **Verify API** — test `llm.lab.alexpires.me/api/*` (no auth yet, same as current direct access)
|
||||
9. **Verify OpenWebUI still works** — confirm dev-01 still connects to gpu-01:11434
|
||||
|
||||
### Phase 2 — Switch to localhost + API key (after validation, decommission OpenWebUI)
|
||||
|
||||
> **Status: COMPLETE** — All steps executed and verified.
|
||||
|
||||
10. **Update host vars** — change `llama_server_bind: "127.0.0.1"`, add `--api-key` to `llama_server_argv_extra` ✅
|
||||
11. **Update firewall** — remove port 11434 rules from `fw_allowed_ports` ✅
|
||||
12. **Run llama_server playbook** — restart container with new binding + API key ✅
|
||||
13. **Run hardening playbook** — `playbook_hardening_cis.yml --limit gpu-01.lab.alexpires.me -t ufw` to apply firewall changes ✅
|
||||
14. **Verify API** — `llm.lab.alexpires.me/api/v1/chat/completions` returns 401 without key, 200 with key ✅
|
||||
15. **Decommission OpenWebUI** — removed from `terraform/apps/dev-01/apps.tf` ✅
|
||||
16. **Remove OpenWebUI DNS records** — removed `open-webui` CNAME from `terraform/apps/dev-01/config.tf` zones ✅
|
||||
17. **Remove OpenWebUI locals** — removed `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` from config.tf ✅
|
||||
18. **Remove `webui_secret_key` variable** — removed from `terraform/apps/dev-01/secrets.tf` ✅
|
||||
19. **Remove `[openwebui]` group** — cleaned up `inventory/hosts` ✅
|
||||
20. **Run `tofu apply`** on `terraform/apps/dev-01/` — destroyed 8 OpenWebUI K8s resources (namespace, deployment, service, ingress, secret, service accounts) ✅
|
||||
|
||||
### Phase 2 — Switch to localhost + API key (after validation, decommission OpenWebUI)
|
||||
|
||||
10. **Update host vars** — change `llama_server_bind: "127.0.0.1"`, add `--api-key` to `llama_server_argv_extra`
|
||||
11. **Update firewall** — remove port 11434 rules from `fw_allowed_ports`
|
||||
12. **Run llama_server playbook** — restart container with new binding + API key
|
||||
13. **Run hardening playbook** — `playbook_hardening_cis.yml --limit gpu-01.lab.alexpires.me -t ufw` to apply firewall changes
|
||||
14. **Verify API** — test `llm.lab.alexpires.me/api/*` requires API key
|
||||
15. **Decommission OpenWebUI** — remove Terraform module from `terraform/apps/dev-01/apps.tf` and `config.tf`
|
||||
16. **Remove OpenWebUI DNS records** — clean up `open-webui` CNAME in `terraform/apps/dev-01/config.tf` zones
|
||||
17. **Remove `[openwebui]` group** — clean up `inventory/hosts`
|
||||
18. **Run `tofu apply`** on `terraform/apps/dev-01/` — destroys OpenWebUI K8s resources
|
||||
|
||||
## Playbooks to run
|
||||
|
||||
### Phase 1
|
||||
|
||||
> **COMPLETE** — All steps executed and verified.
|
||||
|
||||
```sh
|
||||
cd ansible
|
||||
sectool exec ansible-playbook -u $ANSIBLE_USER playbook_nginx_proxy.yml # nginx + oauth2 + certbot (all-in-one)
|
||||
```
|
||||
|
||||
### Phase 2
|
||||
|
||||
> **COMPLETE** — All steps executed and verified.
|
||||
|
||||
```sh
|
||||
cd ansible
|
||||
sectool exec ansible-playbook -u $ANSIBLE_USER playbook_llama_server.yml # container binding + API key
|
||||
sectool exec ansible-playbook -u $ANSIBLE_USER playbook_hardening_cis.yml -t ufw # remove port 11434 rules
|
||||
```
|
||||
|
||||
Then in terraform:
|
||||
```sh
|
||||
cd terraform/apps/dev-01
|
||||
sectool exec tofu plan # review OpenWebUI destruction
|
||||
sectool exec tofu apply --auto-approve # apply
|
||||
```
|
||||
|
||||
## Files to modify
|
||||
|
||||
| File | Phase | Change |
|
||||
|------|-------|--------|
|
||||
| `terraform/apps/dev-01/config.tf` | 1 | Add `llm` A record to zones |
|
||||
| `inventory/hosts` | 1 | Add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` |
|
||||
| `ansible/roles/nginx_proxy/templates/nginx-server.conf.j2` | 1 | Add `no_auth` per-site conditional (3 blocks) |
|
||||
| `ansible/roles/llama_server/defaults/main.yml` | 1 | Add `llama_server_bind` variable (default `0.0.0.0`) |
|
||||
| `ansible/roles/llama_server/tasks/el/main.yml` | 1 | Use `llama_server_bind` in podman port mapping |
|
||||
| `ansible/roles/nginx_proxy/tasks/nginx.yml` | 1 | Add SELinux tasks (`nginx_connect_any` boolean + oauth2-proxy port to `http_port_t`) |
|
||||
| `ansible/host_vars/gpu-01.lab.alexpires.me.yml` | 1 | Add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules |
|
||||
| `ansible/host_vars/gpu-01.lab.alexpires.me.yml` | 2 | Change `llama_server_bind` to `127.0.0.1`, add `--api-key` to `llama_server_argv_extra`, remove port 11434 fw rules ✅ |
|
||||
| `terraform/apps/dev-01/apps.tf` | 2 | Remove `module "open-webui"` block ✅ |
|
||||
| `terraform/apps/dev-01/config.tf` | 2 | Remove `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` locals; remove `open-webui` DNS record from zones ✅ |
|
||||
| `terraform/apps/dev-01/secrets.tf` | 2 | Remove `webui_secret_key` variable ✅ |
|
||||
| `inventory/hosts` | 2 | Remove `[openwebui]` group ✅ |
|
||||
|
||||
## Rollback plan
|
||||
|
||||
### Phase 1 rollback (nginx proxy broken)
|
||||
|
||||
- Remove gpu-01 from `[nginx]` group in `inventory/hosts`
|
||||
- Re-run hardening playbook on gpu-01 to clean up nginx/oauth2-proxy packages
|
||||
- Revert `nginx-server.conf.j2` template changes
|
||||
- llama.cpp continues running on `0.0.0.0:11434` — no impact
|
||||
- OpenWebUI continues to work
|
||||
|
||||
## Issues Encountered & Resolved
|
||||
|
||||
### 1. nginx `copy` module missing `remote_src: true`
|
||||
**File:** `ansible/roles/nginx_proxy/tasks/oauth2_proxy.yml:16`
|
||||
The `copy` module was trying to copy from the controller instead of the remote host. Added `remote_src: true`.
|
||||
|
||||
### 2. SELinux blocking nginx → oauth2-proxy / llama.cpp connections
|
||||
**Symptom:** `connect() to 127.0.0.1:4180 failed (13: Permission denied)` in nginx error log
|
||||
**Fix:**
|
||||
- Set SELinux boolean `nginx_connect_any` to `true` (persistent)
|
||||
- Added port 4180 to `http_port_t` via `semanage port`
|
||||
- Added both as Ansible tasks in `nginx.yml` for persistence
|
||||
|
||||
### 3. Template `no_auth` attribute error
|
||||
**Symptom:** `object of type 'dict' has no attribute 'no_auth'`
|
||||
**Fix:** Changed `{% if not item.no_auth %}` to `{% if not (item.no_auth | default(false) | bool) %}` to handle missing attribute on dicts.
|
||||
|
||||
### 4. Playbook failed on model query after container restart (Phase 2)
|
||||
**Symptom:** `urlopen error [Errno 111] Connection refused` when querying `http://127.0.0.1:11434/models?reload=1`
|
||||
**Cause:** The playbook's "Query llama router model list" task ran before the container was fully started. The container did start correctly with `127.0.0.1:11434` binding and API key active.
|
||||
**Impact:** Non-blocking — the container was running correctly; only the model list query task failed.
|
||||
|
||||
## Known Warnings (non-blocking)
|
||||
|
||||
- `types_hash_bucket_size: 64` — nginx warning, non-blocking
|
||||
- oauth2-proxy PKCE warning (`--code-challenge-method`) — non-blocking, S256 is auto-negotiated
|
||||
|
||||
## Rollback plan
|
||||
|
||||
### Phase 1 rollback (nginx proxy broken)
|
||||
|
||||
- Revert `llama_server_bind` to `0.0.0.0` in host vars
|
||||
- Remove `--api-key` from `llama_server_argv_extra`
|
||||
- Re-add port 11434 firewall rules
|
||||
- Re-run `playbook_llama_server.yml` + `playbook_hardening_cis.yml -t ufw` — container restarts with old settings
|
||||
- OpenWebUI still works (llama.cpp is back on `0.0.0.0` without API key)
|
||||
- Fix issues and re-attempt Phase 2
|
||||
|
||||
### Phase 2 rollback (after OpenWebUI decommission)
|
||||
|
||||
- Restore `llama_server_bind: "0.0.0.0"` in host vars
|
||||
- Remove `--api-key` from `llama_server_argv_extra`
|
||||
- Re-add port 11434 firewall rules
|
||||
- Restore `module "open-webui"` in `terraform/apps/dev-01/apps.tf`
|
||||
- Restore `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` locals in `config.tf`
|
||||
- Restore `open-webui` DNS CNAME record in `config.tf` zones
|
||||
- Restore `webui_secret_key` variable in `secrets.tf`
|
||||
- Re-add `[openwebui]` group in `inventory/hosts`
|
||||
- Run `tofu apply` to recreate OpenWebUI K8s resources
|
||||
@@ -0,0 +1,605 @@
|
||||
# TLS + OAuth2 SSO for dev-02
|
||||
|
||||
## Goal
|
||||
|
||||
Add HTTPS, TLS termination, and OAuth2/OIDC SSO to dev-02 services (code-server + opencode),
|
||||
exposed directly on the internal network for VPN/office access.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────────┐
|
||||
│ dev-02.lab.alexpires.me │
|
||||
│ (Fedora bare metal) │
|
||||
│ │
|
||||
│ ┌──────┐ auth_request ┌───────────┐ │
|
||||
│ │nginx │─────────────────►│oauth2- │ │
|
||||
│ │:443 │◄──────────────────│proxy:4180 │ │
|
||||
│ │TLS │ │(session) │ │
|
||||
│ │SSO │ │ │ │
|
||||
│ └──┬───┘ └─────┬─────┘ │
|
||||
│ │ proxy_pass (trusts auth) │ │
|
||||
│ │ │OIDC │
|
||||
│ ┌──▼──────┐ ┌──────▼────┐ │
|
||||
│ │ide:3000 │ │Gitea IdP │ │
|
||||
│ │(127.0.0.1) │code. │ │
|
||||
│ │code- │ │alexpires. │ │
|
||||
│ │server │ │me (pub) │ │
|
||||
│ └────────┘ └───────────┘ │
|
||||
│ ┌────────┐ │
|
||||
│ │agent:4096│ │
|
||||
│ │(127.0.0.1) │
|
||||
│ │opencode │ │
|
||||
│ └────────┘ │
|
||||
└──────────────────────────────────────────┘
|
||||
│ │
|
||||
HTTPS 443 │ │ OIDC
|
||||
(internal) │ │
|
||||
▼ ▼
|
||||
┌──────────────────────────────────────────┐
|
||||
│ Browser (VPN/office) │ code.alexpires.me (public)
|
||||
│ ide.lab.alexpires.me:443 ──────────► Gitea IdP
|
||||
│ agent.lab.alexpires.me:443 ──────────►
|
||||
└──────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## SSO Flow
|
||||
|
||||
1. User visits `https://ide.lab.alexpires.me:443` (direct, requires VPN/office)
|
||||
2. nginx detects no valid session cookie → `auth_request` to oauth2-proxy
|
||||
3. oauth2-proxy returns 401 → nginx redirects browser to Gitea login
|
||||
4. Browser → `code.alexpires.me` (Gitea, public, always reachable)
|
||||
5. User logs in → Gitea redirects back to `https://ide.lab.alexpires.me/oauth2/callback`
|
||||
6. oauth2-proxy exchanges code → access_token → creates session cookie
|
||||
7. Cookie scoped to `*.lab.alexpires.me` → SSO across lab subdomains
|
||||
8. Subsequent requests: nginx trusts cookie → proxies to backend (no re-login)
|
||||
|
||||
**Access constraint:** SSO only works when the user can reach `lab.alexpires.me` (VPN or office network).
|
||||
This is inherent to the architecture — the OAuth2 callback URL (`ide.lab.alexpires.me/oauth2/callback`)
|
||||
must be reachable by the browser after Gitea redirects back. All `lab.alexpires.me` services
|
||||
follow this same constraint.
|
||||
|
||||
## Certificate Strategy
|
||||
|
||||
- **SAN cert** for `ide.lab.alexpires.me` and `agent.lab.alexpires.me`
|
||||
- Obtained via certbot DNS-01 challenge (cloudns plugin)
|
||||
- Auto-renewal via cron with `--deploy-hook "systemctl reload nginx"`
|
||||
- No port 80/443 exposure needed (DNS-01 challenge doesn't need HTTP)
|
||||
- Certbot stores certs in `/etc/letsencrypt/live/` (symlinked from `/etc/letsencrypt/archive/`)
|
||||
- nginx references:
|
||||
- `ssl_certificate /etc/letsencrypt/live/ide.lab.alexpires.me/fullchain.pem;`
|
||||
- `ssl_certificate_key /etc/letsencrypt/live/ide.lab.alexpires.me/privkey.pem;`
|
||||
|
||||
## Service Binding
|
||||
|
||||
| Service | Before | After |
|
||||
|---------|--------|-------|
|
||||
| code-server | `0.0.0.0:3000` | `127.0.0.1:3000` |
|
||||
| opencode | `0.0.0.0:4096` | `127.0.0.1:4096` |
|
||||
| oauth2-proxy | N/A | `127.0.0.1:4180` |
|
||||
| nginx | N/A | `0.0.0.0:443` (only public port) |
|
||||
|
||||
## Pre-requisites
|
||||
|
||||
Before applying, register an OAuth2 app in Gitea:
|
||||
- **Name:** `lab-services`
|
||||
- **Redirect URI:** `https://ide.lab.alexpires.me/oauth2/callback`
|
||||
- **Scopes:** `openid`, `profile`, `email`
|
||||
- **Result:** `client_id` + `client_secret` → stored in `sectool` / env vars
|
||||
|
||||
Generate a cookie secret:
|
||||
```sh
|
||||
openssl rand -base64 32
|
||||
```
|
||||
Store via `sectool` as `OAUTH2_PROXY_COOKIE_SECRET`, `OAUTH2_PROXY_CLIENT_ID`, `OAUTH2_PROXY_CLIENT_SECRET`.
|
||||
|
||||
## Changes Required
|
||||
|
||||
### 1. New Role: `nginx_proxy`
|
||||
|
||||
```
|
||||
ansible/roles/nginx_proxy/
|
||||
├── defaults/main.yml # certbot, nginx, oauth2 config, upstreams
|
||||
├── tasks/
|
||||
│ ├── main.yml # Entry point (includes certbot, nginx, oauth2_proxy)
|
||||
│ ├── certbot.yml # Install certbot, DNS credentials, obtain cert, cron renewal
|
||||
│ ├── nginx.yml # Install nginx, deploy config, systemd, SELinux context
|
||||
│ └── oauth2_proxy.yml # Download binary, config, systemd
|
||||
├── templates/
|
||||
│ ├── nginx.conf.j2 # Main nginx config (worker processes, logging, etc.)
|
||||
│ ├── nginx-server.conf.j2 # Server blocks (SAN domains, TLS, auth_request, proxy)
|
||||
│ └── oauth2-proxy.cfg.j2 # OAuth2/OIDC config for Gitea
|
||||
└── files/
|
||||
└── .gitkeep
|
||||
```
|
||||
|
||||
#### `defaults/main.yml`
|
||||
|
||||
```yaml
|
||||
nginx_proxy_certbot_enabled: true
|
||||
nginx_proxy_certbot_email: "c.alexandre.pires@alexpires.me"
|
||||
nginx_proxy_certbot_domains:
|
||||
- "ide.lab.alexpires.me"
|
||||
- "agent.lab.alexpires.me"
|
||||
nginx_proxy_certbot_cron_minute: "0"
|
||||
nginx_proxy_certbot_cron_hour: "3"
|
||||
nginx_proxy_certbot_credential_file: "/etc/letsencrypt/cloudns.ini"
|
||||
nginx_proxy_certbot_credential_mode: "0600"
|
||||
nginx_proxy_certbot_nginx_reload: true
|
||||
|
||||
nginx_proxy_nginx_enabled: true
|
||||
nginx_proxy_nginx_bind: "0.0.0.0"
|
||||
nginx_proxy_nginx_port: 443
|
||||
nginx_proxy_nginx_worker_connections: 1024
|
||||
nginx_proxy_nginx_log_format: "main"
|
||||
nginx_proxy_nginx_access_log: "/var/log/nginx/access.log"
|
||||
nginx_proxy_nginx_error_log: "/var/log/nginx/error.log warn"
|
||||
nginx_proxy_nginx_ssl_protocols: "TLSv1.2 TLSv1.3"
|
||||
nginx_proxy_nginx_ssl_ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384"
|
||||
nginx_proxy_nginx_ssl_prefer_server_ciphers: "on"
|
||||
nginx_proxy_nginx_hsts_max_age: "31536000"
|
||||
nginx_proxy_nginx_hsts_include_subdomains: true
|
||||
|
||||
nginx_proxy_oauth2_proxy_enabled: true
|
||||
nginx_proxy_oauth2_proxy_bind: "127.0.0.1"
|
||||
nginx_proxy_oauth2_proxy_port: 4180
|
||||
nginx_proxy_oauth2_proxy_version: "2.13.0"
|
||||
nginx_proxy_oauth2_proxy_cookie_domain: ".lab.alexpires.me"
|
||||
nginx_proxy_oauth2_proxy_cookie_name: "_alexpires_oauth2_proxy"
|
||||
nginx_proxy_oauth2_proxy_cookie_secret: "{{ lookup('env', 'OAUTH2_PROXY_COOKIE_SECRET', default='') }}"
|
||||
nginx_proxy_oauth2_proxy_client_id: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_ID', default='') }}"
|
||||
nginx_proxy_oauth2_proxy_client_secret: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_SECRET', default='') }}"
|
||||
nginx_proxy_oauth2_proxy_redirect_url: "https://ide.lab.alexpires.me/oauth2/callback"
|
||||
nginx_proxy_oauth2_proxy_provider: "gitea"
|
||||
nginx_proxy_oauth2_proxy_provider_url: "https://code.alexpires.me"
|
||||
nginx_proxy_oauth2_proxy_scopes: "openid profile email"
|
||||
nginx_proxy_oauth2_proxy_email_domains: []
|
||||
nginx_proxy_oauth2_proxy_upstream: "static://200"
|
||||
nginx_proxy_oauth2_proxy_skip_jwt: false
|
||||
nginx_proxy_oauth2_proxy_set_basic_auth: false
|
||||
nginx_proxy_oauth2_proxy_set_xauthrequest: true
|
||||
nginx_proxy_oauth2_proxy_ssl_insecure_skip_verify: false
|
||||
nginx_proxy_oauth2_proxy_login_button: "Sign in with Gitea"
|
||||
nginx_proxy_oauth2_proxy_proxy_prefix: "/oauth2"
|
||||
nginx_proxy_oauth2_proxy_silent_login: false
|
||||
nginx_proxy_oauth2_proxy_skip_provider_whoami: false
|
||||
nginx_proxy_oauth2_proxy_cookie_refresh: "0s"
|
||||
nginx_proxy_oauth2_proxy_cookie_expire: "168h"
|
||||
nginx_proxy_oauth2_proxy_session_storage_type: "memory"
|
||||
nginx_proxy_oauth2_proxy_request_logging: true
|
||||
nginx_proxy_oauth2_proxy_log_file: "/var/log/oauth2-proxy.log"
|
||||
nginx_proxy_oauth2_proxy_log_level: "info"
|
||||
|
||||
nginx_proxy_upstreams:
|
||||
ide:
|
||||
host: "127.0.0.1"
|
||||
port: 3000
|
||||
agent:
|
||||
host: "127.0.0.1"
|
||||
port: 4096
|
||||
|
||||
nginx_proxy_cloudns_auth_id: "{{ lookup('env', 'CLOUDNS_AUTH_ID') }}"
|
||||
nginx_proxy_cloudns_auth_password: "{{ lookup('env', 'CLOUDNS_PASSWORD') }}"
|
||||
nginx_proxy_cloudns_nameserver: "109.201.133.111"
|
||||
```
|
||||
|
||||
#### `tasks/certbot.yml`
|
||||
|
||||
- Install certbot + certbot-dns-cloudns via pip
|
||||
- Create CloudDNS credentials file at `/etc/letsencrypt/cloudns.ini`:
|
||||
```ini
|
||||
dns_cloudns_auth_id = {{ nginx_proxy_cloudns_auth_id }}
|
||||
dns_cloudns_auth_password = {{ nginx_proxy_cloudns_auth_password }}
|
||||
```
|
||||
- Permissions: `0600`, owner: `root:root`
|
||||
- Obtain SAN cert:
|
||||
```sh
|
||||
certbot certonly --dns-cloudns \
|
||||
-d ide.lab.alexpires.me \
|
||||
-d agent.lab.alexpires.me \
|
||||
--non-interactive \
|
||||
--agree-tos \
|
||||
-m {{ nginx_proxy_certbot_email }}
|
||||
```
|
||||
- Create cron job for renewal (daily at 3:00 AM):
|
||||
```sh
|
||||
certbot renew --deploy-hook "systemctl reload nginx"
|
||||
```
|
||||
- Certs stored in `/etc/letsencrypt/live/ide.lab.alexpires.me/`
|
||||
|
||||
#### `tasks/nginx.yml`
|
||||
|
||||
- Install nginx package
|
||||
- Render `nginx.conf` (global):
|
||||
```nginx
|
||||
user nginx;
|
||||
worker_processes auto;
|
||||
error_log {{ nginx_proxy_nginx_error_log }};
|
||||
pid /run/nginx.pid;
|
||||
|
||||
events {
|
||||
worker_connections {{ nginx_proxy_nginx_worker_connections }};
|
||||
}
|
||||
|
||||
http {
|
||||
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
|
||||
'$status $body_bytes_sent "$http_referer" '
|
||||
'"$http_user_agent" "$http_x_forwarded_for"';
|
||||
|
||||
access_log {{ nginx_proxy_nginx_access_log }} main;
|
||||
|
||||
sendfile on;
|
||||
tcp_nopush on;
|
||||
tcp_nodelay on;
|
||||
keepalive_timeout 65;
|
||||
types_hash_max_size 2048;
|
||||
|
||||
include /etc/nginx/mime.types;
|
||||
default_type application/octet-stream;
|
||||
|
||||
# WebSocket upgrade mapping
|
||||
map $http_upgrade $connection_upgrade {
|
||||
default upgrade;
|
||||
'' close;
|
||||
}
|
||||
|
||||
include /etc/nginx/conf.d/*.conf;
|
||||
}
|
||||
```
|
||||
- Render `nginx-server.conf` (server blocks):
|
||||
```nginx
|
||||
# IDE (code-server)
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name ide.lab.alexpires.me;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/ide.lab.alexpires.me/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/ide.lab.alexpires.me/privkey.pem;
|
||||
ssl_protocols {{ nginx_proxy_nginx_ssl_protocols }};
|
||||
ssl_ciphers {{ nginx_proxy_nginx_ssl_ciphers }};
|
||||
ssl_prefer_server_ciphers {{ nginx_proxy_nginx_ssl_prefer_server_ciphers }};
|
||||
|
||||
add_header Strict-Transport-Security "max-age={{ nginx_proxy_nginx_hsts_max_age }}; {{ nginx_proxy_nginx_hsts_include_subdomains ? 'includeSubDomains' : '' }}" always;
|
||||
|
||||
# OAuth2 auth_request
|
||||
location = /oauth2/ {
|
||||
proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }};
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Scheme $scheme;
|
||||
proxy_set_header X-Auth-Request-Redirect $request_uri;
|
||||
}
|
||||
|
||||
location = /oauth2/callback {
|
||||
proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }}/oauth2/callback;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Scheme $scheme;
|
||||
proxy_set_header X-Auth-Request-Redirect $request_uri;
|
||||
}
|
||||
|
||||
location /oauth2/ {
|
||||
proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }};
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Scheme $scheme;
|
||||
}
|
||||
|
||||
location / {
|
||||
auth_request /oauth2/;
|
||||
auth_request_set $auth_status $upstream_status;
|
||||
auth_request_set $auth_cookie $upstream_http_set_cookie;
|
||||
|
||||
add_header Set-Cookie $auth_cookie;
|
||||
|
||||
proxy_pass http://127.0.0.1:{{ nginx_proxy_upstreams.ide.port }};
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Port $server_port;
|
||||
|
||||
# WebSocket support
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
|
||||
# Timeouts for code-server
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
}
|
||||
|
||||
# Agent (opencode)
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name agent.lab.alexpires.me;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/ide.lab.alexpires.me/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/ide.lab.alexpires.me/privkey.pem;
|
||||
ssl_protocols {{ nginx_proxy_nginx_ssl_protocols }};
|
||||
ssl_ciphers {{ nginx_proxy_nginx_ssl_ciphers }};
|
||||
ssl_prefer_server_ciphers {{ nginx_proxy_nginx_ssl_prefer_server_ciphers }};
|
||||
|
||||
add_header Strict-Transport-Security "max-age={{ nginx_proxy_nginx_hsts_max_age }}; {{ nginx_proxy_nginx_hsts_include_subdomains ? 'includeSubDomains' : '' }}" always;
|
||||
|
||||
# OAuth2 auth_request
|
||||
location = /oauth2/ {
|
||||
proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }};
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Scheme $scheme;
|
||||
proxy_set_header X-Auth-Request-Redirect $request_uri;
|
||||
}
|
||||
|
||||
location = /oauth2/callback {
|
||||
proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }}/oauth2/callback;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Scheme $scheme;
|
||||
proxy_set_header X-Auth-Request-Redirect $request_uri;
|
||||
}
|
||||
|
||||
location /oauth2/ {
|
||||
proxy_pass http://127.0.0.1:{{ nginx_proxy_oauth2_proxy_port }};
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Scheme $scheme;
|
||||
}
|
||||
|
||||
location / {
|
||||
auth_request /oauth2/;
|
||||
auth_request_set $auth_status $upstream_status;
|
||||
auth_request_set $auth_cookie $upstream_http_set_cookie;
|
||||
|
||||
add_header Set-Cookie $auth_cookie;
|
||||
|
||||
proxy_pass http://127.0.0.1:{{ nginx_proxy_upstreams.agent.port }};
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
proxy_set_header X-Forwarded-Host $host;
|
||||
proxy_set_header X-Forwarded-Port $server_port;
|
||||
|
||||
# WebSocket support
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
|
||||
# Timeouts for opencode
|
||||
proxy_read_timeout 3600s;
|
||||
proxy_send_timeout 3600s;
|
||||
}
|
||||
}
|
||||
```
|
||||
- Enable + start nginx service
|
||||
- Set SELinux context: `semanage fcontext -a -t httpd_cert_t "/etc/letsencrypt(/.*)?"` + `restorecon -Rv /etc/letsencrypt`
|
||||
|
||||
#### `tasks/oauth2_proxy.yml`
|
||||
|
||||
- Download latest release from GitHub (Go binary for linux/amd64):
|
||||
```sh
|
||||
wget -qO /tmp/oauth2-proxy.tar.gz "https://github.com/oauth2-proxy/oauth2-proxy/releases/download/v{{ nginx_proxy_oauth2_proxy_version }}/oauth2-proxy-v{{ nginx_proxy_oauth2_proxy_version }}.linux-amd64.tar.gz"
|
||||
tar xzf /tmp/oauth2-proxy.tar.gz -C /tmp/
|
||||
mv /tmp/oauth2-proxy-v{{ nginx_proxy_oauth2_proxy_version }}.linux-amd64/oauth2-proxy /usr/local/bin/
|
||||
```
|
||||
- Render config file at `/etc/oauth2-proxy/oauth2-proxy.cfg`:
|
||||
```ini
|
||||
provider = "{{ nginx_proxy_oauth2_proxy_provider }}"
|
||||
provider_url = "{{ nginx_proxy_oauth2_proxy_provider_url }}"
|
||||
client_id = "{{ nginx_proxy_oauth2_proxy_client_id }}"
|
||||
client_secret = "{{ nginx_proxy_oauth2_proxy_client_secret }}"
|
||||
cookie_domain = "{{ nginx_proxy_oauth2_proxy_cookie_domain }}"
|
||||
cookie_name = "{{ nginx_proxy_oauth2_proxy_cookie_name }}"
|
||||
cookie_secret = "{{ nginx_proxy_oauth2_proxy_cookie_secret }}"
|
||||
cookie_expire = "{{ nginx_proxy_oauth2_proxy_cookie_expire | default('168h') }}"
|
||||
cookie_refresh = "{{ nginx_proxy_oauth2_proxy_cookie_refresh | default('0s') }}"
|
||||
redirect_url = "{{ nginx_proxy_oauth2_proxy_redirect_url }}"
|
||||
login_url = "{{ nginx_proxy_oauth2_proxy_provider_url }}/login/oauth/authorize"
|
||||
redeem_url = "{{ nginx_proxy_oauth2_proxy_provider_url }}/login/oauth/access_token"
|
||||
openid_connectissuerurl = "{{ nginx_proxy_oauth2_proxy_provider_url }}"
|
||||
scopes = "{{ nginx_proxy_oauth2_proxy_scopes }}"
|
||||
email_domains = {{ nginx_proxy_oauth2_proxy_email_domains | to_json }}
|
||||
upstream = "{{ nginx_proxy_oauth2_proxy_upstream }}"
|
||||
skip_jwt_bearer_tokens = {{ nginx_proxy_oauth2_proxy_skip_jwt | lower }}
|
||||
set_basic_auth = {{ nginx_proxy_oauth2_proxy_set_basic_auth | lower }}
|
||||
set_xauthrequest = {{ nginx_proxy_oauth2_proxy_set_xauthrequest | lower }}
|
||||
ssl_insecure_skip_verify = {{ nginx_proxy_oauth2_proxy_ssl_insecure_skip_verify | lower }}
|
||||
proxy_prefix = "{{ nginx_proxy_oauth2_proxy_proxy_prefix }}"
|
||||
silent_login = {{ nginx_proxy_oauth2_proxy_silent_login | lower }}
|
||||
skip_provider_whoami = {{ nginx_proxy_oauth2_proxy_skip_provider_whoami | lower }}
|
||||
listen = "{{ nginx_proxy_oauth2_proxy_bind }}:{{ nginx_proxy_oauth2_proxy_port }}"
|
||||
{% if nginx_proxy_oauth2_proxy_request_logging | default(true) %}
|
||||
logfile_filename = "{{ nginx_proxy_oauth2_proxy_log_file | default('/var/log/oauth2-proxy.log') }}"
|
||||
{% endif %}
|
||||
{% if nginx_proxy_oauth2_proxy_log_level | default('info') %}
|
||||
log_level = "{{ nginx_proxy_oauth2_proxy_log_level | default('info') }}"
|
||||
{% endif %}
|
||||
cookie_secure = false
|
||||
{% if nginx_proxy_oauth2_proxy_login_button | default('Sign in with Gitea') %}
|
||||
custom_login_message = "Sign in with your Gitea account"
|
||||
{% endif %}
|
||||
```
|
||||
- Create systemd service at `/etc/systemd/system/oauth2-proxy.service`:
|
||||
```ini
|
||||
[Unit]
|
||||
Description=oauth2-proxy for {{ nginx_proxy_oauth2_proxy_cookie_domain }}
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=nginx
|
||||
ExecStart=/usr/local/bin/oauth2-proxy --config /etc/oauth2-proxy/oauth2-proxy.cfg
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
- Enable + start service
|
||||
- Set config file permissions: `0640`, owner `root:nginx` (contains `client_secret`)
|
||||
- Create log directory with proper permissions: `mkdir -p /var/log && chown nginx:nginx /var/log/oauth2-proxy.log`
|
||||
|
||||
### Notes
|
||||
|
||||
- **opencode without credentials** — Verify opencode starts and functions correctly when `OPENCODE_SERVER_USERNAME`/`OPENCODE_SERVER_PASSWORD` are not set. If opencode requires them to start, we may need to keep them but they will be ignored since nginx/oauth2-proxy handles auth.
|
||||
|
||||
### 2. Update `linux_dev_station` Role
|
||||
|
||||
#### `defaults/main.yml`
|
||||
|
||||
- Change `linux_dev_station_code_server_host` default to `127.0.0.1`
|
||||
- Change `linux_dev_station_opencode_server_host` default to `127.0.0.1`
|
||||
|
||||
#### `templates/code-server.service.j2`
|
||||
|
||||
- Change `--bind-addr` to `127.0.0.1:3000`
|
||||
- **Remove** `Environment="PASSWORD={{ linux_dev_station_code_server_password }}"` — SSO handles auth, no password needed
|
||||
|
||||
#### `templates/opencode.service.j2`
|
||||
|
||||
- Change `--hostname` to `127.0.0.1`
|
||||
- **Remove** `Environment="OPENCODE_SERVER_USERNAME={{ linux_dev_station_opencode_server_username }}"` — SSO handles auth
|
||||
- **Remove** `Environment="OPENCODE_SERVER_PASSWORD={{ linux_dev_station_opencode_server_password }}"` — SSO handles auth
|
||||
|
||||
### 3. Update dev-01 DNS & Remove Proxy Config
|
||||
|
||||
#### `terraform/apps/dev-01/config.tf`
|
||||
|
||||
**Add DNS records** for `ide` and `agent` pointing to dev-02 (`10.19.4.210`):
|
||||
|
||||
```hcl
|
||||
{
|
||||
name = "ide"
|
||||
type = "A"
|
||||
content = "10.19.4.210"
|
||||
},
|
||||
{
|
||||
name = "agent"
|
||||
type = "A"
|
||||
content = "10.19.4.210"
|
||||
},
|
||||
```
|
||||
|
||||
**Remove** the `opencode` and `code-server` entries from `proxy_services` (no longer needed, dev-02 handles its own TLS).
|
||||
|
||||
**Remove** the `opencode` and `vscode` CNAME records from the `zones` list:
|
||||
```hcl
|
||||
// Remove these:
|
||||
{
|
||||
name = "opencode"
|
||||
type = "CNAME"
|
||||
content = "dev-01.${local.lab_domain}."
|
||||
},
|
||||
{
|
||||
name = "vscode"
|
||||
type = "CNAME"
|
||||
content = "dev-01.${local.lab_domain}."
|
||||
},
|
||||
```
|
||||
|
||||
### 4. Update Host Vars & Inventory
|
||||
|
||||
#### `ansible/host_vars/dev-02.lab.alexpires.me.yml`
|
||||
|
||||
Add:
|
||||
|
||||
```yaml
|
||||
# Certbot
|
||||
nginx_proxy_certbot_enabled: true
|
||||
nginx_proxy_certbot_email: "c.alexandre.pires@alexpires.me"
|
||||
nginx_proxy_certbot_domains:
|
||||
- "ide.lab.alexpires.me"
|
||||
- "agent.lab.alexpires.me"
|
||||
nginx_proxy_certbot_credential_file: "/etc/letsencrypt/cloudns.ini"
|
||||
nginx_proxy_cloudns_auth_id: "{{ lookup('env', 'CLOUDNS_AUTH_ID') }}"
|
||||
nginx_proxy_cloudns_auth_password: "{{ lookup('env', 'CLOUDNS_PASSWORD') }}"
|
||||
nginx_proxy_cloudns_nameserver: "109.201.133.111"
|
||||
|
||||
# OAuth2 / Gitea
|
||||
nginx_proxy_oauth2_proxy_cookie_secret: "{{ lookup('env', 'OAUTH2_PROXY_COOKIE_SECRET') }}"
|
||||
nginx_proxy_oauth2_proxy_client_id: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_ID') }}"
|
||||
nginx_proxy_oauth2_proxy_client_secret: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_SECRET') }}"
|
||||
```
|
||||
|
||||
#### `inventory/hosts`
|
||||
|
||||
Add `dev-02.lab.alexpires.me` to `[certbot]` group:
|
||||
```ini
|
||||
[certbot]
|
||||
dev-02.lab.alexpires.me
|
||||
```
|
||||
|
||||
### 5. Gitea OAuth2 Application
|
||||
|
||||
Register one OAuth2 app in Gitea:
|
||||
- **Name:** `lab-services`
|
||||
- **Redirect URI:** `https://ide.lab.alexpires.me/oauth2/callback`
|
||||
- **Scopes:** `openid`, `profile`, `email`
|
||||
- **Result:** `client_id` + `client_secret` → stored in `sectool` / env vars
|
||||
|
||||
### 6. Firewall Changes (handled by existing firewall role)
|
||||
|
||||
| Port | Before | After |
|
||||
|------|--------|-------|
|
||||
| 22 | `10.19.4.0/24` + VPN | Same |
|
||||
| 3000 | `10.19.4.0/24` | **CLOSED** |
|
||||
| 4096 | `10.19.4.0/24` | **CLOSED** |
|
||||
| 443 | — | `10.19.4.0/24` + VPN |
|
||||
|
||||
The `linux_dev_station` role already manages firewall rules. Update `fw_allowed_ports` in host_vars:
|
||||
```yaml
|
||||
fw_allowed_ports:
|
||||
- { rule: "allow", port: "22", proto: "tcp", from: "10.19.4.0/24" }
|
||||
- { rule: "allow", port: "22", proto: "tcp", from: "10.5.5.5/32" }
|
||||
- { rule: "allow", port: "443", proto: "tcp", from: "10.19.4.0/24" }
|
||||
- { rule: "allow", port: "443", proto: "tcp", from: "10.5.5.5/32" }
|
||||
```
|
||||
|
||||
## SELinux Considerations
|
||||
|
||||
- Fedora ships with SELinux enforcing by default
|
||||
- nginx needs permission to read certs: `semanage fcontext -a -t httpd_cert_t "/etc/letsencrypt(/.*)?"` + `restorecon -Rv /etc/letsencrypt`
|
||||
- oauth2-proxy runs as `nginx` user — ensure systemd service file sets `User=nginx`
|
||||
- nginx needs outbound HTTP/HTTPS to Gitea and CloudDNS API — these are allowed by default for the `httpd_t` domain
|
||||
|
||||
## Playbook Structure
|
||||
|
||||
```yaml
|
||||
# playbook_linux_dev_station.yml
|
||||
- hosts: linux_dev
|
||||
roles:
|
||||
- linux_dev_station
|
||||
|
||||
# playbook_nginx_proxy.yml
|
||||
- hosts: nginx
|
||||
roles:
|
||||
- nginx_proxy
|
||||
```
|
||||
|
||||
The `nginx_proxy` role is a standalone, generic role — not tied to `linux_dev_station`.
|
||||
|
||||
## Apply Order
|
||||
|
||||
1. Register Gitea OAuth2 app → store creds in `sectool`
|
||||
2. Run `playbook_nginx_proxy.yml` on dev-02 (Ansible)
|
||||
3. Update `fw_allowed_ports` in dev-02 host_vars (close 3000/4096, add 443)
|
||||
4. Update dev-01 DNS + remove proxy config (Terraform)
|
||||
5. Verify: `https://ide.lab.alexpires.me` → SSO via Gitea
|
||||
6. Verify: `https://agent.lab.alexpires.me` → SSO via Gitea
|
||||
|
||||
## Files to Create
|
||||
|
||||
| File | Action |
|
||||
|------|--------|
|
||||
| `ansible/roles/nginx_proxy/` | **NEW** role |
|
||||
| `ansible/playbook_nginx_proxy.yml` | **NEW** standalone playbook (`hosts: certbot`) |
|
||||
| `ansible/roles/linux_dev_station/defaults/main.yml` | Update bind to `127.0.0.1` |
|
||||
| `ansible/roles/linux_dev_station/templates/code-server.service.j2` | `--bind-addr 127.0.0.1:3000` |
|
||||
| `ansible/roles/linux_dev_station/templates/opencode.service.j2` | `--hostname 127.0.0.1` |
|
||||
| `ansible/host_vars/dev-02.lab.alexpires.me.yml` | Add certbot + oauth2 vars |
|
||||
| `inventory/hosts` | Add dev-02 to `[certbot]` |
|
||||
| `terraform/apps/dev-01/config.tf` | Add DNS records for ide/agent, remove proxy config, remove CNAMEs |
|
||||
| `implemented/linux_dev_station.md` | Update report |
|
||||
@@ -3,6 +3,8 @@
|
||||
## Repo Structure
|
||||
|
||||
- **`ansible/`** — Playbooks, roles, inventory. Runs against bare-metal servers.
|
||||
- **`.opencode/plans/`** — Plans under execution
|
||||
- **`implemented/`** — Reports of implemented features
|
||||
- **`terraform/`** — IaC (OpenTofu for K8s app deployments + cloud provisioning).
|
||||
- **`inventory/hosts`** — Ansible inventory. All hosts and groups.
|
||||
- **`requirements/`** — pip requirements files: `ansible.txt`, `dev.txt`, `opentofu.txt`.
|
||||
@@ -26,7 +28,11 @@ export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES # if on a MAC
|
||||
```
|
||||
|
||||
### Running playbooks
|
||||
```sh
|
||||
|
||||
**Note:** ```sectool``` just need to be used when secrets must be injected to the environment, if the command does not requires secrets injection do not use ```sectool```
|
||||
|
||||
```
|
||||
sh
|
||||
cd ansible
|
||||
sectool exec ansible-playbook -u $ANSIBLE_USER playbook_<name>.yml
|
||||
```
|
||||
@@ -64,11 +70,33 @@ Excluded paths: `.git/`, `.github/`, `tests/`. Warns on: `line-length`, `yaml[li
|
||||
- `terraform/iac/cloudns/` — DNS (cloudns)
|
||||
- `terraform/iac/scaleway/` — Scaleway infrastructure
|
||||
|
||||
### Secrets
|
||||
### Running Terraform / OpenTofu
|
||||
|
||||
Each terraform directory has its own `sectool.env`. Run `sectool` to load secrets before `tofu init/plan/apply`:
|
||||
To validate plane (mandatory before apply)
|
||||
```sh
|
||||
sectool env -d terraform/apps/dev-01/ -- tofu plan
|
||||
sectool exec tofu plan
|
||||
```
|
||||
Apply:
|
||||
```sh
|
||||
sectool exec tofu apply --auto-approve
|
||||
```
|
||||
|
||||
## Secrets
|
||||
|
||||
To list secrets:
|
||||
```sh
|
||||
sectool vault list
|
||||
```
|
||||
To set a secret:
|
||||
```sh
|
||||
sectool vault set <secret name> <secret value>
|
||||
```
|
||||
To set a secret:
|
||||
```sh
|
||||
sectool vault get <secret name> <secret value>
|
||||
```
|
||||
|
||||
|
||||
### Module structure
|
||||
- `terraform/modules/apps/` — application modules (Nextcloud, etc.)
|
||||
@@ -112,3 +140,8 @@ sectool exec cntb get instances --oauth2-clientid="$CONTABO_CLIENT_ID" ...
|
||||
- Ansible roles follow galaxy conventions: `defaults/`, `tasks/`, `meta/`, `files/`, `templates/`
|
||||
- `tasks/main.yml` is the entrypoint for every role
|
||||
- Host vars: `ansible/host_vars/<hostname>.yml`
|
||||
|
||||
|
||||
## Troubleshooting and planning
|
||||
|
||||
- When troubleshooting and planning use deepwiki whenever possible to get documentation or information, deepwiki provides information about projects hosted on github to agents.
|
||||
|
||||
@@ -83,7 +83,7 @@ linux_dev_station_opencode_config:
|
||||
name: "alexpires.me"
|
||||
npm: "@ai-sdk/openai-compatible"
|
||||
options:
|
||||
baseURL: "https://open-webui.lab.alexpires.me/api"
|
||||
baseURL: "https://llm.lab.alexpires.me/api/v1"
|
||||
apiKey: "{{ lookup('env', 'OPEN_WEBUI_API_KEY', default='') }}"
|
||||
timeout: 3000000
|
||||
chunkTimeout: 3000000
|
||||
@@ -92,6 +92,12 @@ linux_dev_station_opencode_config:
|
||||
name: "qwen3-6-27b-mtp-q4-0"
|
||||
qwen3-6-35b-a3b-mtp-ud-q4-m:
|
||||
name: "qwen3-6-35b-a3b-mtp-ud-q4-m"
|
||||
scaleway:
|
||||
name: "scaleway"
|
||||
npm: "@ai-sdk/openai-compatible"
|
||||
options:
|
||||
baseURL: "https://api.scaleway.ai/{{ lookup('env', 'SCW_DEFAULT_ORGANIZATION_ID', default='') }}/v1"
|
||||
apiKey: "{{ lookup('env', 'SCW_INFERENCE_API_KEY', default='') }}"
|
||||
mcp:
|
||||
deepwiki:
|
||||
type: "remote"
|
||||
|
||||
@@ -39,8 +39,8 @@ ufw_enable: true
|
||||
fw_allowed_ports:
|
||||
- { rule: "allow", port: "22", proto: "tcp", from: "10.5.5.5/32" } # Wireguard VPN (Laptop)
|
||||
- { rule: "allow", port: "22", proto: "tcp", from: "10.19.4.0/24" } # local network
|
||||
- { rule: "allow", port: "11434", proto: "tcp", from: "10.19.4.0/24" } # Ollama (local network)
|
||||
- { rule: "allow", port: "11434", proto: "tcp", from: "10.5.5.5/32" } # Ollama (Laptop)
|
||||
- { rule: "allow", port: "443", proto: "tcp", from: "10.19.4.0/24" } # HTTPS (nginx reverse proxy)
|
||||
- { rule: "allow", port: "443", proto: "tcp", from: "10.5.5.5/32" } # HTTPS (nginx reverse proxy, VPN)
|
||||
- { rule: "allow", port: "8880", proto: "tcp", from: "10.19.4.0/24" } # Kokoro-FastAPI (local network)
|
||||
- { rule: "allow", port: "8880", proto: "tcp", from: "10.5.5.5/32" } # Kokoro-FastAPI (Laptop)
|
||||
- { rule: "allow", port: "8188", proto: "tcp", from: "10.19.4.0/24" } # ComfyUI (local network)
|
||||
@@ -112,6 +112,7 @@ llama_server_home: "/mnt/data/llamacpp"
|
||||
llama_server_user_pubkey: "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHHOPBR9p9kq5Cqzpe4cr3jHnweaYrHPXv5sXNzt+sCXP54uc5rWUBhxW2OQVvQzJ47dEVhEKi4WSC7LcuKS2G5AQDzWXNiasHvYIYQU3F/EknVCZnsiXYqXphYkJA6rJCNRnISZCIC1mocq6PB7J08ONdRFCvjfUBuVRT8BNGXNmQ/zQ=="
|
||||
|
||||
llama_server_port: 11434
|
||||
llama_server_bind: "127.0.0.1"
|
||||
llama_server_devices:
|
||||
- "nvidia.com/gpu=all"
|
||||
- "/dev/kfd:/dev/kfd:rw"
|
||||
@@ -202,6 +203,8 @@ llama_server_models:
|
||||
|
||||
llama_server_argv_extra:
|
||||
[
|
||||
"--api-key",
|
||||
"{{ lookup('env', 'OPEN_WEBUI_API_KEY', default='') }}",
|
||||
"-t",
|
||||
10,
|
||||
"-np",
|
||||
@@ -249,3 +252,33 @@ auto_suspend_cpu_idle_threshold: 90.0
|
||||
# llama_server_preset_global:
|
||||
# ctx-size: 98304
|
||||
# n-gpu-layers: all
|
||||
|
||||
# nginx_proxy: TLS + OAuth2/OIDC SSO for llm.lab.alexpires.me
|
||||
nginx_proxy_certbot_email: "c.alexandre.pires@alexpires.me"
|
||||
nginx_proxy_oauth2_proxy_enabled: true
|
||||
nginx_proxy_oauth2_proxy_provider: "oidc"
|
||||
nginx_proxy_oauth2_proxy_provider_url: "https://code.alexpires.me"
|
||||
nginx_proxy_oauth2_proxy_redirect_urls:
|
||||
- "https://llm.lab.alexpires.me/oauth2/callback"
|
||||
nginx_proxy_oauth2_proxy_email_domains:
|
||||
- "*"
|
||||
nginx_proxy_oauth2_proxy_cookie_secret: "{{ lookup('env', 'OAUTH2_PROXY_COOKIE_SECRET', default='') }}"
|
||||
nginx_proxy_oauth2_proxy_client_id: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_ID', default='') }}"
|
||||
nginx_proxy_oauth2_proxy_client_secret: "{{ lookup('env', 'OAUTH2_PROXY_CLIENT_SECRET', default='') }}"
|
||||
nginx_proxy_sites:
|
||||
- name: llm
|
||||
server_name: "llm.lab.alexpires.me"
|
||||
upstream:
|
||||
host: "127.0.0.1"
|
||||
port: 11434
|
||||
websocket: true
|
||||
extra_locations:
|
||||
- path: /api/
|
||||
config:
|
||||
proxy_pass: "http://127.0.0.1:11434/"
|
||||
proxy_buffering: "off"
|
||||
proxy_read_timeout: "1800s"
|
||||
proxy_send_timeout: "1800s"
|
||||
proxy_http_version: "1.1"
|
||||
proxy_set_header Upgrade: "$http_upgrade"
|
||||
proxy_set_header Connection: "$connection_upgrade"
|
||||
|
||||
@@ -10,6 +10,7 @@ llama_server_extra_groups: "users,video"
|
||||
llama_server_devices: []
|
||||
llama_server_env: {}
|
||||
llama_server_port: 11434
|
||||
llama_server_bind: "0.0.0.0"
|
||||
llama_server_preset_global: {}
|
||||
llama_server_host: "127.0.0.1"
|
||||
llama_server_argv_extra: []
|
||||
|
||||
@@ -135,7 +135,7 @@
|
||||
state: started
|
||||
device: "{{ llama_server_devices | default(omit) }}"
|
||||
env: "{{ llama_server_env | default(omit) }}"
|
||||
ports: "0.0.0.0:{{ llama_server_port }}:8080/tcp"
|
||||
ports: "{{ llama_server_bind }}:{{ llama_server_port }}:8080/tcp"
|
||||
volumes:
|
||||
- "{{ llama_server_home }}/models:/models:Z"
|
||||
- "{{ llama_server_home }}/.cache:/app/.cache:Z"
|
||||
|
||||
@@ -44,3 +44,18 @@
|
||||
become: true
|
||||
ansible.builtin.command: restorecon -Rv /etc/letsencrypt
|
||||
changed_when: false
|
||||
|
||||
- name: Set SELinux boolean to allow nginx to connect to any port
|
||||
become: true
|
||||
ansible.builtin.command: semanage boolean -m --on nginx_connect_any
|
||||
failed_when: false
|
||||
changed_when: false
|
||||
when: ansible_selinux.status == "enabled"
|
||||
|
||||
- name: Add oauth2-proxy port to SELinux http_port_t
|
||||
become: true
|
||||
ansible.builtin.command: >-
|
||||
semanage port -a -t http_port_t -p tcp {{ nginx_proxy_oauth2_proxy_port }}
|
||||
failed_when: false
|
||||
changed_when: false
|
||||
when: ansible_selinux.status == "enabled"
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
mode: "0755"
|
||||
owner: root
|
||||
group: root
|
||||
remote_src: true
|
||||
|
||||
- name: Create oauth2-proxy config directory
|
||||
become: true
|
||||
|
||||
@@ -13,6 +13,7 @@ server {
|
||||
|
||||
add_header Strict-Transport-Security "max-age={{ nginx_proxy_nginx_hsts_max_age }}; {{ nginx_proxy_nginx_hsts_include_subdomains | ternary('includeSubDomains', '') }}" always;
|
||||
|
||||
{% if not (item.no_auth | default(false) | bool) %}
|
||||
# Redirect unauthenticated requests to OAuth2 sign-in
|
||||
error_page 401 = /oauth2/sign_in;
|
||||
|
||||
@@ -52,13 +53,16 @@ server {
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Scheme $scheme;
|
||||
}
|
||||
{% endif %}
|
||||
|
||||
location / {
|
||||
{% if not (item.no_auth | default(false) | bool) %}
|
||||
auth_request /oauth2/auth;
|
||||
auth_request_set $auth_status $upstream_status;
|
||||
auth_request_set $auth_cookie $upstream_http_set_cookie;
|
||||
|
||||
add_header Set-Cookie $auth_cookie;
|
||||
{% endif %}
|
||||
add_header Strict-Transport-Security "max-age={{ nginx_proxy_nginx_hsts_max_age }}; {{ nginx_proxy_nginx_hsts_include_subdomains | ternary('includeSubDomains', '') }}" always;
|
||||
|
||||
proxy_pass http://{{ item.upstream.host }}:{{ item.upstream.port }};
|
||||
|
||||
@@ -24,3 +24,5 @@ OPEN_WEBUI_API_KEY=$OPEN_WEBUI_API_KEY
|
||||
OAUTH2_PROXY_COOKIE_SECRET=$OAUTH2_PROXY_COOKIE_SECRET
|
||||
OAUTH2_PROXY_CLIENT_ID=$OAUTH2_PROXY_CLIENT_ID
|
||||
OAUTH2_PROXY_CLIENT_SECRET=$OAUTH2_PROXY_CLIENT_SECRET
|
||||
SCW_DEFAULT_ORGANIZATION_ID=$SCALEWAY_ORGANIZATION_ID
|
||||
SCW_INFERENCE_API_KEY=$SCW_INFERENCE_API_KEY
|
||||
+1
-3
@@ -27,6 +27,7 @@ gpu-01.lab.alexpires.me
|
||||
|
||||
[nginx]
|
||||
dev-02.lab.alexpires.me
|
||||
gpu-01.lab.alexpires.me
|
||||
|
||||
[public]
|
||||
prod-01.alexpires.me
|
||||
@@ -92,9 +93,6 @@ gpu-01.lab.alexpires.me
|
||||
[comfyui]
|
||||
gpu-01.lab.alexpires.me
|
||||
|
||||
[openwebui]
|
||||
dev-01.lab.alexpires.me
|
||||
|
||||
[mcp]
|
||||
dev-01.lab.alexpires.me
|
||||
|
||||
|
||||
@@ -1,14 +1,3 @@
|
||||
module "open-webui" {
|
||||
source = "../../modules/apps/open-webui"
|
||||
|
||||
persistent_folder = local.persistent_folder
|
||||
fqdn = local.open_webui_fqdn
|
||||
llm_proxy = local.llm_proxy
|
||||
secret_key = var.webui_secret_key
|
||||
tag = "v0.9.2-slim"
|
||||
log_level = local.open_webui_log_level
|
||||
}
|
||||
|
||||
module "samba" {
|
||||
source = "../../modules/apps/samba"
|
||||
|
||||
|
||||
@@ -6,10 +6,6 @@ locals {
|
||||
lab_domain = "lab.alexpires.me"
|
||||
internal_issuer = "internal-ca"
|
||||
|
||||
# open-webui
|
||||
open_webui_fqdn = "open-webui.${local.lab_domain}"
|
||||
open_webui_log_level = "DEBUG"
|
||||
|
||||
# samba
|
||||
samba_shares = [
|
||||
{
|
||||
@@ -61,12 +57,7 @@ locals {
|
||||
type = "CNAME"
|
||||
content = "dev-01.${local.lab_domain}."
|
||||
},
|
||||
{
|
||||
name = "open-webui"
|
||||
type = "CNAME"
|
||||
content = "dev-01.${local.lab_domain}."
|
||||
},
|
||||
{
|
||||
{
|
||||
name = "smb"
|
||||
type = "CNAME"
|
||||
content = "dev-01.${local.lab_domain}."
|
||||
@@ -116,6 +107,11 @@ locals {
|
||||
type = "A"
|
||||
content = "10.19.4.210"
|
||||
},
|
||||
{
|
||||
name = "llm"
|
||||
type = "A"
|
||||
content = "10.19.4.106"
|
||||
},
|
||||
{
|
||||
name = "ide"
|
||||
type = "A"
|
||||
@@ -180,14 +176,6 @@ locals {
|
||||
}
|
||||
}
|
||||
|
||||
llm_proxy = {
|
||||
enabled = true
|
||||
mac = "04:7c:16:d6:63:66"
|
||||
ip = "10.19.4.106"
|
||||
port = 11434
|
||||
scheme = "http"
|
||||
}
|
||||
|
||||
llm_proxy_1 = {
|
||||
enabled = true
|
||||
mac = "52:54:00:CD:DD:2E"
|
||||
|
||||
@@ -22,12 +22,6 @@ variable "telegram_bot_token" {
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
variable "webui_secret_key" {
|
||||
description = "The secret key for WEBUI"
|
||||
type = string
|
||||
sensitive = true
|
||||
}
|
||||
|
||||
variable "searxng_secret_key" {
|
||||
description = "The secret key for SearXNG"
|
||||
type = string
|
||||
|
||||
@@ -147,28 +147,6 @@ locals {
|
||||
}
|
||||
|
||||
proxy_services = {
|
||||
"open-webui" = {
|
||||
http_port = 8080
|
||||
https_port = 8443
|
||||
tls = true
|
||||
resolver = "10.19.4.136"
|
||||
fqdn = ["ai.alexpires.me"]
|
||||
read_timeout = "1800"
|
||||
send_timeout = "1800"
|
||||
upstream = "https://open-webui.lab.alexpires.me"
|
||||
locations = [
|
||||
{
|
||||
path = "/ws/socket.io/"
|
||||
headers = {
|
||||
"Upgrade" = "$http_upgrade"
|
||||
"Connection" = "upgrade"
|
||||
}
|
||||
},
|
||||
{
|
||||
path = "/"
|
||||
},
|
||||
]
|
||||
},
|
||||
"nextcloud" = {
|
||||
http_port = 8081
|
||||
https_port = 8444
|
||||
|
||||
@@ -170,16 +170,6 @@ locals {
|
||||
type = "TXT"
|
||||
value = "v=DKIM1; h=sha256; k=rsa; p=${var.alexpires_me_scaleway_dkim_value}"
|
||||
},
|
||||
{
|
||||
name = "webdav"
|
||||
type = "CNAME"
|
||||
value = "${local.vps_1.name}.alexpires.me"
|
||||
},
|
||||
{
|
||||
name = "ai"
|
||||
type = "CNAME"
|
||||
value = "${local.vps_1.name}.alexpires.me"
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"provider": "bitwarden",
|
||||
"bitwarden": {
|
||||
"api_url": "https://api.bitwarden.eu",
|
||||
"identity_url": "https://identity.bitwarden.eu",
|
||||
"project": "c6bd56a4-9fec-4388-b74a-b2a100c0ae07"
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user