alexpires.me/a13labs.infra
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
main
a13labs.infra/.opencode/plans/gateway-api-migration.md
T
alexandre.pires 8724360fb2 Enhance infrastructure and application configurations 
- Updated AGENTS.md to include new directories for execution plans and implemented feature reports.
- Modified Ansible host variables for dev-02 and gpu-01 to change API endpoints and add new configurations for Scaleway.
- Adjusted firewall rules in gpu-01 to allow HTTPS traffic instead of the previous Ollama port.
- Added OAuth2 proxy configurations to gpu-01 for enhanced security.
- Removed deprecated open-webui module from Terraform app configurations.
- Updated Terraform configurations to reflect changes in local variables and removed unused secrets.
- Created a new sectool.json file for CloudNS integration with Bitwarden.
- Enhanced SELinux configurations for nginx to allow connections to any port and added oauth2-proxy port.
2026-07-12 16:28:45 -04:00

322 lines
14 KiB
Markdown
Raw Permalink Blame History

# Gateway API Migration & Cloudflare Integration
## Overview
Migrate prod-01 and dev-01 clusters from the deprecated nginx-ingress controller (retired March 2026) to NGINX Gateway Fabric via the Gateway API, improve cross-cluster WireGuard connectivity, and add Cloudflare Tunnel for edge protection.
## Current Architecture
```
Internet → Contabo Public IP → OPNsense → prod-01
↓
nginx-ingress plugin (MicroK8s)
↓
MetalLB (10.2.0.40-49)
↓
┌───────────────┼─────────────────┐
↓ ↓ ↓
[proxy pod] [app pods] [TCP streams]
(nginx + WG) (gitea, m3uproxy, etc) (hostPort)
↓
WireGuard → aristotle (Contabo VPN)
↓
dev-01 backends (open-webui, nextcloud)
```
**Key facts:**
- 14 `kubernetes_ingress_v1` resources across both clusters
- All use `ingress.class: public` + nginx-specific annotations
- Proxy module on prod-01 creates a double-proxy (Ingress → proxy pod → backends)
- TCP/UDP streams use nginx with hostPorts
- DNS: ClouDNS, no Cloudflare involvement
- cert-manager with DNS-01 (dev-01) and HTTP-01 (prod-01) for Let's Encrypt
- **Both clusters are single-node** — no need for MetalLB, NodePort is sufficient
## Decisions Made
| Decision | Choice | Rationale |
|---|---|---|
| Gateway API implementation | **NGINX Gateway Fabric** | Keeps nginx data plane, minimal behavioral changes, conformant v1.4.1 |
| Cross-cluster connectivity | **Improve WireGuard** | Keep through aristotle, allowed_ips to dev-01 only (10.19.4.136/32), remove gpu-01 |
| Ollama removal | **Remove** | gpu-01 runs llamacpp (not ollama), prod-01 ollama proxy is broken, dev-01 DNS CNAME stale |
| Cloudflare scope | **Tunnels only** | Get DDoS+WAF, keep DNS with ClouDNS, least disruptive |
| Proxy module | **All through proxy** | Gateway routes everything to proxy Service, simpler Terraform |
| Website module | **Direct Gateway route** | Keep separate pods, no proxy overhead for static sites |
| TLS termination | **At Cloudflare edge** | Simpler cert management, Cloudflare provides origin cert |
## Target Architecture
```
Internet → Cloudflare Tunnel → Cloudflare Edge (TLS termination)
↓
cloudflared pods (DaemonSet)
↓
Gateway (NGINX Gateway Fabric, NodePort Service)
HTTP listener :80 (from cloudflared)
↓
┌───────────────┼─────────────────┐
↓ ↓ ↓
[proxy Service] [app Services] [TCP/UDP Routes]
↓ ↓ ↓
[proxy pod + WG] [direct pods] [direct pods]
(nginx) (website, etc.) (rustdesk, etc.)
↓
Cross-cluster → dev-01 backends
(open-webui, nextcloud)
```
---
## Phase 1: Gateway API Migration
### 1A. Install NGINX Gateway Fabric (Ansible)
**New file: `ansible/roles/microk8s/tasks/gateway-api.yml`**
```
- Install `snap` package manager (if needed)
- Install helm via snap
- Add nginx-stable helm repo
- Install Gateway API CRDs (standard channel + experimental for TCPRoute/UDPRoute)
- Install NGINX Gateway Fabric via Helm (experimental features enabled)
- Create GatewayClass named `public`
- Create Gateway Service with type = NodePort (single-node, no MetalLB needed)
```
**Modify: `ansible/roles/microk8s/tasks/main.yml`**
- Include `gateway-api.yml` AFTER existing setup (runs alongside nginx-ingress initially)
**Modify: `ansible/roles/microk8s/defaults/main.yml`**
- Add variables: `gateway_fabric_version`, `gateway_fabric_experimental_enabled`
- Remove MetalLB-related vars (`microk8s_lb_provider`, `microk8s_lb_ips`, `microk8s_metallb_version`)
### 1B. Create Gateway Module (Terraform)
**New module: `terraform/modules/utils/gateway/`**
- `main.tf`: Gateway resource with HTTP (:80) listener, `ServiceType: NodePort`
- `variables.tf`: TLS secret names, listener config
- `outputs.tf`: Gateway service name, NodePort port numbers
**Deploy in both clusters** (`prod-01/apps.tf`, `dev-01/apps.tf`)
### 1C. Migrate Terraform Modules (Ingress → Gateway API Resources)
#### Proxy module (`terraform/modules/utils/proxy/main.tf`)
- **Remove**: `kubernetes_ingress_v1.ingress` resource (lines 160-203)
- **Remove**: `host_port` from stream container ports in Deployment
- **Remove**: ollama proxy service definition from prod-01 config.tf
- **Add**: HTTPRoute for cross-cluster services (open-webui, nextcloud)
- **Add**: TCPRoute resources for RustDesk (21115, 21116, 21117 TCP) and Prometheus (9090)
- **Add**: UDPRoute resource for RustDesk (21116 UDP)
- The proxy pod's nginx stream block can be removed (Gateway handles L4 routing)
- **Keep**: WireGuard sidecar, auth sidecar, internal nginx reverse proxy config
#### Website module (`terraform/modules/apps/nginx/main.tf`)
- **Remove**: `kubernetes_ingress_v1.website_ingress` (lines 103-145)
- **Add**: HTTPRoute per website with backendRef → website Service
- Gateway handles TLS termination, website pods receive plain HTTP
- Website pods can drop internal TLS (simplification)
#### Gitea module (`terraform/modules/apps/gitea/ingress.tf`)
- **Remove**: `kubernetes_ingress_v1.gitea_ingress`
- **Add**: HTTPRoute with backendRef → Gitea Service
- Use `BackendTLSPolicy` if Gitea serves HTTPS internally
#### M3UProxy module (`terraform/modules/apps/m3uproxy/main.tf`)
- **Remove**: `kubernetes_ingress_v1.m3uproxy_ingress_http` (lines 241-274)
- **Add**: HTTPRoute with backendRef → m3uproxy Service
#### dev-01 modules
| Module | Current | Change |
|---|---|---|
| `open-webui/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `grafana/grafana.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `mstream/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `searxng/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `vaultwarden/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `terraform-mcp/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `nextcloud/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `proxy/main.tf` | kubernetes_ingress_v1 | HTTPRoute (opencode, vscode) |
#### Annotation → Gateway API Mapping
| Current nginx annotation | Gateway API equivalent |
|---|---|
| `nginx.org/ssl-redirect: true` | HTTPRoute with `RequestRedirect` filter |
| `nginx.org/redirect-to-https: true` | HTTPRoute on HTTP listener with redirect |
| `nginx.ingress.kubernetes.io/backend-protocol: HTTPS` | `BackendTLSPolicy` resource |
| `nginx.ingress.kubernetes.io/proxy-body-size` | `UpstreamSettings` policy (NGF custom) |
| `nginx.ingress.kubernetes.io/proxy-request-buffering` | `ProxySettings` policy (NGF custom) |
| `nginx.ingress.kubernetes.io/proxy-buffer-size` | `UpstreamSettings` policy |
| `cert-manager.io/cluster-issuer` | cert-manager annotation on Gateway |
### 1D. cert-manager Integration
Use **approach B**: Keep existing Certificate resources, reference secrets in Gateway TLS config.
- Each module's `kubernetes_manifest` Certificate resource stays unchanged
- Gateway HTTPS listener references cert secrets via `certificateRefs`
- This avoids alpha cert-manager Gateway integration
### 1E. Migration Execution Order
1. Apply new Ansible role (installs Gateway Fabric alongside nginx-ingress)
2. Deploy Gateway resource to both clusters
3. Migrate modules on **dev-01 first** (lower risk):
- Deploy HTTPRoutes for each module, one at a time
- Verify each service works
- Remove old Ingress resources
4. Migrate modules on **prod-01**:
- Same process, one module at a time
5. Disable nginx-ingress plugin on both clusters
6. Clean up IngressClass, old LoadBalancer Service
---
## Phase 2: Cross-Cluster Connectivity
### 2A. Improve WireGuard
**Modify: `terraform/apps/prod-01/config.tf`** (WireGuard config):
- Change `allowed_ips` from `[10.19.4.106/32, 10.19.4.136/32]` → `["10.19.4.136/32"]` (dev-01 only)
- Remove gpu-01 (10.19.4.106) - ollama proxy being removed, no need to reach gpu-01
- `persistent_keepalive` already set to 25
**Modify: `terraform/modules/utils/proxy/resources/wireguard/wireguard.conf.tpl`**:
- Template `PersistentKeepalive` from config variable
- Template `AllowedIPs` from config list
**Also clean up**:
- Remove `"ollama"` entry from `proxy_services` in prod-01 config.tf
- Remove `ollama` CNAME from dev-01 DNS zones in dev-01 config.tf
**Modify: `terraform/modules/utils/proxy/main.tf`** (WireGuard container):
- Improve liveness probe: `wg show` instead of `wg show | grep -q transfer`
- Add readiness probe
- Add resource limits (memory cap)
### 2B. DNS Resolution
No changes needed. The proxy's per-location resolver in `default.conf.tpl` already handles DNS correctly for cross-cluster backends.
---
## Phase 3: Cloudflare Tunnels
### 3A. Cloudflare Setup
- Create/verify Cloudflare account with `alexpires.me` domain
- Keep DNS with ClouDNS (Cloudflare acts as reverse proxy only)
- Create Cloudflare Tunnel for prod-01
- Configure route rules: `https://*.alexpires.me` → Gateway Service
- Set up origin rules (TLS terminates at Cloudflare edge → HTTP to cloudflared)
### 3B. Deploy cloudflared
**New module: `terraform/modules/utils/cloudflare-tunnel/`**
- DaemonSet of `cloudflare/cloudflared` pods
- ConfigMap with tunnel credentials and route definitions
- Each public FQDN mapped to `http://<gateway-service>.<namespace>:80`
### 3C. Gateway Configuration Changes
- Gateway uses HTTP (:80) NodePort listener for Cloudflare Tunnel traffic
- Cloudflare Tunnel connects to `http://<node_ip>:<nodeport>`
- No HTTPS listener needed since all traffic comes through Cloudflare Tunnel
### 3D. Firewall Changes
- Close UFW ports 80/443 (cloudflared uses ephemeral outbound ports)
- Keep SSH + WireGuard ports open
### 3E. Security Benefits
With Cloudflare in front:
- No open ports on prod-01 (80, 443, etc. closed)
- DDoS protection (automatic with Cloudflare)
- WAF rules (OWASP Top 10, custom rules)
- Rate limiting on public endpoints
- Bot management
---
## Phase 4: Cleanup
- Remove nginx-ingress plugin from MicroK8s (`microk8s.disable ingress`)
- Remove unused IngressClass resources
- Remove existing MetalLB (delete metallb-system namespace, IPAddressPool, L2Advertisement)
- Delete orphaned `tasks/loadbalancer.yml` and `tasks/loadbalancer/metallb.yaml`
- Remove MetalLB vars from `defaults/main.yml`
- Update documentation
---
## File Change Summary
### New Files
| File | Purpose |
|---|---|
| `ansible/roles/microk8s/tasks/gateway-api.yml` | NGINX Gateway Fabric install |
| `terraform/modules/utils/gateway/main.tf` | Gateway + ReferenceGrant resources |
| `terraform/modules/utils/gateway/variables.tf` | Gateway configuration |
| `terraform/modules/utils/gateway/outputs.tf` | Gateway outputs |
| `terraform/modules/utils/cloudflare-tunnel/main.tf` | cloudflared DaemonSet |
| `terraform/modules/utils/cloudflare-tunnel/variables.tf` | Tunnel config |
### Deleted Files
| File | Reason |
|---|---|
| `ansible/roles/microk8s/tasks/loadbalancer.yml` | Not wired into main.yml, was for nginx-ingress LB |
| `ansible/roles/microk8s/tasks/loadbalancer/metallb.yaml` | MetalLB no longer needed (single-node, NodePort) |
### Modified Files
| File | Change |
|---|---|
| `ansible/roles/microk8s/tasks/main.yml` | Include gateway-api.yml |
| `ansible/roles/microk8s/defaults/main.yml` | Add Gateway Fabric defaults, remove MetalLB vars |
| `terraform/modules/utils/proxy/main.tf` | Ingress → HTTPRoute + TCPRoute + UDPRoute |
| `terraform/modules/utils/proxy/variables.tf` | Update for Gateway API |
| `terraform/modules/utils/proxy/resources/wireguard/wireguard.conf.tpl` | Add PersistentKeepalive |
| `terraform/modules/apps/nginx/main.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/gitea/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/m3uproxy/main.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/open-webui/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/grafana/grafana.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/mstream/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/searxng/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/vaultwarden/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/terraform-mcp/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/nextcloud/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/ssh_locker_web/` | Ingress → HTTPRoute |
| `terraform/apps/prod-01/apps.tf` | Add gateway module |
| `terraform/apps/prod-01/config.tf` | WireGuard (remove gpu-01), remove ollama proxy |
| `terraform/apps/dev-01/config.tf` | Remove ollama DNS CNAME |
| `terraform/apps/dev-01/apps.tf` | Add gateway module |
---
## Risk Assessment
| Risk | Mitigation |
|---|---|
| TCPRoute/UDPRoute experimental in NGF | Start with HTTPRoute only, keep proxy streams temporarily |
| Cloudflare Tunnel adds latency | Monitor, acceptable for reverse proxy traffic |
| cert-manager Gateway integration alpha | Use approach B (separate Certificate resources) |
| Downtime during migration | Run both controllers in parallel, switch via DNS TTL |
| Removing gpu-01 from WireGuard allowed_ips | Verify no active proxy depends on gpu-01 reachability |
| Removing existing MetalLB on prod-01 | Do after Cloudflare Tunnel is fully operational |
## Timeline Estimate
| Phase | Duration | Dependencies |
|---|---|---|
| Phase 1 (Gateway API) | 2-3 weeks | None |
| Phase 2 (WireGuard) | 1-2 weeks | Can run parallel with Phase 1 |
| Phase 3 (Cloudflare) | 1-2 weeks | After Phase 1 |
| Phase 4 (Cleanup) | 1 week | After Phase 3 |
Reference in New Issue View Git Blame Copy Permalink