# Gateway API Migration & Cloudflare Integration

## Overview

Migrate prod-01 and dev-01 clusters from the deprecated nginx-ingress controller (retired March 2026) to NGINX Gateway Fabric via the Gateway API, improve cross-cluster WireGuard connectivity, and add Cloudflare Tunnel for edge protection.

## Current Architecture

```
Internet → Contabo Public IP → OPNsense → prod-01
                              ↓
                     nginx-ingress plugin (MicroK8s)
                              ↓
                         MetalLB (10.2.0.40-49)
                              ↓
               ┌───────────────┼─────────────────┐
               ↓               ↓                  ↓
        [proxy pod]      [app pods]        [TCP streams]
      (nginx + WG)  (gitea, m3uproxy, etc)  (hostPort)
          ↓
    WireGuard → aristotle (Contabo VPN)
          ↓
      dev-01 backends (open-webui, nextcloud)
```

**Key facts:**
- 14 `kubernetes_ingress_v1` resources across both clusters
- All use `ingress.class: public` + nginx-specific annotations
- Proxy module on prod-01 creates a double-proxy (Ingress → proxy pod → backends)
- TCP/UDP streams use nginx with hostPorts
- DNS: ClouDNS, no Cloudflare involvement
- cert-manager with DNS-01 (dev-01) and HTTP-01 (prod-01) for Let's Encrypt
- **Both clusters are single-node** — no need for MetalLB, NodePort is sufficient

## Decisions Made

| Decision | Choice | Rationale |
|---|---|---|
| Gateway API implementation | **NGINX Gateway Fabric** | Keeps nginx data plane, minimal behavioral changes, conformant v1.4.1 |
| Cross-cluster connectivity | **Improve WireGuard** | Keep through aristotle, allowed_ips to dev-01 only (10.19.4.136/32), remove gpu-01 |
| Ollama removal | **Remove** | gpu-01 runs llamacpp (not ollama), prod-01 ollama proxy is broken, dev-01 DNS CNAME stale |
| Cloudflare scope | **Tunnels only** | Get DDoS+WAF, keep DNS with ClouDNS, least disruptive |
| Proxy module | **All through proxy** | Gateway routes everything to proxy Service, simpler Terraform |
| Website module | **Direct Gateway route** | Keep separate pods, no proxy overhead for static sites |
| TLS termination | **At Cloudflare edge** | Simpler cert management, Cloudflare provides origin cert |

## Target Architecture

```
Internet → Cloudflare Tunnel → Cloudflare Edge (TLS termination)
                              ↓
                     cloudflared pods (DaemonSet)
                              ↓
                     Gateway (NGINX Gateway Fabric, NodePort Service)
                     HTTP listener :80 (from cloudflared)
                              ↓
               ┌───────────────┼─────────────────┐
               ↓               ↓                  ↓
        [proxy Service]  [app Services]    [TCP/UDP Routes]
               ↓               ↓                  ↓
      [proxy pod + WG]   [direct pods]      [direct pods]
          (nginx)   (website, etc.)       (rustdesk, etc.)
               ↓
         Cross-cluster → dev-01 backends
         (open-webui, nextcloud)
```

---

## Phase 1: Gateway API Migration

### 1A. Install NGINX Gateway Fabric (Ansible)

**New file: `ansible/roles/microk8s/tasks/gateway-api.yml`**

```
- Install `snap` package manager (if needed)
- Install helm via snap
- Add nginx-stable helm repo
- Install Gateway API CRDs (standard channel + experimental for TCPRoute/UDPRoute)
- Install NGINX Gateway Fabric via Helm (experimental features enabled)
- Create GatewayClass named `public`
- Create Gateway Service with type = NodePort (single-node, no MetalLB needed)
```

**Modify: `ansible/roles/microk8s/tasks/main.yml`**
- Include `gateway-api.yml` AFTER existing setup (runs alongside nginx-ingress initially)

**Modify: `ansible/roles/microk8s/defaults/main.yml`**
- Add variables: `gateway_fabric_version`, `gateway_fabric_experimental_enabled`
- Remove MetalLB-related vars (`microk8s_lb_provider`, `microk8s_lb_ips`, `microk8s_metallb_version`)

### 1B. Create Gateway Module (Terraform)

**New module: `terraform/modules/utils/gateway/`**
- `main.tf`: Gateway resource with HTTP (:80) listener, `ServiceType: NodePort`
- `variables.tf`: TLS secret names, listener config
- `outputs.tf`: Gateway service name, NodePort port numbers

**Deploy in both clusters** (`prod-01/apps.tf`, `dev-01/apps.tf`)

### 1C. Migrate Terraform Modules (Ingress → Gateway API Resources)

#### Proxy module (`terraform/modules/utils/proxy/main.tf`)

- **Remove**: `kubernetes_ingress_v1.ingress` resource (lines 160-203)
- **Remove**: `host_port` from stream container ports in Deployment
- **Remove**: ollama proxy service definition from prod-01 config.tf
- **Add**: HTTPRoute for cross-cluster services (open-webui, nextcloud)
- **Add**: TCPRoute resources for RustDesk (21115, 21116, 21117 TCP) and Prometheus (9090)
- **Add**: UDPRoute resource for RustDesk (21116 UDP)
- The proxy pod's nginx stream block can be removed (Gateway handles L4 routing)
- **Keep**: WireGuard sidecar, auth sidecar, internal nginx reverse proxy config

#### Website module (`terraform/modules/apps/nginx/main.tf`)

- **Remove**: `kubernetes_ingress_v1.website_ingress` (lines 103-145)
- **Add**: HTTPRoute per website with backendRef → website Service
- Gateway handles TLS termination, website pods receive plain HTTP
- Website pods can drop internal TLS (simplification)

#### Gitea module (`terraform/modules/apps/gitea/ingress.tf`)

- **Remove**: `kubernetes_ingress_v1.gitea_ingress`
- **Add**: HTTPRoute with backendRef → Gitea Service
- Use `BackendTLSPolicy` if Gitea serves HTTPS internally

#### M3UProxy module (`terraform/modules/apps/m3uproxy/main.tf`)

- **Remove**: `kubernetes_ingress_v1.m3uproxy_ingress_http` (lines 241-274)
- **Add**: HTTPRoute with backendRef → m3uproxy Service

#### dev-01 modules

| Module | Current | Change |
|---|---|---|
| `open-webui/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `grafana/grafana.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `mstream/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `searxng/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `vaultwarden/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `terraform-mcp/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `nextcloud/ingress.tf` | kubernetes_ingress_v1 | HTTPRoute |
| `proxy/main.tf` | kubernetes_ingress_v1 | HTTPRoute (opencode, vscode) |

#### Annotation → Gateway API Mapping

| Current nginx annotation | Gateway API equivalent |
|---|---|
| `nginx.org/ssl-redirect: true` | HTTPRoute with `RequestRedirect` filter |
| `nginx.org/redirect-to-https: true` | HTTPRoute on HTTP listener with redirect |
| `nginx.ingress.kubernetes.io/backend-protocol: HTTPS` | `BackendTLSPolicy` resource |
| `nginx.ingress.kubernetes.io/proxy-body-size` | `UpstreamSettings` policy (NGF custom) |
| `nginx.ingress.kubernetes.io/proxy-request-buffering` | `ProxySettings` policy (NGF custom) |
| `nginx.ingress.kubernetes.io/proxy-buffer-size` | `UpstreamSettings` policy |
| `cert-manager.io/cluster-issuer` | cert-manager annotation on Gateway |

### 1D. cert-manager Integration

Use **approach B**: Keep existing Certificate resources, reference secrets in Gateway TLS config.

- Each module's `kubernetes_manifest` Certificate resource stays unchanged
- Gateway HTTPS listener references cert secrets via `certificateRefs`
- This avoids alpha cert-manager Gateway integration

### 1E. Migration Execution Order

1. Apply new Ansible role (installs Gateway Fabric alongside nginx-ingress)
2. Deploy Gateway resource to both clusters
3. Migrate modules on **dev-01 first** (lower risk):
   - Deploy HTTPRoutes for each module, one at a time
   - Verify each service works
   - Remove old Ingress resources
4. Migrate modules on **prod-01**:
   - Same process, one module at a time
5. Disable nginx-ingress plugin on both clusters
6. Clean up IngressClass, old LoadBalancer Service

---

## Phase 2: Cross-Cluster Connectivity

### 2A. Improve WireGuard

**Modify: `terraform/apps/prod-01/config.tf`** (WireGuard config):
- Change `allowed_ips` from `[10.19.4.106/32, 10.19.4.136/32]` → `["10.19.4.136/32"]` (dev-01 only)
- Remove gpu-01 (10.19.4.106) - ollama proxy being removed, no need to reach gpu-01
- `persistent_keepalive` already set to 25

**Modify: `terraform/modules/utils/proxy/resources/wireguard/wireguard.conf.tpl`**:
- Template `PersistentKeepalive` from config variable
- Template `AllowedIPs` from config list

**Also clean up**:
- Remove `"ollama"` entry from `proxy_services` in prod-01 config.tf
- Remove `ollama` CNAME from dev-01 DNS zones in dev-01 config.tf

**Modify: `terraform/modules/utils/proxy/main.tf`** (WireGuard container):
- Improve liveness probe: `wg show` instead of `wg show | grep -q transfer`
- Add readiness probe
- Add resource limits (memory cap)

### 2B. DNS Resolution

No changes needed. The proxy's per-location resolver in `default.conf.tpl` already handles DNS correctly for cross-cluster backends.

---

## Phase 3: Cloudflare Tunnels

### 3A. Cloudflare Setup

- Create/verify Cloudflare account with `alexpires.me` domain
- Keep DNS with ClouDNS (Cloudflare acts as reverse proxy only)
- Create Cloudflare Tunnel for prod-01
- Configure route rules: `https://*.alexpires.me` → Gateway Service
- Set up origin rules (TLS terminates at Cloudflare edge → HTTP to cloudflared)

### 3B. Deploy cloudflared

**New module: `terraform/modules/utils/cloudflare-tunnel/`**
- DaemonSet of `cloudflare/cloudflared` pods
- ConfigMap with tunnel credentials and route definitions
- Each public FQDN mapped to `http://<gateway-service>.<namespace>:80`

### 3C. Gateway Configuration Changes

- Gateway uses HTTP (:80) NodePort listener for Cloudflare Tunnel traffic
- Cloudflare Tunnel connects to `http://<node_ip>:<nodeport>`
- No HTTPS listener needed since all traffic comes through Cloudflare Tunnel

### 3D. Firewall Changes

- Close UFW ports 80/443 (cloudflared uses ephemeral outbound ports)
- Keep SSH + WireGuard ports open

### 3E. Security Benefits

With Cloudflare in front:
- No open ports on prod-01 (80, 443, etc. closed)
- DDoS protection (automatic with Cloudflare)
- WAF rules (OWASP Top 10, custom rules)
- Rate limiting on public endpoints
- Bot management

---

## Phase 4: Cleanup

- Remove nginx-ingress plugin from MicroK8s (`microk8s.disable ingress`)
- Remove unused IngressClass resources
- Remove existing MetalLB (delete metallb-system namespace, IPAddressPool, L2Advertisement)
- Delete orphaned `tasks/loadbalancer.yml` and `tasks/loadbalancer/metallb.yaml`
- Remove MetalLB vars from `defaults/main.yml`
- Update documentation

---

## File Change Summary

### New Files
| File | Purpose |
|---|---|
| `ansible/roles/microk8s/tasks/gateway-api.yml` | NGINX Gateway Fabric install |
| `terraform/modules/utils/gateway/main.tf` | Gateway + ReferenceGrant resources |
| `terraform/modules/utils/gateway/variables.tf` | Gateway configuration |
| `terraform/modules/utils/gateway/outputs.tf` | Gateway outputs |
| `terraform/modules/utils/cloudflare-tunnel/main.tf` | cloudflared DaemonSet |
| `terraform/modules/utils/cloudflare-tunnel/variables.tf` | Tunnel config |

### Deleted Files
| File | Reason |
|---|---|
| `ansible/roles/microk8s/tasks/loadbalancer.yml` | Not wired into main.yml, was for nginx-ingress LB |
| `ansible/roles/microk8s/tasks/loadbalancer/metallb.yaml` | MetalLB no longer needed (single-node, NodePort) |

### Modified Files
| File | Change |
|---|---|
| `ansible/roles/microk8s/tasks/main.yml` | Include gateway-api.yml |
| `ansible/roles/microk8s/defaults/main.yml` | Add Gateway Fabric defaults, remove MetalLB vars |
| `terraform/modules/utils/proxy/main.tf` | Ingress → HTTPRoute + TCPRoute + UDPRoute |
| `terraform/modules/utils/proxy/variables.tf` | Update for Gateway API |
| `terraform/modules/utils/proxy/resources/wireguard/wireguard.conf.tpl` | Add PersistentKeepalive |
| `terraform/modules/apps/nginx/main.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/gitea/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/m3uproxy/main.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/open-webui/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/grafana/grafana.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/mstream/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/searxng/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/vaultwarden/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/terraform-mcp/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/nextcloud/ingress.tf` | Ingress → HTTPRoute |
| `terraform/modules/apps/ssh_locker_web/` | Ingress → HTTPRoute |
| `terraform/apps/prod-01/apps.tf` | Add gateway module |
| `terraform/apps/prod-01/config.tf` | WireGuard (remove gpu-01), remove ollama proxy |
| `terraform/apps/dev-01/config.tf` | Remove ollama DNS CNAME |
| `terraform/apps/dev-01/apps.tf` | Add gateway module |

---

## Risk Assessment

| Risk | Mitigation |
|---|---|
| TCPRoute/UDPRoute experimental in NGF | Start with HTTPRoute only, keep proxy streams temporarily |
| Cloudflare Tunnel adds latency | Monitor, acceptable for reverse proxy traffic |
| cert-manager Gateway integration alpha | Use approach B (separate Certificate resources) |
| Downtime during migration | Run both controllers in parallel, switch via DNS TTL |
| Removing gpu-01 from WireGuard allowed_ips | Verify no active proxy depends on gpu-01 reachability |
| Removing existing MetalLB on prod-01 | Do after Cloudflare Tunnel is fully operational |

## Timeline Estimate

| Phase | Duration | Dependencies |
|---|---|---|
| Phase 1 (Gateway API) | 2-3 weeks | None |
| Phase 2 (WireGuard) | 1-2 weeks | Can run parallel with Phase 1 |
| Phase 3 (Cloudflare) | 1-2 weeks | After Phase 1 |
| Phase 4 (Cleanup) | 1 week | After Phase 3 |