a13labs.infra/.opencode/plans/gateway-api-migration.md

Gateway API Migration & Cloudflare Integration

Overview

Migrate prod-01 and dev-01 clusters from the deprecated nginx-ingress controller (retired March 2026) to NGINX Gateway Fabric via the Gateway API, improve cross-cluster WireGuard connectivity, and add Cloudflare Tunnel for edge protection.

Current Architecture

Internet → Contabo Public IP → OPNsense → prod-01
                              ↓
                     nginx-ingress plugin (MicroK8s)
                              ↓
                         MetalLB (10.2.0.40-49)
                              ↓
               ┌───────────────┼─────────────────┐
               ↓               ↓                  ↓
        [proxy pod]      [app pods]        [TCP streams]
      (nginx + WG)  (gitea, m3uproxy, etc)  (hostPort)
          ↓
    WireGuard → aristotle (Contabo VPN)
          ↓
      dev-01 backends (open-webui, nextcloud)

Key facts:

• 14 kubernetes_ingress_v1 resources across both clusters
• All use ingress.class: public + nginx-specific annotations
• Proxy module on prod-01 creates a double-proxy (Ingress → proxy pod → backends)
• TCP/UDP streams use nginx with hostPorts
• DNS: ClouDNS, no Cloudflare involvement
• cert-manager with DNS-01 (dev-01) and HTTP-01 (prod-01) for Let's Encrypt
• Both clusters are single-node — no need for MetalLB, NodePort is sufficient

Decisions Made

Decision	Choice	Rationale
Gateway API implementation	NGINX Gateway Fabric	Keeps nginx data plane, minimal behavioral changes, conformant v1.4.1
Cross-cluster connectivity	Improve WireGuard	Keep through aristotle, allowed_ips to dev-01 only (10.19.4.136/32), remove gpu-01
Ollama removal	Remove	gpu-01 runs llamacpp (not ollama), prod-01 ollama proxy is broken, dev-01 DNS CNAME stale
Cloudflare scope	Tunnels only	Get DDoS+WAF, keep DNS with ClouDNS, least disruptive
Proxy module	All through proxy	Gateway routes everything to proxy Service, simpler Terraform
Website module	Direct Gateway route	Keep separate pods, no proxy overhead for static sites
TLS termination	At Cloudflare edge	Simpler cert management, Cloudflare provides origin cert

Target Architecture

Internet → Cloudflare Tunnel → Cloudflare Edge (TLS termination)
                              ↓
                     cloudflared pods (DaemonSet)
                              ↓
                     Gateway (NGINX Gateway Fabric, NodePort Service)
                     HTTP listener :80 (from cloudflared)
                              ↓
               ┌───────────────┼─────────────────┐
               ↓               ↓                  ↓
        [proxy Service]  [app Services]    [TCP/UDP Routes]
               ↓               ↓                  ↓
      [proxy pod + WG]   [direct pods]      [direct pods]
          (nginx)   (website, etc.)       (rustdesk, etc.)
               ↓
         Cross-cluster → dev-01 backends
         (open-webui, nextcloud)

---

Phase 1: Gateway API Migration

1A. Install NGINX Gateway Fabric (Ansible)

New file: ansible/roles/microk8s/tasks/gateway-api.yml

- Install `snap` package manager (if needed)
- Install helm via snap
- Add nginx-stable helm repo
- Install Gateway API CRDs (standard channel + experimental for TCPRoute/UDPRoute)
- Install NGINX Gateway Fabric via Helm (experimental features enabled)
- Create GatewayClass named `public`
- Create Gateway Service with type = NodePort (single-node, no MetalLB needed)

Modify: ansible/roles/microk8s/tasks/main.yml

• Include gateway-api.yml AFTER existing setup (runs alongside nginx-ingress initially)

Modify: ansible/roles/microk8s/defaults/main.yml

• Add variables: gateway_fabric_version, gateway_fabric_experimental_enabled
• Remove MetalLB-related vars (microk8s_lb_provider, microk8s_lb_ips, microk8s_metallb_version)

1B. Create Gateway Module (Terraform)

New module: terraform/modules/utils/gateway/

• main.tf: Gateway resource with HTTP (:80) listener, ServiceType: NodePort
• variables.tf: TLS secret names, listener config
• outputs.tf: Gateway service name, NodePort port numbers

Deploy in both clusters (prod-01/apps.tf, dev-01/apps.tf)

1C. Migrate Terraform Modules (Ingress → Gateway API Resources)

Proxy module (terraform/modules/utils/proxy/main.tf)

• Remove: kubernetes_ingress_v1.ingress resource (lines 160-203)
• Remove: host_port from stream container ports in Deployment
• Remove: ollama proxy service definition from prod-01 config.tf
• Add: HTTPRoute for cross-cluster services (open-webui, nextcloud)
• Add: TCPRoute resources for RustDesk (21115, 21116, 21117 TCP) and Prometheus (9090)
• Add: UDPRoute resource for RustDesk (21116 UDP)
• The proxy pod's nginx stream block can be removed (Gateway handles L4 routing)
• Keep: WireGuard sidecar, auth sidecar, internal nginx reverse proxy config

Website module (terraform/modules/apps/nginx/main.tf)

• Remove: kubernetes_ingress_v1.website_ingress (lines 103-145)
• Add: HTTPRoute per website with backendRef → website Service
• Gateway handles TLS termination, website pods receive plain HTTP
• Website pods can drop internal TLS (simplification)

Gitea module (terraform/modules/apps/gitea/ingress.tf)

• Remove: kubernetes_ingress_v1.gitea_ingress
• Add: HTTPRoute with backendRef → Gitea Service
• Use BackendTLSPolicy if Gitea serves HTTPS internally

M3UProxy module (terraform/modules/apps/m3uproxy/main.tf)

• Remove: kubernetes_ingress_v1.m3uproxy_ingress_http (lines 241-274)
• Add: HTTPRoute with backendRef → m3uproxy Service

dev-01 modules

Module	Current	Change
open-webui/ingress.tf	kubernetes_ingress_v1	HTTPRoute
grafana/grafana.tf	kubernetes_ingress_v1	HTTPRoute
mstream/ingress.tf	kubernetes_ingress_v1	HTTPRoute
searxng/ingress.tf	kubernetes_ingress_v1	HTTPRoute
vaultwarden/ingress.tf	kubernetes_ingress_v1	HTTPRoute
terraform-mcp/ingress.tf	kubernetes_ingress_v1	HTTPRoute
nextcloud/ingress.tf	kubernetes_ingress_v1	HTTPRoute
proxy/main.tf	kubernetes_ingress_v1	HTTPRoute (opencode, vscode)

Annotation → Gateway API Mapping

Current nginx annotation	Gateway API equivalent
nginx.org/ssl-redirect: true	HTTPRoute with RequestRedirect filter
nginx.org/redirect-to-https: true	HTTPRoute on HTTP listener with redirect
nginx.ingress.kubernetes.io/backend-protocol: HTTPS	BackendTLSPolicy resource
nginx.ingress.kubernetes.io/proxy-body-size	UpstreamSettings policy (NGF custom)
nginx.ingress.kubernetes.io/proxy-request-buffering	ProxySettings policy (NGF custom)
nginx.ingress.kubernetes.io/proxy-buffer-size	UpstreamSettings policy
cert-manager.io/cluster-issuer	cert-manager annotation on Gateway

1D. cert-manager Integration

Use approach B: Keep existing Certificate resources, reference secrets in Gateway TLS config.

• Each module's kubernetes_manifest Certificate resource stays unchanged
• Gateway HTTPS listener references cert secrets via certificateRefs
• This avoids alpha cert-manager Gateway integration

1E. Migration Execution Order

1. Apply new Ansible role (installs Gateway Fabric alongside nginx-ingress)
2. Deploy Gateway resource to both clusters
3. Migrate modules on dev-01 first (lower risk):
   • Deploy HTTPRoutes for each module, one at a time
   • Verify each service works
   • Remove old Ingress resources
4. Migrate modules on prod-01:
   • Same process, one module at a time
5. Disable nginx-ingress plugin on both clusters
6. Clean up IngressClass, old LoadBalancer Service

---

Phase 2: Cross-Cluster Connectivity

2A. Improve WireGuard

Modify: terraform/apps/prod-01/config.tf (WireGuard config):

• Change allowed_ips from [10.19.4.106/32, 10.19.4.136/32] → ["10.19.4.136/32"] (dev-01 only)
• Remove gpu-01 (10.19.4.106) - ollama proxy being removed, no need to reach gpu-01
• persistent_keepalive already set to 25

Modify: terraform/modules/utils/proxy/resources/wireguard/wireguard.conf.tpl:

• Template PersistentKeepalive from config variable
• Template AllowedIPs from config list

Also clean up:

• Remove "ollama" entry from proxy_services in prod-01 config.tf
• Remove ollama CNAME from dev-01 DNS zones in dev-01 config.tf

Modify: terraform/modules/utils/proxy/main.tf (WireGuard container):

• Improve liveness probe: wg show instead of wg show | grep -q transfer
• Add readiness probe
• Add resource limits (memory cap)

2B. DNS Resolution

No changes needed. The proxy's per-location resolver in default.conf.tpl already handles DNS correctly for cross-cluster backends.

---

Phase 3: Cloudflare Tunnels

3A. Cloudflare Setup

• Create/verify Cloudflare account with alexpires.me domain
• Keep DNS with ClouDNS (Cloudflare acts as reverse proxy only)
• Create Cloudflare Tunnel for prod-01
• Configure route rules: https://*.alexpires.me → Gateway Service
• Set up origin rules (TLS terminates at Cloudflare edge → HTTP to cloudflared)

3B. Deploy cloudflared

New module: terraform/modules/utils/cloudflare-tunnel/

• DaemonSet of cloudflare/cloudflared pods
• ConfigMap with tunnel credentials and route definitions
• Each public FQDN mapped to http://<gateway-service>.<namespace>:80

3C. Gateway Configuration Changes

• Gateway uses HTTP (:80) NodePort listener for Cloudflare Tunnel traffic
• Cloudflare Tunnel connects to http://<node_ip>:<nodeport>
• No HTTPS listener needed since all traffic comes through Cloudflare Tunnel

3D. Firewall Changes

• Close UFW ports 80/443 (cloudflared uses ephemeral outbound ports)
• Keep SSH + WireGuard ports open

3E. Security Benefits

With Cloudflare in front:

• No open ports on prod-01 (80, 443, etc. closed)
• DDoS protection (automatic with Cloudflare)
• WAF rules (OWASP Top 10, custom rules)
• Rate limiting on public endpoints
• Bot management

---

Phase 4: Cleanup

• Remove nginx-ingress plugin from MicroK8s (microk8s.disable ingress)
• Remove unused IngressClass resources
• Remove existing MetalLB (delete metallb-system namespace, IPAddressPool, L2Advertisement)
• Delete orphaned tasks/loadbalancer.yml and tasks/loadbalancer/metallb.yaml
• Remove MetalLB vars from defaults/main.yml
• Update documentation

---

File Change Summary

New Files

File	Purpose
ansible/roles/microk8s/tasks/gateway-api.yml	NGINX Gateway Fabric install
terraform/modules/utils/gateway/main.tf	Gateway + ReferenceGrant resources
terraform/modules/utils/gateway/variables.tf	Gateway configuration
terraform/modules/utils/gateway/outputs.tf	Gateway outputs
terraform/modules/utils/cloudflare-tunnel/main.tf	cloudflared DaemonSet
terraform/modules/utils/cloudflare-tunnel/variables.tf	Tunnel config

Deleted Files

File	Reason
ansible/roles/microk8s/tasks/loadbalancer.yml	Not wired into main.yml, was for nginx-ingress LB
ansible/roles/microk8s/tasks/loadbalancer/metallb.yaml	MetalLB no longer needed (single-node, NodePort)

Modified Files

File	Change
ansible/roles/microk8s/tasks/main.yml	Include gateway-api.yml
ansible/roles/microk8s/defaults/main.yml	Add Gateway Fabric defaults, remove MetalLB vars
terraform/modules/utils/proxy/main.tf	Ingress → HTTPRoute + TCPRoute + UDPRoute
terraform/modules/utils/proxy/variables.tf	Update for Gateway API
terraform/modules/utils/proxy/resources/wireguard/wireguard.conf.tpl	Add PersistentKeepalive
terraform/modules/apps/nginx/main.tf	Ingress → HTTPRoute
terraform/modules/apps/gitea/ingress.tf	Ingress → HTTPRoute
terraform/modules/apps/m3uproxy/main.tf	Ingress → HTTPRoute
terraform/modules/apps/open-webui/ingress.tf	Ingress → HTTPRoute
terraform/modules/apps/grafana/grafana.tf	Ingress → HTTPRoute
terraform/modules/apps/mstream/ingress.tf	Ingress → HTTPRoute
terraform/modules/apps/searxng/ingress.tf	Ingress → HTTPRoute
terraform/modules/apps/vaultwarden/ingress.tf	Ingress → HTTPRoute
terraform/modules/apps/terraform-mcp/ingress.tf	Ingress → HTTPRoute
terraform/modules/apps/nextcloud/ingress.tf	Ingress → HTTPRoute
terraform/modules/apps/ssh_locker_web/	Ingress → HTTPRoute
terraform/apps/prod-01/apps.tf	Add gateway module
terraform/apps/prod-01/config.tf	WireGuard (remove gpu-01), remove ollama proxy
terraform/apps/dev-01/config.tf	Remove ollama DNS CNAME
terraform/apps/dev-01/apps.tf	Add gateway module

---

Risk Assessment

Risk	Mitigation
TCPRoute/UDPRoute experimental in NGF	Start with HTTPRoute only, keep proxy streams temporarily
Cloudflare Tunnel adds latency	Monitor, acceptable for reverse proxy traffic
cert-manager Gateway integration alpha	Use approach B (separate Certificate resources)
Downtime during migration	Run both controllers in parallel, switch via DNS TTL
Removing gpu-01 from WireGuard allowed_ips	Verify no active proxy depends on gpu-01 reachability
Removing existing MetalLB on prod-01	Do after Cloudflare Tunnel is fully operational

Timeline Estimate

Phase	Duration	Dependencies
Phase 1 (Gateway API)	2-3 weeks	None
Phase 2 (WireGuard)	1-2 weeks	Can run parallel with Phase 1
Phase 3 (Cloudflare)	1-2 weeks	After Phase 1
Phase 4 (Cleanup)	1 week	After Phase 3