Enhance infrastructure and application configurations

- Updated AGENTS.md to include new directories for execution plans and implemented feature reports. - Modified Ansible host variables for dev-02 and gpu-01 to change API endpoints and add new configurations for Scaleway. - Adjusted firewall rules in gpu-01 to allow HTTPS traffic instead of the previous Ollama port. - Added OAuth2 proxy configurations to gpu-01 for enhanced security. - Removed deprecated open-webui module from Terraform app configurations. - Updated Terraform configurations to reflect changes in local variables and removed unused secrets. - Created a new sectool.json file for CloudNS integration with Bitwarden. - Enhanced SELinux configurations for nginx to allow connections to any port and added oauth2-proxy port.
2026-07-12 16:28:45 -04:00
parent a4a8554fb6
commit 8724360fb2
23 changed files with 2106 additions and 77 deletions
@@ -0,0 +1,234 @@
+# Plan: Replace OpenWebUI with llama.cpp behind nginx proxy
+
+## Current State
+
+- **llama.cpp** runs on `gpu-01.lab.alexpires.me` via Podman on port `11434` (bound to `127.0.0.1` with API key)
+- **nginx proxy** role runs on `dev-02.lab.alexpires.me` and `gpu-01.lab.alexpires.me`
+- **OpenWebUI** has been decommissioned (removed from K8s, DNS, inventory)
+- **DNS** for `lab.alexpires.me` is managed via Terraform (`terraform/apps/dev-01/config.tf` zones)
+- **oauth2-proxy** template uses OIDC provider (`code.alexpires.me`), `set_xauthrequest = true`
+
+## Goals
+
+- Deploy llama.cpp behind nginx proxy at `llm.lab.alexpires.me` ✅
+- API paths (`/api/*`) accessible without OAuth2 (token auth enforced via `--api-key`) ✅
+- All other paths (chat UI, etc.) protected with OAuth2 via oauth2-proxy ✅
+- Issue SSL certificate for `llm.lab.alexpires.me` via CloudNS DNS ✅
+- OpenWebUI decommissioned ✅
+
+## Important Constraint
+
+> **Resolved.** Phase 2 completed: llama.cpp now bound to `127.0.0.1` with API key. OpenWebUI decommissioned.
+
+## Changes Required
+
+> All changes have been implemented. See execution order below for completed steps.
+
+### 1. Add DNS record for llm.lab.alexpires.me ✅
+
+**File:** `terraform/apps/dev-01/config.tf`
+
+Add A record to the `lab.alexpires.me` zone:
+```hcl
+{
+  name    = "llm"
+  type    = "A"
+  content = "10.19.4.106"
+}
+```
+
+Run `tofu plan/apply` on `terraform/apps/dev-01/` before Phase 1 so DNS resolves.
+
+### 2. Add gpu-01 to `[nginx]` inventory group ✅
+
+### 3. Update nginx-server.conf.j2 — per-site `no_auth` conditional ✅
+
+### 4. Configure gpu-01 host vars for nginx proxy ✅
+
+### 5. Add firewall rules for HTTPS ✅
+
+### 6. Update llama_server role — support configurable bind address ✅
+
+### 7. Two-phase transition ✅
+
+**Phase 1 — Deploy nginx proxy (llama.cpp still on 0.0.0.0, no API key)**
+
+- Keep `llama_server_bind: "0.0.0.0"` (current behavior, default)
+- Do NOT set API key
+- nginx proxies to `127.0.0.1:11434` (container is bound to `0.0.0.0:11434`, reachable from localhost)
+- OpenWebUI continues to work (still connects to `gpu-01:11434` directly)
+- Test: `llm.lab.alexpires.me/` requires OAuth2 login, `llm.lab.alexpires.me/api/*` is open
+
+**Phase 2 — Switch to localhost binding + enable API key (after validation)**
+
+- Change `llama_server_bind: "127.0.0.1"` in host vars
+- Add `--api-key` to `llama_server_argv_extra` in host vars:
+  ```yaml
+  llama_server_argv_extra:
+    - "--api-key"
+    - "{{ lookup('env', 'LLAMA_SERVER_API_KEY') }}"
+    # ... existing args ...
+  ```
+- OpenWebUI will break (can no longer reach llama.cpp on `gpu-01:11434`) — decommission OpenWebUI
+- Verify all consumers migrate to `llm.lab.alexpires.me/api/*` with `Authorization: Bearer <key>`
+- Remove old firewall rules for port 11434 (no longer externally reachable)
+
+## Execution Order
+
+### Phase 1 — Deploy nginx proxy (llama.cpp unchanged, OpenWebUI still works)
+
+> **Status: COMPLETE** — All steps executed and verified.
+
+1. **Update Terraform DNS** — add `llm` A record in `terraform/apps/dev-01/config.tf`, run `tofu apply` ✅
+2. **Update inventory** — add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` group ✅
+3. **Update nginx-server.conf.j2** — add `no_auth` per-site conditional (template stays generic) ✅
+4. **Update llama_server role** — add `llama_server_bind` variable (default `0.0.0.0`) ✅
+5. **Update host vars** — add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules ✅
+6. **Run nginx proxy playbook** — installs nginx + oauth2-proxy + certbot + deploys config ✅
+7. **Fix SELinux** — add `nginx_connect_any` boolean + port 4180 to `http_port_t` (manually applied, added to role for persistence) ✅
+8. **Verify chat UI** — `llm.lab.alexpires.me/` returns oauth2-proxy login page (200) ✅
+9. **Verify API** — `llm.lab.alexpires.me/api/v1/models` returns 200 with models list ✅
+10. **Verify OpenWebUI still works** — llama.cpp still bound to `0.0.0.0:11434` ✅
+
+1. **Update Terraform DNS** — add `llm` A record in `terraform/apps/dev-01/config.tf`, run `tofu apply`
+2. **Update inventory** — add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` group
+3. **Update nginx-server.conf.j2** — add `no_auth` per-site conditional (template stays generic)
+4. **Update llama_server role** — add `llama_server_bind` variable (default `0.0.0.0`)
+5. **Update host vars** — add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules
+6. **Run nginx proxy playbook** — installs nginx + oauth2-proxy + certbot + deploys config (all in one run)
+7. **Verify chat UI** — test `llm.lab.alexpires.me/` with OAuth2 login
+8. **Verify API** — test `llm.lab.alexpires.me/api/*` (no auth yet, same as current direct access)
+9. **Verify OpenWebUI still works** — confirm dev-01 still connects to gpu-01:11434
+
+### Phase 2 — Switch to localhost + API key (after validation, decommission OpenWebUI)
+
+> **Status: COMPLETE** — All steps executed and verified.
+
+10. **Update host vars** — change `llama_server_bind: "127.0.0.1"`, add `--api-key` to `llama_server_argv_extra` ✅
+11. **Update firewall** — remove port 11434 rules from `fw_allowed_ports` ✅
+12. **Run llama_server playbook** — restart container with new binding + API key ✅
+13. **Run hardening playbook** — `playbook_hardening_cis.yml --limit gpu-01.lab.alexpires.me -t ufw` to apply firewall changes ✅
+14. **Verify API** — `llm.lab.alexpires.me/api/v1/chat/completions` returns 401 without key, 200 with key ✅
+15. **Decommission OpenWebUI** — removed from `terraform/apps/dev-01/apps.tf` ✅
+16. **Remove OpenWebUI DNS records** — removed `open-webui` CNAME from `terraform/apps/dev-01/config.tf` zones ✅
+17. **Remove OpenWebUI locals** — removed `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` from config.tf ✅
+18. **Remove `webui_secret_key` variable** — removed from `terraform/apps/dev-01/secrets.tf` ✅
+19. **Remove `[openwebui]` group** — cleaned up `inventory/hosts` ✅
+20. **Run `tofu apply`** on `terraform/apps/dev-01/` — destroyed 8 OpenWebUI K8s resources (namespace, deployment, service, ingress, secret, service accounts) ✅
+
+### Phase 2 — Switch to localhost + API key (after validation, decommission OpenWebUI)
+
+10. **Update host vars** — change `llama_server_bind: "127.0.0.1"`, add `--api-key` to `llama_server_argv_extra`
+11. **Update firewall** — remove port 11434 rules from `fw_allowed_ports`
+12. **Run llama_server playbook** — restart container with new binding + API key
+13. **Run hardening playbook** — `playbook_hardening_cis.yml --limit gpu-01.lab.alexpires.me -t ufw` to apply firewall changes
+14. **Verify API** — test `llm.lab.alexpires.me/api/*` requires API key
+15. **Decommission OpenWebUI** — remove Terraform module from `terraform/apps/dev-01/apps.tf` and `config.tf`
+16. **Remove OpenWebUI DNS records** — clean up `open-webui` CNAME in `terraform/apps/dev-01/config.tf` zones
+17. **Remove `[openwebui]` group** — clean up `inventory/hosts`
+18. **Run `tofu apply`** on `terraform/apps/dev-01/` — destroys OpenWebUI K8s resources
+
+## Playbooks to run
+
+### Phase 1
+
+> **COMPLETE** — All steps executed and verified.
+
+```sh
+cd ansible
+sectool exec ansible-playbook -u $ANSIBLE_USER playbook_nginx_proxy.yml    # nginx + oauth2 + certbot (all-in-one)
+```
+
+### Phase 2
+
+> **COMPLETE** — All steps executed and verified.
+
+```sh
+cd ansible
+sectool exec ansible-playbook -u $ANSIBLE_USER playbook_llama_server.yml                # container binding + API key
+sectool exec ansible-playbook -u $ANSIBLE_USER playbook_hardening_cis.yml -t ufw        # remove port 11434 rules
+```
+
+Then in terraform:
+```sh
+cd terraform/apps/dev-01
+sectool exec tofu plan    # review OpenWebUI destruction
+sectool exec tofu apply --auto-approve   # apply
+```
+
+## Files to modify
+
+| File | Phase | Change |
+|------|-------|--------|
+| `terraform/apps/dev-01/config.tf` | 1 | Add `llm` A record to zones |
+| `inventory/hosts` | 1 | Add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` |
+| `ansible/roles/nginx_proxy/templates/nginx-server.conf.j2` | 1 | Add `no_auth` per-site conditional (3 blocks) |
+| `ansible/roles/llama_server/defaults/main.yml` | 1 | Add `llama_server_bind` variable (default `0.0.0.0`) |
+| `ansible/roles/llama_server/tasks/el/main.yml` | 1 | Use `llama_server_bind` in podman port mapping |
+| `ansible/roles/nginx_proxy/tasks/nginx.yml` | 1 | Add SELinux tasks (`nginx_connect_any` boolean + oauth2-proxy port to `http_port_t`) |
+| `ansible/host_vars/gpu-01.lab.alexpires.me.yml` | 1 | Add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules |
+| `ansible/host_vars/gpu-01.lab.alexpires.me.yml` | 2 | Change `llama_server_bind` to `127.0.0.1`, add `--api-key` to `llama_server_argv_extra`, remove port 11434 fw rules ✅ |
+| `terraform/apps/dev-01/apps.tf` | 2 | Remove `module "open-webui"` block ✅ |
+| `terraform/apps/dev-01/config.tf` | 2 | Remove `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` locals; remove `open-webui` DNS record from zones ✅ |
+| `terraform/apps/dev-01/secrets.tf` | 2 | Remove `webui_secret_key` variable ✅ |
+| `inventory/hosts` | 2 | Remove `[openwebui]` group ✅ |
+
+## Rollback plan
+
+### Phase 1 rollback (nginx proxy broken)
+
+- Remove gpu-01 from `[nginx]` group in `inventory/hosts`
+- Re-run hardening playbook on gpu-01 to clean up nginx/oauth2-proxy packages
+- Revert `nginx-server.conf.j2` template changes
+- llama.cpp continues running on `0.0.0.0:11434` — no impact
+- OpenWebUI continues to work
+
+## Issues Encountered & Resolved
+
+### 1. nginx `copy` module missing `remote_src: true`
+**File:** `ansible/roles/nginx_proxy/tasks/oauth2_proxy.yml:16`
+The `copy` module was trying to copy from the controller instead of the remote host. Added `remote_src: true`.
+
+### 2. SELinux blocking nginx → oauth2-proxy / llama.cpp connections
+**Symptom:** `connect() to 127.0.0.1:4180 failed (13: Permission denied)` in nginx error log
+**Fix:** 
+- Set SELinux boolean `nginx_connect_any` to `true` (persistent)
+- Added port 4180 to `http_port_t` via `semanage port`
+- Added both as Ansible tasks in `nginx.yml` for persistence
+
+### 3. Template `no_auth` attribute error
+**Symptom:** `object of type 'dict' has no attribute 'no_auth'`
+**Fix:** Changed `{% if not item.no_auth %}` to `{% if not (item.no_auth | default(false) | bool) %}` to handle missing attribute on dicts.
+
+### 4. Playbook failed on model query after container restart (Phase 2)
+**Symptom:** `urlopen error [Errno 111] Connection refused` when querying `http://127.0.0.1:11434/models?reload=1`
+**Cause:** The playbook's "Query llama router model list" task ran before the container was fully started. The container did start correctly with `127.0.0.1:11434` binding and API key active.
+**Impact:** Non-blocking — the container was running correctly; only the model list query task failed.
+
+## Known Warnings (non-blocking)
+
+- `types_hash_bucket_size: 64` — nginx warning, non-blocking
+- oauth2-proxy PKCE warning (`--code-challenge-method`) — non-blocking, S256 is auto-negotiated
+
+## Rollback plan
+
+### Phase 1 rollback (nginx proxy broken)
+
+- Revert `llama_server_bind` to `0.0.0.0` in host vars
+- Remove `--api-key` from `llama_server_argv_extra`
+- Re-add port 11434 firewall rules
+- Re-run `playbook_llama_server.yml` + `playbook_hardening_cis.yml -t ufw` — container restarts with old settings
+- OpenWebUI still works (llama.cpp is back on `0.0.0.0` without API key)
+- Fix issues and re-attempt Phase 2
+
+### Phase 2 rollback (after OpenWebUI decommission)
+
+- Restore `llama_server_bind: "0.0.0.0"` in host vars
+- Remove `--api-key` from `llama_server_argv_extra`
+- Re-add port 11434 firewall rules
+- Restore `module "open-webui"` in `terraform/apps/dev-01/apps.tf`
+- Restore `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` locals in `config.tf`
+- Restore `open-webui` DNS CNAME record in `config.tf` zones
+- Restore `webui_secret_key` variable in `secrets.tf`
+- Re-add `[openwebui]` group in `inventory/hosts`
+- Run `tofu apply` to recreate OpenWebUI K8s resources