# Plan: Replace OpenWebUI with llama.cpp behind nginx proxy

## Current State

- **llama.cpp** runs on `gpu-01.lab.alexpires.me` via Podman on port `11434` (bound to `127.0.0.1` with API key)
- **nginx proxy** role runs on `dev-02.lab.alexpires.me` and `gpu-01.lab.alexpires.me`
- **OpenWebUI** has been decommissioned (removed from K8s, DNS, inventory)
- **DNS** for `lab.alexpires.me` is managed via Terraform (`terraform/apps/dev-01/config.tf` zones)
- **oauth2-proxy** template uses OIDC provider (`code.alexpires.me`), `set_xauthrequest = true`

## Goals

- Deploy llama.cpp behind nginx proxy at `llm.lab.alexpires.me` ✅
- API paths (`/api/*`) accessible without OAuth2 (token auth enforced via `--api-key`) ✅
- All other paths (chat UI, etc.) protected with OAuth2 via oauth2-proxy ✅
- Issue SSL certificate for `llm.lab.alexpires.me` via CloudNS DNS ✅
- OpenWebUI decommissioned ✅

## Important Constraint

> **Resolved.** Phase 2 completed: llama.cpp now bound to `127.0.0.1` with API key. OpenWebUI decommissioned.

## Changes Required

> All changes have been implemented. See execution order below for completed steps.

### 1. Add DNS record for llm.lab.alexpires.me ✅

**File:** `terraform/apps/dev-01/config.tf`

Add A record to the `lab.alexpires.me` zone:
```hcl
{
  name    = "llm"
  type    = "A"
  content = "10.19.4.106"
}
```

Run `tofu plan/apply` on `terraform/apps/dev-01/` before Phase 1 so DNS resolves.

### 2. Add gpu-01 to `[nginx]` inventory group ✅

### 3. Update nginx-server.conf.j2 — per-site `no_auth` conditional ✅

### 4. Configure gpu-01 host vars for nginx proxy ✅

### 5. Add firewall rules for HTTPS ✅

### 6. Update llama_server role — support configurable bind address ✅

### 7. Two-phase transition ✅

**Phase 1 — Deploy nginx proxy (llama.cpp still on 0.0.0.0, no API key)**

- Keep `llama_server_bind: "0.0.0.0"` (current behavior, default)
- Do NOT set API key
- nginx proxies to `127.0.0.1:11434` (container is bound to `0.0.0.0:11434`, reachable from localhost)
- OpenWebUI continues to work (still connects to `gpu-01:11434` directly)
- Test: `llm.lab.alexpires.me/` requires OAuth2 login, `llm.lab.alexpires.me/api/*` is open

**Phase 2 — Switch to localhost binding + enable API key (after validation)**

- Change `llama_server_bind: "127.0.0.1"` in host vars
- Add `--api-key` to `llama_server_argv_extra` in host vars:
  ```yaml
  llama_server_argv_extra:
    - "--api-key"
    - "{{ lookup('env', 'LLAMA_SERVER_API_KEY') }}"
    # ... existing args ...
  ```
- OpenWebUI will break (can no longer reach llama.cpp on `gpu-01:11434`) — decommission OpenWebUI
- Verify all consumers migrate to `llm.lab.alexpires.me/api/*` with `Authorization: Bearer <key>`
- Remove old firewall rules for port 11434 (no longer externally reachable)

## Execution Order

### Phase 1 — Deploy nginx proxy (llama.cpp unchanged, OpenWebUI still works)

> **Status: COMPLETE** — All steps executed and verified.

1. **Update Terraform DNS** — add `llm` A record in `terraform/apps/dev-01/config.tf`, run `tofu apply` ✅
2. **Update inventory** — add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` group ✅
3. **Update nginx-server.conf.j2** — add `no_auth` per-site conditional (template stays generic) ✅
4. **Update llama_server role** — add `llama_server_bind` variable (default `0.0.0.0`) ✅
5. **Update host vars** — add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules ✅
6. **Run nginx proxy playbook** — installs nginx + oauth2-proxy + certbot + deploys config ✅
7. **Fix SELinux** — add `nginx_connect_any` boolean + port 4180 to `http_port_t` (manually applied, added to role for persistence) ✅
8. **Verify chat UI** — `llm.lab.alexpires.me/` returns oauth2-proxy login page (200) ✅
9. **Verify API** — `llm.lab.alexpires.me/api/v1/models` returns 200 with models list ✅
10. **Verify OpenWebUI still works** — llama.cpp still bound to `0.0.0.0:11434` ✅

1. **Update Terraform DNS** — add `llm` A record in `terraform/apps/dev-01/config.tf`, run `tofu apply`
2. **Update inventory** — add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` group
3. **Update nginx-server.conf.j2** — add `no_auth` per-site conditional (template stays generic)
4. **Update llama_server role** — add `llama_server_bind` variable (default `0.0.0.0`)
5. **Update host vars** — add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules
6. **Run nginx proxy playbook** — installs nginx + oauth2-proxy + certbot + deploys config (all in one run)
7. **Verify chat UI** — test `llm.lab.alexpires.me/` with OAuth2 login
8. **Verify API** — test `llm.lab.alexpires.me/api/*` (no auth yet, same as current direct access)
9. **Verify OpenWebUI still works** — confirm dev-01 still connects to gpu-01:11434

### Phase 2 — Switch to localhost + API key (after validation, decommission OpenWebUI)

> **Status: COMPLETE** — All steps executed and verified.

10. **Update host vars** — change `llama_server_bind: "127.0.0.1"`, add `--api-key` to `llama_server_argv_extra` ✅
11. **Update firewall** — remove port 11434 rules from `fw_allowed_ports` ✅
12. **Run llama_server playbook** — restart container with new binding + API key ✅
13. **Run hardening playbook** — `playbook_hardening_cis.yml --limit gpu-01.lab.alexpires.me -t ufw` to apply firewall changes ✅
14. **Verify API** — `llm.lab.alexpires.me/api/v1/chat/completions` returns 401 without key, 200 with key ✅
15. **Decommission OpenWebUI** — removed from `terraform/apps/dev-01/apps.tf` ✅
16. **Remove OpenWebUI DNS records** — removed `open-webui` CNAME from `terraform/apps/dev-01/config.tf` zones ✅
17. **Remove OpenWebUI locals** — removed `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` from config.tf ✅
18. **Remove `webui_secret_key` variable** — removed from `terraform/apps/dev-01/secrets.tf` ✅
19. **Remove `[openwebui]` group** — cleaned up `inventory/hosts` ✅
20. **Run `tofu apply`** on `terraform/apps/dev-01/` — destroyed 8 OpenWebUI K8s resources (namespace, deployment, service, ingress, secret, service accounts) ✅

### Phase 2 — Switch to localhost + API key (after validation, decommission OpenWebUI)

10. **Update host vars** — change `llama_server_bind: "127.0.0.1"`, add `--api-key` to `llama_server_argv_extra`
11. **Update firewall** — remove port 11434 rules from `fw_allowed_ports`
12. **Run llama_server playbook** — restart container with new binding + API key
13. **Run hardening playbook** — `playbook_hardening_cis.yml --limit gpu-01.lab.alexpires.me -t ufw` to apply firewall changes
14. **Verify API** — test `llm.lab.alexpires.me/api/*` requires API key
15. **Decommission OpenWebUI** — remove Terraform module from `terraform/apps/dev-01/apps.tf` and `config.tf`
16. **Remove OpenWebUI DNS records** — clean up `open-webui` CNAME in `terraform/apps/dev-01/config.tf` zones
17. **Remove `[openwebui]` group** — clean up `inventory/hosts`
18. **Run `tofu apply`** on `terraform/apps/dev-01/` — destroys OpenWebUI K8s resources

## Playbooks to run

### Phase 1

> **COMPLETE** — All steps executed and verified.

```sh
cd ansible
sectool exec ansible-playbook -u $ANSIBLE_USER playbook_nginx_proxy.yml    # nginx + oauth2 + certbot (all-in-one)
```

### Phase 2

> **COMPLETE** — All steps executed and verified.

```sh
cd ansible
sectool exec ansible-playbook -u $ANSIBLE_USER playbook_llama_server.yml                # container binding + API key
sectool exec ansible-playbook -u $ANSIBLE_USER playbook_hardening_cis.yml -t ufw        # remove port 11434 rules
```

Then in terraform:
```sh
cd terraform/apps/dev-01
sectool exec tofu plan    # review OpenWebUI destruction
sectool exec tofu apply --auto-approve   # apply
```

## Files to modify

| File | Phase | Change |
|------|-------|--------|
| `terraform/apps/dev-01/config.tf` | 1 | Add `llm` A record to zones |
| `inventory/hosts` | 1 | Add gpu-01 to `[nginx]` group, clean up duplicate `[certbot]` |
| `ansible/roles/nginx_proxy/templates/nginx-server.conf.j2` | 1 | Add `no_auth` per-site conditional (3 blocks) |
| `ansible/roles/llama_server/defaults/main.yml` | 1 | Add `llama_server_bind` variable (default `0.0.0.0`) |
| `ansible/roles/llama_server/tasks/el/main.yml` | 1 | Use `llama_server_bind` in podman port mapping |
| `ansible/roles/nginx_proxy/tasks/nginx.yml` | 1 | Add SELinux tasks (`nginx_connect_any` boolean + oauth2-proxy port to `http_port_t`) |
| `ansible/host_vars/gpu-01.lab.alexpires.me.yml` | 1 | Add nginx sites (with `extra_locations` for `/api/`), OAuth2 creds, firewall rules |
| `ansible/host_vars/gpu-01.lab.alexpires.me.yml` | 2 | Change `llama_server_bind` to `127.0.0.1`, add `--api-key` to `llama_server_argv_extra`, remove port 11434 fw rules ✅ |
| `terraform/apps/dev-01/apps.tf` | 2 | Remove `module "open-webui"` block ✅ |
| `terraform/apps/dev-01/config.tf` | 2 | Remove `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` locals; remove `open-webui` DNS record from zones ✅ |
| `terraform/apps/dev-01/secrets.tf` | 2 | Remove `webui_secret_key` variable ✅ |
| `inventory/hosts` | 2 | Remove `[openwebui]` group ✅ |

## Rollback plan

### Phase 1 rollback (nginx proxy broken)

- Remove gpu-01 from `[nginx]` group in `inventory/hosts`
- Re-run hardening playbook on gpu-01 to clean up nginx/oauth2-proxy packages
- Revert `nginx-server.conf.j2` template changes
- llama.cpp continues running on `0.0.0.0:11434` — no impact
- OpenWebUI continues to work

## Issues Encountered & Resolved

### 1. nginx `copy` module missing `remote_src: true`
**File:** `ansible/roles/nginx_proxy/tasks/oauth2_proxy.yml:16`
The `copy` module was trying to copy from the controller instead of the remote host. Added `remote_src: true`.

### 2. SELinux blocking nginx → oauth2-proxy / llama.cpp connections
**Symptom:** `connect() to 127.0.0.1:4180 failed (13: Permission denied)` in nginx error log
**Fix:** 
- Set SELinux boolean `nginx_connect_any` to `true` (persistent)
- Added port 4180 to `http_port_t` via `semanage port`
- Added both as Ansible tasks in `nginx.yml` for persistence

### 3. Template `no_auth` attribute error
**Symptom:** `object of type 'dict' has no attribute 'no_auth'`
**Fix:** Changed `{% if not item.no_auth %}` to `{% if not (item.no_auth | default(false) | bool) %}` to handle missing attribute on dicts.

### 4. Playbook failed on model query after container restart (Phase 2)
**Symptom:** `urlopen error [Errno 111] Connection refused` when querying `http://127.0.0.1:11434/models?reload=1`
**Cause:** The playbook's "Query llama router model list" task ran before the container was fully started. The container did start correctly with `127.0.0.1:11434` binding and API key active.
**Impact:** Non-blocking — the container was running correctly; only the model list query task failed.

## Known Warnings (non-blocking)

- `types_hash_bucket_size: 64` — nginx warning, non-blocking
- oauth2-proxy PKCE warning (`--code-challenge-method`) — non-blocking, S256 is auto-negotiated

## Rollback plan

### Phase 1 rollback (nginx proxy broken)

- Revert `llama_server_bind` to `0.0.0.0` in host vars
- Remove `--api-key` from `llama_server_argv_extra`
- Re-add port 11434 firewall rules
- Re-run `playbook_llama_server.yml` + `playbook_hardening_cis.yml -t ufw` — container restarts with old settings
- OpenWebUI still works (llama.cpp is back on `0.0.0.0` without API key)
- Fix issues and re-attempt Phase 2

### Phase 2 rollback (after OpenWebUI decommission)

- Restore `llama_server_bind: "0.0.0.0"` in host vars
- Remove `--api-key` from `llama_server_argv_extra`
- Re-add port 11434 firewall rules
- Restore `module "open-webui"` in `terraform/apps/dev-01/apps.tf`
- Restore `open_webui_fqdn`, `open_webui_log_level`, `llm_proxy` locals in `config.tf`
- Restore `open-webui` DNS CNAME record in `config.tf` zones
- Restore `webui_secret_key` variable in `secrets.tf`
- Re-add `[openwebui]` group in `inventory/hosts`
- Run `tofu apply` to recreate OpenWebUI K8s resources