feat: Add SearXNG module and update infrastructure
- Introduced a new SearXNG module with configuration for Kubernetes deployment, service, and ingress. - Updated existing Terraform app modules to include SearXNG in both dev-01 and prod-01 environments. - Added necessary variables and secrets for SearXNG in the respective Terraform files. - Updated Dockerfiles for llama.cpp and ollama to use latest versions. - Enhanced security and hardening defaults in Kubernetes deployments. - Improved documentation for Copilot instructions and Terraform workflows.
This commit is contained in:
@@ -0,0 +1,301 @@
|
||||
---
|
||||
name: terraform-develop-app
|
||||
description: >
|
||||
Use when developing, deploying, or debugging Terraform app modules in this
|
||||
repository. Covers module scaffolding, secrets injection with sectool and
|
||||
TF_VAR, OpenTofu command workflow, Kubernetes runtime debugging with kubectl,
|
||||
hardening defaults for pod and container security contexts, and known quirks
|
||||
such as the SearXNG IPv6 Granian bind issue. Also covers integrating modules
|
||||
into dev-01 and prod-01 app roots including DNS records, sectool.env mapping,
|
||||
and targeted apply recovery patterns.
|
||||
---
|
||||
|
||||
# Terraform App Module Skill
|
||||
|
||||
## Purpose
|
||||
|
||||
Use this skill when developing, deploying, or debugging Terraform app modules in this repository. It covers module authoring conventions, secrets injection with sectool, OpenTofu command workflow, Kubernetes runtime debugging, and known behavioral quirks discovered in production.
|
||||
|
||||
---
|
||||
|
||||
## Repository Layout
|
||||
|
||||
```
|
||||
terraform/
|
||||
apps/
|
||||
dev-01/ # dev cluster root
|
||||
prod-01/ # prod cluster root
|
||||
modules/
|
||||
apps/ # reusable app modules
|
||||
db/ # database modules
|
||||
utils/ # utility modules
|
||||
```
|
||||
|
||||
App roots are the deployment entry points. Reusable logic lives in modules and is consumed by app roots.
|
||||
|
||||
Use `terraform/modules/apps/template` as the baseline shape when creating a new module.
|
||||
|
||||
---
|
||||
|
||||
## Cluster References
|
||||
|
||||
| Name | Folder | Kubernetes context |
|
||||
|--------|--------------------------|----------------------|
|
||||
| dev-01 | terraform/apps/dev-01 | microk8s-dev-01 |
|
||||
| pro-01 | terraform/apps/prod-01 | microk8s-prod-01 |
|
||||
|
||||
When planning or reviewing changes, always consider both clusters. The production folder is named `prod-01`; discussions may call it `pro-01`.
|
||||
|
||||
---
|
||||
|
||||
## Creating a New App Module
|
||||
|
||||
Create `terraform/modules/apps/<name>/` with these files:
|
||||
|
||||
### provider.tf
|
||||
|
||||
```hcl
|
||||
terraform {
|
||||
required_version = "~>1.8"
|
||||
required_providers {
|
||||
kubernetes = {
|
||||
source = "hashicorp/kubernetes"
|
||||
version = "2.36.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### variables.tf
|
||||
|
||||
Declare at minimum:
|
||||
|
||||
```hcl
|
||||
variable "persistent_folder" {
|
||||
description = "Path for persistent data on the host"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "fqdn" {
|
||||
description = "FQDN for the service"
|
||||
type = string
|
||||
}
|
||||
|
||||
variable "tag" {
|
||||
description = "Image tag to deploy"
|
||||
type = string
|
||||
default = "x.y.z" # always pin, never use latest
|
||||
}
|
||||
```
|
||||
|
||||
Add sensitive variables at module level only when the module itself needs to inject them into a Kubernetes secret. If the secret is passed from the app root, declare it sensitive there.
|
||||
|
||||
### config.tf
|
||||
|
||||
Use locals to normalize inputs. Do not append the app name to `persistent_folder` if the caller already passes an app-specific path:
|
||||
|
||||
```hcl
|
||||
locals {
|
||||
app_folder = var.persistent_folder
|
||||
app_version = var.tag
|
||||
app_fqdn = var.fqdn
|
||||
}
|
||||
```
|
||||
|
||||
### main.tf resource order
|
||||
|
||||
1. `kubernetes_namespace`
|
||||
2. `kubernetes_service_account` — with `automount_service_account_token = false`
|
||||
3. `kubernetes_config_map` (non-sensitive config)
|
||||
4. `kubernetes_secret` (sensitive data)
|
||||
5. `kubernetes_deployment`
|
||||
6. `kubernetes_service`
|
||||
|
||||
### ingress.tf (when HTTP/HTTPS exposed)
|
||||
|
||||
```hcl
|
||||
resource "kubernetes_ingress_v1" "<name>" {
|
||||
metadata {
|
||||
annotations = {
|
||||
"kubernetes.io/ingress.class" = "public"
|
||||
"cert-manager.io/cluster-issuer" = var.issuer
|
||||
"kubernetes.io/tls-acme" = "true"
|
||||
"nginx.ingress.kubernetes.io/ssl-redirect" = "true"
|
||||
}
|
||||
}
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
Accept `issuer` as a variable (default `"letsencrypt-prod"`) so the same module works for both clusters and internal CA scenarios.
|
||||
|
||||
---
|
||||
|
||||
## Hardening Defaults
|
||||
|
||||
Apply these to every new deployment unless there is a documented exception:
|
||||
|
||||
```hcl
|
||||
spec {
|
||||
automount_service_account_token = false
|
||||
|
||||
security_context {
|
||||
run_as_non_root = true
|
||||
run_as_user = <uid>
|
||||
run_as_group = <gid>
|
||||
fs_group = <gid>
|
||||
seccomp_profile {
|
||||
type = "RuntimeDefault"
|
||||
}
|
||||
}
|
||||
|
||||
container {
|
||||
security_context {
|
||||
allow_privilege_escalation = false
|
||||
privileged = false
|
||||
read_only_root_filesystem = true
|
||||
capabilities {
|
||||
drop = ["ALL"]
|
||||
}
|
||||
}
|
||||
|
||||
# provide writable scratch via emptyDir, not root filesystem
|
||||
volume_mount {
|
||||
name = "<name>-tmp"
|
||||
mount_path = "/tmp"
|
||||
}
|
||||
}
|
||||
|
||||
volume {
|
||||
name = "<name>-tmp"
|
||||
empty_dir {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integrating a Module into an App Root
|
||||
|
||||
Do this for each cluster root:
|
||||
|
||||
1. Add a `module` block in `apps.tf`:
|
||||
```hcl
|
||||
module "myapp" {
|
||||
source = "../../modules/apps/myapp"
|
||||
persistent_folder = local.myapp_folder
|
||||
fqdn = local.myapp_fqdn
|
||||
tag = "x.y.z"
|
||||
secret_key = var.myapp_secret_key
|
||||
}
|
||||
```
|
||||
|
||||
2. Add locals in `config.tf`:
|
||||
```hcl
|
||||
myapp_folder = "/mnt/md0/myapp"
|
||||
myapp_fqdn = "myapp.${local.lab_domain}"
|
||||
```
|
||||
|
||||
3. Add a DNS CNAME record inside `local.zones[0].records`:
|
||||
```hcl
|
||||
{ name = "myapp", type = "CNAME", content = "dev-01.${local.lab_domain}." }
|
||||
```
|
||||
|
||||
4. Add sensitive variable in `secrets.tf`:
|
||||
```hcl
|
||||
variable "myapp_secret_key" {
|
||||
description = "Secret key for MyApp"
|
||||
type = string
|
||||
sensitive = true
|
||||
}
|
||||
```
|
||||
|
||||
5. Map the env variable in `sectool.env`:
|
||||
```
|
||||
TF_VAR_myapp_secret_key=$MYAPP_SECRET_KEY
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Secrets Pattern (sectool + TF_VAR)
|
||||
|
||||
Secrets are sourced from Bitwarden via `sectool` using `sectool.json` at the repository root.
|
||||
|
||||
Flow:
|
||||
1. Store secret value in Bitwarden under the project referenced in `sectool.json`.
|
||||
2. Declare the variable in `secrets.tf` with `sensitive = true`.
|
||||
3. Map `TF_VAR_<name>=$ENV_VAR_NAME` in `terraform/apps/<cluster>/sectool.env`.
|
||||
4. Run all OpenTofu commands via `sectool exec` so variables are injected automatically.
|
||||
|
||||
---
|
||||
|
||||
## OpenTofu Workflow
|
||||
|
||||
Run from the relevant app root (e.g., `terraform/apps/dev-01`):
|
||||
|
||||
```bash
|
||||
# Initialise
|
||||
sectool -f ../../../sectool.json exec tofu init
|
||||
|
||||
# Format and validate
|
||||
tofu fmt
|
||||
sectool -f ../../../sectool.json exec tofu validate
|
||||
|
||||
# Plan
|
||||
sectool -f ../../../sectool.json exec tofu plan
|
||||
|
||||
# Apply
|
||||
sectool -f ../../../sectool.json exec tofu apply -auto-approve
|
||||
|
||||
# Targeted apply (emergency recovery only — always follow with a full plan)
|
||||
sectool -f ../../../sectool.json exec tofu apply -auto-approve -target=module.<name>
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Runtime Debugging Playbook
|
||||
|
||||
When a pod is in CrashLoopBackOff:
|
||||
|
||||
```bash
|
||||
# 1. Inspect state and events
|
||||
kubectl -n <ns> get pods -o wide
|
||||
kubectl -n <ns> describe pod <pod>
|
||||
|
||||
# 2. Check current and previous logs
|
||||
kubectl -n <ns> logs <pod> --all-containers=true --tail=200
|
||||
kubectl -n <ns> logs <pod> --previous --all-containers=true --tail=200
|
||||
|
||||
# 3. Validate rollout after a fix
|
||||
kubectl -n <ns> rollout status deployment/<name> --timeout=120s
|
||||
|
||||
# 4. Test a runtime env override before persisting in Terraform
|
||||
kubectl -n <ns> set env deployment/<name> KEY=value
|
||||
# verify recovery, then add KEY to the module ConfigMap and re-apply
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known Quirks
|
||||
|
||||
### SearXNG — IPv6 bind failure
|
||||
|
||||
The `searxng/searxng` image defaults to `GRANIAN_HOST=::` (IPv6). On nodes without IPv6 socket support this causes an immediate crash:
|
||||
|
||||
```
|
||||
RuntimeError: Address family not supported by protocol (os error 97)
|
||||
```
|
||||
|
||||
Fix: set `GRANIAN_HOST = "0.0.0.0"` in the module ConfigMap.
|
||||
|
||||
---
|
||||
|
||||
## Review Checklist
|
||||
|
||||
Before proposing or applying any Terraform change:
|
||||
|
||||
- [ ] Changes scoped to both dev-01 and prod-01 where relevant.
|
||||
- [ ] Sensitive inputs declared in `secrets.tf` + mapped in `sectool.env`.
|
||||
- [ ] Image tag is pinned (not `latest`).
|
||||
- [ ] Hardening defaults present (non-root, drop-all capabilities, read-only root fs).
|
||||
- [ ] Commands in examples use `sectool -f ../../../sectool.json exec tofu`.
|
||||
@@ -0,0 +1,115 @@
|
||||
---
|
||||
name: terraform-update-app-version
|
||||
description: >
|
||||
Use when updating a container image version for an app managed by Terraform in
|
||||
this repository. Covers how to find where the tag is set, update dev-01 and
|
||||
prod-01 app roots safely, run sectool-wrapped OpenTofu validation and plan,
|
||||
apply changes, and verify rollout with kubectl.
|
||||
---
|
||||
|
||||
# Terraform Update App Version Skill
|
||||
|
||||
## Purpose
|
||||
|
||||
Use this skill when you need to update the deployed container image version for an app module.
|
||||
|
||||
This repository usually sets the image tag in the app root module call (`terraform/apps/<cluster>/apps.tf`) and passes it to the module variable `tag`, then to `local.app_version` inside the module.
|
||||
|
||||
Always use pinned tags. Do not use `latest`.
|
||||
|
||||
---
|
||||
|
||||
## Scope
|
||||
|
||||
Update both clusters unless the request explicitly says otherwise:
|
||||
|
||||
- dev-01: `terraform/apps/dev-01`
|
||||
- pro-01 folder: `terraform/apps/prod-01`
|
||||
|
||||
Note: production is often called "pro-01" in discussion, but the folder name is `prod-01`.
|
||||
|
||||
---
|
||||
|
||||
## Where to Change the Version
|
||||
|
||||
Common location:
|
||||
|
||||
- `terraform/apps/<cluster>/apps.tf`
|
||||
|
||||
Pattern:
|
||||
|
||||
```hcl
|
||||
module "<app_name>" {
|
||||
source = "../../modules/apps/<app_name>"
|
||||
...
|
||||
tag = "<image-tag>"
|
||||
}
|
||||
```
|
||||
|
||||
Module wiring pattern:
|
||||
|
||||
- `terraform/modules/apps/<app_name>/variables.tf` defines `variable "tag"`
|
||||
- `terraform/modules/apps/<app_name>/config.tf` typically sets `local.app_version = var.tag`
|
||||
- `terraform/modules/apps/<app_name>/main.tf` uses `${local.app_version}` in image reference
|
||||
|
||||
If tag is not in the app root, check the module defaults and root overrides before editing.
|
||||
|
||||
---
|
||||
|
||||
## Update Workflow
|
||||
|
||||
Run from each app root directory (`terraform/apps/dev-01` and `terraform/apps/prod-01`) as needed.
|
||||
|
||||
1. Edit `apps.tf` and set the new pinned tag.
|
||||
2. Validate configuration:
|
||||
|
||||
```bash
|
||||
tofu fmt
|
||||
sectool -f ../../../sectool.json exec tofu validate
|
||||
```
|
||||
|
||||
3. Review plan:
|
||||
|
||||
```bash
|
||||
sectool -f ../../../sectool.json exec tofu plan
|
||||
```
|
||||
|
||||
4. Apply:
|
||||
|
||||
```bash
|
||||
sectool -f ../../../sectool.json exec tofu apply -auto-approve
|
||||
```
|
||||
|
||||
5. Verify rollout:
|
||||
|
||||
```bash
|
||||
kubectl --context microk8s-<cluster> -n <namespace> rollout status deployment/<deployment> --timeout=120s
|
||||
kubectl --context microk8s-<cluster> -n <namespace> get pods -o wide
|
||||
kubectl --context microk8s-<cluster> -n <namespace> logs deployment/<deployment> --all-containers=true --tail=200
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Emergency Recovery
|
||||
|
||||
If immediate recovery is required, a targeted apply can be used temporarily:
|
||||
|
||||
```bash
|
||||
sectool -f ../../../sectool.json exec tofu apply -auto-approve -target=module.<app_name>
|
||||
```
|
||||
|
||||
Afterward, always run a full plan to detect drift:
|
||||
|
||||
```bash
|
||||
sectool -f ../../../sectool.json exec tofu plan
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Review Checklist
|
||||
|
||||
- [ ] New image tag is pinned and explicit.
|
||||
- [ ] Both cluster roots were considered (dev-01 and prod-01).
|
||||
- [ ] Commands were run with `sectool -f ../../../sectool.json exec tofu`.
|
||||
- [ ] Rollout and pod health were verified with kubectl.
|
||||
- [ ] Any runtime hotfix used during debugging was persisted back into Terraform.
|
||||
Reference in New Issue
Block a user