a13labs.infra/.opencode/skills/terraform-develop-app/SKILL.md

---
name: terraform-develop-app
description: >
  Use when developing, deploying, or debugging Terraform app modules in this
  repository. Covers module scaffolding, secrets injection with sectool and
  TF_VAR, OpenTofu command workflow, Kubernetes runtime debugging with kubectl,
  hardening defaults for pod and container security contexts, and known quirks
  such as the SearXNG IPv6 Granian bind issue. Also covers integrating modules
  into dev-01 and prod-01 app roots including DNS records, sectool.env mapping,
  and targeted apply recovery patterns.
---

# Terraform App Module Skill

## Purpose

Use this skill when developing, deploying, or debugging Terraform app modules in this repository. It covers module authoring conventions, secrets injection with sectool, OpenTofu command workflow, Kubernetes runtime debugging, and known behavioral quirks discovered in production.

---

## Repository Layout

```
terraform/
  apps/
    dev-01/          # dev cluster root
    prod-01/         # prod cluster root
  modules/
    apps/            # reusable app modules
    db/              # database modules
    utils/           # utility modules
```

App roots are the deployment entry points. Reusable logic lives in modules and is consumed by app roots.

Use `terraform/modules/apps/template` as the baseline shape when creating a new module.

---

## Cluster References

| Name   | Folder                   | Kubernetes context   |
|--------|--------------------------|----------------------|
| dev-01 | terraform/apps/dev-01    | microk8s-dev-01      |
| pro-01 | terraform/apps/prod-01   | microk8s-prod-01     |

When planning or reviewing changes, always consider both clusters. The production folder is named `prod-01`; discussions may call it `pro-01`.

---

## Creating a New App Module

Create `terraform/modules/apps/<name>/` with these files:

### provider.tf

```hcl
terraform {
  required_version = "~>1.8"
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.36.0"
    }
  }
}
```

### variables.tf

Declare at minimum:

```hcl
variable "persistent_folder" {
  description = "Path for persistent data on the host"
  type        = string
}

variable "fqdn" {
  description = "FQDN for the service"
  type        = string
}

variable "tag" {
  description = "Image tag to deploy"
  type        = string
  default     = "x.y.z"   # always pin, never use latest
}
```

Add sensitive variables at module level only when the module itself needs to inject them into a Kubernetes secret. If the secret is passed from the app root, declare it sensitive there.

### config.tf

Use locals to normalize inputs. Do not append the app name to `persistent_folder` if the caller already passes an app-specific path:

```hcl
locals {
  app_folder  = var.persistent_folder
  app_version = var.tag
  app_fqdn    = var.fqdn
}
```

### main.tf resource order

1. `kubernetes_namespace`
2. `kubernetes_service_account` — with `automount_service_account_token = false`
3. `kubernetes_config_map` (non-sensitive config)
4. `kubernetes_secret` (sensitive data)
5. `kubernetes_deployment`
6. `kubernetes_service`

### ingress.tf (when HTTP/HTTPS exposed)

```hcl
resource "kubernetes_ingress_v1" "<name>" {
  metadata {
    annotations = {
      "kubernetes.io/ingress.class"              = "public"
      "cert-manager.io/cluster-issuer"           = var.issuer
      "kubernetes.io/tls-acme"                   = "true"
      "nginx.ingress.kubernetes.io/ssl-redirect" = "true"
    }
  }
  ...
}
```

Accept `issuer` as a variable (default `"letsencrypt-prod"`) so the same module works for both clusters and internal CA scenarios.

---

## Hardening Defaults

Apply these to every new deployment unless there is a documented exception:

```hcl
spec {
  automount_service_account_token = false

  security_context {
    run_as_non_root = true
    run_as_user     = <uid>
    run_as_group    = <gid>
    fs_group        = <gid>
    seccomp_profile {
      type = "RuntimeDefault"
    }
  }

  container {
    security_context {
      allow_privilege_escalation = false
      privileged                 = false
      read_only_root_filesystem  = true
      capabilities {
        drop = ["ALL"]
      }
    }

    # provide writable scratch via emptyDir, not root filesystem
    volume_mount {
      name       = "<name>-tmp"
      mount_path = "/tmp"
    }
  }

  volume {
    name = "<name>-tmp"
    empty_dir {}
  }
}
```

---

## Integrating a Module into an App Root

Do this for each cluster root:

1. Add a `module` block in `apps.tf`:
   ```hcl
   module "myapp" {
     source            = "../../modules/apps/myapp"
     persistent_folder = local.myapp_folder
     fqdn              = local.myapp_fqdn
     tag               = "x.y.z"
     secret_key        = var.myapp_secret_key
   }
   ```

2. Add locals in `config.tf`:
   ```hcl
   myapp_folder = "/mnt/md0/myapp"
   myapp_fqdn   = "myapp.${local.lab_domain}"
   ```

3. Add a DNS CNAME record inside `local.zones[0].records`:
   ```hcl
   { name = "myapp", type = "CNAME", content = "dev-01.${local.lab_domain}." }
   ```

4. Add sensitive variable in `secrets.tf`:
   ```hcl
   variable "myapp_secret_key" {
     description = "Secret key for MyApp"
     type        = string
     sensitive   = true
   }
   ```

5. Map the env variable in `sectool.env`:
   ```
   TF_VAR_myapp_secret_key=$MYAPP_SECRET_KEY
   ```

---

## Secrets Pattern (sectool + TF_VAR)

Secrets are sourced from Bitwarden via `sectool` using `sectool.json` at the repository root.

Flow:
1. Store secret value in Bitwarden under the project referenced in `sectool.json`.
2. Declare the variable in `secrets.tf` with `sensitive = true`.
3. Map `TF_VAR_<name>=$ENV_VAR_NAME` in `terraform/apps/<cluster>/sectool.env`.
4. Run all OpenTofu commands via `sectool exec` so variables are injected automatically.

---

## OpenTofu Workflow

Run from the relevant app root (e.g., `terraform/apps/dev-01`):

```bash
# Initialise
sectool -f ../../../sectool.json exec tofu init

# Format and validate
tofu fmt
sectool -f ../../../sectool.json exec tofu validate

# Plan
sectool -f ../../../sectool.json exec tofu plan

# Apply
sectool -f ../../../sectool.json exec tofu apply -auto-approve

# Targeted apply (emergency recovery only — always follow with a full plan)
sectool -f ../../../sectool.json exec tofu apply -auto-approve -target=module.<name>
```

---

## Runtime Debugging Playbook

When a pod is in CrashLoopBackOff:

```bash
# 1. Inspect state and events
kubectl -n <ns> get pods -o wide
kubectl -n <ns> describe pod <pod>

# 2. Check current and previous logs
kubectl -n <ns> logs <pod> --all-containers=true --tail=200
kubectl -n <ns> logs <pod> --previous --all-containers=true --tail=200

# 3. Validate rollout after a fix
kubectl -n <ns> rollout status deployment/<name> --timeout=120s

# 4. Test a runtime env override before persisting in Terraform
kubectl -n <ns> set env deployment/<name> KEY=value
# verify recovery, then add KEY to the module ConfigMap and re-apply
```

---

## Known Quirks

### SearXNG — IPv6 bind failure

The `searxng/searxng` image defaults to `GRANIAN_HOST=::` (IPv6). On nodes without IPv6 socket support this causes an immediate crash:

```
RuntimeError: Address family not supported by protocol (os error 97)
```

Fix: set `GRANIAN_HOST = "0.0.0.0"` in the module ConfigMap.

---

## Review Checklist

Before proposing or applying any Terraform change:

- [ ] Changes scoped to both dev-01 and prod-01 where relevant.
- [ ] Sensitive inputs declared in `secrets.tf` + mapped in `sectool.env`.
- [ ] Image tag is pinned (not `latest`).
- [ ] Hardening defaults present (non-root, drop-all capabilities, read-only root fs).
- [ ] Commands in examples use `sectool -f ../../../sectool.json exec tofu`.