a13labs.infra/.opencode/skills/terraform-develop-app/SKILL.md

name, description

name

description

terraform-develop-app

Use when developing, deploying, or debugging Terraform app modules in this repository. Covers module scaffolding, secrets injection with sectool and TF_VAR, OpenTofu command workflow, Kubernetes runtime debugging with kubectl, hardening defaults for pod and container security contexts, and known quirks such as the SearXNG IPv6 Granian bind issue. Also covers integrating modules into dev-01 and prod-01 app roots including DNS records, sectool.env mapping, and targeted apply recovery patterns.

Terraform App Module Skill

Purpose

Use this skill when developing, deploying, or debugging Terraform app modules in this repository. It covers module authoring conventions, secrets injection with sectool, OpenTofu command workflow, Kubernetes runtime debugging, and known behavioral quirks discovered in production.

---

Repository Layout

terraform/
  apps/
    dev-01/          # dev cluster root
    prod-01/         # prod cluster root
  modules/
    apps/            # reusable app modules
    db/              # database modules
    utils/           # utility modules

App roots are the deployment entry points. Reusable logic lives in modules and is consumed by app roots.

Use terraform/modules/apps/template as the baseline shape when creating a new module.

---

Cluster References

Name	Folder	Kubernetes context
dev-01	terraform/apps/dev-01	microk8s-dev-01
pro-01	terraform/apps/prod-01	microk8s-prod-01

When planning or reviewing changes, always consider both clusters. The production folder is named prod-01; discussions may call it pro-01.

---

Creating a New App Module

Create terraform/modules/apps/<name>/ with these files:

provider.tf

terraform {
  required_version = "~>1.8"
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "2.36.0"
    }
  }
}

variables.tf

Declare at minimum:

variable "persistent_folder" {
  description = "Path for persistent data on the host"
  type        = string
}

variable "fqdn" {
  description = "FQDN for the service"
  type        = string
}

variable "tag" {
  description = "Image tag to deploy"
  type        = string
  default     = "x.y.z"   # always pin, never use latest
}

Add sensitive variables at module level only when the module itself needs to inject them into a Kubernetes secret. If the secret is passed from the app root, declare it sensitive there.

config.tf

Use locals to normalize inputs. Do not append the app name to persistent_folder if the caller already passes an app-specific path:

locals {
  app_folder  = var.persistent_folder
  app_version = var.tag
  app_fqdn    = var.fqdn
}

main.tf resource order

1. kubernetes_namespace
2. kubernetes_service_account — with automount_service_account_token = false
3. kubernetes_config_map (non-sensitive config)
4. kubernetes_secret (sensitive data)
5. kubernetes_deployment
6. kubernetes_service

ingress.tf (when HTTP/HTTPS exposed)

resource "kubernetes_ingress_v1" "<name>" {
  metadata {
    annotations = {
      "kubernetes.io/ingress.class"              = "public"
      "cert-manager.io/cluster-issuer"           = var.issuer
      "kubernetes.io/tls-acme"                   = "true"
      "nginx.ingress.kubernetes.io/ssl-redirect" = "true"
    }
  }
  ...
}

Accept issuer as a variable (default "letsencrypt-prod") so the same module works for both clusters and internal CA scenarios.

---

Hardening Defaults

Apply these to every new deployment unless there is a documented exception:

spec {
  automount_service_account_token = false

  security_context {
    run_as_non_root = true
    run_as_user     = <uid>
    run_as_group    = <gid>
    fs_group        = <gid>
    seccomp_profile {
      type = "RuntimeDefault"
    }
  }

  container {
    security_context {
      allow_privilege_escalation = false
      privileged                 = false
      read_only_root_filesystem  = true
      capabilities {
        drop = ["ALL"]
      }
    }

    # provide writable scratch via emptyDir, not root filesystem
    volume_mount {
      name       = "<name>-tmp"
      mount_path = "/tmp"
    }
  }

  volume {
    name = "<name>-tmp"
    empty_dir {}
  }
}

---

Integrating a Module into an App Root

Do this for each cluster root:

1. Add a module block in apps.tf:

   module "myapp" {
     source            = "../../modules/apps/myapp"
     persistent_folder = local.myapp_folder
     fqdn              = local.myapp_fqdn
     tag               = "x.y.z"
     secret_key        = var.myapp_secret_key
   }
   

2. Add locals in config.tf:

   myapp_folder = "/mnt/md0/myapp"
   myapp_fqdn   = "myapp.${local.lab_domain}"
   

3. Add a DNS CNAME record inside local.zones[0].records:

   { name = "myapp", type = "CNAME", content = "dev-01.${local.lab_domain}." }
   

4. Add sensitive variable in secrets.tf:

   variable "myapp_secret_key" {
     description = "Secret key for MyApp"
     type        = string
     sensitive   = true
   }
   

5. Map the env variable in sectool.env:

   TF_VAR_myapp_secret_key=$MYAPP_SECRET_KEY
   

---

Secrets Pattern (sectool + TF_VAR)

Secrets are sourced from Bitwarden via sectool using sectool.json at the repository root.

Flow:

1. Store secret value in Bitwarden under the project referenced in sectool.json.
2. Declare the variable in secrets.tf with sensitive = true.
3. Map TF_VAR_<name>=$ENV_VAR_NAME in terraform/apps/<cluster>/sectool.env.
4. Run all OpenTofu commands via sectool exec so variables are injected automatically.

---

OpenTofu Workflow

Run from the relevant app root (e.g., terraform/apps/dev-01):

# Initialise
sectool -f ../../../sectool.json exec tofu init

# Format and validate
tofu fmt
sectool -f ../../../sectool.json exec tofu validate

# Plan
sectool -f ../../../sectool.json exec tofu plan

# Apply
sectool -f ../../../sectool.json exec tofu apply -auto-approve

# Targeted apply (emergency recovery only — always follow with a full plan)
sectool -f ../../../sectool.json exec tofu apply -auto-approve -target=module.<name>

---

Runtime Debugging Playbook

When a pod is in CrashLoopBackOff:

# 1. Inspect state and events
kubectl -n <ns> get pods -o wide
kubectl -n <ns> describe pod <pod>

# 2. Check current and previous logs
kubectl -n <ns> logs <pod> --all-containers=true --tail=200
kubectl -n <ns> logs <pod> --previous --all-containers=true --tail=200

# 3. Validate rollout after a fix
kubectl -n <ns> rollout status deployment/<name> --timeout=120s

# 4. Test a runtime env override before persisting in Terraform
kubectl -n <ns> set env deployment/<name> KEY=value
# verify recovery, then add KEY to the module ConfigMap and re-apply

---

Known Quirks

SearXNG — IPv6 bind failure

The searxng/searxng image defaults to GRANIAN_HOST=:: (IPv6). On nodes without IPv6 socket support this causes an immediate crash:

RuntimeError: Address family not supported by protocol (os error 97)

Fix: set GRANIAN_HOST = "0.0.0.0" in the module ConfigMap.

---

Review Checklist

Before proposing or applying any Terraform change:

• Changes scoped to both dev-01 and prod-01 where relevant.
• Sensitive inputs declared in secrets.tf + mapped in sectool.env.
• Image tag is pinned (not latest).
• Hardening defaults present (non-root, drop-all capabilities, read-only root fs).
• Commands in examples use sectool -f ../../../sectool.json exec tofu.