a13labs.infra/.github/copilot-instructions.md

Copilot Instructions for a13labs.infra

This repository manages infrastructure mostly with Ansible and OpenTofu/Terraform. These instructions define how Terraform app modules should be developed, deployed, and debugged.

Scope and Structure

• App roots:
  • terraform/apps/dev-01
  • terraform/apps/prod-01
• Reusable modules:
  • terraform/modules/apps
  • terraform/modules/db
  • terraform/modules/utils
• New app modules should be created under terraform/modules/apps/.
• Use terraform/modules/apps/template as the base shape when creating a new app module.

Cluster References

Always cover both clusters in planning and reviews:

• dev-01
  • App root: terraform/apps/dev-01
  • Kubernetes context: microk8s-dev-01
• pro-01
  • App root: terraform/apps/prod-01
  • Kubernetes context: microk8s-prod-01

Note: The production folder name is prod-01, while discussions may refer to it as pro-01.

Module Development Standards

When adding a new app module in terraform/modules/apps/:

1. Create the standard files:
   • provider.tf
   • variables.tf
   • config.tf
   • main.tf
   • ingress.tf (if exposed over HTTP/HTTPS)
2. Provider constraints should match repository conventions:
   • required_version ~>1.8
   • kubernetes provider version 2.36.0
3. Keep names consistent across namespace, service account, deployment labels, and service selectors.
4. Use locals in config.tf for derived paths and normalized values.
5. Treat secrets as Terraform inputs (sensitive variables), not hardcoded literals.
6. Prefer explicit image tags instead of latest for predictable rollouts.

Security and Hardening Defaults

For Kubernetes deployments, prefer hardened defaults unless an app requires exceptions:

• Disable service account token automount at both SA and pod level.
• Use pod security context with non-root UID/GID and RuntimeDefault seccomp.
• Use container security context:
  • allow_privilege_escalation = false
  • privileged = false
  • drop all capabilities
  • read_only_root_filesystem = true
• Provide writable mounts only where required (for example cache path and /tmp via emptyDir).

Integrating a Module into an App Root

For each cluster root (dev-01 and pro-01):

1. Register module in apps.tf.
2. Add needed locals in config.tf (FQDN, path, issuer, feature toggles).
3. Add DNS record entries when the app is externally reachable.
4. Add sensitive input variables in secrets.tf.
5. Map TF_VAR entries in sectool.env.

Secrets Pattern (sectool + TF_VAR)

Secret flow is:

1. Declare sensitive variables in secrets.tf.
2. Map TF_VAR_ in terraform/apps//sectool.env.
3. Keep value source in vault/environment, not in repository.
4. Run OpenTofu through sectool so variables are injected.

Example workflow from an app root:

• sectool -f ../../../sectool.json exec tofu init
• sectool -f ../../../sectool.json exec tofu validate
• sectool -f ../../../sectool.json exec tofu plan
• sectool -f ../../../sectool.json exec tofu apply -auto-approve

OpenTofu and Deployment Workflow

Preferred command flow per cluster root:

1. Format and validate first:
   • tofu fmt
   • sectool -f ../../../sectool.json exec tofu validate
2. Plan before apply:
   • sectool -f ../../../sectool.json exec tofu plan
3. Apply:
   • sectool -f ../../../sectool.json exec tofu apply -auto-approve

Targeted apply can be used for emergency recovery, then follow with a full plan to detect drift.

Runtime Debugging Playbook (kubectl)

When a deployment fails (CrashLoopBackOff):

1. Inspect pod state and events:
   • kubectl -n get pods -o wide
   • kubectl -n describe pod
2. Check logs:
   • kubectl -n logs --all-containers=true --tail=200
   • kubectl -n logs --previous --all-containers=true --tail=200
3. Validate rollout:
   • kubectl -n rollout status deployment/ --timeout=120s
4. If needed, test temporary env override with kubectl set env, verify recovery, then persist in Terraform module.

Known SearXNG Behavior in This Repo

• SearXNG may fail on startup with IPv6 bind on hosts without IPv6 socket support.
• Set GRANIAN_HOST to 0.0.0.0 in module-managed configuration.
• Keep image tag pinned in app roots to avoid unexpected upstream behavior changes.

Review Checklist for Copilot Changes

Before proposing Terraform changes:

1. Confirm changes cover both dev-01 and pro-01 impact.
2. Confirm secrets are wired via sensitive variables + sectool mapping.
3. Confirm image tags are pinned.
4. Confirm deployment hardening defaults are retained.
5. Confirm commands use sectool + tofu in examples and runbooks.