Add GPU monitoring services and configurations
- Create systemd service templates for llama_exporter, nvidia_exporter, and podman. - Add Prometheus configuration template for GPU metrics scraping. - Introduce variables for GPU monitoring role in main.yml. - Implement tasks for syncing llama models, including user setup and package installation. - Update podman tasks to ensure proper sudo access and lingering for the podman user. - Modify inventory to include gpu_monitoring group. - Add Terraform modules for deploying Vaultwarden, including ingress and service configurations. - Create Grafana dashboard for real-time GPU monitoring metrics. - Update secrets and environment files to include Vaultwarden admin token.
This commit is contained in:
@@ -0,0 +1,486 @@
|
||||
---
|
||||
name: ansible-develop
|
||||
description: >
|
||||
Use when developing, writing, or debugging Ansible playbooks and roles in this
|
||||
repository. Covers playbook and role conventions, inventory group targeting,
|
||||
molecule-podman testing, variable scoping (defaults/vars/host_vars), Jinja2
|
||||
templates, tag-based execution, and known patterns such as the podman shared
|
||||
tasks helper and dual Debian/RedHat OS-family blocks.
|
||||
---
|
||||
|
||||
# Ansible Playbook & Role Skill
|
||||
|
||||
## Purpose
|
||||
|
||||
Use this skill when creating new playbooks or roles in `ansible/`. It covers
|
||||
project conventions for structure, testing, variable management, secrets, and
|
||||
execution patterns used across all existing automation.
|
||||
|
||||
---
|
||||
|
||||
## Repository Layout
|
||||
|
||||
```
|
||||
ansible/
|
||||
ansible.cfg # inventory = ../inventory/hosts
|
||||
requirements.yml # collections: ansible.posix, community.general,
|
||||
# containers.podman, community.libvirt, ...
|
||||
.yamllint.yml # 120-char line limit, extended default rules
|
||||
playbook_*.yml # top-level entry points (run from repo root)
|
||||
tasks/ # shared task fragments (e.g. tasks/podman.yml)
|
||||
roles/
|
||||
<name>/
|
||||
defaults/main.yml # public defaults (overridable by callers)
|
||||
vars/main.yml # internal values (non-overridable)
|
||||
tasks/main.yml # entry point; include_ or block per sub-feature
|
||||
handlers/main.yml # notify handlers (services, reloads)
|
||||
templates/ # Jinja2 files (.j2 extension)
|
||||
files/ # static files copied as-is
|
||||
meta/main.yml # galaxy metadata, min_ansible_version, dependencies
|
||||
molecule/default/ # tests: converge.yml, prepare.yml, verify.yml, molecule.yml
|
||||
tests/inventory # small static inventory for role tests
|
||||
tests/test.yml # molecule verifier
|
||||
.yamllint.yml # (optional) role-level yamllint overrides
|
||||
README.md # role documentation
|
||||
inventory/
|
||||
hosts # host groups (ini-format inventory)
|
||||
host_vars/ # per-host YAML variable files
|
||||
```
|
||||
|
||||
Playbooks live in `ansible/` and are run from the repository root. The
|
||||
inventory path is relative: `../inventory/hosts` (as declared in
|
||||
`ansible/ansible.cfg`).
|
||||
|
||||
---
|
||||
|
||||
## Inventory & Group Targeting
|
||||
|
||||
The inventory at `inventory/hosts` uses **ini-format groups**. Playbooks target
|
||||
groups via `hosts: <groupname>`. Group names follow kebab-case convention.
|
||||
|
||||
Existing groups:
|
||||
|
||||
| Group | Hosts |
|
||||
|-------------------|------------------------------------------------|
|
||||
| `all` | prod-01, dev-01, vh-01, gpu-01 |
|
||||
| `hardening` | prod-01, dev-01, vh-01, gpu-01 |
|
||||
| `ubuntu` | prod-01, dev-01 |
|
||||
| `fedora` | vh-01, gpu-01 |
|
||||
| `microk8s` | prod-01, dev-01 |
|
||||
| `monitoring` | prod-01 |
|
||||
| `duo` | prod-01, dev-01, vh-01, gpu-01 |
|
||||
| `virtualization` | vh-01 |
|
||||
| `ollama` | gpu-01 |
|
||||
| `comfyui` | gpu-01 |
|
||||
| `microk8s` | prod-01, dev-01 |
|
||||
| `certbot` | vh-01, gpu-01 |
|
||||
| `nvidia` | gpu-01 |
|
||||
| `amd` | gpu-01 |
|
||||
| `public` | prod-01 |
|
||||
| `private` | dev-01, vh-01, gpu-01 |
|
||||
| `windows` | ci-01 |
|
||||
|
||||
When adding a new playbook, **add a new group** in `inventory/hosts` and target
|
||||
it from the playbook.
|
||||
|
||||
---
|
||||
|
||||
## Playbook Patterns
|
||||
|
||||
Two patterns are used. Choose the one that fits the task.
|
||||
|
||||
### Pattern 1 — Role-based (preferred for reusable logic)
|
||||
|
||||
Use when the task is encapsulated in a role or may be reused across playbooks.
|
||||
|
||||
```yaml
|
||||
---
|
||||
- name: My feature setup
|
||||
hosts: mygroup
|
||||
gather_facts: true
|
||||
|
||||
roles:
|
||||
- role: "my_role"
|
||||
tags:
|
||||
- roles
|
||||
- roles::my_role
|
||||
```
|
||||
|
||||
### Pattern 2 — Inline tasks
|
||||
|
||||
Use for single-shot playbooks that configure one specific host with custom logic
|
||||
(e.g., virt_host, encryption).
|
||||
|
||||
```yaml
|
||||
---
|
||||
- name: My inline setup
|
||||
hosts: mygroup
|
||||
become: true
|
||||
gather_facts: true
|
||||
|
||||
tasks:
|
||||
- name: Ensure the system is Debian
|
||||
ansible.builtin.assert:
|
||||
that:
|
||||
- ansible_os_family == 'Debian'
|
||||
fail_msg: "This playbook only supports Debian hosts."
|
||||
|
||||
- name: Install required packages
|
||||
become: true
|
||||
ansible.builtin.apt:
|
||||
name:
|
||||
- package-one
|
||||
- package-two
|
||||
state: present
|
||||
update_cache: true
|
||||
```
|
||||
|
||||
### Shared task fragments
|
||||
|
||||
Complex operations (e.g. podman pod/container lifecycle) go into `tasks/` and
|
||||
are included via `ansible.builtin.include_tasks`. See `tasks/podman.yml` for
|
||||
the pattern: required-var checks, OS-family `set_fact`, `become_user` blocks,
|
||||
and loop-based resource creation.
|
||||
|
||||
---
|
||||
|
||||
## Role Development
|
||||
|
||||
### Minimal role directory structure
|
||||
|
||||
```
|
||||
roles/my_role/
|
||||
defaults/main.yml # public defaults
|
||||
tasks/main.yml # entry point
|
||||
handlers/main.yml # notify handlers
|
||||
templates/ # Jinja2 files
|
||||
meta/main.yml # galaxy metadata
|
||||
molecule/default/ # tests
|
||||
```
|
||||
|
||||
### defaults/main.yml
|
||||
|
||||
Declare **all** public variables here with safe defaults. This is how callers
|
||||
override behavior.
|
||||
|
||||
```yaml
|
||||
---
|
||||
my_feature_enabled: true
|
||||
my_feature_port: 8080
|
||||
my_feature_users: []
|
||||
```
|
||||
|
||||
Use `lookup('env', 'VAR_NAME')` for secrets with empty default:
|
||||
|
||||
```yaml
|
||||
my_api_key: "{{ lookup('env', 'MY_API_KEY') | default('') }}"
|
||||
```
|
||||
|
||||
### tasks/main.yml
|
||||
|
||||
Use `block` + `when` for OS-family conditional logic:
|
||||
|
||||
```yaml
|
||||
---
|
||||
- name: Install packages (Debian)
|
||||
when: ansible_os_family == 'Debian'
|
||||
block:
|
||||
- name: Install deps
|
||||
become: true
|
||||
ansible.builtin.apt:
|
||||
name: "{{ item }}"
|
||||
state: present
|
||||
update_cache: true
|
||||
loop:
|
||||
- package-one
|
||||
|
||||
- name: Install packages (RedHat)
|
||||
when: ansible_os_family == 'RedHat'
|
||||
block:
|
||||
- name: Install deps
|
||||
become: true
|
||||
ansible.builtin.dnf:
|
||||
name: "{{ item }}"
|
||||
state: present
|
||||
loop:
|
||||
- package-one
|
||||
```
|
||||
|
||||
Use `include_tasks` for sub-features, tagged separately:
|
||||
|
||||
```yaml
|
||||
- name: Include volume provisioning
|
||||
ansible.builtin.include_tasks: volume.yml
|
||||
tags:
|
||||
- roles::my_role::volume
|
||||
|
||||
- name: Include KMS setup
|
||||
ansible.builtin.include_tasks: kms.yml
|
||||
tags:
|
||||
- roles::my_role::kms
|
||||
```
|
||||
|
||||
### meta/main.yml
|
||||
|
||||
Always declare `galaxy_info` with namespace `a13labs`, author
|
||||
`Alexandre Pires`, and company `A13Labs`:
|
||||
|
||||
```yaml
|
||||
---
|
||||
galaxy_info:
|
||||
author: Alexandre Pires
|
||||
description: Role to do something
|
||||
company: A13Labs
|
||||
role_name: my_role
|
||||
namespace: a13labs
|
||||
license: MIT
|
||||
min_ansible_version: "2.1"
|
||||
```
|
||||
|
||||
### Tag conventions
|
||||
|
||||
Use nested tags following the pattern `roles::<role_name>::<feature>`. Top-level
|
||||
playbooks use `roles` and `roles::<role_name>`.
|
||||
|
||||
---
|
||||
|
||||
## Molecule Testing (Podman)
|
||||
|
||||
Every role should have a `molecule/default/` test scenario using the Podman
|
||||
driver.
|
||||
|
||||
### molecule.yml
|
||||
|
||||
```yaml
|
||||
---
|
||||
driver:
|
||||
name: podman
|
||||
platforms:
|
||||
- name: instance
|
||||
image: ubuntu:latest
|
||||
pre_build_image: true
|
||||
volumes:
|
||||
- /sys/fs/cgroup:/sys/fs/cgroup:ro
|
||||
privileged: true
|
||||
provisioner:
|
||||
name: ansible
|
||||
env:
|
||||
MY_ROLE_VAR: "test_value"
|
||||
playbooks:
|
||||
converge: converge.yml
|
||||
prepare: prepare.yml
|
||||
verifier:
|
||||
name: ansible
|
||||
```
|
||||
|
||||
### prepare.yml
|
||||
|
||||
Must install python3 and sudo using raw (facts are unavailable in containers):
|
||||
|
||||
```yaml
|
||||
---
|
||||
- name: Prepare
|
||||
hosts: all
|
||||
gather_facts: false
|
||||
tasks:
|
||||
- name: Update cache
|
||||
ansible.builtin.raw: apt update
|
||||
|
||||
- name: Install required packages
|
||||
ansible.builtin.raw: apt install -y python3 sudo
|
||||
```
|
||||
|
||||
### converge.yml
|
||||
|
||||
```yaml
|
||||
---
|
||||
- name: Converge
|
||||
hosts: all
|
||||
gather_facts: true
|
||||
tasks:
|
||||
- name: Include my_role
|
||||
ansible.builtin.include_role:
|
||||
name: "my_role"
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
Run from the role directory:
|
||||
|
||||
```bash
|
||||
cd ansible/roles/my_role
|
||||
molecule test
|
||||
```
|
||||
|
||||
This runs: prepare -> converge -> verify -> destroy.
|
||||
|
||||
**Important**: Most containers lack systemd. Guard service-related tasks:
|
||||
`when: ansible_service_mgr == "systemd"`.
|
||||
|
||||
---
|
||||
|
||||
## Secrets & Environment Variables
|
||||
|
||||
Secrets are injected via `sectool` using `sectool.json` at the repository root.
|
||||
The `ANSIBLE_USER` env var must be set before opening opencode.
|
||||
|
||||
Secrets in roles use `lookup('env', 'VAR_NAME')` — `sectool` injects them
|
||||
automatically when playbooks are run through `sectool exec`.
|
||||
|
||||
Pattern in role defaults:
|
||||
|
||||
```yaml
|
||||
api_key: "{{ lookup('env', 'MY_API_KEY') | default('') }}"
|
||||
```
|
||||
|
||||
Pattern in host_vars (for host-level secrets):
|
||||
|
||||
```yaml
|
||||
cloudns_auth_id: "{{ lookup('env', 'CLOUDNS_AUTH_ID') }}"
|
||||
cloudns_auth_password: "{{ lookup('env', 'CLOUDNS_PASSWORD') }}"
|
||||
```
|
||||
|
||||
Never hardcode secrets. Always use `lookup('env', ...)` with a `| default('')`
|
||||
fallback.
|
||||
|
||||
---
|
||||
|
||||
## Execution
|
||||
|
||||
All Ansible commands must be run from the **`ansible/` directory** using
|
||||
`sectool` to inject secrets:
|
||||
|
||||
```bash
|
||||
cd ansible/
|
||||
|
||||
# Role-based playbook
|
||||
sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_myfeature.yml
|
||||
|
||||
# Tag-based partial run
|
||||
sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_myfeature.yml --tags roles::my_role::volume
|
||||
|
||||
# Limit to specific group
|
||||
sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_myfeature.yml --limit mygroup
|
||||
|
||||
# Role tests (molecule runs independently, no sectool wrapper needed)
|
||||
cd ansible/roles/my_role && molecule test
|
||||
```
|
||||
|
||||
`$ANSIBLE_USER` must be set in the shell before opening opencode.
|
||||
|
||||
---
|
||||
|
||||
## Linting
|
||||
|
||||
### yamllint
|
||||
|
||||
```bash
|
||||
cd ansible
|
||||
yamllint -c .yamllint.yml .
|
||||
```
|
||||
|
||||
Rules: 120-char line width (warning), forbid implicit/explicit octal, extended
|
||||
default ruleset.
|
||||
|
||||
### ansible-lint
|
||||
|
||||
If available, run from the `ansible/` directory:
|
||||
|
||||
```bash
|
||||
cd ansible
|
||||
ansible-lint
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Known Patterns
|
||||
|
||||
### Dual OS-family support
|
||||
|
||||
Always support both Debian and RedHat families when packages differ:
|
||||
|
||||
```yaml
|
||||
- name: Set binary path (Debian)
|
||||
when: ansible_os_family == 'Debian'
|
||||
ansible.builtin.set_fact:
|
||||
podman_binary: /usr/bin/podman
|
||||
|
||||
- name: Set binary path (RedHat)
|
||||
when: ansible_os_family == 'RedHat'
|
||||
ansible.builtin.set_fact:
|
||||
podman_binary: /usr/sbin/podman
|
||||
```
|
||||
|
||||
### become_user blocks
|
||||
|
||||
When a task needs to run as a different user (e.g. podman operations), use a
|
||||
`become: true` + `become_user` block:
|
||||
|
||||
```yaml
|
||||
- name: Run tasks as podman user
|
||||
become: true
|
||||
become_user: "{{ podman_user }}"
|
||||
block:
|
||||
- name: Build images
|
||||
containers.podman.podman_image:
|
||||
...
|
||||
```
|
||||
|
||||
### systemd service generation
|
||||
|
||||
Use `ansible.builtin.template` to generate `.service` files from `.j2`
|
||||
templates, then `daemon_reload` + `systemd` enable:
|
||||
|
||||
```yaml
|
||||
- name: Create systemd service file
|
||||
become: true
|
||||
ansible.builtin.template:
|
||||
src: myservice.service.j2
|
||||
dest: "/etc/systemd/system/my-service.service"
|
||||
mode: "0644"
|
||||
notify: Reload daemon
|
||||
|
||||
handlers:
|
||||
- name: Reload daemon
|
||||
changed_when: false
|
||||
ansible.builtin.systemd:
|
||||
daemon_reload: true
|
||||
```
|
||||
|
||||
### nmcli bridge creation
|
||||
|
||||
Network bridge configuration uses `nmcli connection add` with `async` +
|
||||
`wait_for` to handle reconnection after interface changes:
|
||||
|
||||
```yaml
|
||||
- name: Create bridge
|
||||
ansible.builtin.command: nmcli connection add ...
|
||||
register: bridge_created
|
||||
async: 10
|
||||
poll: 0
|
||||
|
||||
- name: Wait for reconnection
|
||||
ansible.builtin.wait_for:
|
||||
host: "{{ bridge_ip }}"
|
||||
port: 22
|
||||
timeout: 30
|
||||
delegate_to: localhost
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Review Checklist
|
||||
|
||||
Before proposing a new playbook or role:
|
||||
|
||||
- [ ] Playbook targets a defined group in `inventory/hosts` (add one if needed).
|
||||
- [ ] Role has `defaults/main.yml` with all public variables + safe defaults.
|
||||
- [ ] Secrets use `lookup('env', 'VAR') | default('')`, never hardcoded.
|
||||
- [ ] Dual OS-family support when package names differ.
|
||||
- [ ] Tasks tagged with `roles::<role_name>::<feature>`.
|
||||
- [ ] Molecule `molecule/default/` scenario with prepare + converge.
|
||||
- [ ] `molecule.yml` uses Podman driver with `privileged: true` + cgroup mount.
|
||||
- [ ] Line length within 120 characters.
|
||||
- [ ] `yamllint -c ansible/.yamllint.yml ansible/` passes.
|
||||
- [ ] Service tasks guarded by `when: ansible_service_mgr == "systemd"` in
|
||||
molecule containers.
|
||||
Reference in New Issue
Block a user