diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..a7c0782 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,114 @@ +# AGENTS.md - A13Labs Infrastructure + +## Repo Structure + +- **`ansible/`** — Playbooks, roles, inventory. Runs against bare-metal servers. +- **`terraform/`** — IaC (OpenTofu for K8s app deployments + cloud provisioning). +- **`inventory/hosts`** — Ansible inventory. All hosts and groups. +- **`requirements/`** — pip requirements files: `ansible.txt`, `dev.txt`, `opentofu.txt`. +- **`sectool`** — Secret manager (Bitwarden, configured in `sectool.json`). Both root and `ansible/` have their own `sectool.json` + `sectool.env`. + +## Key Facts + +- **No `Makefile`, no `justfile`, no pre-commit config.** Commands are run directly. +- **All GitHub workflows are `.disabled`** — CI is not active. Verification is local-only. +- **`ansible/ansible.cfg`** points to `../inventory/hosts` (relative to `ansible/` dir). Always run playbooks from the `ansible/` directory, or use `-i` to override. +- **YAML linting:** `ansible-lint` config at `ansible/.ansible-lint`. `.yamllint.yml` at `ansible/.yamllint.yml` (120 char line warning). +- **No `.pre-commit-config.yaml` exists** — linting is manual or CI-only. + +## Ansible + +### Required Environment vars + +```sh +export ANSIBLE_USER=provision +export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES # if on a MAC +``` + +### Running playbooks +```sh +cd ansible +sectool exec ansible-playbook -u $ANSIBLE_USER playbook_.yml +``` + +### Role testing (Molecule + Podman - Linux Only) +```sh +cd ansible/roles/ +molecule test # create -> converge -> verify -> destroy +molecule converge # run converge only +molecule destroy # clean up +``` + +**Molecule gotchas:** +- Always run `molecule` commands **from the role directory**, not from repo root. +- Molecule containers use `podman` driver. No systemd by default — guard service tasks with `when: ansible_service_mgr == "systemd"`. +- `molecule/default/prepare.yml` must install python3 and sudo via `raw:` before Ansible can run Python modules. +- `provisioner.env.ANSIBLE_ROLES_PATH` should point up to the parent (e.g., `../../../`). + +### Ansible-lint +```sh +ansible-lint # runs from repo root or ansible/ +``` +Excluded paths: `.git/`, `.github/`, `tests/`. Warns on: `line-length`, `yaml[line-length]`, `var-naming[no-role-prefix]`. + +## Terraform / OpenTofu + +### App deployments (K8s) +- `terraform/apps/dev-01/` — dev Kubernetes cluster (microk8s) +- `terraform/apps/prod-01/` — prod Kubernetes cluster (microk8s) +- Uses Kubernetes + TLS providers. `provider.tf` sets `insecure = true`. + +### Cloud IaC +- `terraform/iac/aws/` — AWS infrastructure +- `terraform/iac/contabo/` — Contabo VPS +- `terraform/iac/cloudns/` — DNS (cloudns) +- `terraform/iac/scaleway/` — Scaleway infrastructure + +### Secrets +Each terraform directory has its own `sectool.env`. Run `sectool` to load secrets before `tofu init/plan/apply`: +```sh +sectool env -d terraform/apps/dev-01/ -- tofu plan +``` + +### Module structure +- `terraform/modules/apps/` — application modules (Nextcloud, etc.) +- `terraform/modules/db/` — database modules (MariaDB with backups) +- `terraform/modules/utils/` — utilities (exporters, cert-checker, prometheus, trivy, etc.) + +## Scripts (`scripts/`) + +One-off utility scripts — not library code: +- `install-opnsense.sh` — boots OPNsense on a VM via virt-install (one-time router setup) +- `chronograf.sh` — `kubectl port-forward` wrapper for Chronograf pod +- `proxy-pt.sh` — `kubectl port-forward` wrapper for m3uproxy pod +- `steamos` — launches SteamOS gaming session with sunshine +- `cpu_power_monitor.py` — RAPL power monitoring (requires root) + +## Environment Setup + +```sh +# Ansible + dev tools +pip install -r requirements/ansible.txt -r requirements/dev.txt + +# Terraform/OpenTofu deps +pip install -r requirements/opentofu.txt +``` + +## Contabo CLI + +```sh +sectool exec cntb get instances --oauth2-clientid="$CONTABO_CLIENT_ID" ... +``` + +## Windows hosts + +- Use `ConfigureRemotingForAnsible.ps1` in `scripts/` to enable WinRM on Windows targets. +- On MSYS2: install packages listed in `Readme.MD` (go, python, cryptography, paramiko, rpds-py, etc.). +- Python venv: `python -m venv .venv --system-site-packages` + +## Style Conventions + +- YAML: 120 char line length (warning threshold) +- Ansible roles follow galaxy conventions: `defaults/`, `tasks/`, `meta/`, `files/`, `templates/` +- `tasks/main.yml` is the entrypoint for every role +- Host vars: `ansible/host_vars/.yml` diff --git a/ansible/host_vars/gpu-01.lab.alexpires.me.yml b/ansible/host_vars/gpu-01.lab.alexpires.me.yml index 0e78812..16e3a2f 100644 --- a/ansible/host_vars/gpu-01.lab.alexpires.me.yml +++ b/ansible/host_vars/gpu-01.lab.alexpires.me.yml @@ -132,103 +132,103 @@ llama_server_models: quant: "Q8_0" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "gemma-4-26b-a4b-ud-iq4-xs" model_id: "unsloth/gemma-4-26B-A4B-it-GGUF" quant: "UD-IQ4_XS" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "gemma-4-31b-iq4-xs" model_id: "unsloth/gemma-4-31B-it-GGUF" quant: "IQ4_XS" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-5-9b-iq4-xs" model_id: "unsloth/Qwen3.5-9B-GGUF" quant: "IQ4_XS" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-5-9b-q8-0" model_id: "unsloth/Qwen3.5-9B-GGUF" quant: "Q8_0" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-5-27b-mtp-iq4-xs" model_id: "unsloth/Qwen3.5-27B-MTP-GGUF" quant: "IQ4_XS" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-5-35b-a3b-ud-iq4-xs" model_id: "unsloth/Qwen3.5-35B-A3B-GGUF" quant: "UD-IQ4_XS" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-5-35b-a3b-q4-k-m" model_id: "unsloth/Qwen3.5-35B-A3B-GGUF" quant: "Q4_K_M" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-6-27b-mtp-q4-0" model_id: "unsloth/Qwen3.6-27B-MTP-GGUF" quant: "Q4_0" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-6-27b-mtp-ud-q4-0-k-xl" model_id: "unsloth/Qwen3.6-27B-MTP-GGUF" quant: "UD-Q4_K_XL" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-6-27b-mtp-ud-q4-xl" model_id: "unsloth/Qwen3.6-27B-MTP-GGUF" quant: "UD-Q4_K_XL" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-6-35b-a3b-mtp-ud-iq4-xs" model_id: "unsloth/Qwen3.6-35B-A3B-MTP-GGUF" quant: "UD-IQ4_XS" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-6-35b-a3b-mtp-ud-q4-m" model_id: "unsloth/Qwen3.6-35B-A3B-MTP-GGUF" quant: "UD-Q4_K_M" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-6-35b-a3b-ud-q4-k-m" model_id: "unsloth/Qwen3.6-35B-A3B-GGUF" quant: "UD-Q4_K_M" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "qwen3-6-35b-a3b-claude47-distilled-iq4-xs" model_id: "lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-IQ4_XS-GGUF" quant: "IQ4_XS" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "ministral-3-14b-instruct-iq4-xs" model_id: "unsloth/Ministral-3-14B-Instruct-2512-GGUF" quant: "IQ4_XS" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "granite-4-1-30b-iq4-xs" model_id: "unsloth/granite-4.1-30b-GGUF" quant: "IQ4_XS" revision: "main" preset: - ctx-size: 131072 + ctx-size: 98304 - name: "devstral-small-2-24b-instruct-iq4-xs" model_id: "unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF" quant: "IQ4_XS" @@ -239,7 +239,7 @@ llama_server_argv_extra: "-t", 10, "-np", - 1, + -1, "-b", 1024, "-ub", @@ -257,7 +257,7 @@ llama_server_argv_extra: "-dev", "Vulkan1,Vulkan2", "-ts", - "8,12", + "12,16", "-fit", "off", "-mg", @@ -265,8 +265,13 @@ llama_server_argv_extra: "--mmap", ] +auto_suspend_enabled: true +auto_suspend_check_interval: 1min +auto_suspend_idle_timeout_cycles: 2 +auto_suspend_cpu_idle_threshold: 90.0 +auto_suspend_gpu_util_threshold: 10.0 # # Optional global model preset options applied under [*] in models.ini. # # Keys map to llama.cpp CLI args without leading dashes, short args, or LLAMA_ARG_* env names. # llama_server_preset_global: -# ctx-size: 131072 +# ctx-size: 98304 # n-gpu-layers: all diff --git a/ansible/roles/auto_suspend/README.md b/ansible/roles/auto_suspend/README.md new file mode 100644 index 0000000..e80adc0 --- /dev/null +++ b/ansible/roles/auto_suspend/README.md @@ -0,0 +1,388 @@ +# auto_suspend — Idle-based System Suspend + +Auto-suspend monitors a system's CPU and GPU utilization, applies exponential moving average (EMA) smoothing, and suspends the machine when idle for a configured period. + +**Target platform:** RHEL/EL 9 with SELinux enforcing (Fedora/RHEL datacenter GPUs) + +--- + +## Goal + +Automatically put idle GPU workstations into suspend (RAM sleep) to save power and reduce heat, while ensuring no user session or active workload is interrupted. + +The system checks every 5 minutes. It suspends only after `idle_timeout_cycles` consecutive checks confirm the system is idle, using EMA-smoothed metrics to avoid false positives from transient spikes. + +--- + +## Architecture + +``` +┌─────────────┐ every 5 min ┌──────────────────┐ +│ timer │ ────────────────────▶ │ service │ +│ auto-suspend│ │ auto-suspend │ +└─────────────┘ └────────┬─────────┘ + │ + measures: cpu, nvidia, amd + │ + EMA smoothing + warmup check + │ + evaluate idle? + ╱ ╲ + yes ╱ ╲ no + │ │ + ▼ ▼ + idle_counter++ idle_counter=0 + │ + counter ≥ 4? + ╱ ╲ + yes ╱ ╲ no + │ │ + ▼ ▼ + systemctl re-persist + suspend state.json +``` + +### Key design decisions + +- **State persists across timer invocations** — `state.json` carries EMA averages, idle counter, and sample count between checks. The system can accumulate 20+ minutes of data across reboots (state survives suspend). +- **Any busy GPU prevents suspend** — max utilization across all NVIDIA + AMD GPUs is used. One busy GPU = no suspend. +- **Reset on resume** — after `systemctl suspend`, state is fully reset so the warmup period starts fresh. +- **GPU presence is optional** — `None` from a GPU driver means "no GPU of this type", which does NOT block suspend. + +--- + +## Directory Structure + +``` +ansible/roles/auto_suspend/ +├── defaults/main.yml # Configurable parameters (tunable via inventory/host_vars) +├── meta/main.yml # Galaxy metadata +└── tasks/ +│ └── main.yml # Ansible tasks (dependencies, scripts, systemd, SELinux) +└── files/ + ├── auto_suspend.py # Main Python daemon (522 lines) + ├── test_auto_suspend.py # Unit tests (48 tests, stdlib only) + └── fixtures/ # Real command samples from gpu-01 + ├── amd-smi-monitor--json.txt + ├── mpstat--1--1.idle.txt + ├── nvidia-smi--query-gpu.utilization.gpu.csv.noheader.idle.txt + └── who.with-users.txt +``` + +--- + +## Playbook Usage + +```yaml +# ansible/playbook_auto_suspend.yml +- name: Setup Auto Suspend + hosts: auto_suspend + roles: + - auto_suspend +``` + +Deploy via: +```bash +cd ansible/ +sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_auto_suspend.yml +``` + +The `auto_suspend` inventory group currently contains only `gpu-01.lab.alexpires.me`. + +--- + +## Configuration (defaults/main.yml) + +| Variable | Default | Description | +|---|---|---| +| `auto_suspend_alpha` | `0.3` | EMA weight for current sample (0.3 = 30% current, 70% historical) | +| `auto_suspend_min_samples` | `3` | Warmup checks before trust (3 × 5 min = 15 min minimum) | +| `auto_suspend_idle_timeout_cycles` | `4` | Consecutive idle checks before suspend (4 × 5 min = 20 min total) | +| `auto_suspend_cpu_idle_threshold` | `90.0` | CPU idle % must exceed this to count as idle | +| `auto_suspend_gpu_util_threshold` | `5.0` | GPU util % must be below this to count as idle | +| `auto_suspend_state_dir` | `/var/lib/auto_suspend` | Directory for state.json persistence | +| `auto_suspend_log_file` | `/var/log/auto_suspend.log` | Log file path (RotatingFileHandler, 1 MB × 3 backups) | +| `auto_suspend_grub_config_path_bios` | `/boot/grub2/grub.cfg` | GRUB config output path for BIOS systems | + +All variables can be overridden via inventory, group_vars, or host_vars. + +--- + +## Deployed Artifacts + +| Path | Type | Purpose | +|---|---|---| +| `/usr/local/bin/auto_suspend.py` | Python script | Main daemon, copied from `files/auto_suspend.py` | +| `/etc/systemd/system/auto-suspend.service` | systemd oneshot | Runs the check + suspend logic | +| `/etc/systemd/system/auto-suspend.timer` | systemd timer | Triggers service every 5 minutes (5 min after boot) | +| `/etc/systemd/sleep.conf.d/mem-deep.conf` | sysctl override | Sets `MemorySleepMode=deep` | +| `/var/lib/auto_suspend/state.json` | JSON | EMA state, idle counter, sample count | +| `/var/log/auto_suspend.log` | log | Rotating log (1 MB, 3 backups) | + +--- + +## auto_suspend.py — Script Reference + +### CLI Arguments + +All arguments map directly to `defaults/main.yml` variables: + +``` +--alpha 0.3 EMA smoothing factor +--min-samples 3 Warmup period +--idle-timeout-cycles 4 Consecutive idle checks needed +--cpu-idle-threshold 90.0 CPU idle % threshold +--gpu-util-threshold 5.0 GPU util % threshold +--state-dir /var/lib/auto_suspend +--log-file /var/log/auto_suspend.log +--debug Enable debug-level logging +--dry-run Run check without persisting state or suspending +``` + +### Public Functions + +| Function | Returns | Purpose | +|---|---|---| +| `setup_logging(log_file, debug)` | `logging.Logger` | Configures file + stderr handlers | +| `fresh_state()` | `dict` | Returns default state with all values at initial values | +| `load_state(state_dir, logger)` | `dict` | Loads state from disk, returns fresh on any error | +| `save_state(state_dir, state, logger)` | `None` | Atomic write via temp file + os.replace | +| `measure_cpu(logger)` | `float \| None` | Runs `mpstat 1 1`, returns %idle | +| `measure_gpu_nvidia(logger)` | `float \| None` | Runs `nvidia-smi`, returns max util across all GPUs | +| `measure_gpu_amd(logger)` | `float \| None` | Runs `amd-smi monitor --json`, returns gfx.value of GPU 0 | +| `get_active_users(logger)` | `int` | Parses `who` output, returns non-empty line count | +| `get_why_not_idle(cpu_avg, gpu_avg, sample_count, idle_counter, has_users, cpu_threshold, gpu_threshold)` | `list[str]` | Returns human-readable reasons why system is not idle | +| `update_averages(state, cpu_n, gpu_n, alpha)` | `dict` | EMA update: returns new state with averaged values | +| `evaluate(state, has_users, cpu_threshold, gpu_threshold)` | `bool` | Gate checks: warmup → users → CPU → GPU | +| `signal_handler(signum, frame)` | `NoReturn` | SIGTERM/SIGINT handler, calls `sys.exit(0)` | +| `main()` | `NoReturn` | Entry point: measures, evaluates, updates state, suspends | + +### State Schema (`state.json`) + +```json +{ + "cpu_idle_n": null, // latest CPU idle % (current sample) + "cpu_idle_n1": null, // latest CPU idle % (previous sample, n-1) + "gpu_util_n": null, // latest GPU util % (current sample) + "gpu_util_n1": null, // latest GPU util % (previous sample, n-1) + "cpu_avg": null, // EMA-smoothed CPU idle average + "gpu_avg": null, // EMA-smoothed GPU utilization average + "idle_counter": 0, // consecutive idle cycles + "sample_count": 0 // total samples collected +} +``` + +### EMA Formula + +``` +EMA_new = alpha * current_sample + (1 - alpha) * previous_EMA +``` + +With `alpha=0.3`: 30% weight on the current reading, 70% on the historical average. `round(..., 2)` keeps precision to 2 decimal places. + +### GPU Detection + +**NVIDIA** — `nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits` +- Parses each line as float, returns `max()` across all GPUs +- Returns `None` if nvidia-smi is missing, fails, or times out +- `None` means no NVIDIA GPU present (does NOT block suspend) + +**AMD** — `amd-smi monitor --json` +- Parses `data[0]["gfx"]["value"]` from first GPU +- Handles both `{"value": N, "unit": "%"}` and raw numeric `gfx` formats +- Returns `None` if amd-smi is missing, fails, or times out + +**Combined** — `max()` across all non-None GPU measurements. Any busy GPU prevents suspend. + +### Suspend Decision Logic (evaluate) + +Four gates in order: + +1. **Warmup** — If `sample_count < min_samples`, return `False` (skip evaluation) +2. **Active users** — If `who` returns non-empty output, return `False` +3. **CPU idle** — If `cpu_avg > cpu_threshold` is False, return `False` +4. **GPU idle** — If GPU is present (`gpu_avg is not None`) and `gpu_avg >= gpu_threshold`, return `False` + +Only when all gates pass does `idle_counter` increment. When `idle_counter >= idle_timeout_cycles`, `systemctl suspend` is called. On failure to suspend, `idle_counter` resets to 0. + +### Dry-Run Mode + +`--dry-run` runs all measurement, EMA, and evaluation logic but: +- Does NOT persist state to disk +- Does NOT call `systemctl suspend` +- Prints `Would suspend: idle timeout reached (idle_counter=X/Y)` instead + +--- + +## Unit Tests (`test_auto_suspend.py`) + +48 tests covering all public functions using `unittest` + `unittest.mock`. + +### Test Classes + +| Class | Tests | Coverage | +|---|---|---| +| `TestFreshState` | 1 | Returns correct defaults | +| `TestLoadState` | 4 | Missing file, valid JSON, invalid JSON, missing keys merged | +| `TestSaveState` | 3 | Atomic write, parent dir creation, no leftover tmp | +| `TestMeasureCpu` | 4 | Idle, busy, non-zero exit, timeout | +| `TestMeasureGpuNvidia` | 6 | Idle, busy single, multi-GPU max, file not found, timeout, non-zero exit | +| `TestMeasureGpuAmd` | 7 | Busy, idle, medium, file not found, short output, empty array, non-zero exit | +| `TestGetActiveUsers` | 3 | No users, with users, timeout | +| `TestUpdateAverages` | 4 | First sample, EMA calc, n→n1 shift, None preservation | +| `TestEvaluate` | 10 | Idle system, warmup skip, users, CPU not idle, GPU busy, GPU None, at threshold CPU, at threshold GPU, custom thresholds | +| `TestGetWhyNotIdle` | 5 | Users, CPU low, GPU high, warming up, multiple reasons | +| `TestSetupLogging` | 1 | Returns logger with handlers | +| `TestSignalHandler` | 1 | Returns exit code 0 | + +### Running Tests + +```bash +cd ansible/roles/auto_suspend +python3 -m pytest files/test_auto_suspend.py -v +# or +python3 files/test_auto_suspend.py +``` + +Tests use `unittest.mock.patch("auto_suspend.subprocess.run")` to intercept all subprocess calls. Fixture data comes from real command outputs on gpu-01. + +--- + +## Making Changes + +### Modifying the Python Script (`files/auto_suspend.py`) + +1. Edit `files/auto_suspend.py` +2. Add or update tests in `files/test_auto_suspend.py` — mock `subprocess.run` with `MagicMock(returncode=..., stdout=..., stderr=...)` +3. Add new fixtures to `files/fixtures/` if you need new command output samples +4. Run `python3 -m pytest files/test_auto_suspend.py -v` — all 48 tests must pass +5. Deploy to gpu-01 and verify with `journalctl -u auto-suspend.service` + +### Modifying Ansible Tasks (`tasks/main.yml`) + +The role uses inline content for systemd unit files (no `templates/` directory). To change systemd configuration, edit the inline YAML in the relevant `ansible.builtin.copy` task. + +Key tasks in order: +1. Install dependencies (`sysstat`, `python3`) +2. Enable sysstat service +3. Set GRUB `mem_sleep_default=deep` (BIOS only) +4. Create state directory + log file +5. Copy Python script +6. Set SELinux context (`bin_t` + `restorecon`) +7. Create systemd service (inline) +8. Create sleep.conf.d deep sleep override +9. Create systemd timer (inline) +10. Reload systemd daemon +11. Enable + start timer +12. Verify with `--dry-run` + +### Adding a New GPU Vendor + +1. Add `measure_gpu_(logger)` in `auto_suspend.py` following the existing pattern (return `float` utilization or `None`) +2. Wire it into `main()`: + ```python + gpu_ = measure_gpu_(logger) + gpu_values = [v for v in [gpu_nvidia, gpu_amd, gpu_] if v is not None] + gpu_util = max(gpu_values) if gpu_values else None + ``` +3. Add CLI argument if vendor-specific threshold is needed +4. Add fixtures for test data +5. Add `TestMeasureGpu` test class + +### Overriding Defaults + +All defaults in `defaults/main.yml` can be overridden via: +- Inventory: `group_vars/auto_suspend.yml` +- Host vars: `host_vars/gpu-01.lab.alexpires.me.yml` +- Command line: `ansible-playbook ... -e "auto_suspend_alpha=0.5"` + +--- + +## SELinux + +On EL 9 with SELinux enforcing, the Python script in `/usr/local/bin/` needs `bin_t` context: + +```yaml +- name: Set SELinux file context on script + community.general.sefcontext: + path: /usr/local/bin/auto_suspend.py + setype: bin_t + state: present + failed_when: false + +- name: Restore SELinux context on script + ansible.builtin.command: restorecon -v /usr/local/bin/auto_suspend.py + failed_when: false +``` + +Both tasks use `failed_when: false` to tolerate systems where SELinux is disabled or the context is already correct. + +Requires `community.general` Ansible collection. + +--- + +## Systemd Configuration + +### auto-suspend.service +- `Type=oneshot` — runs once per timer tick +- `Restart=on-failure`, `RestartSec=30` — retries on error +- Passes all config as CLI flags (not via environment) + +### auto-suspend.timer +- `OnBootSec=5min` — first run 5 minutes after boot +- `OnUnitActiveSec=5min` — subsequent runs every 5 minutes +- `Unit=auto-suspend.service` + +### sleep.conf.d +`/etc/systemd/sleep.conf.d/mem-deep.conf` sets `MemorySleepMode=deep` to ensure deep sleep state (lowest power consumption). + +### GRUB +The role ensures `mem_sleep_default=deep` is present in `GRUB_CMDLINE_LINUX` and regenerates the GRUB config. This is BIOS-specific (UEFI is not currently supported). + +--- + +## Hardware Target + +The role was developed for `gpu-01.lab.alexpires.me`: +- Fedora 42 (kernel 6.17.11), 12-core x86_64 +- NVIDIA GPU + AMD GPU (dual-GPU system) +- SELinux enforcing +- Used for AI inference (Ollama, ComfyUI, llama.cpp, Qwen) +- Network-accessible workstation + +--- + +## Troubleshooting + +### Check service status +```bash +systemctl status auto-suspend.service +systemctl status auto-suspend.timer +``` + +### View logs +```bash +journalctl -u auto-suspend.service --no-pager +cat /var/log/auto_suspend.log +``` + +### Run a manual check +```bash +/usr/local/bin/auto_suspend.py --dry-run --debug +``` + +### Check state +```bash +cat /var/lib/auto_suspend/state.json +``` + +### Force a timer run +```bash +systemctl start auto-suspend.service +``` + +### Re-deploy after changes +```bash +cd ansible/ +sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_auto_suspend.yml +``` diff --git a/ansible/roles/auto_suspend/defaults/main.yml b/ansible/roles/auto_suspend/defaults/main.yml index 950d827..025fcd8 100644 --- a/ansible/roles/auto_suspend/defaults/main.yml +++ b/ansible/roles/auto_suspend/defaults/main.yml @@ -8,3 +8,4 @@ auto_suspend_cpu_idle_threshold: 90.0 auto_suspend_gpu_util_threshold: 5.0 auto_suspend_state_dir: /var/lib/auto_suspend auto_suspend_log_file: /var/log/auto_suspend.log +auto_suspend_check_interval: 5min diff --git a/ansible/roles/auto_suspend/files/auto_suspend.py b/ansible/roles/auto_suspend/files/auto_suspend.py index cb8732d..d5f2936 100644 --- a/ansible/roles/auto_suspend/files/auto_suspend.py +++ b/ansible/roles/auto_suspend/files/auto_suspend.py @@ -11,7 +11,6 @@ across timer invocations. State is fully reset on suspend resume. """ import argparse -import csv import json import logging import logging.handlers @@ -142,11 +141,17 @@ def measure_cpu(logger): Runs ``mpstat 1 1`` and extracts the last field of the "Average:" line, which is ``%idle``. Returns a float, or None on failure. + + Uses ``LC_ALL=C`` to force English locale output regardless of system + locale (avoids issues like Portuguese ``Média:`` with comma decimals). """ + env = dict(os.environ) + env["LC_ALL"] = "C" try: result = subprocess.run( ["mpstat", "1", "1"], capture_output=True, text=True, timeout=15, + env=env, ) if result.returncode != 0: logger.warning("mpstat returned non-zero: %s", result.stderr.strip()) @@ -161,6 +166,7 @@ def measure_cpu(logger): logger.debug("CPU idle: %.1f%%", idle_pct) return idle_pct + logger.warning("mpstat output did not contain expected data line") return None except subprocess.TimeoutExpired: logger.warning("mpstat timed out") @@ -198,7 +204,7 @@ def measure_gpu_nvidia(logger): return None max_util = max(utils) - logger.debug("NVIDIA GPU utilization: max=%.1f%%", max_util) + logger.info("NVIDIA GPU utilization: max=%.1f%%", max_util) return max_util except FileNotFoundError: logger.debug("nvidia-smi not found") @@ -215,51 +221,45 @@ def measure_gpu_amd(logger): """ Measure GPU utilization via amd-smi. - Parses the ``gfx`` column from ``amd-smi monitor --csv`` by header name - (not by fixed column index). Returns None if amd-smi is unavailable. + Parses the ``gfx.value`` field from ``amd-smi monitor --json``. + Returns the utilization of the first GPU, or None if amd-smi is unavailable. """ try: result = subprocess.run( - ["amd-smi", "monitor", "--csv"], + ["amd-smi", "monitor", "--json"], capture_output=True, text=True, timeout=15, ) if result.returncode != 0: logger.debug("amd-smi returned %d — no AMD GPU", result.returncode) return None - lines = result.stdout.strip().splitlines() - if len(lines) < 2: - logger.debug("amd-smi output too short") + raw = result.stdout.strip() + + data = json.loads(result.stdout) + if not data or not isinstance(data, list): + logger.debug("amd-smi JSON output empty or invalid") return None - # Filter out non-data lines (comments, annotations with wrong field count) - csv_reader = csv.reader(lines) - header = next(csv_reader) - for fields in csv_reader: - if len(fields) != len(header): - continue - row = dict(zip(header, fields)) - try: - # Try both "gfx" and "gfx_util" column names - gfx_val = row.get("gfx") or row.get("gfx_util") - if gfx_val is None: - logger.debug("amd-smi CSV has no 'gfx' column; columns: %s", - list(row.keys())) - return None - util = float(gfx_val) - logger.debug("AMD GPU utilization: %.1f%%", util) - return util - except (ValueError, TypeError): - continue - - return None + gpu = data[0] + gfx = gpu.get("gfx") + if isinstance(gfx, dict): + gfx_val = gfx.get("value") + else: + gfx_val = gfx + if gfx_val is None: + logger.debug("amd-smi JSON has no 'gfx.value' field; fields: %s", + list(gpu.keys())) + return None + util = float(gfx_val) + logger.info("AMD GPU utilization: %.1f%% (gfx=%s)", util, gfx_val) + return util except FileNotFoundError: logger.debug("amd-smi not found") return None except subprocess.TimeoutExpired: logger.warning("amd-smi timed out") return None - except BaseException as exc: + except (json.JSONDecodeError, ValueError, TypeError) as exc: logger.warning("Failed to parse amd-smi output: %s", exc) return None @@ -339,24 +339,25 @@ def update_averages(state, cpu_n, gpu_n, alpha, logger): # Evaluation # --------------------------------------------------------------------------- -def get_why_not_idle(state, has_users, logger): +def get_why_not_idle(state, has_users, cpu_idle_threshold=DEFAULT_CPU_IDLE_THRESHOLD, + gpu_util_threshold=DEFAULT_GPU_UTIL_THRESHOLD, min_samples=DEFAULT_MIN_SAMPLES, logger=None): """Return a human-readable reason why the system is not idle.""" reasons = [] if has_users: reasons.append("users logged in") if state["cpu_avg"] is not None: - if state["cpu_avg"] <= DEFAULT_CPU_IDLE_THRESHOLD: + if state["cpu_avg"] <= cpu_idle_threshold: reasons.append( - f"CPU idle avg {state['cpu_avg']:.1f}% <= threshold {DEFAULT_CPU_IDLE_THRESHOLD:.0f}%" + f"CPU idle avg {state['cpu_avg']:.1f}% <= threshold {cpu_idle_threshold:.0f}%" ) if state["gpu_avg"] is not None: - if state["gpu_avg"] >= DEFAULT_GPU_UTIL_THRESHOLD: + if state["gpu_avg"] >= gpu_util_threshold: reasons.append( - f"GPU util avg {state['gpu_avg']:.1f}% >= threshold {DEFAULT_GPU_UTIL_THRESHOLD:.0f}%" + f"GPU util avg {state['gpu_avg']:.1f}% >= threshold {gpu_util_threshold:.0f}%" ) if not reasons: - if state["sample_count"] < DEFAULT_MIN_SAMPLES: - reasons.append(f"warming up ({state['sample_count']}/{DEFAULT_MIN_SAMPLES} samples)") + if state["sample_count"] < min_samples: + reasons.append(f"warming up ({state['sample_count']}/{min_samples} samples)") else: reasons.append("unknown") return "; ".join(reasons) @@ -491,17 +492,17 @@ def main(): try: result = subprocess.run( - ["systemctl", "suspend"], + ["shutdown", "now"], capture_output=True, text=True, timeout=10, ) if result.returncode != 0: - logger.error("systemctl suspend failed: %s", result.stderr.strip()) + logger.error("shutdown failed: %s", result.stderr.strip()) else: - logger.info("systemctl suspend succeeded") + logger.info("shutdown succeeded") except subprocess.TimeoutExpired: - logger.error("systemctl suspend timed out") + logger.error("shutdown timed out") except FileNotFoundError: - logger.error("systemctl not found — cannot suspend") + logger.error("shutdown not found — cannot suspend") except BaseException as exc: logger.error("Failed to suspend: %s", exc) @@ -511,7 +512,7 @@ def main(): else: if not args.dry_run: state["idle_counter"] = 0 - reason = get_why_not_idle(state, has_users, logger) + reason = get_why_not_idle(state, has_users, args.cpu_idle_threshold, args.gpu_util_threshold, args.min_samples, logger) logger.info("Not idle (%s) — reset counter", reason) # --- Save and exit --- diff --git a/ansible/roles/auto_suspend/files/fixtures/amd-smi-monitor--csv.txt b/ansible/roles/auto_suspend/files/fixtures/amd-smi-monitor--csv.txt deleted file mode 100644 index d18fea6..0000000 --- a/ansible/roles/auto_suspend/files/fixtures/amd-smi-monitor--csv.txt +++ /dev/null @@ -1,3 +0,0 @@ -gpu,power_usage,hotspot_temperature,memory_temperature,gfx,gfx_clock,mem,mem_clock,encoder,decoder,vclock,dclock,single_bit_ecc,double_bit_ecc,pcie_replay,vram_used,vram_total,pcie_bw,pviol,tviol,phot_tviol,vr_tviol,hbm_tviol -0,20,52,60,100,1200,0,96,N/A,0.0,25,25,0,0,N/A,11913,16334,N/A,N/A,N/A,N/A,N/A,N/A - diff --git a/ansible/roles/auto_suspend/files/fixtures/amd-smi-monitor--json.txt b/ansible/roles/auto_suspend/files/fixtures/amd-smi-monitor--json.txt new file mode 100644 index 0000000..7d1b071 --- /dev/null +++ b/ansible/roles/auto_suspend/files/fixtures/amd-smi-monitor--json.txt @@ -0,0 +1,27 @@ +[ + { + "gpu": 0, + "power_usage": {"value": 20, "unit": "W"}, + "hotspot_temperature": {"value": 52, "unit": "C"}, + "memory_temperature": {"value": 60, "unit": "C"}, + "gfx": {"value": 100, "unit": "%"}, + "gfx_clock": {"value": 1200, "unit": "MHz"}, + "mem": {"value": 0, "unit": "%"}, + "mem_clock": {"value": 96, "unit": "MHz"}, + "encoder": "N/A", + "decoder": {"value": 0.0, "unit": "%"}, + "vclock": {"value": 25, "unit": "MHz"}, + "dclock": {"value": 25, "unit": "MHz"}, + "single_bit_ecc": 0, + "double_bit_ecc": 0, + "pcie_replay": "N/A", + "vram_used": {"value": 11913, "unit": "MB"}, + "vram_total": {"value": 16334, "unit": "MB"}, + "pcie_bw": {"value": "N/A", "unit": "Mb/s"}, + "pviol": {"value": "N/A", "unit": "%"}, + "tviol": {"value": "N/A", "unit": "%"}, + "phot_tviol": {"value": "N/A", "unit": "%"}, + "vr_tviol": {"value": "N/A", "unit": "%"}, + "hbm_tviol": {"value": "N/A", "unit": "%"} + } +] diff --git a/ansible/roles/auto_suspend/files/test_auto_suspend.py b/ansible/roles/auto_suspend/files/test_auto_suspend.py index 8502540..4fa535c 100644 --- a/ansible/roles/auto_suspend/files/test_auto_suspend.py +++ b/ansible/roles/auto_suspend/files/test_auto_suspend.py @@ -48,13 +48,9 @@ NVIDIA_BUSY_4GAPS = """ 0 NVIDIA_BUSY_SINGLE = """ 95 """ NVIDIA_NONE = "" -AMD_SMII_GFX = _load_fixture("amd-smi-monitor--csv.txt") -AMD_SMII_GFX_0 = """gpu,power_usage,hotspot_temperature,memory_temperature,gfx,gfx_clock,mem,mem_clock,encoder,decoder,vclock,dclock,single_bit_ecc,double_bit_ecc,pcie_replay,vram_used,vram_total,pcie_bw,pviol,tviol,phot_tviol,vr_tviol,hbm_tviol -0,20,52,62,0,1200,0,96,N/A,0.0,25,25,0,0,N/A,11928,16334,N/A,N/A,N/A,N/A,N/A,N/A -""" -AMD_SMII_GFX_80 = """gpu,power_usage,hotspot_temperature,memory_temperature,gfx,gfx_clock,mem,mem_clock,encoder,decoder,vclock,dclock,single_bit_ecc,double_bit_ecc,pcie_replay,vram_used,vram_total,pcie_bw,pviol,tviol,phot_tviol,vr_tviol,hbm_tviol -0,180,78,82,80,1800,90,200,N/A,45.0,25,25,0,0,N/A,14000,16334,N/A,N/A,N/A,N/A,N/A,N/A -""" +AMD_SMII_JSON = _load_fixture("amd-smi-monitor--json.txt") +AMD_SMII_GFX_0 = '[{"gpu":0,"gfx":{"value":0,"unit":"%"}}]' +AMD_SMII_GFX_80 = '[{"gpu":0,"gfx":{"value":80,"unit":"%"}}]' WHO_WITH_USERS = _load_fixture("who.with-users.txt") WHO_EMPTY = "" @@ -65,10 +61,6 @@ MPSTAT_HIGH = """Linux 6.17.11-200.fc42.x86_64 (gpu-01) 06/08/26 _x86_64_ (12 Average: all 85.30 0.00 8.20 0.50 0.00 0.10 0.00 0.00 0.00 5.90 """ -AMD_SMII_ONLY_HEADER = """gpu,power_usage,hotspot_temperature,memory_temperature,gfx -0,20,52,62,0 -""" - # --------------------------------------------------------------------------- # Tests @@ -194,6 +186,22 @@ class TestMeasureCpu(unittest.TestCase): self.assertIsNone(result) logger.warning.assert_called_with("mpstat timed out") + def test_portuguese_locale(self): + """Verify mpstat with C locale (LC_ALL=C) parses correctly. + + The measure_cpu function forces LC_ALL=C so mpstat always outputs + in English regardless of system locale. + """ + from auto_suspend import measure_cpu + logger = MagicMock() + with patch("auto_suspend.subprocess.run") as run: + run.return_value = _make_mock_result(MPSTAT_IDLE) + result = measure_cpu(logger) + # Verify subprocess was called with LC_ALL=C + call_kwargs = run.call_args + self.assertEqual(call_kwargs[1]["env"]["LC_ALL"], "C") + self.assertAlmostEqual(result, 99.58, places=1) + class TestMeasureGpuNvidia(unittest.TestCase): @@ -253,7 +261,7 @@ class TestMeasureGpuAmd(unittest.TestCase): from auto_suspend import measure_gpu_amd logger = MagicMock() with patch("auto_suspend.subprocess.run") as run: - run.return_value = _make_mock_result(AMD_SMII_GFX) + run.return_value = _make_mock_result(AMD_SMII_JSON) result = measure_gpu_amd(logger) self.assertAlmostEqual(result, 100.0, places=1) @@ -289,6 +297,14 @@ class TestMeasureGpuAmd(unittest.TestCase): result = measure_gpu_amd(logger) self.assertIsNone(result) + def test_empty_array(self): + from auto_suspend import measure_gpu_amd + logger = MagicMock() + with patch("auto_suspend.subprocess.run") as run: + run.return_value = _make_mock_result("[]") + result = measure_gpu_amd(logger) + self.assertIsNone(result) + def test_returncode_nonzero(self): from auto_suspend import measure_gpu_amd logger = MagicMock() @@ -371,6 +387,77 @@ class TestUpdateAverages(unittest.TestCase): self.assertAlmostEqual(state["gpu_avg"], 5.0) self.assertEqual(state["sample_count"], 2) + def test_ema_decay_idle_gpu(self): + """ + Simulate GPU going from 100% to 0% idle and verify EMA decays. + + This validates that the EMA calculation correctly handles the scenario + where a busy GPU becomes idle — the average should converge toward + idle values after enough consecutive idle samples. + """ + from auto_suspend import fresh_state, update_averages + logger = MagicMock() + state = fresh_state() + + # Phase 1: GPU was busy (simulate 10 consecutive 100% samples) + for _ in range(10): + state = update_averages(state, cpu_n=95.0, gpu_n=100.0, alpha=0.3, logger=logger) + + self.assertAlmostEqual(state["gpu_avg"], 100.0, places=1) + self.assertEqual(state["sample_count"], 10) + + # Phase 2: GPU goes idle (0%) for 20 consecutive samples + for _ in range(20): + state = update_averages(state, cpu_n=99.0, gpu_n=0.0, alpha=0.3, logger=logger) + + # After 20 idle samples from 100%, EMA should be very close to 0 + # EMA = 0.7^20 * 100 ≈ 0.08% + self.assertAlmostEqual(state["gpu_avg"], 0.08, places=1) + self.assertEqual(state["sample_count"], 30) + + def test_ema_convergence_multiple_values(self): + """ + Verify EMA converges to the steady-state value regardless of start. + + With alpha=0.3 and 100 identical samples, the average should equal + the sample value (since EMA_new = 0.3*x + 0.7*x = x). + """ + from auto_suspend import fresh_state, update_averages + logger = MagicMock() + state = fresh_state() + + gpu_idle = 2.0 + for _ in range(50): + state = update_averages(state, cpu_n=95.0, gpu_n=gpu_idle, alpha=0.3, logger=logger) + + self.assertAlmostEqual(state["gpu_avg"], gpu_idle, places=2) + self.assertAlmostEqual(state["cpu_avg"], 95.0, places=2) + + def test_ema_with_missing_gpu_sample(self): + """ + Verify None GPU sample preserves the average. + + When measure_gpu_amd returns None (e.g., tool unavailable), the + average should NOT be updated — it should keep the last value. + """ + from auto_suspend import fresh_state, update_averages + logger = MagicMock() + state = fresh_state() + + # Seed with a GPU average + state = update_averages(state, cpu_n=95.0, gpu_n=10.0, alpha=0.3, logger=logger) + self.assertAlmostEqual(state["gpu_avg"], 10.0) + + # Now GPU measurement fails (None) — average should be preserved + state = update_averages(state, cpu_n=95.0, gpu_n=None, alpha=0.3, logger=logger) + self.assertAlmostEqual(state["gpu_avg"], 10.0) + self.assertEqual(state["sample_count"], 2) + + # After GPU returns, EMA should continue from the preserved value + state = update_averages(state, cpu_n=95.0, gpu_n=0.0, alpha=0.3, logger=logger) + # EMA = 0.3 * 0 + 0.7 * 10 = 7.0 + self.assertAlmostEqual(state["gpu_avg"], 7.0) + class TestEvaluate(unittest.TestCase): diff --git a/ansible/roles/auto_suspend/tasks/main.yml b/ansible/roles/auto_suspend/tasks/main.yml index 825c8bb..d147ed4 100644 --- a/ansible/roles/auto_suspend/tasks/main.yml +++ b/ansible/roles/auto_suspend/tasks/main.yml @@ -143,11 +143,11 @@ dest: /etc/systemd/system/auto-suspend.timer content: | [Unit] - Description=Run idle suspend check every 5 minutes + Description=Run idle suspend check every {{ auto_suspend_check_interval }} [Timer] - OnBootSec=5min - OnUnitActiveSec=5min + OnBootSec={{ auto_suspend_check_interval }} + OnUnitActiveSec={{ auto_suspend_check_interval }} Unit=auto-suspend.service [Install]