auto_suspend: busctl->shutdown now, 1min timer config, AMD JSON parsing, locale fix, unit tests
This commit is contained in:
@@ -0,0 +1,114 @@
|
||||
# AGENTS.md - A13Labs Infrastructure
|
||||
|
||||
## Repo Structure
|
||||
|
||||
- **`ansible/`** — Playbooks, roles, inventory. Runs against bare-metal servers.
|
||||
- **`terraform/`** — IaC (OpenTofu for K8s app deployments + cloud provisioning).
|
||||
- **`inventory/hosts`** — Ansible inventory. All hosts and groups.
|
||||
- **`requirements/`** — pip requirements files: `ansible.txt`, `dev.txt`, `opentofu.txt`.
|
||||
- **`sectool`** — Secret manager (Bitwarden, configured in `sectool.json`). Both root and `ansible/` have their own `sectool.json` + `sectool.env`.
|
||||
|
||||
## Key Facts
|
||||
|
||||
- **No `Makefile`, no `justfile`, no pre-commit config.** Commands are run directly.
|
||||
- **All GitHub workflows are `.disabled`** — CI is not active. Verification is local-only.
|
||||
- **`ansible/ansible.cfg`** points to `../inventory/hosts` (relative to `ansible/` dir). Always run playbooks from the `ansible/` directory, or use `-i` to override.
|
||||
- **YAML linting:** `ansible-lint` config at `ansible/.ansible-lint`. `.yamllint.yml` at `ansible/.yamllint.yml` (120 char line warning).
|
||||
- **No `.pre-commit-config.yaml` exists** — linting is manual or CI-only.
|
||||
|
||||
## Ansible
|
||||
|
||||
### Required Environment vars
|
||||
|
||||
```sh
|
||||
export ANSIBLE_USER=provision
|
||||
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES # if on a MAC
|
||||
```
|
||||
|
||||
### Running playbooks
|
||||
```sh
|
||||
cd ansible
|
||||
sectool exec ansible-playbook -u $ANSIBLE_USER playbook_<name>.yml
|
||||
```
|
||||
|
||||
### Role testing (Molecule + Podman - Linux Only)
|
||||
```sh
|
||||
cd ansible/roles/<role_name>
|
||||
molecule test # create -> converge -> verify -> destroy
|
||||
molecule converge # run converge only
|
||||
molecule destroy # clean up
|
||||
```
|
||||
|
||||
**Molecule gotchas:**
|
||||
- Always run `molecule` commands **from the role directory**, not from repo root.
|
||||
- Molecule containers use `podman` driver. No systemd by default — guard service tasks with `when: ansible_service_mgr == "systemd"`.
|
||||
- `molecule/default/prepare.yml` must install python3 and sudo via `raw:` before Ansible can run Python modules.
|
||||
- `provisioner.env.ANSIBLE_ROLES_PATH` should point up to the parent (e.g., `../../../`).
|
||||
|
||||
### Ansible-lint
|
||||
```sh
|
||||
ansible-lint # runs from repo root or ansible/
|
||||
```
|
||||
Excluded paths: `.git/`, `.github/`, `tests/`. Warns on: `line-length`, `yaml[line-length]`, `var-naming[no-role-prefix]`.
|
||||
|
||||
## Terraform / OpenTofu
|
||||
|
||||
### App deployments (K8s)
|
||||
- `terraform/apps/dev-01/` — dev Kubernetes cluster (microk8s)
|
||||
- `terraform/apps/prod-01/` — prod Kubernetes cluster (microk8s)
|
||||
- Uses Kubernetes + TLS providers. `provider.tf` sets `insecure = true`.
|
||||
|
||||
### Cloud IaC
|
||||
- `terraform/iac/aws/` — AWS infrastructure
|
||||
- `terraform/iac/contabo/` — Contabo VPS
|
||||
- `terraform/iac/cloudns/` — DNS (cloudns)
|
||||
- `terraform/iac/scaleway/` — Scaleway infrastructure
|
||||
|
||||
### Secrets
|
||||
Each terraform directory has its own `sectool.env`. Run `sectool` to load secrets before `tofu init/plan/apply`:
|
||||
```sh
|
||||
sectool env -d terraform/apps/dev-01/ -- tofu plan
|
||||
```
|
||||
|
||||
### Module structure
|
||||
- `terraform/modules/apps/` — application modules (Nextcloud, etc.)
|
||||
- `terraform/modules/db/` — database modules (MariaDB with backups)
|
||||
- `terraform/modules/utils/` — utilities (exporters, cert-checker, prometheus, trivy, etc.)
|
||||
|
||||
## Scripts (`scripts/`)
|
||||
|
||||
One-off utility scripts — not library code:
|
||||
- `install-opnsense.sh` — boots OPNsense on a VM via virt-install (one-time router setup)
|
||||
- `chronograf.sh` — `kubectl port-forward` wrapper for Chronograf pod
|
||||
- `proxy-pt.sh` — `kubectl port-forward` wrapper for m3uproxy pod
|
||||
- `steamos` — launches SteamOS gaming session with sunshine
|
||||
- `cpu_power_monitor.py` — RAPL power monitoring (requires root)
|
||||
|
||||
## Environment Setup
|
||||
|
||||
```sh
|
||||
# Ansible + dev tools
|
||||
pip install -r requirements/ansible.txt -r requirements/dev.txt
|
||||
|
||||
# Terraform/OpenTofu deps
|
||||
pip install -r requirements/opentofu.txt
|
||||
```
|
||||
|
||||
## Contabo CLI
|
||||
|
||||
```sh
|
||||
sectool exec cntb get instances --oauth2-clientid="$CONTABO_CLIENT_ID" ...
|
||||
```
|
||||
|
||||
## Windows hosts
|
||||
|
||||
- Use `ConfigureRemotingForAnsible.ps1` in `scripts/` to enable WinRM on Windows targets.
|
||||
- On MSYS2: install packages listed in `Readme.MD` (go, python, cryptography, paramiko, rpds-py, etc.).
|
||||
- Python venv: `python -m venv .venv --system-site-packages`
|
||||
|
||||
## Style Conventions
|
||||
|
||||
- YAML: 120 char line length (warning threshold)
|
||||
- Ansible roles follow galaxy conventions: `defaults/`, `tasks/`, `meta/`, `files/`, `templates/`
|
||||
- `tasks/main.yml` is the entrypoint for every role
|
||||
- Host vars: `ansible/host_vars/<hostname>.yml`
|
||||
@@ -132,103 +132,103 @@ llama_server_models:
|
||||
quant: "Q8_0"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "gemma-4-26b-a4b-ud-iq4-xs"
|
||||
model_id: "unsloth/gemma-4-26B-A4B-it-GGUF"
|
||||
quant: "UD-IQ4_XS"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "gemma-4-31b-iq4-xs"
|
||||
model_id: "unsloth/gemma-4-31B-it-GGUF"
|
||||
quant: "IQ4_XS"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-5-9b-iq4-xs"
|
||||
model_id: "unsloth/Qwen3.5-9B-GGUF"
|
||||
quant: "IQ4_XS"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-5-9b-q8-0"
|
||||
model_id: "unsloth/Qwen3.5-9B-GGUF"
|
||||
quant: "Q8_0"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-5-27b-mtp-iq4-xs"
|
||||
model_id: "unsloth/Qwen3.5-27B-MTP-GGUF"
|
||||
quant: "IQ4_XS"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-5-35b-a3b-ud-iq4-xs"
|
||||
model_id: "unsloth/Qwen3.5-35B-A3B-GGUF"
|
||||
quant: "UD-IQ4_XS"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-5-35b-a3b-q4-k-m"
|
||||
model_id: "unsloth/Qwen3.5-35B-A3B-GGUF"
|
||||
quant: "Q4_K_M"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-6-27b-mtp-q4-0"
|
||||
model_id: "unsloth/Qwen3.6-27B-MTP-GGUF"
|
||||
quant: "Q4_0"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-6-27b-mtp-ud-q4-0-k-xl"
|
||||
model_id: "unsloth/Qwen3.6-27B-MTP-GGUF"
|
||||
quant: "UD-Q4_K_XL"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-6-27b-mtp-ud-q4-xl"
|
||||
model_id: "unsloth/Qwen3.6-27B-MTP-GGUF"
|
||||
quant: "UD-Q4_K_XL"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-6-35b-a3b-mtp-ud-iq4-xs"
|
||||
model_id: "unsloth/Qwen3.6-35B-A3B-MTP-GGUF"
|
||||
quant: "UD-IQ4_XS"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-6-35b-a3b-mtp-ud-q4-m"
|
||||
model_id: "unsloth/Qwen3.6-35B-A3B-MTP-GGUF"
|
||||
quant: "UD-Q4_K_M"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-6-35b-a3b-ud-q4-k-m"
|
||||
model_id: "unsloth/Qwen3.6-35B-A3B-GGUF"
|
||||
quant: "UD-Q4_K_M"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "qwen3-6-35b-a3b-claude47-distilled-iq4-xs"
|
||||
model_id: "lordx64/Qwen3.6-35B-A3B-Claude-4.7-Opus-Reasoning-Distilled-IQ4_XS-GGUF"
|
||||
quant: "IQ4_XS"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "ministral-3-14b-instruct-iq4-xs"
|
||||
model_id: "unsloth/Ministral-3-14B-Instruct-2512-GGUF"
|
||||
quant: "IQ4_XS"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "granite-4-1-30b-iq4-xs"
|
||||
model_id: "unsloth/granite-4.1-30b-GGUF"
|
||||
quant: "IQ4_XS"
|
||||
revision: "main"
|
||||
preset:
|
||||
ctx-size: 131072
|
||||
ctx-size: 98304
|
||||
- name: "devstral-small-2-24b-instruct-iq4-xs"
|
||||
model_id: "unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF"
|
||||
quant: "IQ4_XS"
|
||||
@@ -239,7 +239,7 @@ llama_server_argv_extra:
|
||||
"-t",
|
||||
10,
|
||||
"-np",
|
||||
1,
|
||||
-1,
|
||||
"-b",
|
||||
1024,
|
||||
"-ub",
|
||||
@@ -257,7 +257,7 @@ llama_server_argv_extra:
|
||||
"-dev",
|
||||
"Vulkan1,Vulkan2",
|
||||
"-ts",
|
||||
"8,12",
|
||||
"12,16",
|
||||
"-fit",
|
||||
"off",
|
||||
"-mg",
|
||||
@@ -265,8 +265,13 @@ llama_server_argv_extra:
|
||||
"--mmap",
|
||||
]
|
||||
|
||||
auto_suspend_enabled: true
|
||||
auto_suspend_check_interval: 1min
|
||||
auto_suspend_idle_timeout_cycles: 2
|
||||
auto_suspend_cpu_idle_threshold: 90.0
|
||||
auto_suspend_gpu_util_threshold: 10.0
|
||||
# # Optional global model preset options applied under [*] in models.ini.
|
||||
# # Keys map to llama.cpp CLI args without leading dashes, short args, or LLAMA_ARG_* env names.
|
||||
# llama_server_preset_global:
|
||||
# ctx-size: 131072
|
||||
# ctx-size: 98304
|
||||
# n-gpu-layers: all
|
||||
|
||||
@@ -0,0 +1,388 @@
|
||||
# auto_suspend — Idle-based System Suspend
|
||||
|
||||
Auto-suspend monitors a system's CPU and GPU utilization, applies exponential moving average (EMA) smoothing, and suspends the machine when idle for a configured period.
|
||||
|
||||
**Target platform:** RHEL/EL 9 with SELinux enforcing (Fedora/RHEL datacenter GPUs)
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
Automatically put idle GPU workstations into suspend (RAM sleep) to save power and reduce heat, while ensuring no user session or active workload is interrupted.
|
||||
|
||||
The system checks every 5 minutes. It suspends only after `idle_timeout_cycles` consecutive checks confirm the system is idle, using EMA-smoothed metrics to avoid false positives from transient spikes.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────┐ every 5 min ┌──────────────────┐
|
||||
│ timer │ ────────────────────▶ │ service │
|
||||
│ auto-suspend│ │ auto-suspend │
|
||||
└─────────────┘ └────────┬─────────┘
|
||||
│
|
||||
measures: cpu, nvidia, amd
|
||||
│
|
||||
EMA smoothing + warmup check
|
||||
│
|
||||
evaluate idle?
|
||||
╱ ╲
|
||||
yes ╱ ╲ no
|
||||
│ │
|
||||
▼ ▼
|
||||
idle_counter++ idle_counter=0
|
||||
│
|
||||
counter ≥ 4?
|
||||
╱ ╲
|
||||
yes ╱ ╲ no
|
||||
│ │
|
||||
▼ ▼
|
||||
systemctl re-persist
|
||||
suspend state.json
|
||||
```
|
||||
|
||||
### Key design decisions
|
||||
|
||||
- **State persists across timer invocations** — `state.json` carries EMA averages, idle counter, and sample count between checks. The system can accumulate 20+ minutes of data across reboots (state survives suspend).
|
||||
- **Any busy GPU prevents suspend** — max utilization across all NVIDIA + AMD GPUs is used. One busy GPU = no suspend.
|
||||
- **Reset on resume** — after `systemctl suspend`, state is fully reset so the warmup period starts fresh.
|
||||
- **GPU presence is optional** — `None` from a GPU driver means "no GPU of this type", which does NOT block suspend.
|
||||
|
||||
---
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
ansible/roles/auto_suspend/
|
||||
├── defaults/main.yml # Configurable parameters (tunable via inventory/host_vars)
|
||||
├── meta/main.yml # Galaxy metadata
|
||||
└── tasks/
|
||||
│ └── main.yml # Ansible tasks (dependencies, scripts, systemd, SELinux)
|
||||
└── files/
|
||||
├── auto_suspend.py # Main Python daemon (522 lines)
|
||||
├── test_auto_suspend.py # Unit tests (48 tests, stdlib only)
|
||||
└── fixtures/ # Real command samples from gpu-01
|
||||
├── amd-smi-monitor--json.txt
|
||||
├── mpstat--1--1.idle.txt
|
||||
├── nvidia-smi--query-gpu.utilization.gpu.csv.noheader.idle.txt
|
||||
└── who.with-users.txt
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Playbook Usage
|
||||
|
||||
```yaml
|
||||
# ansible/playbook_auto_suspend.yml
|
||||
- name: Setup Auto Suspend
|
||||
hosts: auto_suspend
|
||||
roles:
|
||||
- auto_suspend
|
||||
```
|
||||
|
||||
Deploy via:
|
||||
```bash
|
||||
cd ansible/
|
||||
sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_auto_suspend.yml
|
||||
```
|
||||
|
||||
The `auto_suspend` inventory group currently contains only `gpu-01.lab.alexpires.me`.
|
||||
|
||||
---
|
||||
|
||||
## Configuration (defaults/main.yml)
|
||||
|
||||
| Variable | Default | Description |
|
||||
|---|---|---|
|
||||
| `auto_suspend_alpha` | `0.3` | EMA weight for current sample (0.3 = 30% current, 70% historical) |
|
||||
| `auto_suspend_min_samples` | `3` | Warmup checks before trust (3 × 5 min = 15 min minimum) |
|
||||
| `auto_suspend_idle_timeout_cycles` | `4` | Consecutive idle checks before suspend (4 × 5 min = 20 min total) |
|
||||
| `auto_suspend_cpu_idle_threshold` | `90.0` | CPU idle % must exceed this to count as idle |
|
||||
| `auto_suspend_gpu_util_threshold` | `5.0` | GPU util % must be below this to count as idle |
|
||||
| `auto_suspend_state_dir` | `/var/lib/auto_suspend` | Directory for state.json persistence |
|
||||
| `auto_suspend_log_file` | `/var/log/auto_suspend.log` | Log file path (RotatingFileHandler, 1 MB × 3 backups) |
|
||||
| `auto_suspend_grub_config_path_bios` | `/boot/grub2/grub.cfg` | GRUB config output path for BIOS systems |
|
||||
|
||||
All variables can be overridden via inventory, group_vars, or host_vars.
|
||||
|
||||
---
|
||||
|
||||
## Deployed Artifacts
|
||||
|
||||
| Path | Type | Purpose |
|
||||
|---|---|---|
|
||||
| `/usr/local/bin/auto_suspend.py` | Python script | Main daemon, copied from `files/auto_suspend.py` |
|
||||
| `/etc/systemd/system/auto-suspend.service` | systemd oneshot | Runs the check + suspend logic |
|
||||
| `/etc/systemd/system/auto-suspend.timer` | systemd timer | Triggers service every 5 minutes (5 min after boot) |
|
||||
| `/etc/systemd/sleep.conf.d/mem-deep.conf` | sysctl override | Sets `MemorySleepMode=deep` |
|
||||
| `/var/lib/auto_suspend/state.json` | JSON | EMA state, idle counter, sample count |
|
||||
| `/var/log/auto_suspend.log` | log | Rotating log (1 MB, 3 backups) |
|
||||
|
||||
---
|
||||
|
||||
## auto_suspend.py — Script Reference
|
||||
|
||||
### CLI Arguments
|
||||
|
||||
All arguments map directly to `defaults/main.yml` variables:
|
||||
|
||||
```
|
||||
--alpha 0.3 EMA smoothing factor
|
||||
--min-samples 3 Warmup period
|
||||
--idle-timeout-cycles 4 Consecutive idle checks needed
|
||||
--cpu-idle-threshold 90.0 CPU idle % threshold
|
||||
--gpu-util-threshold 5.0 GPU util % threshold
|
||||
--state-dir /var/lib/auto_suspend
|
||||
--log-file /var/log/auto_suspend.log
|
||||
--debug Enable debug-level logging
|
||||
--dry-run Run check without persisting state or suspending
|
||||
```
|
||||
|
||||
### Public Functions
|
||||
|
||||
| Function | Returns | Purpose |
|
||||
|---|---|---|
|
||||
| `setup_logging(log_file, debug)` | `logging.Logger` | Configures file + stderr handlers |
|
||||
| `fresh_state()` | `dict` | Returns default state with all values at initial values |
|
||||
| `load_state(state_dir, logger)` | `dict` | Loads state from disk, returns fresh on any error |
|
||||
| `save_state(state_dir, state, logger)` | `None` | Atomic write via temp file + os.replace |
|
||||
| `measure_cpu(logger)` | `float \| None` | Runs `mpstat 1 1`, returns %idle |
|
||||
| `measure_gpu_nvidia(logger)` | `float \| None` | Runs `nvidia-smi`, returns max util across all GPUs |
|
||||
| `measure_gpu_amd(logger)` | `float \| None` | Runs `amd-smi monitor --json`, returns gfx.value of GPU 0 |
|
||||
| `get_active_users(logger)` | `int` | Parses `who` output, returns non-empty line count |
|
||||
| `get_why_not_idle(cpu_avg, gpu_avg, sample_count, idle_counter, has_users, cpu_threshold, gpu_threshold)` | `list[str]` | Returns human-readable reasons why system is not idle |
|
||||
| `update_averages(state, cpu_n, gpu_n, alpha)` | `dict` | EMA update: returns new state with averaged values |
|
||||
| `evaluate(state, has_users, cpu_threshold, gpu_threshold)` | `bool` | Gate checks: warmup → users → CPU → GPU |
|
||||
| `signal_handler(signum, frame)` | `NoReturn` | SIGTERM/SIGINT handler, calls `sys.exit(0)` |
|
||||
| `main()` | `NoReturn` | Entry point: measures, evaluates, updates state, suspends |
|
||||
|
||||
### State Schema (`state.json`)
|
||||
|
||||
```json
|
||||
{
|
||||
"cpu_idle_n": null, // latest CPU idle % (current sample)
|
||||
"cpu_idle_n1": null, // latest CPU idle % (previous sample, n-1)
|
||||
"gpu_util_n": null, // latest GPU util % (current sample)
|
||||
"gpu_util_n1": null, // latest GPU util % (previous sample, n-1)
|
||||
"cpu_avg": null, // EMA-smoothed CPU idle average
|
||||
"gpu_avg": null, // EMA-smoothed GPU utilization average
|
||||
"idle_counter": 0, // consecutive idle cycles
|
||||
"sample_count": 0 // total samples collected
|
||||
}
|
||||
```
|
||||
|
||||
### EMA Formula
|
||||
|
||||
```
|
||||
EMA_new = alpha * current_sample + (1 - alpha) * previous_EMA
|
||||
```
|
||||
|
||||
With `alpha=0.3`: 30% weight on the current reading, 70% on the historical average. `round(..., 2)` keeps precision to 2 decimal places.
|
||||
|
||||
### GPU Detection
|
||||
|
||||
**NVIDIA** — `nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits`
|
||||
- Parses each line as float, returns `max()` across all GPUs
|
||||
- Returns `None` if nvidia-smi is missing, fails, or times out
|
||||
- `None` means no NVIDIA GPU present (does NOT block suspend)
|
||||
|
||||
**AMD** — `amd-smi monitor --json`
|
||||
- Parses `data[0]["gfx"]["value"]` from first GPU
|
||||
- Handles both `{"value": N, "unit": "%"}` and raw numeric `gfx` formats
|
||||
- Returns `None` if amd-smi is missing, fails, or times out
|
||||
|
||||
**Combined** — `max()` across all non-None GPU measurements. Any busy GPU prevents suspend.
|
||||
|
||||
### Suspend Decision Logic (evaluate)
|
||||
|
||||
Four gates in order:
|
||||
|
||||
1. **Warmup** — If `sample_count < min_samples`, return `False` (skip evaluation)
|
||||
2. **Active users** — If `who` returns non-empty output, return `False`
|
||||
3. **CPU idle** — If `cpu_avg > cpu_threshold` is False, return `False`
|
||||
4. **GPU idle** — If GPU is present (`gpu_avg is not None`) and `gpu_avg >= gpu_threshold`, return `False`
|
||||
|
||||
Only when all gates pass does `idle_counter` increment. When `idle_counter >= idle_timeout_cycles`, `systemctl suspend` is called. On failure to suspend, `idle_counter` resets to 0.
|
||||
|
||||
### Dry-Run Mode
|
||||
|
||||
`--dry-run` runs all measurement, EMA, and evaluation logic but:
|
||||
- Does NOT persist state to disk
|
||||
- Does NOT call `systemctl suspend`
|
||||
- Prints `Would suspend: idle timeout reached (idle_counter=X/Y)` instead
|
||||
|
||||
---
|
||||
|
||||
## Unit Tests (`test_auto_suspend.py`)
|
||||
|
||||
48 tests covering all public functions using `unittest` + `unittest.mock`.
|
||||
|
||||
### Test Classes
|
||||
|
||||
| Class | Tests | Coverage |
|
||||
|---|---|---|
|
||||
| `TestFreshState` | 1 | Returns correct defaults |
|
||||
| `TestLoadState` | 4 | Missing file, valid JSON, invalid JSON, missing keys merged |
|
||||
| `TestSaveState` | 3 | Atomic write, parent dir creation, no leftover tmp |
|
||||
| `TestMeasureCpu` | 4 | Idle, busy, non-zero exit, timeout |
|
||||
| `TestMeasureGpuNvidia` | 6 | Idle, busy single, multi-GPU max, file not found, timeout, non-zero exit |
|
||||
| `TestMeasureGpuAmd` | 7 | Busy, idle, medium, file not found, short output, empty array, non-zero exit |
|
||||
| `TestGetActiveUsers` | 3 | No users, with users, timeout |
|
||||
| `TestUpdateAverages` | 4 | First sample, EMA calc, n→n1 shift, None preservation |
|
||||
| `TestEvaluate` | 10 | Idle system, warmup skip, users, CPU not idle, GPU busy, GPU None, at threshold CPU, at threshold GPU, custom thresholds |
|
||||
| `TestGetWhyNotIdle` | 5 | Users, CPU low, GPU high, warming up, multiple reasons |
|
||||
| `TestSetupLogging` | 1 | Returns logger with handlers |
|
||||
| `TestSignalHandler` | 1 | Returns exit code 0 |
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
cd ansible/roles/auto_suspend
|
||||
python3 -m pytest files/test_auto_suspend.py -v
|
||||
# or
|
||||
python3 files/test_auto_suspend.py
|
||||
```
|
||||
|
||||
Tests use `unittest.mock.patch("auto_suspend.subprocess.run")` to intercept all subprocess calls. Fixture data comes from real command outputs on gpu-01.
|
||||
|
||||
---
|
||||
|
||||
## Making Changes
|
||||
|
||||
### Modifying the Python Script (`files/auto_suspend.py`)
|
||||
|
||||
1. Edit `files/auto_suspend.py`
|
||||
2. Add or update tests in `files/test_auto_suspend.py` — mock `subprocess.run` with `MagicMock(returncode=..., stdout=..., stderr=...)`
|
||||
3. Add new fixtures to `files/fixtures/` if you need new command output samples
|
||||
4. Run `python3 -m pytest files/test_auto_suspend.py -v` — all 48 tests must pass
|
||||
5. Deploy to gpu-01 and verify with `journalctl -u auto-suspend.service`
|
||||
|
||||
### Modifying Ansible Tasks (`tasks/main.yml`)
|
||||
|
||||
The role uses inline content for systemd unit files (no `templates/` directory). To change systemd configuration, edit the inline YAML in the relevant `ansible.builtin.copy` task.
|
||||
|
||||
Key tasks in order:
|
||||
1. Install dependencies (`sysstat`, `python3`)
|
||||
2. Enable sysstat service
|
||||
3. Set GRUB `mem_sleep_default=deep` (BIOS only)
|
||||
4. Create state directory + log file
|
||||
5. Copy Python script
|
||||
6. Set SELinux context (`bin_t` + `restorecon`)
|
||||
7. Create systemd service (inline)
|
||||
8. Create sleep.conf.d deep sleep override
|
||||
9. Create systemd timer (inline)
|
||||
10. Reload systemd daemon
|
||||
11. Enable + start timer
|
||||
12. Verify with `--dry-run`
|
||||
|
||||
### Adding a New GPU Vendor
|
||||
|
||||
1. Add `measure_gpu_<vendor>(logger)` in `auto_suspend.py` following the existing pattern (return `float` utilization or `None`)
|
||||
2. Wire it into `main()`:
|
||||
```python
|
||||
gpu_<vendor> = measure_gpu_<vendor>(logger)
|
||||
gpu_values = [v for v in [gpu_nvidia, gpu_amd, gpu_<vendor>] if v is not None]
|
||||
gpu_util = max(gpu_values) if gpu_values else None
|
||||
```
|
||||
3. Add CLI argument if vendor-specific threshold is needed
|
||||
4. Add fixtures for test data
|
||||
5. Add `TestMeasureGpu<Vendor>` test class
|
||||
|
||||
### Overriding Defaults
|
||||
|
||||
All defaults in `defaults/main.yml` can be overridden via:
|
||||
- Inventory: `group_vars/auto_suspend.yml`
|
||||
- Host vars: `host_vars/gpu-01.lab.alexpires.me.yml`
|
||||
- Command line: `ansible-playbook ... -e "auto_suspend_alpha=0.5"`
|
||||
|
||||
---
|
||||
|
||||
## SELinux
|
||||
|
||||
On EL 9 with SELinux enforcing, the Python script in `/usr/local/bin/` needs `bin_t` context:
|
||||
|
||||
```yaml
|
||||
- name: Set SELinux file context on script
|
||||
community.general.sefcontext:
|
||||
path: /usr/local/bin/auto_suspend.py
|
||||
setype: bin_t
|
||||
state: present
|
||||
failed_when: false
|
||||
|
||||
- name: Restore SELinux context on script
|
||||
ansible.builtin.command: restorecon -v /usr/local/bin/auto_suspend.py
|
||||
failed_when: false
|
||||
```
|
||||
|
||||
Both tasks use `failed_when: false` to tolerate systems where SELinux is disabled or the context is already correct.
|
||||
|
||||
Requires `community.general` Ansible collection.
|
||||
|
||||
---
|
||||
|
||||
## Systemd Configuration
|
||||
|
||||
### auto-suspend.service
|
||||
- `Type=oneshot` — runs once per timer tick
|
||||
- `Restart=on-failure`, `RestartSec=30` — retries on error
|
||||
- Passes all config as CLI flags (not via environment)
|
||||
|
||||
### auto-suspend.timer
|
||||
- `OnBootSec=5min` — first run 5 minutes after boot
|
||||
- `OnUnitActiveSec=5min` — subsequent runs every 5 minutes
|
||||
- `Unit=auto-suspend.service`
|
||||
|
||||
### sleep.conf.d
|
||||
`/etc/systemd/sleep.conf.d/mem-deep.conf` sets `MemorySleepMode=deep` to ensure deep sleep state (lowest power consumption).
|
||||
|
||||
### GRUB
|
||||
The role ensures `mem_sleep_default=deep` is present in `GRUB_CMDLINE_LINUX` and regenerates the GRUB config. This is BIOS-specific (UEFI is not currently supported).
|
||||
|
||||
---
|
||||
|
||||
## Hardware Target
|
||||
|
||||
The role was developed for `gpu-01.lab.alexpires.me`:
|
||||
- Fedora 42 (kernel 6.17.11), 12-core x86_64
|
||||
- NVIDIA GPU + AMD GPU (dual-GPU system)
|
||||
- SELinux enforcing
|
||||
- Used for AI inference (Ollama, ComfyUI, llama.cpp, Qwen)
|
||||
- Network-accessible workstation
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Check service status
|
||||
```bash
|
||||
systemctl status auto-suspend.service
|
||||
systemctl status auto-suspend.timer
|
||||
```
|
||||
|
||||
### View logs
|
||||
```bash
|
||||
journalctl -u auto-suspend.service --no-pager
|
||||
cat /var/log/auto_suspend.log
|
||||
```
|
||||
|
||||
### Run a manual check
|
||||
```bash
|
||||
/usr/local/bin/auto_suspend.py --dry-run --debug
|
||||
```
|
||||
|
||||
### Check state
|
||||
```bash
|
||||
cat /var/lib/auto_suspend/state.json
|
||||
```
|
||||
|
||||
### Force a timer run
|
||||
```bash
|
||||
systemctl start auto-suspend.service
|
||||
```
|
||||
|
||||
### Re-deploy after changes
|
||||
```bash
|
||||
cd ansible/
|
||||
sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_auto_suspend.yml
|
||||
```
|
||||
@@ -8,3 +8,4 @@ auto_suspend_cpu_idle_threshold: 90.0
|
||||
auto_suspend_gpu_util_threshold: 5.0
|
||||
auto_suspend_state_dir: /var/lib/auto_suspend
|
||||
auto_suspend_log_file: /var/log/auto_suspend.log
|
||||
auto_suspend_check_interval: 5min
|
||||
|
||||
@@ -11,7 +11,6 @@ across timer invocations. State is fully reset on suspend resume.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import logging
|
||||
import logging.handlers
|
||||
@@ -142,11 +141,17 @@ def measure_cpu(logger):
|
||||
|
||||
Runs ``mpstat 1 1`` and extracts the last field of the "Average:" line,
|
||||
which is ``%idle``. Returns a float, or None on failure.
|
||||
|
||||
Uses ``LC_ALL=C`` to force English locale output regardless of system
|
||||
locale (avoids issues like Portuguese ``Média:`` with comma decimals).
|
||||
"""
|
||||
env = dict(os.environ)
|
||||
env["LC_ALL"] = "C"
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["mpstat", "1", "1"],
|
||||
capture_output=True, text=True, timeout=15,
|
||||
env=env,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
logger.warning("mpstat returned non-zero: %s", result.stderr.strip())
|
||||
@@ -161,6 +166,7 @@ def measure_cpu(logger):
|
||||
logger.debug("CPU idle: %.1f%%", idle_pct)
|
||||
return idle_pct
|
||||
|
||||
logger.warning("mpstat output did not contain expected data line")
|
||||
return None
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("mpstat timed out")
|
||||
@@ -198,7 +204,7 @@ def measure_gpu_nvidia(logger):
|
||||
return None
|
||||
|
||||
max_util = max(utils)
|
||||
logger.debug("NVIDIA GPU utilization: max=%.1f%%", max_util)
|
||||
logger.info("NVIDIA GPU utilization: max=%.1f%%", max_util)
|
||||
return max_util
|
||||
except FileNotFoundError:
|
||||
logger.debug("nvidia-smi not found")
|
||||
@@ -215,51 +221,45 @@ def measure_gpu_amd(logger):
|
||||
"""
|
||||
Measure GPU utilization via amd-smi.
|
||||
|
||||
Parses the ``gfx`` column from ``amd-smi monitor --csv`` by header name
|
||||
(not by fixed column index). Returns None if amd-smi is unavailable.
|
||||
Parses the ``gfx.value`` field from ``amd-smi monitor --json``.
|
||||
Returns the utilization of the first GPU, or None if amd-smi is unavailable.
|
||||
"""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["amd-smi", "monitor", "--csv"],
|
||||
["amd-smi", "monitor", "--json"],
|
||||
capture_output=True, text=True, timeout=15,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
logger.debug("amd-smi returned %d — no AMD GPU", result.returncode)
|
||||
return None
|
||||
|
||||
lines = result.stdout.strip().splitlines()
|
||||
if len(lines) < 2:
|
||||
logger.debug("amd-smi output too short")
|
||||
raw = result.stdout.strip()
|
||||
|
||||
data = json.loads(result.stdout)
|
||||
if not data or not isinstance(data, list):
|
||||
logger.debug("amd-smi JSON output empty or invalid")
|
||||
return None
|
||||
|
||||
# Filter out non-data lines (comments, annotations with wrong field count)
|
||||
csv_reader = csv.reader(lines)
|
||||
header = next(csv_reader)
|
||||
for fields in csv_reader:
|
||||
if len(fields) != len(header):
|
||||
continue
|
||||
row = dict(zip(header, fields))
|
||||
try:
|
||||
# Try both "gfx" and "gfx_util" column names
|
||||
gfx_val = row.get("gfx") or row.get("gfx_util")
|
||||
if gfx_val is None:
|
||||
logger.debug("amd-smi CSV has no 'gfx' column; columns: %s",
|
||||
list(row.keys()))
|
||||
return None
|
||||
util = float(gfx_val)
|
||||
logger.debug("AMD GPU utilization: %.1f%%", util)
|
||||
return util
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
|
||||
return None
|
||||
gpu = data[0]
|
||||
gfx = gpu.get("gfx")
|
||||
if isinstance(gfx, dict):
|
||||
gfx_val = gfx.get("value")
|
||||
else:
|
||||
gfx_val = gfx
|
||||
if gfx_val is None:
|
||||
logger.debug("amd-smi JSON has no 'gfx.value' field; fields: %s",
|
||||
list(gpu.keys()))
|
||||
return None
|
||||
util = float(gfx_val)
|
||||
logger.info("AMD GPU utilization: %.1f%% (gfx=%s)", util, gfx_val)
|
||||
return util
|
||||
except FileNotFoundError:
|
||||
logger.debug("amd-smi not found")
|
||||
return None
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.warning("amd-smi timed out")
|
||||
return None
|
||||
except BaseException as exc:
|
||||
except (json.JSONDecodeError, ValueError, TypeError) as exc:
|
||||
logger.warning("Failed to parse amd-smi output: %s", exc)
|
||||
return None
|
||||
|
||||
@@ -339,24 +339,25 @@ def update_averages(state, cpu_n, gpu_n, alpha, logger):
|
||||
# Evaluation
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def get_why_not_idle(state, has_users, logger):
|
||||
def get_why_not_idle(state, has_users, cpu_idle_threshold=DEFAULT_CPU_IDLE_THRESHOLD,
|
||||
gpu_util_threshold=DEFAULT_GPU_UTIL_THRESHOLD, min_samples=DEFAULT_MIN_SAMPLES, logger=None):
|
||||
"""Return a human-readable reason why the system is not idle."""
|
||||
reasons = []
|
||||
if has_users:
|
||||
reasons.append("users logged in")
|
||||
if state["cpu_avg"] is not None:
|
||||
if state["cpu_avg"] <= DEFAULT_CPU_IDLE_THRESHOLD:
|
||||
if state["cpu_avg"] <= cpu_idle_threshold:
|
||||
reasons.append(
|
||||
f"CPU idle avg {state['cpu_avg']:.1f}% <= threshold {DEFAULT_CPU_IDLE_THRESHOLD:.0f}%"
|
||||
f"CPU idle avg {state['cpu_avg']:.1f}% <= threshold {cpu_idle_threshold:.0f}%"
|
||||
)
|
||||
if state["gpu_avg"] is not None:
|
||||
if state["gpu_avg"] >= DEFAULT_GPU_UTIL_THRESHOLD:
|
||||
if state["gpu_avg"] >= gpu_util_threshold:
|
||||
reasons.append(
|
||||
f"GPU util avg {state['gpu_avg']:.1f}% >= threshold {DEFAULT_GPU_UTIL_THRESHOLD:.0f}%"
|
||||
f"GPU util avg {state['gpu_avg']:.1f}% >= threshold {gpu_util_threshold:.0f}%"
|
||||
)
|
||||
if not reasons:
|
||||
if state["sample_count"] < DEFAULT_MIN_SAMPLES:
|
||||
reasons.append(f"warming up ({state['sample_count']}/{DEFAULT_MIN_SAMPLES} samples)")
|
||||
if state["sample_count"] < min_samples:
|
||||
reasons.append(f"warming up ({state['sample_count']}/{min_samples} samples)")
|
||||
else:
|
||||
reasons.append("unknown")
|
||||
return "; ".join(reasons)
|
||||
@@ -491,17 +492,17 @@ def main():
|
||||
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["systemctl", "suspend"],
|
||||
["shutdown", "now"],
|
||||
capture_output=True, text=True, timeout=10,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
logger.error("systemctl suspend failed: %s", result.stderr.strip())
|
||||
logger.error("shutdown failed: %s", result.stderr.strip())
|
||||
else:
|
||||
logger.info("systemctl suspend succeeded")
|
||||
logger.info("shutdown succeeded")
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.error("systemctl suspend timed out")
|
||||
logger.error("shutdown timed out")
|
||||
except FileNotFoundError:
|
||||
logger.error("systemctl not found — cannot suspend")
|
||||
logger.error("shutdown not found — cannot suspend")
|
||||
except BaseException as exc:
|
||||
logger.error("Failed to suspend: %s", exc)
|
||||
|
||||
@@ -511,7 +512,7 @@ def main():
|
||||
else:
|
||||
if not args.dry_run:
|
||||
state["idle_counter"] = 0
|
||||
reason = get_why_not_idle(state, has_users, logger)
|
||||
reason = get_why_not_idle(state, has_users, args.cpu_idle_threshold, args.gpu_util_threshold, args.min_samples, logger)
|
||||
logger.info("Not idle (%s) — reset counter", reason)
|
||||
|
||||
# --- Save and exit ---
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
gpu,power_usage,hotspot_temperature,memory_temperature,gfx,gfx_clock,mem,mem_clock,encoder,decoder,vclock,dclock,single_bit_ecc,double_bit_ecc,pcie_replay,vram_used,vram_total,pcie_bw,pviol,tviol,phot_tviol,vr_tviol,hbm_tviol
|
||||
0,20,52,60,100,1200,0,96,N/A,0.0,25,25,0,0,N/A,11913,16334,N/A,N/A,N/A,N/A,N/A,N/A
|
||||
|
||||
@@ -0,0 +1,27 @@
|
||||
[
|
||||
{
|
||||
"gpu": 0,
|
||||
"power_usage": {"value": 20, "unit": "W"},
|
||||
"hotspot_temperature": {"value": 52, "unit": "C"},
|
||||
"memory_temperature": {"value": 60, "unit": "C"},
|
||||
"gfx": {"value": 100, "unit": "%"},
|
||||
"gfx_clock": {"value": 1200, "unit": "MHz"},
|
||||
"mem": {"value": 0, "unit": "%"},
|
||||
"mem_clock": {"value": 96, "unit": "MHz"},
|
||||
"encoder": "N/A",
|
||||
"decoder": {"value": 0.0, "unit": "%"},
|
||||
"vclock": {"value": 25, "unit": "MHz"},
|
||||
"dclock": {"value": 25, "unit": "MHz"},
|
||||
"single_bit_ecc": 0,
|
||||
"double_bit_ecc": 0,
|
||||
"pcie_replay": "N/A",
|
||||
"vram_used": {"value": 11913, "unit": "MB"},
|
||||
"vram_total": {"value": 16334, "unit": "MB"},
|
||||
"pcie_bw": {"value": "N/A", "unit": "Mb/s"},
|
||||
"pviol": {"value": "N/A", "unit": "%"},
|
||||
"tviol": {"value": "N/A", "unit": "%"},
|
||||
"phot_tviol": {"value": "N/A", "unit": "%"},
|
||||
"vr_tviol": {"value": "N/A", "unit": "%"},
|
||||
"hbm_tviol": {"value": "N/A", "unit": "%"}
|
||||
}
|
||||
]
|
||||
@@ -48,13 +48,9 @@ NVIDIA_BUSY_4GAPS = """ 0
|
||||
NVIDIA_BUSY_SINGLE = """ 95
|
||||
"""
|
||||
NVIDIA_NONE = ""
|
||||
AMD_SMII_GFX = _load_fixture("amd-smi-monitor--csv.txt")
|
||||
AMD_SMII_GFX_0 = """gpu,power_usage,hotspot_temperature,memory_temperature,gfx,gfx_clock,mem,mem_clock,encoder,decoder,vclock,dclock,single_bit_ecc,double_bit_ecc,pcie_replay,vram_used,vram_total,pcie_bw,pviol,tviol,phot_tviol,vr_tviol,hbm_tviol
|
||||
0,20,52,62,0,1200,0,96,N/A,0.0,25,25,0,0,N/A,11928,16334,N/A,N/A,N/A,N/A,N/A,N/A
|
||||
"""
|
||||
AMD_SMII_GFX_80 = """gpu,power_usage,hotspot_temperature,memory_temperature,gfx,gfx_clock,mem,mem_clock,encoder,decoder,vclock,dclock,single_bit_ecc,double_bit_ecc,pcie_replay,vram_used,vram_total,pcie_bw,pviol,tviol,phot_tviol,vr_tviol,hbm_tviol
|
||||
0,180,78,82,80,1800,90,200,N/A,45.0,25,25,0,0,N/A,14000,16334,N/A,N/A,N/A,N/A,N/A,N/A
|
||||
"""
|
||||
AMD_SMII_JSON = _load_fixture("amd-smi-monitor--json.txt")
|
||||
AMD_SMII_GFX_0 = '[{"gpu":0,"gfx":{"value":0,"unit":"%"}}]'
|
||||
AMD_SMII_GFX_80 = '[{"gpu":0,"gfx":{"value":80,"unit":"%"}}]'
|
||||
WHO_WITH_USERS = _load_fixture("who.with-users.txt")
|
||||
WHO_EMPTY = ""
|
||||
|
||||
@@ -65,10 +61,6 @@ MPSTAT_HIGH = """Linux 6.17.11-200.fc42.x86_64 (gpu-01) 06/08/26 _x86_64_ (12
|
||||
Average: all 85.30 0.00 8.20 0.50 0.00 0.10 0.00 0.00 0.00 5.90
|
||||
"""
|
||||
|
||||
AMD_SMII_ONLY_HEADER = """gpu,power_usage,hotspot_temperature,memory_temperature,gfx
|
||||
0,20,52,62,0
|
||||
"""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tests
|
||||
@@ -194,6 +186,22 @@ class TestMeasureCpu(unittest.TestCase):
|
||||
self.assertIsNone(result)
|
||||
logger.warning.assert_called_with("mpstat timed out")
|
||||
|
||||
def test_portuguese_locale(self):
|
||||
"""Verify mpstat with C locale (LC_ALL=C) parses correctly.
|
||||
|
||||
The measure_cpu function forces LC_ALL=C so mpstat always outputs
|
||||
in English regardless of system locale.
|
||||
"""
|
||||
from auto_suspend import measure_cpu
|
||||
logger = MagicMock()
|
||||
with patch("auto_suspend.subprocess.run") as run:
|
||||
run.return_value = _make_mock_result(MPSTAT_IDLE)
|
||||
result = measure_cpu(logger)
|
||||
# Verify subprocess was called with LC_ALL=C
|
||||
call_kwargs = run.call_args
|
||||
self.assertEqual(call_kwargs[1]["env"]["LC_ALL"], "C")
|
||||
self.assertAlmostEqual(result, 99.58, places=1)
|
||||
|
||||
|
||||
class TestMeasureGpuNvidia(unittest.TestCase):
|
||||
|
||||
@@ -253,7 +261,7 @@ class TestMeasureGpuAmd(unittest.TestCase):
|
||||
from auto_suspend import measure_gpu_amd
|
||||
logger = MagicMock()
|
||||
with patch("auto_suspend.subprocess.run") as run:
|
||||
run.return_value = _make_mock_result(AMD_SMII_GFX)
|
||||
run.return_value = _make_mock_result(AMD_SMII_JSON)
|
||||
result = measure_gpu_amd(logger)
|
||||
self.assertAlmostEqual(result, 100.0, places=1)
|
||||
|
||||
@@ -289,6 +297,14 @@ class TestMeasureGpuAmd(unittest.TestCase):
|
||||
result = measure_gpu_amd(logger)
|
||||
self.assertIsNone(result)
|
||||
|
||||
def test_empty_array(self):
|
||||
from auto_suspend import measure_gpu_amd
|
||||
logger = MagicMock()
|
||||
with patch("auto_suspend.subprocess.run") as run:
|
||||
run.return_value = _make_mock_result("[]")
|
||||
result = measure_gpu_amd(logger)
|
||||
self.assertIsNone(result)
|
||||
|
||||
def test_returncode_nonzero(self):
|
||||
from auto_suspend import measure_gpu_amd
|
||||
logger = MagicMock()
|
||||
@@ -371,6 +387,77 @@ class TestUpdateAverages(unittest.TestCase):
|
||||
self.assertAlmostEqual(state["gpu_avg"], 5.0)
|
||||
self.assertEqual(state["sample_count"], 2)
|
||||
|
||||
def test_ema_decay_idle_gpu(self):
|
||||
"""
|
||||
Simulate GPU going from 100% to 0% idle and verify EMA decays.
|
||||
|
||||
This validates that the EMA calculation correctly handles the scenario
|
||||
where a busy GPU becomes idle — the average should converge toward
|
||||
idle values after enough consecutive idle samples.
|
||||
"""
|
||||
from auto_suspend import fresh_state, update_averages
|
||||
logger = MagicMock()
|
||||
state = fresh_state()
|
||||
|
||||
# Phase 1: GPU was busy (simulate 10 consecutive 100% samples)
|
||||
for _ in range(10):
|
||||
state = update_averages(state, cpu_n=95.0, gpu_n=100.0, alpha=0.3, logger=logger)
|
||||
|
||||
self.assertAlmostEqual(state["gpu_avg"], 100.0, places=1)
|
||||
self.assertEqual(state["sample_count"], 10)
|
||||
|
||||
# Phase 2: GPU goes idle (0%) for 20 consecutive samples
|
||||
for _ in range(20):
|
||||
state = update_averages(state, cpu_n=99.0, gpu_n=0.0, alpha=0.3, logger=logger)
|
||||
|
||||
# After 20 idle samples from 100%, EMA should be very close to 0
|
||||
# EMA = 0.7^20 * 100 ≈ 0.08%
|
||||
self.assertAlmostEqual(state["gpu_avg"], 0.08, places=1)
|
||||
self.assertEqual(state["sample_count"], 30)
|
||||
|
||||
def test_ema_convergence_multiple_values(self):
|
||||
"""
|
||||
Verify EMA converges to the steady-state value regardless of start.
|
||||
|
||||
With alpha=0.3 and 100 identical samples, the average should equal
|
||||
the sample value (since EMA_new = 0.3*x + 0.7*x = x).
|
||||
"""
|
||||
from auto_suspend import fresh_state, update_averages
|
||||
logger = MagicMock()
|
||||
state = fresh_state()
|
||||
|
||||
gpu_idle = 2.0
|
||||
for _ in range(50):
|
||||
state = update_averages(state, cpu_n=95.0, gpu_n=gpu_idle, alpha=0.3, logger=logger)
|
||||
|
||||
self.assertAlmostEqual(state["gpu_avg"], gpu_idle, places=2)
|
||||
self.assertAlmostEqual(state["cpu_avg"], 95.0, places=2)
|
||||
|
||||
def test_ema_with_missing_gpu_sample(self):
|
||||
"""
|
||||
Verify None GPU sample preserves the average.
|
||||
|
||||
When measure_gpu_amd returns None (e.g., tool unavailable), the
|
||||
average should NOT be updated — it should keep the last value.
|
||||
"""
|
||||
from auto_suspend import fresh_state, update_averages
|
||||
logger = MagicMock()
|
||||
state = fresh_state()
|
||||
|
||||
# Seed with a GPU average
|
||||
state = update_averages(state, cpu_n=95.0, gpu_n=10.0, alpha=0.3, logger=logger)
|
||||
self.assertAlmostEqual(state["gpu_avg"], 10.0)
|
||||
|
||||
# Now GPU measurement fails (None) — average should be preserved
|
||||
state = update_averages(state, cpu_n=95.0, gpu_n=None, alpha=0.3, logger=logger)
|
||||
self.assertAlmostEqual(state["gpu_avg"], 10.0)
|
||||
self.assertEqual(state["sample_count"], 2)
|
||||
|
||||
# After GPU returns, EMA should continue from the preserved value
|
||||
state = update_averages(state, cpu_n=95.0, gpu_n=0.0, alpha=0.3, logger=logger)
|
||||
# EMA = 0.3 * 0 + 0.7 * 10 = 7.0
|
||||
self.assertAlmostEqual(state["gpu_avg"], 7.0)
|
||||
|
||||
|
||||
class TestEvaluate(unittest.TestCase):
|
||||
|
||||
|
||||
@@ -143,11 +143,11 @@
|
||||
dest: /etc/systemd/system/auto-suspend.timer
|
||||
content: |
|
||||
[Unit]
|
||||
Description=Run idle suspend check every 5 minutes
|
||||
Description=Run idle suspend check every {{ auto_suspend_check_interval }}
|
||||
|
||||
[Timer]
|
||||
OnBootSec=5min
|
||||
OnUnitActiveSec=5min
|
||||
OnBootSec={{ auto_suspend_check_interval }}
|
||||
OnUnitActiveSec={{ auto_suspend_check_interval }}
|
||||
Unit=auto-suspend.service
|
||||
|
||||
[Install]
|
||||
|
||||
Reference in New Issue
Block a user