389 lines
15 KiB
Markdown
389 lines
15 KiB
Markdown
# auto_suspend — Idle-based System Suspend
|
||
|
||
Auto-suspend monitors a system's CPU and GPU utilization, applies exponential moving average (EMA) smoothing, and suspends the machine when idle for a configured period.
|
||
|
||
**Target platform:** RHEL/EL 9 with SELinux enforcing (Fedora/RHEL datacenter GPUs)
|
||
|
||
---
|
||
|
||
## Goal
|
||
|
||
Automatically put idle GPU workstations into suspend (RAM sleep) to save power and reduce heat, while ensuring no user session or active workload is interrupted.
|
||
|
||
The system checks every 5 minutes. It suspends only after `idle_timeout_cycles` consecutive checks confirm the system is idle, using EMA-smoothed metrics to avoid false positives from transient spikes.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```
|
||
┌─────────────┐ every 5 min ┌──────────────────┐
|
||
│ timer │ ────────────────────▶ │ service │
|
||
│ auto-suspend│ │ auto-suspend │
|
||
└─────────────┘ └────────┬─────────┘
|
||
│
|
||
measures: cpu, nvidia, amd
|
||
│
|
||
EMA smoothing + warmup check
|
||
│
|
||
evaluate idle?
|
||
╱ ╲
|
||
yes ╱ ╲ no
|
||
│ │
|
||
▼ ▼
|
||
idle_counter++ idle_counter=0
|
||
│
|
||
counter ≥ 4?
|
||
╱ ╲
|
||
yes ╱ ╲ no
|
||
│ │
|
||
▼ ▼
|
||
systemctl re-persist
|
||
suspend state.json
|
||
```
|
||
|
||
### Key design decisions
|
||
|
||
- **State persists across timer invocations** — `state.json` carries EMA averages, idle counter, and sample count between checks. The system can accumulate 20+ minutes of data across reboots (state survives suspend).
|
||
- **Any busy GPU prevents suspend** — max utilization across all NVIDIA + AMD GPUs is used. One busy GPU = no suspend.
|
||
- **Reset on resume** — after `systemctl suspend`, state is fully reset so the warmup period starts fresh.
|
||
- **GPU presence is optional** — `None` from a GPU driver means "no GPU of this type", which does NOT block suspend.
|
||
|
||
---
|
||
|
||
## Directory Structure
|
||
|
||
```
|
||
ansible/roles/auto_suspend/
|
||
├── defaults/main.yml # Configurable parameters (tunable via inventory/host_vars)
|
||
├── meta/main.yml # Galaxy metadata
|
||
└── tasks/
|
||
│ └── main.yml # Ansible tasks (dependencies, scripts, systemd, SELinux)
|
||
└── files/
|
||
├── auto_suspend.py # Main Python daemon (522 lines)
|
||
├── test_auto_suspend.py # Unit tests (48 tests, stdlib only)
|
||
└── fixtures/ # Real command samples from gpu-01
|
||
├── amd-smi-monitor--json.txt
|
||
├── mpstat--1--1.idle.txt
|
||
├── nvidia-smi--query-gpu.utilization.gpu.csv.noheader.idle.txt
|
||
└── who.with-users.txt
|
||
```
|
||
|
||
---
|
||
|
||
## Playbook Usage
|
||
|
||
```yaml
|
||
# ansible/playbook_auto_suspend.yml
|
||
- name: Setup Auto Suspend
|
||
hosts: auto_suspend
|
||
roles:
|
||
- auto_suspend
|
||
```
|
||
|
||
Deploy via:
|
||
```bash
|
||
cd ansible/
|
||
sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_auto_suspend.yml
|
||
```
|
||
|
||
The `auto_suspend` inventory group currently contains only `gpu-01.lab.alexpires.me`.
|
||
|
||
---
|
||
|
||
## Configuration (defaults/main.yml)
|
||
|
||
| Variable | Default | Description |
|
||
|---|---|---|
|
||
| `auto_suspend_alpha` | `0.3` | EMA weight for current sample (0.3 = 30% current, 70% historical) |
|
||
| `auto_suspend_min_samples` | `3` | Warmup checks before trust (3 × 5 min = 15 min minimum) |
|
||
| `auto_suspend_idle_timeout_cycles` | `4` | Consecutive idle checks before suspend (4 × 5 min = 20 min total) |
|
||
| `auto_suspend_cpu_idle_threshold` | `90.0` | CPU idle % must exceed this to count as idle |
|
||
| `auto_suspend_gpu_util_threshold` | `5.0` | GPU util % must be below this to count as idle |
|
||
| `auto_suspend_state_dir` | `/var/lib/auto_suspend` | Directory for state.json persistence |
|
||
| `auto_suspend_log_file` | `/var/log/auto_suspend.log` | Log file path (RotatingFileHandler, 1 MB × 3 backups) |
|
||
| `auto_suspend_grub_config_path_bios` | `/boot/grub2/grub.cfg` | GRUB config output path for BIOS systems |
|
||
|
||
All variables can be overridden via inventory, group_vars, or host_vars.
|
||
|
||
---
|
||
|
||
## Deployed Artifacts
|
||
|
||
| Path | Type | Purpose |
|
||
|---|---|---|
|
||
| `/usr/local/bin/auto_suspend.py` | Python script | Main daemon, copied from `files/auto_suspend.py` |
|
||
| `/etc/systemd/system/auto-suspend.service` | systemd oneshot | Runs the check + suspend logic |
|
||
| `/etc/systemd/system/auto-suspend.timer` | systemd timer | Triggers service every 5 minutes (5 min after boot) |
|
||
| `/etc/systemd/sleep.conf.d/mem-deep.conf` | sysctl override | Sets `MemorySleepMode=deep` |
|
||
| `/var/lib/auto_suspend/state.json` | JSON | EMA state, idle counter, sample count |
|
||
| `/var/log/auto_suspend.log` | log | Rotating log (1 MB, 3 backups) |
|
||
|
||
---
|
||
|
||
## auto_suspend.py — Script Reference
|
||
|
||
### CLI Arguments
|
||
|
||
All arguments map directly to `defaults/main.yml` variables:
|
||
|
||
```
|
||
--alpha 0.3 EMA smoothing factor
|
||
--min-samples 3 Warmup period
|
||
--idle-timeout-cycles 4 Consecutive idle checks needed
|
||
--cpu-idle-threshold 90.0 CPU idle % threshold
|
||
--gpu-util-threshold 5.0 GPU util % threshold
|
||
--state-dir /var/lib/auto_suspend
|
||
--log-file /var/log/auto_suspend.log
|
||
--debug Enable debug-level logging
|
||
--dry-run Run check without persisting state or suspending
|
||
```
|
||
|
||
### Public Functions
|
||
|
||
| Function | Returns | Purpose |
|
||
|---|---|---|
|
||
| `setup_logging(log_file, debug)` | `logging.Logger` | Configures file + stderr handlers |
|
||
| `fresh_state()` | `dict` | Returns default state with all values at initial values |
|
||
| `load_state(state_dir, logger)` | `dict` | Loads state from disk, returns fresh on any error |
|
||
| `save_state(state_dir, state, logger)` | `None` | Atomic write via temp file + os.replace |
|
||
| `measure_cpu(logger)` | `float \| None` | Runs `mpstat 1 1`, returns %idle |
|
||
| `measure_gpu_nvidia(logger)` | `float \| None` | Runs `nvidia-smi`, returns max util across all GPUs |
|
||
| `measure_gpu_amd(logger)` | `float \| None` | Runs `amd-smi monitor --json`, returns gfx.value of GPU 0 |
|
||
| `get_active_users(logger)` | `int` | Parses `who` output, returns non-empty line count |
|
||
| `get_why_not_idle(cpu_avg, gpu_avg, sample_count, idle_counter, has_users, cpu_threshold, gpu_threshold)` | `list[str]` | Returns human-readable reasons why system is not idle |
|
||
| `update_averages(state, cpu_n, gpu_n, alpha)` | `dict` | EMA update: returns new state with averaged values |
|
||
| `evaluate(state, has_users, cpu_threshold, gpu_threshold)` | `bool` | Gate checks: warmup → users → CPU → GPU |
|
||
| `signal_handler(signum, frame)` | `NoReturn` | SIGTERM/SIGINT handler, calls `sys.exit(0)` |
|
||
| `main()` | `NoReturn` | Entry point: measures, evaluates, updates state, suspends |
|
||
|
||
### State Schema (`state.json`)
|
||
|
||
```json
|
||
{
|
||
"cpu_idle_n": null, // latest CPU idle % (current sample)
|
||
"cpu_idle_n1": null, // latest CPU idle % (previous sample, n-1)
|
||
"gpu_util_n": null, // latest GPU util % (current sample)
|
||
"gpu_util_n1": null, // latest GPU util % (previous sample, n-1)
|
||
"cpu_avg": null, // EMA-smoothed CPU idle average
|
||
"gpu_avg": null, // EMA-smoothed GPU utilization average
|
||
"idle_counter": 0, // consecutive idle cycles
|
||
"sample_count": 0 // total samples collected
|
||
}
|
||
```
|
||
|
||
### EMA Formula
|
||
|
||
```
|
||
EMA_new = alpha * current_sample + (1 - alpha) * previous_EMA
|
||
```
|
||
|
||
With `alpha=0.3`: 30% weight on the current reading, 70% on the historical average. `round(..., 2)` keeps precision to 2 decimal places.
|
||
|
||
### GPU Detection
|
||
|
||
**NVIDIA** — `nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits`
|
||
- Parses each line as float, returns `max()` across all GPUs
|
||
- Returns `None` if nvidia-smi is missing, fails, or times out
|
||
- `None` means no NVIDIA GPU present (does NOT block suspend)
|
||
|
||
**AMD** — `amd-smi monitor --json`
|
||
- Parses `data[0]["gfx"]["value"]` from first GPU
|
||
- Handles both `{"value": N, "unit": "%"}` and raw numeric `gfx` formats
|
||
- Returns `None` if amd-smi is missing, fails, or times out
|
||
|
||
**Combined** — `max()` across all non-None GPU measurements. Any busy GPU prevents suspend.
|
||
|
||
### Suspend Decision Logic (evaluate)
|
||
|
||
Four gates in order:
|
||
|
||
1. **Warmup** — If `sample_count < min_samples`, return `False` (skip evaluation)
|
||
2. **Active users** — If `who` returns non-empty output, return `False`
|
||
3. **CPU idle** — If `cpu_avg > cpu_threshold` is False, return `False`
|
||
4. **GPU idle** — If GPU is present (`gpu_avg is not None`) and `gpu_avg >= gpu_threshold`, return `False`
|
||
|
||
Only when all gates pass does `idle_counter` increment. When `idle_counter >= idle_timeout_cycles`, `systemctl suspend` is called. On failure to suspend, `idle_counter` resets to 0.
|
||
|
||
### Dry-Run Mode
|
||
|
||
`--dry-run` runs all measurement, EMA, and evaluation logic but:
|
||
- Does NOT persist state to disk
|
||
- Does NOT call `systemctl suspend`
|
||
- Prints `Would suspend: idle timeout reached (idle_counter=X/Y)` instead
|
||
|
||
---
|
||
|
||
## Unit Tests (`test_auto_suspend.py`)
|
||
|
||
48 tests covering all public functions using `unittest` + `unittest.mock`.
|
||
|
||
### Test Classes
|
||
|
||
| Class | Tests | Coverage |
|
||
|---|---|---|
|
||
| `TestFreshState` | 1 | Returns correct defaults |
|
||
| `TestLoadState` | 4 | Missing file, valid JSON, invalid JSON, missing keys merged |
|
||
| `TestSaveState` | 3 | Atomic write, parent dir creation, no leftover tmp |
|
||
| `TestMeasureCpu` | 4 | Idle, busy, non-zero exit, timeout |
|
||
| `TestMeasureGpuNvidia` | 6 | Idle, busy single, multi-GPU max, file not found, timeout, non-zero exit |
|
||
| `TestMeasureGpuAmd` | 7 | Busy, idle, medium, file not found, short output, empty array, non-zero exit |
|
||
| `TestGetActiveUsers` | 3 | No users, with users, timeout |
|
||
| `TestUpdateAverages` | 4 | First sample, EMA calc, n→n1 shift, None preservation |
|
||
| `TestEvaluate` | 10 | Idle system, warmup skip, users, CPU not idle, GPU busy, GPU None, at threshold CPU, at threshold GPU, custom thresholds |
|
||
| `TestGetWhyNotIdle` | 5 | Users, CPU low, GPU high, warming up, multiple reasons |
|
||
| `TestSetupLogging` | 1 | Returns logger with handlers |
|
||
| `TestSignalHandler` | 1 | Returns exit code 0 |
|
||
|
||
### Running Tests
|
||
|
||
```bash
|
||
cd ansible/roles/auto_suspend
|
||
python3 -m pytest files/test_auto_suspend.py -v
|
||
# or
|
||
python3 files/test_auto_suspend.py
|
||
```
|
||
|
||
Tests use `unittest.mock.patch("auto_suspend.subprocess.run")` to intercept all subprocess calls. Fixture data comes from real command outputs on gpu-01.
|
||
|
||
---
|
||
|
||
## Making Changes
|
||
|
||
### Modifying the Python Script (`files/auto_suspend.py`)
|
||
|
||
1. Edit `files/auto_suspend.py`
|
||
2. Add or update tests in `files/test_auto_suspend.py` — mock `subprocess.run` with `MagicMock(returncode=..., stdout=..., stderr=...)`
|
||
3. Add new fixtures to `files/fixtures/` if you need new command output samples
|
||
4. Run `python3 -m pytest files/test_auto_suspend.py -v` — all 48 tests must pass
|
||
5. Deploy to gpu-01 and verify with `journalctl -u auto-suspend.service`
|
||
|
||
### Modifying Ansible Tasks (`tasks/main.yml`)
|
||
|
||
The role uses inline content for systemd unit files (no `templates/` directory). To change systemd configuration, edit the inline YAML in the relevant `ansible.builtin.copy` task.
|
||
|
||
Key tasks in order:
|
||
1. Install dependencies (`sysstat`, `python3`)
|
||
2. Enable sysstat service
|
||
3. Set GRUB `mem_sleep_default=deep` (BIOS only)
|
||
4. Create state directory + log file
|
||
5. Copy Python script
|
||
6. Set SELinux context (`bin_t` + `restorecon`)
|
||
7. Create systemd service (inline)
|
||
8. Create sleep.conf.d deep sleep override
|
||
9. Create systemd timer (inline)
|
||
10. Reload systemd daemon
|
||
11. Enable + start timer
|
||
12. Verify with `--dry-run`
|
||
|
||
### Adding a New GPU Vendor
|
||
|
||
1. Add `measure_gpu_<vendor>(logger)` in `auto_suspend.py` following the existing pattern (return `float` utilization or `None`)
|
||
2. Wire it into `main()`:
|
||
```python
|
||
gpu_<vendor> = measure_gpu_<vendor>(logger)
|
||
gpu_values = [v for v in [gpu_nvidia, gpu_amd, gpu_<vendor>] if v is not None]
|
||
gpu_util = max(gpu_values) if gpu_values else None
|
||
```
|
||
3. Add CLI argument if vendor-specific threshold is needed
|
||
4. Add fixtures for test data
|
||
5. Add `TestMeasureGpu<Vendor>` test class
|
||
|
||
### Overriding Defaults
|
||
|
||
All defaults in `defaults/main.yml` can be overridden via:
|
||
- Inventory: `group_vars/auto_suspend.yml`
|
||
- Host vars: `host_vars/gpu-01.lab.alexpires.me.yml`
|
||
- Command line: `ansible-playbook ... -e "auto_suspend_alpha=0.5"`
|
||
|
||
---
|
||
|
||
## SELinux
|
||
|
||
On EL 9 with SELinux enforcing, the Python script in `/usr/local/bin/` needs `bin_t` context:
|
||
|
||
```yaml
|
||
- name: Set SELinux file context on script
|
||
community.general.sefcontext:
|
||
path: /usr/local/bin/auto_suspend.py
|
||
setype: bin_t
|
||
state: present
|
||
failed_when: false
|
||
|
||
- name: Restore SELinux context on script
|
||
ansible.builtin.command: restorecon -v /usr/local/bin/auto_suspend.py
|
||
failed_when: false
|
||
```
|
||
|
||
Both tasks use `failed_when: false` to tolerate systems where SELinux is disabled or the context is already correct.
|
||
|
||
Requires `community.general` Ansible collection.
|
||
|
||
---
|
||
|
||
## Systemd Configuration
|
||
|
||
### auto-suspend.service
|
||
- `Type=oneshot` — runs once per timer tick
|
||
- `Restart=on-failure`, `RestartSec=30` — retries on error
|
||
- Passes all config as CLI flags (not via environment)
|
||
|
||
### auto-suspend.timer
|
||
- `OnBootSec=5min` — first run 5 minutes after boot
|
||
- `OnUnitActiveSec=5min` — subsequent runs every 5 minutes
|
||
- `Unit=auto-suspend.service`
|
||
|
||
### sleep.conf.d
|
||
`/etc/systemd/sleep.conf.d/mem-deep.conf` sets `MemorySleepMode=deep` to ensure deep sleep state (lowest power consumption).
|
||
|
||
### GRUB
|
||
The role ensures `mem_sleep_default=deep` is present in `GRUB_CMDLINE_LINUX` and regenerates the GRUB config. This is BIOS-specific (UEFI is not currently supported).
|
||
|
||
---
|
||
|
||
## Hardware Target
|
||
|
||
The role was developed for `gpu-01.lab.alexpires.me`:
|
||
- Fedora 42 (kernel 6.17.11), 12-core x86_64
|
||
- NVIDIA GPU + AMD GPU (dual-GPU system)
|
||
- SELinux enforcing
|
||
- Used for AI inference (Ollama, ComfyUI, llama.cpp, Qwen)
|
||
- Network-accessible workstation
|
||
|
||
---
|
||
|
||
## Troubleshooting
|
||
|
||
### Check service status
|
||
```bash
|
||
systemctl status auto-suspend.service
|
||
systemctl status auto-suspend.timer
|
||
```
|
||
|
||
### View logs
|
||
```bash
|
||
journalctl -u auto-suspend.service --no-pager
|
||
cat /var/log/auto_suspend.log
|
||
```
|
||
|
||
### Run a manual check
|
||
```bash
|
||
/usr/local/bin/auto_suspend.py --dry-run --debug
|
||
```
|
||
|
||
### Check state
|
||
```bash
|
||
cat /var/lib/auto_suspend/state.json
|
||
```
|
||
|
||
### Force a timer run
|
||
```bash
|
||
systemctl start auto-suspend.service
|
||
```
|
||
|
||
### Re-deploy after changes
|
||
```bash
|
||
cd ansible/
|
||
sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_auto_suspend.yml
|
||
```
|