alexpires.me/a13labs.infra
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
86decddcd199afe6acc5dd936a0a46d138a4e5ab
a13labs.infra/ansible/roles/auto_suspend/README.md
T

15 KiB
Raw Blame History

auto_suspend — Idle-based System Suspend

Auto-suspend monitors a system's CPU and GPU utilization, applies exponential moving average (EMA) smoothing, and suspends the machine when idle for a configured period.

Target platform: RHEL/EL 9 with SELinux enforcing (Fedora/RHEL datacenter GPUs)

Goal

Automatically put idle GPU workstations into suspend (RAM sleep) to save power and reduce heat, while ensuring no user session or active workload is interrupted.

The system checks every 5 minutes. It suspends only after idle_timeout_cycles consecutive checks confirm the system is idle, using EMA-smoothed metrics to avoid false positives from transient spikes.

Architecture

┌─────────────┐     every 5 min      ┌──────────────────┐
│  timer      │ ────────────────────▶ │  service         │
│ auto-suspend│                       │ auto-suspend     │
└─────────────┘                       └────────┬─────────┘
                                                │
                                    measures: cpu, nvidia, amd
                                                │
                                    EMA smoothing + warmup check
                                                │
                                    evaluate idle?
                                          ╱    ╲
                                     yes ╱      ╲ no
                                       │          │
                                       ▼          ▼
                              idle_counter++   idle_counter=0
                                       │
                               counter ≥ 4?
                                    ╱      ╲
                                yes ╱        ╲ no
                                  │          │
                                  ▼          ▼
                            systemctl     re-persist
                            suspend        state.json

Key design decisions

  • State persists across timer invocations — state.json carries EMA averages, idle counter, and sample count between checks. The system can accumulate 20+ minutes of data across reboots (state survives suspend).
  • Any busy GPU prevents suspend — max utilization across all NVIDIA + AMD GPUs is used. One busy GPU = no suspend.
  • Reset on resume — after systemctl suspend, state is fully reset so the warmup period starts fresh.
  • GPU presence is optional — None from a GPU driver means "no GPU of this type", which does NOT block suspend.

Directory Structure

ansible/roles/auto_suspend/
├── defaults/main.yml           # Configurable parameters (tunable via inventory/host_vars)
├── meta/main.yml               # Galaxy metadata
└── tasks/
│   └── main.yml                # Ansible tasks (dependencies, scripts, systemd, SELinux)
└── files/
    ├── auto_suspend.py         # Main Python daemon (522 lines)
    ├── test_auto_suspend.py    # Unit tests (48 tests, stdlib only)
    └── fixtures/               # Real command samples from gpu-01
        ├── amd-smi-monitor--json.txt
        ├── mpstat--1--1.idle.txt
        ├── nvidia-smi--query-gpu.utilization.gpu.csv.noheader.idle.txt
        └── who.with-users.txt

Playbook Usage

# ansible/playbook_auto_suspend.yml
- name: Setup Auto Suspend
  hosts: auto_suspend
  roles:
    - auto_suspend

Deploy via:

cd ansible/
sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_auto_suspend.yml

The auto_suspend inventory group currently contains only gpu-01.lab.alexpires.me.

Configuration (defaults/main.yml)

Variable Default Description
auto_suspend_alpha 0.3 EMA weight for current sample (0.3 = 30% current, 70% historical)
auto_suspend_min_samples 3 Warmup checks before trust (3 × 5 min = 15 min minimum)
auto_suspend_idle_timeout_cycles 4 Consecutive idle checks before suspend (4 × 5 min = 20 min total)
auto_suspend_cpu_idle_threshold 90.0 CPU idle % must exceed this to count as idle
auto_suspend_gpu_util_threshold 5.0 GPU util % must be below this to count as idle
auto_suspend_state_dir /var/lib/auto_suspend Directory for state.json persistence
auto_suspend_log_file /var/log/auto_suspend.log Log file path (RotatingFileHandler, 1 MB × 3 backups)
auto_suspend_grub_config_path_bios /boot/grub2/grub.cfg GRUB config output path for BIOS systems

All variables can be overridden via inventory, group_vars, or host_vars.

Deployed Artifacts

Path Type Purpose
/usr/local/bin/auto_suspend.py Python script Main daemon, copied from files/auto_suspend.py
/etc/systemd/system/auto-suspend.service systemd oneshot Runs the check + suspend logic
/etc/systemd/system/auto-suspend.timer systemd timer Triggers service every 5 minutes (5 min after boot)
/etc/systemd/sleep.conf.d/mem-deep.conf sysctl override Sets MemorySleepMode=deep
/var/lib/auto_suspend/state.json JSON EMA state, idle counter, sample count
/var/log/auto_suspend.log log Rotating log (1 MB, 3 backups)

auto_suspend.py — Script Reference

CLI Arguments

All arguments map directly to defaults/main.yml variables:

--alpha 0.3                    EMA smoothing factor
--min-samples 3                Warmup period
--idle-timeout-cycles 4        Consecutive idle checks needed
--cpu-idle-threshold 90.0      CPU idle % threshold
--gpu-util-threshold 5.0       GPU util % threshold
--state-dir /var/lib/auto_suspend
--log-file /var/log/auto_suspend.log
--debug                        Enable debug-level logging
--dry-run                      Run check without persisting state or suspending

Public Functions

Function Returns Purpose
setup_logging(log_file, debug) logging.Logger Configures file + stderr handlers
fresh_state() dict Returns default state with all values at initial values
load_state(state_dir, logger) dict Loads state from disk, returns fresh on any error
save_state(state_dir, state, logger) None Atomic write via temp file + os.replace
measure_cpu(logger) float | None Runs mpstat 1 1, returns %idle
measure_gpu_nvidia(logger) float | None Runs nvidia-smi, returns max util across all GPUs
measure_gpu_amd(logger) float | None Runs amd-smi monitor --json, returns gfx.value of GPU 0
get_active_users(logger) int Parses who output, returns non-empty line count
get_why_not_idle(cpu_avg, gpu_avg, sample_count, idle_counter, has_users, cpu_threshold, gpu_threshold) list[str] Returns human-readable reasons why system is not idle
update_averages(state, cpu_n, gpu_n, alpha) dict EMA update: returns new state with averaged values
evaluate(state, has_users, cpu_threshold, gpu_threshold) bool Gate checks: warmup → users → CPU → GPU
signal_handler(signum, frame) NoReturn SIGTERM/SIGINT handler, calls sys.exit(0)
main() NoReturn Entry point: measures, evaluates, updates state, suspends

State Schema (state.json)

{
  "cpu_idle_n": null,           // latest CPU idle % (current sample)
  "cpu_idle_n1": null,          // latest CPU idle % (previous sample, n-1)
  "gpu_util_n": null,           // latest GPU util % (current sample)
  "gpu_util_n1": null,         // latest GPU util % (previous sample, n-1)
  "cpu_avg": null,              // EMA-smoothed CPU idle average
  "gpu_avg": null,              // EMA-smoothed GPU utilization average
  "idle_counter": 0,            // consecutive idle cycles
  "sample_count": 0             // total samples collected
}

EMA Formula

EMA_new = alpha * current_sample + (1 - alpha) * previous_EMA

With alpha=0.3: 30% weight on the current reading, 70% on the historical average. round(..., 2) keeps precision to 2 decimal places.

GPU Detection

NVIDIA — nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits

  • Parses each line as float, returns max() across all GPUs
  • Returns None if nvidia-smi is missing, fails, or times out
  • None means no NVIDIA GPU present (does NOT block suspend)

AMD — amd-smi monitor --json

  • Parses data[0]["gfx"]["value"] from first GPU
  • Handles both {"value": N, "unit": "%"} and raw numeric gfx formats
  • Returns None if amd-smi is missing, fails, or times out

Combined — max() across all non-None GPU measurements. Any busy GPU prevents suspend.

Suspend Decision Logic (evaluate)

Four gates in order:

  1. Warmup — If sample_count < min_samples, return False (skip evaluation)
  2. Active users — If who returns non-empty output, return False
  3. CPU idle — If cpu_avg > cpu_threshold is False, return False
  4. GPU idle — If GPU is present (gpu_avg is not None) and gpu_avg >= gpu_threshold, return False

Only when all gates pass does idle_counter increment. When idle_counter >= idle_timeout_cycles, systemctl suspend is called. On failure to suspend, idle_counter resets to 0.

Dry-Run Mode

--dry-run runs all measurement, EMA, and evaluation logic but:

  • Does NOT persist state to disk
  • Does NOT call systemctl suspend
  • Prints Would suspend: idle timeout reached (idle_counter=X/Y) instead

Unit Tests (test_auto_suspend.py)

48 tests covering all public functions using unittest + unittest.mock.

Test Classes

Class Tests Coverage
TestFreshState 1 Returns correct defaults
TestLoadState 4 Missing file, valid JSON, invalid JSON, missing keys merged
TestSaveState 3 Atomic write, parent dir creation, no leftover tmp
TestMeasureCpu 4 Idle, busy, non-zero exit, timeout
TestMeasureGpuNvidia 6 Idle, busy single, multi-GPU max, file not found, timeout, non-zero exit
TestMeasureGpuAmd 7 Busy, idle, medium, file not found, short output, empty array, non-zero exit
TestGetActiveUsers 3 No users, with users, timeout
TestUpdateAverages 4 First sample, EMA calc, n→n1 shift, None preservation
TestEvaluate 10 Idle system, warmup skip, users, CPU not idle, GPU busy, GPU None, at threshold CPU, at threshold GPU, custom thresholds
TestGetWhyNotIdle 5 Users, CPU low, GPU high, warming up, multiple reasons
TestSetupLogging 1 Returns logger with handlers
TestSignalHandler 1 Returns exit code 0

Running Tests

cd ansible/roles/auto_suspend
python3 -m pytest files/test_auto_suspend.py -v
# or
python3 files/test_auto_suspend.py

Tests use unittest.mock.patch("auto_suspend.subprocess.run") to intercept all subprocess calls. Fixture data comes from real command outputs on gpu-01.

Making Changes

Modifying the Python Script (files/auto_suspend.py)

  1. Edit files/auto_suspend.py
  2. Add or update tests in files/test_auto_suspend.py — mock subprocess.run with MagicMock(returncode=..., stdout=..., stderr=...)
  3. Add new fixtures to files/fixtures/ if you need new command output samples
  4. Run python3 -m pytest files/test_auto_suspend.py -v — all 48 tests must pass
  5. Deploy to gpu-01 and verify with journalctl -u auto-suspend.service

Modifying Ansible Tasks (tasks/main.yml)

The role uses inline content for systemd unit files (no templates/ directory). To change systemd configuration, edit the inline YAML in the relevant ansible.builtin.copy task.

Key tasks in order:

  1. Install dependencies (sysstat, python3)
  2. Enable sysstat service
  3. Set GRUB mem_sleep_default=deep (BIOS only)
  4. Create state directory + log file
  5. Copy Python script
  6. Set SELinux context (bin_t + restorecon)
  7. Create systemd service (inline)
  8. Create sleep.conf.d deep sleep override
  9. Create systemd timer (inline)
  10. Reload systemd daemon
  11. Enable + start timer
  12. Verify with --dry-run

Adding a New GPU Vendor

  1. Add measure_gpu_<vendor>(logger) in auto_suspend.py following the existing pattern (return float utilization or None)
  2. Wire it into main(): 
    gpu_<vendor> = measure_gpu_<vendor>(logger)
gpu_values = [v for v in [gpu_nvidia, gpu_amd, gpu_<vendor>] if v is not None]
gpu_util = max(gpu_values) if gpu_values else None
  3. Add CLI argument if vendor-specific threshold is needed
  4. Add fixtures for test data
  5. Add TestMeasureGpu<Vendor> test class

Overriding Defaults

All defaults in defaults/main.yml can be overridden via:

  • Inventory: group_vars/auto_suspend.yml
  • Host vars: host_vars/gpu-01.lab.alexpires.me.yml
  • Command line: ansible-playbook ... -e "auto_suspend_alpha=0.5"

SELinux

On EL 9 with SELinux enforcing, the Python script in /usr/local/bin/ needs bin_t context:

- name: Set SELinux file context on script
  community.general.sefcontext:
    path: /usr/local/bin/auto_suspend.py
    setype: bin_t
    state: present
  failed_when: false

- name: Restore SELinux context on script
  ansible.builtin.command: restorecon -v /usr/local/bin/auto_suspend.py
  failed_when: false

Both tasks use failed_when: false to tolerate systems where SELinux is disabled or the context is already correct.

Requires community.general Ansible collection.

Systemd Configuration

auto-suspend.service

  • Type=oneshot — runs once per timer tick
  • Restart=on-failure, RestartSec=30 — retries on error
  • Passes all config as CLI flags (not via environment)

auto-suspend.timer

  • OnBootSec=5min — first run 5 minutes after boot
  • OnUnitActiveSec=5min — subsequent runs every 5 minutes
  • Unit=auto-suspend.service

sleep.conf.d

/etc/systemd/sleep.conf.d/mem-deep.conf sets MemorySleepMode=deep to ensure deep sleep state (lowest power consumption).

GRUB

The role ensures mem_sleep_default=deep is present in GRUB_CMDLINE_LINUX and regenerates the GRUB config. This is BIOS-specific (UEFI is not currently supported).

Hardware Target

The role was developed for gpu-01.lab.alexpires.me:

  • Fedora 42 (kernel 6.17.11), 12-core x86_64
  • NVIDIA GPU + AMD GPU (dual-GPU system)
  • SELinux enforcing
  • Used for AI inference (Ollama, ComfyUI, llama.cpp, Qwen)
  • Network-accessible workstation

Troubleshooting

Check service status

systemctl status auto-suspend.service
systemctl status auto-suspend.timer

View logs

journalctl -u auto-suspend.service --no-pager
cat /var/log/auto_suspend.log

Run a manual check

/usr/local/bin/auto_suspend.py --dry-run --debug

Check state

cat /var/lib/auto_suspend/state.json

Force a timer run

systemctl start auto-suspend.service

Re-deploy after changes

cd ansible/
sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_auto_suspend.yml
Reference in New Issue View Git Blame Copy Permalink