a13labs.infra/ansible/roles/auto_suspend

auto_suspend — Idle-based System Suspend

Auto-suspend monitors a system's CPU and GPU utilization, applies exponential moving average (EMA) smoothing, and suspends the machine when idle for a configured period.

Target platform: RHEL/EL 9 with SELinux enforcing (Fedora/RHEL datacenter GPUs)

---

Goal

Automatically put idle GPU workstations into suspend (RAM sleep) to save power and reduce heat, while ensuring no user session or active workload is interrupted.

The system checks every 5 minutes. It suspends only after idle_timeout_cycles consecutive checks confirm the system is idle, using EMA-smoothed metrics to avoid false positives from transient spikes.

---

Architecture

┌─────────────┐     every 5 min      ┌──────────────────┐
│  timer      │ ────────────────────▶ │  service         │
│ auto-suspend│                       │ auto-suspend     │
└─────────────┘                       └────────┬─────────┘
                                                │
                                    measures: cpu, nvidia, amd
                                                │
                                    EMA smoothing + warmup check
                                                │
                                    evaluate idle?
                                          ╱    ╲
                                     yes ╱      ╲ no
                                       │          │
                                       ▼          ▼
                              idle_counter++   idle_counter=0
                                       │
                               counter ≥ 4?
                                    ╱      ╲
                                yes ╱        ╲ no
                                  │          │
                                  ▼          ▼
                            systemctl     re-persist
                            suspend        state.json

Key design decisions

• State persists across timer invocations — state.json carries EMA averages, idle counter, and sample count between checks. The system can accumulate 20+ minutes of data across reboots (state survives suspend).
• Any busy GPU prevents suspend — max utilization across all NVIDIA + AMD GPUs is used. One busy GPU = no suspend.
• Reset on resume — after systemctl suspend, state is fully reset so the warmup period starts fresh.
• GPU presence is optional — None from a GPU driver means "no GPU of this type", which does NOT block suspend.

---

Directory Structure

ansible/roles/auto_suspend/
├── defaults/main.yml           # Configurable parameters (tunable via inventory/host_vars)
├── meta/main.yml               # Galaxy metadata
└── tasks/
│   └── main.yml                # Ansible tasks (dependencies, scripts, systemd, SELinux)
└── files/
    ├── auto_suspend.py         # Main Python daemon (522 lines)
    ├── test_auto_suspend.py    # Unit tests (48 tests, stdlib only)
    └── fixtures/               # Real command samples from gpu-01
        ├── amd-smi-monitor--json.txt
        ├── mpstat--1--1.idle.txt
        ├── nvidia-smi--query-gpu.utilization.gpu.csv.noheader.idle.txt
        └── who.with-users.txt

---

Playbook Usage

# ansible/playbook_auto_suspend.yml
- name: Setup Auto Suspend
  hosts: auto_suspend
  roles:
    - auto_suspend

Deploy via:

cd ansible/
sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_auto_suspend.yml

The auto_suspend inventory group currently contains only gpu-01.lab.alexpires.me.

---

Configuration (defaults/main.yml)

Variable	Default	Description
auto_suspend_alpha	0.3	EMA weight for current sample (0.3 = 30% current, 70% historical)
auto_suspend_min_samples	3	Warmup checks before trust (3 × 5 min = 15 min minimum)
auto_suspend_idle_timeout_cycles	4	Consecutive idle checks before suspend (4 × 5 min = 20 min total)
auto_suspend_cpu_idle_threshold	90.0	CPU idle % must exceed this to count as idle
auto_suspend_gpu_util_threshold	5.0	GPU util % must be below this to count as idle
auto_suspend_state_dir	/var/lib/auto_suspend	Directory for state.json persistence
auto_suspend_log_file	/var/log/auto_suspend.log	Log file path (RotatingFileHandler, 1 MB × 3 backups)
auto_suspend_grub_config_path_bios	/boot/grub2/grub.cfg	GRUB config output path for BIOS systems

All variables can be overridden via inventory, group_vars, or host_vars.

---

Deployed Artifacts

Path	Type	Purpose
/usr/local/bin/auto_suspend.py	Python script	Main daemon, copied from files/auto_suspend.py
/etc/systemd/system/auto-suspend.service	systemd oneshot	Runs the check + suspend logic
/etc/systemd/system/auto-suspend.timer	systemd timer	Triggers service every 5 minutes (5 min after boot)
/etc/systemd/sleep.conf.d/mem-deep.conf	sysctl override	Sets MemorySleepMode=deep
/var/lib/auto_suspend/state.json	JSON	EMA state, idle counter, sample count
/var/log/auto_suspend.log	log	Rotating log (1 MB, 3 backups)

---

auto_suspend.py — Script Reference

CLI Arguments

All arguments map directly to defaults/main.yml variables:

--alpha 0.3                    EMA smoothing factor
--min-samples 3                Warmup period
--idle-timeout-cycles 4        Consecutive idle checks needed
--cpu-idle-threshold 90.0      CPU idle % threshold
--gpu-util-threshold 5.0       GPU util % threshold
--state-dir /var/lib/auto_suspend
--log-file /var/log/auto_suspend.log
--debug                        Enable debug-level logging
--dry-run                      Run check without persisting state or suspending

Public Functions

Function	Returns	Purpose
setup_logging(log_file, debug)	logging.Logger	Configures file + stderr handlers
fresh_state()	dict	Returns default state with all values at initial values
load_state(state_dir, logger)	dict	Loads state from disk, returns fresh on any error
save_state(state_dir, state, logger)	None	Atomic write via temp file + os.replace
measure_cpu(logger)	float | None	Runs mpstat 1 1, returns %idle
measure_gpu_nvidia(logger)	float | None	Runs nvidia-smi, returns max util across all GPUs
measure_gpu_amd(logger)	float | None	Runs amd-smi monitor --json, returns gfx.value of GPU 0
get_active_users(logger)	int	Parses who output, returns non-empty line count
get_why_not_idle(cpu_avg, gpu_avg, sample_count, idle_counter, has_users, cpu_threshold, gpu_threshold)	list[str]	Returns human-readable reasons why system is not idle
update_averages(state, cpu_n, gpu_n, alpha)	dict	EMA update: returns new state with averaged values
evaluate(state, has_users, cpu_threshold, gpu_threshold)	bool	Gate checks: warmup → users → CPU → GPU
signal_handler(signum, frame)	NoReturn	SIGTERM/SIGINT handler, calls sys.exit(0)
main()	NoReturn	Entry point: measures, evaluates, updates state, suspends

State Schema (state.json)

{
  "cpu_idle_n": null,           // latest CPU idle % (current sample)
  "cpu_idle_n1": null,          // latest CPU idle % (previous sample, n-1)
  "gpu_util_n": null,           // latest GPU util % (current sample)
  "gpu_util_n1": null,         // latest GPU util % (previous sample, n-1)
  "cpu_avg": null,              // EMA-smoothed CPU idle average
  "gpu_avg": null,              // EMA-smoothed GPU utilization average
  "idle_counter": 0,            // consecutive idle cycles
  "sample_count": 0             // total samples collected
}

EMA Formula

EMA_new = alpha * current_sample + (1 - alpha) * previous_EMA

With alpha=0.3: 30% weight on the current reading, 70% on the historical average. round(..., 2) keeps precision to 2 decimal places.

GPU Detection

NVIDIA — nvidia-smi --query-gpu=utilization.gpu --format=csv,noheader,nounits

• Parses each line as float, returns max() across all GPUs
• Returns None if nvidia-smi is missing, fails, or times out
• None means no NVIDIA GPU present (does NOT block suspend)

AMD — amd-smi monitor --json

• Parses data[0]["gfx"]["value"] from first GPU
• Handles both {"value": N, "unit": "%"} and raw numeric gfx formats
• Returns None if amd-smi is missing, fails, or times out

Combined — max() across all non-None GPU measurements. Any busy GPU prevents suspend.

Suspend Decision Logic (evaluate)

Four gates in order:

1. Warmup — If sample_count < min_samples, return False (skip evaluation)
2. Active users — If who returns non-empty output, return False
3. CPU idle — If cpu_avg > cpu_threshold is False, return False
4. GPU idle — If GPU is present (gpu_avg is not None) and gpu_avg >= gpu_threshold, return False

Only when all gates pass does idle_counter increment. When idle_counter >= idle_timeout_cycles, systemctl suspend is called. On failure to suspend, idle_counter resets to 0.

Dry-Run Mode

--dry-run runs all measurement, EMA, and evaluation logic but:

• Does NOT persist state to disk
• Does NOT call systemctl suspend
• Prints Would suspend: idle timeout reached (idle_counter=X/Y) instead

---

Unit Tests (test_auto_suspend.py)

48 tests covering all public functions using unittest + unittest.mock.

Test Classes

Class	Tests	Coverage
TestFreshState	1	Returns correct defaults
TestLoadState	4	Missing file, valid JSON, invalid JSON, missing keys merged
TestSaveState	3	Atomic write, parent dir creation, no leftover tmp
TestMeasureCpu	4	Idle, busy, non-zero exit, timeout
TestMeasureGpuNvidia	6	Idle, busy single, multi-GPU max, file not found, timeout, non-zero exit
TestMeasureGpuAmd	7	Busy, idle, medium, file not found, short output, empty array, non-zero exit
TestGetActiveUsers	3	No users, with users, timeout
TestUpdateAverages	4	First sample, EMA calc, n→n1 shift, None preservation
TestEvaluate	10	Idle system, warmup skip, users, CPU not idle, GPU busy, GPU None, at threshold CPU, at threshold GPU, custom thresholds
TestGetWhyNotIdle	5	Users, CPU low, GPU high, warming up, multiple reasons
TestSetupLogging	1	Returns logger with handlers
TestSignalHandler	1	Returns exit code 0

Running Tests

cd ansible/roles/auto_suspend
python3 -m pytest files/test_auto_suspend.py -v
# or
python3 files/test_auto_suspend.py

Tests use unittest.mock.patch("auto_suspend.subprocess.run") to intercept all subprocess calls. Fixture data comes from real command outputs on gpu-01.

---

Making Changes

Modifying the Python Script (files/auto_suspend.py)

1. Edit files/auto_suspend.py
2. Add or update tests in files/test_auto_suspend.py — mock subprocess.run with MagicMock(returncode=..., stdout=..., stderr=...)
3. Add new fixtures to files/fixtures/ if you need new command output samples
4. Run python3 -m pytest files/test_auto_suspend.py -v — all 48 tests must pass
5. Deploy to gpu-01 and verify with journalctl -u auto-suspend.service

Modifying Ansible Tasks (tasks/main.yml)

The role uses inline content for systemd unit files (no templates/ directory). To change systemd configuration, edit the inline YAML in the relevant ansible.builtin.copy task.

Key tasks in order:

1. Install dependencies (sysstat, python3)
2. Enable sysstat service
3. Set GRUB mem_sleep_default=deep (BIOS only)
4. Create state directory + log file
5. Copy Python script
6. Set SELinux context (bin_t + restorecon)
7. Create systemd service (inline)
8. Create sleep.conf.d deep sleep override
9. Create systemd timer (inline)
10. Reload systemd daemon
11. Enable + start timer
12. Verify with --dry-run

Adding a New GPU Vendor

1. Add measure_gpu_<vendor>(logger) in auto_suspend.py following the existing pattern (return float utilization or None)
2. Wire it into main():

   gpu_<vendor> = measure_gpu_<vendor>(logger)
   gpu_values = [v for v in [gpu_nvidia, gpu_amd, gpu_<vendor>] if v is not None]
   gpu_util = max(gpu_values) if gpu_values else None
   

3. Add CLI argument if vendor-specific threshold is needed
4. Add fixtures for test data
5. Add TestMeasureGpu<Vendor> test class

Overriding Defaults

All defaults in defaults/main.yml can be overridden via:

• Inventory: group_vars/auto_suspend.yml
• Host vars: host_vars/gpu-01.lab.alexpires.me.yml
• Command line: ansible-playbook ... -e "auto_suspend_alpha=0.5"

---

SELinux

On EL 9 with SELinux enforcing, the Python script in /usr/local/bin/ needs bin_t context:

- name: Set SELinux file context on script
  community.general.sefcontext:
    path: /usr/local/bin/auto_suspend.py
    setype: bin_t
    state: present
  failed_when: false

- name: Restore SELinux context on script
  ansible.builtin.command: restorecon -v /usr/local/bin/auto_suspend.py
  failed_when: false

Both tasks use failed_when: false to tolerate systems where SELinux is disabled or the context is already correct.

Requires community.general Ansible collection.

---

Systemd Configuration

auto-suspend.service

• Type=oneshot — runs once per timer tick
• Restart=on-failure, RestartSec=30 — retries on error
• Passes all config as CLI flags (not via environment)

auto-suspend.timer

• OnBootSec=5min — first run 5 minutes after boot
• OnUnitActiveSec=5min — subsequent runs every 5 minutes
• Unit=auto-suspend.service

sleep.conf.d

/etc/systemd/sleep.conf.d/mem-deep.conf sets MemorySleepMode=deep to ensure deep sleep state (lowest power consumption).

GRUB

The role ensures mem_sleep_default=deep is present in GRUB_CMDLINE_LINUX and regenerates the GRUB config. This is BIOS-specific (UEFI is not currently supported).

---

Hardware Target

The role was developed for gpu-01.lab.alexpires.me:

• Fedora 42 (kernel 6.17.11), 12-core x86_64
• NVIDIA GPU + AMD GPU (dual-GPU system)
• SELinux enforcing
• Used for AI inference (Ollama, ComfyUI, llama.cpp, Qwen)
• Network-accessible workstation

---

Troubleshooting

Check service status

systemctl status auto-suspend.service
systemctl status auto-suspend.timer

View logs

journalctl -u auto-suspend.service --no-pager
cat /var/log/auto_suspend.log

Run a manual check

/usr/local/bin/auto_suspend.py --dry-run --debug

Check state

cat /var/lib/auto_suspend/state.json

Force a timer run

systemctl start auto-suspend.service

Re-deploy after changes

cd ansible/
sectool -f ../sectool.json exec ansible-playbook -u $ANSIBLE_USER playbook_auto_suspend.yml