Linux guests may soon get a way to behave better on a crowded host. The steal governor, a small driver that watches steal time and narrows the set of vCPUs a guest prefers to run on, was merged into tip’s sched/core this morning. Peter Zijlstra committed v14 of the series, which puts it on course for the Linux 7.4 merge window.
The problem
CPU steal time is the slice of the clock where a guest wanted to run and the hypervisor had lent the physical core to someone else. When several busy VMs share a host, that preemption stops being just lost time: a vCPU can be taken away while it holds a lock, in the middle of a critical section, or with caches and TLB warmed up, and the other guests wait on it. That is the noisy neighbour problem, and past a point, handing a guest more vCPUs makes the pile slower rather than faster.
The mechanism
The driver, drivers/virt/steal_governor.c, samples system-wide steal every second by default (interval_ms is tunable from 100 ms to 100 s) and compares the steal ratio against two thresholds. Above 5% it removes one core from the preferred set; at or below 2% it adds one back; in between it does nothing. Preferred CPUs are a scheduler concept this series introduces: the preferred mask is always a subset of the active mask, a policy hint rather than a hotplug: nobody’s CPUs are taken away, the scheduler only tries to keep work on the preferred ones. The shrink is core-granular, because SMT is the common case and some hypervisors schedule by core; the first housekeeping core stays protected, so a guest never folds itself to nothing. The loop re-checks its invariants every pass and shuts down gracefully if hotplug ever breaks them.
The module is STEAL_GOVERNOR, recommended built as =m since it is pointless on hardware that cannot see steal; it selects PREFERRED_CPU for the scheduler, and it refuses to load in a Xen Dom0.
Cooperation is the design
There is no communication between VMs, and none is needed: steal time is the shared signal, produced by the hypervisor, observed by everyone. When all the guests on a host run the governor, each notices the contention and folds a little, contention drops, and the thresholds let them unfold again. The documentation is honest that the scheme is cooperative: a VM that does not participate may end up with more CPUs than its neighbours, so the guidance is to enable it consistently across the VMs sharing a CPU pool.
What it is good for
The wins are concentrated where preemption hurts beyond the lost time: lock-holder stalls, critical sections, threads that talk to each other, and cache or TLB disruption. Databases are the usual example. Pure CPU-time workloads with independent workers may see a small regression from the extra scheduling discipline, and the docs say the default thresholds are a starting point to tune per system, not a law.
My read
The series is in sched/core now, on course for the 7.4 window, and it is the rare kernel feature whose value grows with every VM that turns it on. The docs are upfront that turning it on everywhere is the intended use, an honest ask: cooperative schemes are only as good as their participation. For anyone whose dashboards live one noisy neighbour from a bad afternoon, it is worth watching.
Source: the v14 cover letter and Phoronix’s write-up.