100% CPU Idle? Your Process Is Actually Doing Work

1. Busy Waiting — The Most Common Culprit

Last month I had a client whose file server would randomly peg one core at 100% while the system felt totally idle. No disk I/O, no network traffic, just a process called python3 spinning hot. Turns out it was a monitoring script using a tight loop instead of sleeping.

Busy waiting is when a process checks a condition repeatedly without pausing. It looks like "idle" because the process isn't doing real work—it's just spinning.

How to Identify It

Run top and note the PID of the high-CPU process. Then use strace -p PID. If you see a flood of repeated system calls like futex, nanosleep returning immediately, or read with EAGAIN, it's busy waiting.

sudo strace -p 12345 -c -S calls

This counts system calls. If you see thousands per second, that's busy waiting.

Fix It

• If it's your code: add sleep(0.1) or use select() with a timeout.
• If it's a third-party tool: check for a config option like poll_interval or frequency. Set it to at least 1 second.
• For a systemd service: add Restart=on-failure and WatchdogSec=3 to kill stuck services.

I once fixed a Nagios plugin that did while true; do ; done in bash. The developer forgot a sleep. That's the classic rookie mistake.

2. Kernel Thread Stuck in an Infinite Loop

This one is trickier. A kernel thread (like [kworker] or [irq]) shows 100% CPU in top, but the system stays responsive. It's not idle—the kernel is doing work, but it's useless work.

How to Identify It

Use perf top. If you see a kernel function like delay_tsc, native_safe_halt, or do_idle at the top, you're looking at a buggy idle driver.

sudo perf top -K -p $(pgrep kworker)

Look for functions with unusual names like acpi_idle_enter_simple or intel_idle_fixup. Those are signs of a driver fighting with power management.

Fix It

• Update your kernel. Many idle bugs were fixed in 5.10+ for Intel and AMD.
• Try booting with idle=halt or processor.max_cstate=1 to bypass the broken driver. This is safe for most servers.
• If it's a virtual machine, set the CPU model to host-passthrough in QEMU.

Had a client on an old Xeon E5-2690 v3 where intel_idle.max_cstate=0 fixed a 100% CPU kworker. The machine was a database server, and that one stuck thread was eating a whole core. After the fix, CPU dropped to 10%.

3. GPU Driver Spinning on Polling

NVIDIA and AMD drivers sometimes poll for GPU status too aggressively. You'll see nvidia-smi or amdgpu in top with 100% CPU, but the GPU is idle.

How to Identify It

Check ps aux | grep nvidia or lsmod | grep amdgpu. Then run strace -p PID and look for repeated ioctl calls.

sudo strace -p $(pgrep nvidia-smi) -e ioctl -c

If you see thousands of ioctl calls per second, the driver is polling too fast.

Fix It

• Update the driver. NVIDIA's 470+ series fixed this for many cards.
• Set environment variable __GL_SYNC_DISPLAY_DEVICE=0 for OpenGL apps.
• For AMD: add amdgpu.powerplay=0 to kernel boot parameters.
• If you don't need GPU compute, blacklist the kernel module: echo 'blacklist nvidia' | sudo tee /etc/modprobe.d/blacklist-nvidia.conf. Reboot and the CPU usage goes to zero.

A client of mine ran a headless server with an NVIDIA T4 for ML inference. The driver was polling the GPU every millisecond even when idle. Setting nvidia-persistenced to run with --no-persistence fixed it.

Quick-Reference Summary Table

Remember: if the system feels idle but a process is at 100%, it's not a ghost. It's doing something—just not useful work. Strace and perf are your friends. Don't reboot until you've checked what's spinning.