Fixing Kernel Panic from Stack Overflow on Linux

Quick answer: Boot with panic=10 sysrq_always_enabled=1 and check /var/log/kern.log for the culprit module or process with a deep call trace.

What's a kernel stack overflow and why should you care?

Last week I had a client running a small file server on Ubuntu 22.04. The machine would crash twice a day, always with a kernel panic that mentioned stack overflow. The owner was losing money every time. I've seen this a bunch of times, and it's rarely a hardware problem. It's almost always a buggy kernel module or a process that's eating too much stack space—like a recursive loop gone wild or a network driver that's leaking memory.

The kernel stack is a limited area (usually 8KB or 16KB on x86_64). When a function calls itself too deep or a driver uses a lot of local variables, it overflows. The kernel stops everything and panics. No warning, no graceful shutdown. Just a frozen screen with scary text.

Step-by-step fix: Find and stop the stack overflow

1. Boot with crash recovery options. At the GRUB menu, press 'e' and add panic=10 sysrq_always_enabled=1 to the kernel line. This reboots the machine 10 seconds after a panic and enables the SysRq magic keys. Had a client last month whose server stayed dead for hours because nobody enabled this.
2. Look at the logs. After it crashes again, check /var/log/kern.log or dmesg for the panic output. You'll see a line like "Kernel panic - not syncing: stack overflow" followed by a call trace. The call trace lists the functions that led to the overflow. The last one before the panic is often the bad one.
3. Identify the module or process. In the call trace, look for module names in brackets like [tun] or [usb_storage] or a process name like sendmail. Common culprits are network drivers, USB storage drivers, or custom kernel modules. In my client's case, it was a third-party RAID driver that was buggy.
4. Disable the bad module. If it's a module, blacklist it by editing /etc/modprobe.d/blacklist.conf and adding blacklist bad_module_name. Then update initramfs with update-initramfs -u and reboot. If it's a process, kill it or uninstall the software that runs it.

Alternative fixes if the main one doesn't work

Increase stack size (rarely works, but try)

You can add vmalloc=256M or stack_guard_gap=128 to the kernel boot parameters. This gives more room for stack operations. But don't expect miracles. The real fix is the module.

Use a different kernel version

Sometimes a kernel update introduces the bug. Boot an older kernel from GRUB (select 'Advanced options' and pick an older version). If the panic stops, pin that kernel version until the bug is fixed upstream.

Run a crash dump analysis

If logs aren't enough, enable kdump. Install kdump-tools, configure /etc/default/kdump-tools with KDUMP_COREDIR=/var/crash, then reboot. After the next panic, you'll have a vmcore file that you can analyze with crash tool. This shows the exact memory state.

Prevention tips so you don't see this again

• Always test kernel updates on one machine first before rolling out to the whole fleet. Had a client who updated all 20 servers at once and got 20 crashes in one day.
• Keep an eye on custom modules—especially drivers from hardware vendors. Those are often the first to break with new kernels.
• Set panic=10 and sysrq_always_enabled=1 in /etc/default/grub permanently. It's a no-brainer that saves you a lot of headache.
• Monitor stack usage with tools like perf or trace-cmd if you suspect a process is getting too deep. It's rare, but catching it early beats a panic.

One last thing: if you see the panic only under heavy load, check your network drivers. I've seen this happen with Realtek r8169 and some Broadcom cards. They have a history of stack issues in certain kernels.