SAN Path Failover Didn't Trigger: What I Fixed for a Client
When a SAN path goes down but failover doesn't kick in, your server drops offline. Here's the real cause and fix, from a real client mess.
When This Happens
You're running a VMware cluster or a Windows Server 2019 file server connected to a SAN via Fibre Channel or iSCSI. One of the two paths to your storage array goes down — maybe an HBA driver craps out, or a switch port goes wonky. Your server should transparently fail over to the second path. But it doesn't. Instead, the server freezes, or the drive drops offline, or you see I/O errors in the event log for thirty seconds before it recovers. I had a client last month whose entire SQL Server database went read-only because the SAN failover didn't trip in time. The error they saw was 'The device, \Device\Harddisk1\DR1, is not ready for access yet' in System Event ID 7. On Linux, you'd see 'multipathd: checker failed path checker' followed by a delayed recovery.
Root Cause
The most common reason this happens is that the operating system's disk timeout value is too long. By default, Windows sets DiskTimeoutValue in the registry to 60 seconds. That means the OS will wait a full minute before deciding a path is dead. But your SAN fabric might take 30 seconds to detect the link down and start flapping. If you have multipathing software like MPIO or Linux multipathd, it's supposed to handle this faster — but the disk's timeout trumps everything. I've also seen cases where the multipath configuration has the wrong path group policy (like 'round-robin' instead of 'failover') or the device mapper doesn't have the right udev rules to reassign the path properly. On VMware ESXi, it's often the disk.terminateOnTimeout setting or the wrong SATP rule that keeps the dead path active.
The Fix
Step 1: Check Your Current Disk Timeout
On Windows, open an admin PowerShell and run:
Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Disk' -Name TimeOutValue/sys/block/sdX/device/timeout for each SAN disk. It's usually 30 or 60. On ESXi, it's in advanced settings: disk.terminateOnTimeout (should be 1) and disk.terminateOnTimeout related to the SATP.Step 2: Reduce the Timeout to 15-20 Seconds
On Windows, set it to 15 seconds:
Set-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Services\Disk' -Name TimeOutValue -Value 15echo 15 > /sys/block/sdX/device/timeout/etc/udev/rules.d/99-san-timeout.rules with ACTION=="add|change", SUBSYSTEM=="block", ATTR{device/timeout}="15". Then reload: udevadm control --reload-rules && udevadm trigger. On ESXi, go to Configure > Advanced Settings, search for 'diskTimeout' and set disk.terminateOnTimeout to 1 and lower the timeout (default is 30).Step 3: Fix the MPIO or Multipath Config
Windows MPIO: Open MPIO control panel, go to DSM Discovery, make sure your storage array's vendor ID is listed. If not, add it manually. Then check the policy: for failover, you want 'Fail Over Only'. You can set this via PowerShell:
Set-MSDSMGlobalDefaultLoadBalancePolicy -Policy FOOSet-MPIOSetting -PathId X -LoadBalancePolicy FOO. Linux: Check /etc/multipath.conf. Make sure path_grouping_policy is failover if you want one active path. If you're using multibus, you might need path_selector set to round-robin 0. The key: ensure failback immediate is set so when the primary path recovers, it takes over right away. Then run multipath -r and check multipath -ll to see active paths.Step 4: Verify HBA and Driver Settings
On Windows, check the HBA driver properties: set 'Link Down Timeout' to 5 seconds (or whatever your array vendor recommends). Common HBAs like Emulex or QLogic have this in the device properties under 'Advanced' in Device Manager. On Linux, use lspci -vvv -s 05:00.0 to find the HBA PCI slot, then check driver parameters: cat /sys/module/qla2xxx/parameters/ql2xmaintimer. You want it low, like 2. On ESXi, check the HBA firmware and driver version — old drivers ignored link-down events.
Step 5: Test the Failover
Pull the primary FC cable or disable the iSCSI target's IP. Watch the logs: Windows Event ID 7 should appear within 15 seconds, then Event ID 1066 (MPIO path recovered). Linux: tail -f /var/log/messages should show multipathd: sda: path checkers start and then multipathd: sda: reinstated. ESXi: tail -f /var/log/vmkernel should show lost path and restored path quickly.
If It Still Fails
If failover still doesn't trigger after these steps, you've got a deeper issue. First, check your SAN fabric: did the switch actually propagate the link-down to the target? Use the array's management interface to see if the host port is dead. Second, verify your multipath daemon is running: Windows MPIO should have the disk claimable, Linux systemctl status multipathd should show active. Third, look for a bug in your specific HBA driver — I've seen old Brocade HBAs that won't fail over if there's a CRC mismatch on the cable. And finally, if you're using iSCSI, check the iSCSI initiator's login timeout and session recovery settings. Some arrays require a specific 'Error Recovery Level' setting. If you've done all this and it still drops, call your storage vendor. But nine times out of ten, it's that damn disk timeout setting.
Was this solution helpful?