Azure AKS Node Drain Stuck Fix: 3 Real Causes

1. Pod Disruption Budget (PDB) Blocking the Drain

This is the #1 reason your AKS node drain gets stuck. A PodDisruptionBudget says 'only N pods can be down at once.' If your node has the last running pod of a deployment with minAvailable: 1, the drain stops. You'll see output like:

error when evicting pods - the request would violate the PodDisruptionBudget

Real trigger: You're running 3 replicas of your frontend across 4 nodes. One node goes down for maintenance. The PDB says 'keep 2 running.' But 2 are already on the down node. The drain freezes.

Fix: Check the PDB first. Run:

kubectl get pdb --all-namespaces

Look for any PDB with minAvailable or maxUnavailable that's too tight. If you see one blocking, temporarily delete the PDB (don't worry, it'll recreate if you have Helm or GitOps) or scale up your deployment replicas first. Example:

kubectl delete pdb frontend-pdb -n default

Then try the drain again. After the node drains and you're done, reapply the PDB from your YAML or Helm chart. The drain should complete in under a minute.

What to expect: After deleting the PDB, the node drain resumes. Pods get evicted one by one. You'll see 'evicted' messages in the kubectl output. The drain finishes with 'node drained' message.

2. PDBs with Zero Replicas or Stuck Pods

Sometimes the PDB itself is fine, but the pods it protects have issues. Common scenario: you scaled down a deployment to 0 replicas but the PDB still expects 1 pod to be available. Or a pod is in Terminating state with a finalizer that won't clear.

Real trigger: You scaled a StatefulSet to 0 for testing. The PDB from an old version still says minAvailable: 1. Now the node drain tries to evict that non-existent pod and blocks forever.

Fix:

1. Run kubectl get pods --all-namespaces | grep -i terminating to find stuck pods.
2. If you see a pod stuck in Terminating for over 5 minutes, force-delete it: kubectl delete pod <pod-name> -n <namespace> --grace-period=0 --force. After that, the drain moves on.
3. Also check the PDB again. If the deployment has 0 replicas but the PDB wants 1, delete the PDB or update it to match. The command to edit is kubectl edit pdb <pdb-name> -n <namespace>. Change minAvailable to 0 or remove it.

What to expect: Forcing a pod delete removes it instantly — no grace period. The drain then sees the pod is gone and continues. If you edited the PDB, the drain picks up the new rules in seconds.

3. Persistent Volume (PV) / Volume Attachment Issues

AKS nodes with attached Azure Disks or Files can block drains. The kubelet won't evict a pod if it can't unmount the volume. This happens when the volume is still mounted on the node but the pod is gone, or if the CSI driver has a glitch.

Real trigger: A pod using an Azure Disk (managed-csi) crashed. The disk is still attached to the node as a mount, but the pod's container is dead. The drain attempts to detach the disk, but the CSI driver reports 'volume still in use.' Drain stalls.

Fix:

1. First, check if the node has any lingering volumes: kubectl describe node <node-name>. Look under 'Volumes in Use' — if you see a disk that shouldn't be there, note the PV name.
2. Try draining with the --delete-emptydir-data flag (if any pods use emptyDir volumes that can be recreated). But for Azure Disks, the real fix is to force the detach. Run: kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data --force. The --force flag tells kubelet to evict even if there are volume issues. It's safe for production if you accept the risk of data loss on the disk (which you shouldn't if it's critical).
3. If that doesn't work, you need to cordon the node first: kubectl cordon <node-name>. Then manually delete the pods on that node: kubectl delete pod --field-selector=spec.nodeName=<node-name> --all-namespaces. After all pods are gone, you can safely drain the node (it'll just evict daemonsets). Then go into the Azure portal, find the node's VM, and detach the disk from the VM's 'Disks' blade. Wait 2 minutes, then retry drain.

What to expect: After detaching the disk manually, the drain completes. The node goes to 'SchedulingDisabled' and then 'Ready,SchedulingDisabled' after drain. You can then delete the node in AKS or reboot it.

Quick Reference Summary

If you're still stuck after these steps, check your AKS version. Older versions (before 1.24) had known bugs with the CSI driver. Upgrade to at least 1.27. Also make sure the node is actually healthy — run kubectl get nodes to confirm it's not in NotReady state. A broken node won't drain either.