AKS Node Drain Stuck: Quick Fix and Why It Works

Node Drain Hanging? Here's the Fix

Yeah, I know. You ran kubectl drain on an AKS node and it's sitting there for 10 minutes. Nothing moves. You're thinking about rebooting it. Don't. The culprit here is almost always a PodDisruptionBudget (PDB) or a pod that just won't die.

Step 1: Check if a PDB is Blocking You

Run this first:

kubectl get pdb --all-namespaces

Look for any PDB with minAvailable or maxUnavailable set to a number that can't be met. For example, if you have 2 replicas of an app and minAvailable: 2, you can't evict a single pod. So drain blocks.

Step 2: The Fast Fix

Don't bother editing the PDB — that's slow. Instead, drain the node with --ignore-daemonsets and --delete-emptydir-data flags. But the real trick is --force:

kubectl drain  --ignore-daemonsets --delete-emptydir-data --force

That forces eviction even if PDB says no. It works 90% of the time on AKS.

Step 3: If Still Stuck — Kill the Stubborn Pods

Sometimes a pod just won't leave. Check with:

kubectl get pods --field-selector spec.nodeName= --all-namespaces

Then delete the pod that won't evict:

kubectl delete pod  --namespace  --grace-period=0 --force

Then retry the drain command. It'll fly through.

Why Did That Work?

When you drain a node, Kubernetes checks each pod's PDB before evicting. If the PDB says "I need 2 pods running always" and you only have 2, Kubernetes says "nope" and waits. The --force flag skips that check. It's brutal but necessary when you're in a maintenance window and the clock is ticking.

The --grace-period=0 --force delete is the nuclear option. It kills the pod immediately, no waiting for the container to shut down cleanly. That's fine for stateless apps. For stateful ones, you want to be careful — but in a drain scenario, you're already past that point.

Less Common Variations

Variation 1: Node Has Taints That Pods Don't Tolerate

If the node has taints that prevent pods from scheduling elsewhere, drain can hang. Check taints with:

kubectl describe node  | grep Taints

Fix by adding tolerations to your pods or removing the taint temporarily. Remove with:

kubectl taint nodes  :-

Variation 2: DaemonSet Pods Won't Evict

DaemonSet pods are ignored by default with --ignore-daemonsets. If you need those gone too (rare), you can manually cordon the node and then delete them:

kubectl cordon 
kubectl delete pod -n kube-system --field-selector spec.nodeName= --all

Variation 3: AKS Agent Pool Upgrade Fails

Sometimes the issue isn't your manual drain but an AKS upgrade that hangs. Same root cause: PDBs. The fix is the same — find and remove the blocking pods. But you can also try pausing the upgrade and retrying after deleting pods:

az aks upgrade --resource-group myRG --name myAKS --kubernetes-version 1.27.3 --no-wait

Then check the operation status. If stuck, cancel it and fix pods first.

Prevention: Stop This From Happening Again

Here's what I do after fixing this three times in one month:

• Audit your PDBs. Make sure minAvailable is set to a number that leaves room for eviction. For 3 replicas, set it to 2. For 2 replicas, set it to 1. Never set it equal to replica count unless you're okay with manual intervention.
• Use pod disruption budgets properly. They're for protecting apps, not blocking maintenance. Test your drain on a test cluster first.
• Set up a maintenance window. If you're doing node pool upgrades, schedule them during low traffic and monitor the drain logs. AKS has built-in upgrade notifications — use them.
• Monitor with kubectl events. Before a drain, check if pods are stuck. Run kubectl get events --watch to see what's blocking.

One last thing: never run kubectl drain --force in production without understanding which pods will get killed. For critical stateful apps like databases, drain with --force only as a last resort. In those cases, scale down the app first, then drain.