Fix AKS node pool upgrade stuck in Canceled state

Quick answer for pros

Run az aks nodepool upgrade --resource-group --cluster-name --name --kubernetes-version --force-upgrade --no-wait. Then check state with az aks nodepool show -g -c -n --query provisioningState. Expect it to go to 'Upgrading' then 'Succeeded' within 15 minutes.

Why does this happen?

You triggered a node pool upgrade in AKS (maybe through the portal or CLI), then canceled it mid-way. Or the upgrade timed out. AKS marks the provisioning state as 'Canceled' but doesn't fully clean up. The node pool gets stuck – it won't let you run a new upgrade because the system thinks one is still in progress. This bug is common in AKS versions 1.24 through 1.27. I've seen it happen when people accidentally close their terminal during an upgrade or when Azure's internal orchestrator hiccups.

The real problem: the ARM resource manager sees the state as 'Canceled' but the underlying fleet (the actual nodes) never got the cancel signal. So the upgrade operation stays open. You can't cancel it again. You can't delete the pool if it's your system pool. You're stuck.

Step-by-step fix

Install or update Azure CLI. Open a terminal. Run az --version. If below 2.55, run az upgrade. After upgrade, run az --version again – you should see version >= 2.55. The --force-upgrade flag needs this version.
Sign in to Azure. Run az login. A browser opens. Log in with your Azure admin account. After login, you'll see subscription info in the terminal.
Get your cluster details. Run az aks show -g myResourceGroup -n myCluster --query nodeResourceGroup. Note the node resource group name (it's usually MC_myResourceGroup_myCluster_region). Also run az aks nodepool list -g myResourceGroup --cluster-name myCluster --query [].name. Write down the node pool name that's stuck.
Force the upgrade. Run this command, replacing placeholders with your values:
```
az aks nodepool upgrade --resource-group myResourceGroup --cluster-name myCluster --name myNodePool --kubernetes-version 1.27.3 --force-upgrade --no-wait
```
Use the exact Kubernetes version you want (check available versions with az aks get-versions -l westus --query orchestrators[].orchestratorVersion). The --no-wait flag returns immediately. The --force-upgrade flag tells AKS to ignore the stuck state and push through.
Check the state. Run az aks nodepool show -g myResourceGroup --cluster-name myCluster -n myNodePool --query provisioningState. You should see "Upgrading" after a minute. Run it again every 30 seconds. After 5–15 minutes, it should show "Succeeded".
Verify nodes. Run kubectl get nodes (make sure you're connected to the cluster). You should see all nodes with the new Kubernetes version in the VERSION column. No nodes should be in NotReady state for more than 5 minutes.

Alternative fixes if the main one fails

Fix 1: Scale the node pool to 0 and back. This sometimes clears the stuck state. Run az aks nodepool scale -g myResourceGroup --cluster-name myCluster -n myNodePool --node-count 0 --no-wait. Wait until provisioning state shows "Succeeded" (it scales down). Then scale back: az aks nodepool scale -g myResourceGroup --cluster-name myCluster -n myNodePool --node-count 3 --no-wait. Then run the force upgrade again.

Fix 2: Use the Azure portal. Go to your AKS cluster. Click 'Node pools' in the left menu. Click the stuck pool. At the top, click 'Upgrade version'. Select the same version it's already on (or one higher). Check the box that says 'Force upgrade' (if it appears – this flag only shows on portal if the pool is stuck). Click 'Upgrade'. This does the same as the CLI command.

Fix 3: Delete the node pool (only if it's not the system pool). If the stuck pool is a user pool, you can delete it and recreate. Run az aks nodepool delete -g myResourceGroup --cluster-name myCluster -n myNodePool. Wait for deletion. Then create it again with az aks nodepool add -g myResourceGroup --cluster-name myCluster -n myNodePool --node-count 3 --kubernetes-version 1.27.3. This is the nuclear option – you lose any workloads running on those nodes, so drain them first if possible.

Fix 4: Open a support ticket. If none of the above works (and you've waited 30 minutes after each attempt), open a support request in the Azure portal. Tell them: 'Node pool stuck in Canceled provisioning state. Force upgrade flag did not work. Need backend state reset.' They can reset the provisioning state on the ARM side. This usually takes 4–24 hours.

Prevention tip

Don't cancel an AKS upgrade once it starts. If you need to stop it, wait for it to finish, then roll back to the previous version. To roll back, run az aks nodepool upgrade -g myResourceGroup --cluster-name myCluster -n myNodePool --kubernetes-version 1.26.6 (use the version you were on before). The upgrade will complete normally. Then you can plan a better time.

Also, always use --max-surge during upgrades. Set it to 1 or 2. This prevents resource contention. Example: az aks nodepool upgrade -g myResourceGroup --cluster-name myCluster -n myNodePool --kubernetes-version 1.27.3 --max-surge 1. This gives you a buffer node so the upgrade doesn't time out.

If you're using Terraform or ARM templates, set force_upgrade = true in the node pool resource. That way, if an upgrade ever gets stuck, a new deployment will force it through automatically. I've seen this save teams hours of manual fixing.