Kubernetes etcd Cluster Unreachable – 3 Real Fixes

1. Network or Endpoint Misconfiguration (Most Common)

What's actually happening here is your kube-apiserver can't connect to any etcd instance. The error in the apiserver logs looks like etcdserver: request timed out or connection refused. This happens a lot after a reboot or when etcd runs as a static pod and the IP changes.

The quickest fix: check what endpoints the apiserver is trying to hit. On a kubeadm cluster, look at /etc/kubernetes/manifests/kube-apiserver.yaml for the --etcd-servers flag. It should list every etcd member's IP and port (default 2379). If you have 3 control plane nodes, you'll see something like:

--etcd-servers=https://192.168.1.10:2379,https://192.168.1.11:2379,https://192.168.1.12:2379

If those IPs are wrong or the etcd pods aren't running, fix the IPs first. To check if etcd is running on a node:

kubectl get pods -n kube-system | grep etcd
# should show something like etcd-node1

If the pod isn't there, etcd might be down. On each control plane node, run:

sudo crictl ps | grep etcd
# no output means it's not running

Fix 1a: Update the apiserver manifest
If the IPs changed (like after a network reconfiguration), edit the manifest file and restart kubelet:

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
# update --etcd-servers to current IPs
sudo systemctl restart kubelet

Fix 1b: Check firewall rules
etcd communicates on port 2379 (client) and 2380 (peer). If a firewall blocks 2379 between apiserver and etcd nodes, you get timeouts. On Ubuntu with ufw:

sudo ufw allow 2379/tcp
sudo ufw allow 2380/tcp

On RHEL with firewalld:

sudo firewall-cmd --add-port=2379/tcp --permanent
sudo firewall-cmd --add-port=2380/tcp --permanent
sudo firewall-cmd --reload

Real-world trigger: This bites you after a VM migration or when you change a node's hostname without updating the etcd member list. The IP changes, but the apiserver config still points to the old one.

2. TLS Certificate Mismatch (Second Most Common)

The reason this fix comes second: even if endpoints are right, a cert error breaks the handshake. The apiserver logs show something like tls: first record does not look like a TLS handshake or x509: certificate is valid for node1, not node2.

What's happening is the etcd server certificate doesn't include the IP or DNS name the apiserver is using to connect. Or the client cert the apiserver presents is wrong.

Fix 2a: Verify the etcd server cert
On each control plane node, check the etcd cert:

sudo openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text -noout | grep -E '(Subject|Issuer|DNS|IP)'

Look at the Subject Alternative Name section. It should list the correct IPs and DNS names of that node. If you see the wrong IP, you need to regenerate the certs (run kubeadm init phase certs etcd-server with the right --cert-dir).

Fix 2b: Check the client cert the apiserver uses
The apiserver connects to etcd using a client cert. Look at the --etcd-certfile and --etcd-keyfile in the apiserver manifest. Make sure they point to the right files:

--etcd-certfile=/etc/kubernetes/pki/etcd/server.crt
--etcd-keyfile=/etc/kubernetes/pki/etcd/server.key
# or sometimes:
--etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
--etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key

If these files are missing or corrupted, regenerate them with kubeadm init phase certs apiserver-etcd-client.

Fix 2c: Verify the CA matches
The apiserver and etcd must trust the same CA. Check the --etcd-cafile flag in the apiserver. It should point to /etc/kubernetes/pki/etcd/ca.crt. If this file doesn't match the CA used to sign the etcd certs, you get a cert error.

Real-world trigger: This happens when you copy certs between nodes manually but forget to update the server cert's SAN list for each node's specific IP.

3. Cluster Membership Drift (Rare but Nasty)

This one's tricky. What's actually happening is the etcd cluster itself is healthy, but a node's view of the cluster members is wrong. The logs show etcdserver: unhealthy cluster or rafthttp: failed to dial on port 2380.

Fix 3a: List the current members
If you can reach any etcd instance (even on one node), run:

sudo etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt --key=/etc/kubernetes/pki/etcd/healthcheck-client.key member list

You'll see a list of members with IDs, names, and peer URLs (port 2380). If a node's peer URL doesn't match its current IP, you need to update it.

Fix 3b: Update a member's peer URL
Suppose node2's IP changed from 192.168.1.11 to 192.168.1.21. Run:

sudo etcdctl member update  --peer-urls=https://192.168.1.21:2380

Replace <member-id> with the hex ID from the list command.

Fix 3c: Remove a stale member
If a node is gone forever (dead hardware), remove it:

sudo etcdctl member remove 

Then add the replacement node with etcdctl member add. But be careful: you must match the new node's peer URL exactly.

Real-world trigger: This happens when you replace a control plane node but don't clean up the old etcd member. The new node tries to join, but the old member still points to a dead IP.

Quick-Reference Summary

If you tried all three and it's still broken, check the etcd data directory (/var/lib/etcd) isn't full. A full disk kills etcd silently. Run df -h /var/lib/etcd to check.