SDN controller cluster split-brain fix: OVSDB sync failure

Quick answer

On each controller node, run ovs-appctl cluster/status ovsdb — if it shows "disconnected" or "timeout", restart ovsdb-server on the node that still has the most recent data, then restart the other nodes one by one.

What's happening here

You're running an SDN controller cluster — maybe OpenDaylight, ONOS, or something built on Open vSwitch. After a network hiccup or a node reboot, the OVSDB cluster loses sync. The symptom: some VMs can't reach each other, tunnels flap, and ovs-appctl cluster/status shows "disconnected" for all nodes. What's actually happening is that the Raft consensus among ovsdb-server instances breaks. OVSDB uses Raft to elect a leader and replicate the database. When UDP 6644 (the cluster port) gets blocked or the leader goes down before followers catch up, the cluster splits into fragments that each think they're the leader. This is a split-brain situation, but not the classic one — no data conflict, just sync paralysis.

I've seen this happen on Ubuntu 22.04 with OVS 3.0. The trigger: someone restarted ovsdb-server on node-2 while node-1 was doing a kernel upgrade. The 30-second timeout in the default Raft configuration couldn't handle the network blip.

Fix steps

1. Check cluster status on each node.
   Run ovs-appctl cluster/status ovsdb on every controller node. Look for the line that says "status: timeout" or "status: disconnected". Also note the "term" and "vote" numbers — if they differ wildly between nodes, you've got a split.
2. Identify the node with the most recent data.
   Check ls -l /var/log/openvswitch/ovsdb-server.log — the node with the newest log file usually has the latest committed entries. Or run ovsdb-client dump on each node — compare the "_version" column of the Open_vSwitch table. The one with the highest timestamp is your survivor.
3. Stop ovsdb-server on all nodes except the survivor.
   On the non-survivors: systemctl stop ovsdb-server. On the survivor, do nothing yet.
4. Restart ovsdb-server on the survivor.
   systemctl restart ovsdb-server on the survivor. Wait 10 seconds. Then check its status: ovs-appctl cluster/status ovsdb. It should show "status: leader" or "status: follower" — not "timeout".
5. Bring up the other nodes one by one.
   Start ovsdb-server on the next node: systemctl start ovsdb-server. Wait for it to sync — ovs-appctl cluster/status ovsdb should show "status: follower" within 30 seconds. Then move to the next node. Do not start two at once.
6. Verify the cluster is whole.
   Run ovs-appctl cluster/status ovsdb on all nodes. All should show the same "term" and "election timer". Also check ovsdb-client list-dbs — the Open_vSwitch database should be identical across nodes.

If the main fix doesn't work

Sometimes the database on the survivor is corrupted. Check the log: grep -i error /var/log/openvswitch/ovsdb-server.log. If you see "schema version mismatch" or "logically inconsistent", you'll need to rebuild from backup.

Alternative: Force a leader election by killing ovsdb-server on all nodes, then restarting them simultaneously. This only works if the Raft log is small. Run killall -9 ovsdb-server, then systemctl start ovsdb-server on all nodes at once (within 5 seconds). This is risky — you might lose uncommitted transactions.

Another option: Disable cluster mode temporarily. Edit /etc/default/openvswitch and set OVSDB_CLUSTER_MODE=standalone. Restart ovsdb-server. Then flip it back to cluster and restart. This forces a fresh cluster formation from the survivor's database.

Prevention tip

Set a higher election timeout. The default is 1000ms — too short for production networks with latency. Add this to /etc/openvswitch/ovsdb-server.conf:

election-timer=5000

That gives 5 seconds for leader election. It's a tradeoff — slower failover but fewer split-brain events. Also monitor UDP 6644 with a simple cron job:

*/5 * * * * /usr/bin/ovs-appctl cluster/status ovsdb | grep -q "status: leader" || logger "OVSDB cluster issue detected"

This won't fix everything, but it'll catch split-brain before your VMs start crying.