Fix TRK_VOLUME_NOT_FOUND (0X0DEAD102) on TruCluster servers

Quick answer

Run cnxvol -d to delete the stale volume entry, then cnxvol -a to re-add it.

Why this happens

You're running a TruCluster server (Tru64 UNIX 5.1B or later) and you see the TRK_VOLUME_NOT_FOUND (0X0DEAD102) error in the logs or during a clu_volume operation. This error comes from the cluster's internal volume tracking system — the ServerVolumeTable. Every member of the cluster keeps a table of all volumes (disk devices) it knows about. When a volume gets removed from the cluster (maybe a disk died, or you removed it with clu_remove_member or clu_delete_service), the volume ID stays in the table on some members but not on others. The member that can't find the ID throws this error. It's not a hardware failure — it's a stale entry in the cluster's brain.

I've seen this mostly after a disk replacement. Someone swaps a failed SCSI disk, assigns it a new device name, and adds it back to the cluster without cleaning up the old volume entry first. The cluster sees the new disk as a different volume, but the old volume ID is still listed in the ServerVolumeTable on one or more members. When a cluster operation tries to reference that old ID, you get 0X0DEAD102.

Fix steps

Here's the exact procedure. Do this on any one cluster member — the change propagates to all members.

1. Identify the stale volume ID. Run cnxconfig -l to list all volumes in the cluster's volume table. Look for an entry with a VolumeID that doesn't match any active disk. Note that ID number.
2. Check which member has the stale entry. Run cnxvol -s on each member. Compare the output. The member that reports the error likely still has the old volume ID. Write down the VolumeID from that listing.
3. Delete the stale volume. On the member where the error occurs, run cnxvol -d . Replace with the number you noted. You'll see a confirmation message like Volume deleted from ServerVolumeTable.
4. Re-add the new volume. If you replaced the disk, add the new volume with cnxvol -a . For example, cnxvol -a /dev/disk/dsk20c. The system will assign a new VolumeID automatically.
5. Verify the fix. Run cnxconfig -l again. The stale VolumeID should be gone. If you added a new volume, you'll see it listed with a new ID. Then try whatever operation triggered the original error — it should succeed.

After each cnxvol command, wait a few seconds for the change to propagate across all cluster members. You can check propagation by running cnxvol -s on another member and seeing the updated table.

Alternative fix: If cnxvol fails

Sometimes cnxvol -d won't work because the volume is still referenced by a running service. If you get an error like Volume still in use, you need to stop the service first. Run clu_delete_service to remove the service that's using that volume. Then repeat the cnxvol -d step. After that, you can re-create the service with clu_add_service and assign the new volume.

Another scenario: if the volume ID is missing from all members but the error persists, you might have a corrupted ServerVolumeTable. In that case, stop the cluster membership service on all members (clu_quorum_stop), then restart it (clu_quorum_start). This rebuilds the table from scratch. I've only needed to do this once in 15 years, but it works.

Prevention tips

To avoid seeing this error again, always remove a volume from the cluster before you physically replace a disk. Use cnxvol -d to delete the old volume entry. Then swap the disk, assign the same device name (if possible — on Tru64 you can use rm and mknod to recreate the device file with the same name), and then add it back with cnxvol -a. If you must use a different device name, do the delete before you remove the old disk.

Also, keep a log of volume IDs for your cluster. Print out cnxconfig -l output after every cluster change. That way when 0X0DEAD102 pops up, you can spot the stale entry in seconds.

One more thing: if you're running TruCluster 5.1B on Alpha hardware, the ServerVolumeTable is stored in /etc/cluster/clu_config. Back up that file before any disk changes. A simple cp /etc/cluster/clu_config /etc/cluster/clu_config.backup takes two seconds and has saved my neck more than once.