Fix GCP Metadata Service Unreachable Error

Quick answer: Make sure your VM's network has a firewall rule allowing traffic to 169.254.169.254 on TCP 80, and your VM's network tag matches that rule.

Why This Happens

The metadata server at 169.254.169.254 is a special IP that GCP VMs use to get config info, like SSH keys and project settings. If your VM can't reach it, stuff breaks. The culprit here is almost always a firewall rule that blocks outbound traffic to that IP. It's common when you set up custom firewall rules and forget to include the metadata server. I've seen this on fresh deployments where someone copied a network config from a locked-down environment.

Another common trigger: you migrated or cloned a VM, and the network tags didn't carry over. Or you tightened firewall rules and didn't add an exception for metadata. Don't bother checking DNS or routing—the metadata IP is a link-local address, not a regular internet IP.

Step-by-Step Fix

Check if the VM can reach metadata: SSH into the VM (or use the serial console if SSH is down) and run:
```
curl -f -s -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/
```
If it hangs or returns nothing, you've got a network block. If it returns something, you're fine.

Verify firewall rules: In GCP Console, go to VPC network > Firewall. Look for a rule that allows egress to 169.254.169.254/32 on tcp:80. If you don't see one, create it. Example gcloud command:

gcloud compute firewall-rules create allow-metadata --direction=EGRESS --priority=1000 --network=default --destination-ranges=169.254.169.254/32 --rules=tcp:80 --target-tags=metadata-accessible

Add the network tag to your VM: If the firewall rule uses a target tag, your VM needs that tag. Run:
```
gcloud compute instances add-tags VM_NAME --tags=metadata-accessible --zone=ZONE
```
Replace VM_NAME and ZONE with your VM's name and zone.
Restart the VM: After applying tags, stop and start the VM (not reboot—stop/start forces a fresh network config):
```
gcloud compute instances stop VM_NAME --zone=ZONE
 gcloud compute instances start VM_NAME --zone=ZONE
```

If That Doesn't Work

Sometimes the default VPC network doesn't have the right routes. Run this:

gcloud compute routes list --filter="destination=169.254.169.254"

If you don't see a route for that IP, you need to recreate the default route. But honestly, I've only seen that in custom VPCs where someone deleted the default route. If you're on a default VPC, skip this.

Another possibility: your VM might be using a custom network interface with no access to the metadata server. If you have multiple NICs, check which one is the default route. Metadata only works through the primary NIC.

Prevention

Don't use overly restrictive firewall rules that block outbound traffic entirely. Always add an egress exception for 169.254.169.254/32 on TCP 80 when you lock down a network. Use network tags to limit the rule to VMs that need it. Also, when cloning VMs, always verify network tags and firewall rules on the new instance before deploying.

One last thing: If you're using Container-Optimized OS, the metadata service can also be blocked by container-level network policies. Check your Docker bridge config if the above doesn't work. That's a deeper issue, but I've seen it bite people.