Fix ERROR_IPSEC_IKE_INVALID_SIG (0x3633) on Windows

ERROR_IPSEC_IKE_INVALID_SIG means the IPsec peer's certificate signature didn't verify. The cert is expired, the chain is broken, or the CA isn't trusted on one side.

You're standing up a site-to-site IPsec tunnel between two Windows Server 2019 boxes, or maybe a Windows 11 client VPN using IKEv2 with machine certificates. The tunnel refuses to come up. Event Viewer on the responder shows Event ID 4653 or 5451, and the failure code is 0x3633 — ERROR_IPSEC_IKE_INVALID_SIG. That exact code also shows up when you run Get-NetIPsecMainModeSA and see the SA stuck in a failed state. The signature of the certificate the peer sent didn't verify against the public key in that same certificate, or against the chain you're being asked to trust.

What's actually happening here

During IKE main mode (or IKEv2's IKE_SA_INIT / IKE_AUTH exchange), each side signs a hash of the negotiation using the private key tied to its certificate. The receiver then validates that signature with the public key from the certificate, and walks the chain up to a trusted root. If any of these checks fails — the signature itself is malformed, the certificate is expired, the issuing CA isn't in the local trust store, or the certificate's EKU doesn't include IPsec — Windows reports 0x3633.

The misleading part: 0x3633 sounds like a cryptographic corruption issue. Usually it's not. It's almost always a chain problem. The peer's cert is fine; your machine just doesn't trust who issued it, or the cert's validity window has closed, or you're using a self-signed cert that wasn't pushed to the other side's Root store. I've seen this exact code triggered by a CA that was renewed at the root level — the old intermediate was still being sent by the peer, but the new root was installed locally, so the chain broke silently.

The fix, in order

Confirm which cert is actually being sent. Don't trust the GUI. On the peer (the one whose cert isn't validating), run:
```
certutil -store My
certutil -store My <thumbprint>
```
Check the output for the EKU. You need IP security IKE intermediate (OID 1.3.6.1.5.5.7.3.17) or at minimum Server Authentication. A cert issued with only Client Authentication will sign fine but the receiver will reject it. That's a common cause of 0x3633.
Verify the full chain on the receiving machine. Copy the peer's public cert (.cer) over and run:
```
certutil -verify -urlfetch peer.cer
```
The -urlfetch flag forces CRL and AIA fetching, which is where chain failures usually hide. If you see "Revocation Status Unknown" or "A certificate chain could not be built to a trusted root," that's your answer. Install the missing root or intermediate in the Local Computer store under Trusted Root Certification Authorities or Intermediate Certification Authorities.
Check expiration on both ends. This sounds obvious, but the Windows cert UI lies about "expires soon" on machine certs. Run:
```
Get-ChildItem Cert:\LocalMachine\My | Where-Object {$_.HasPrivateKey} | Format-List Subject, NotBefore, NotAfter, Thumbprint
```
If NotAfter is in the past, autoenrollment failed somewhere. Reissue from your CA and rebind the IPsec policy to the new thumbprint.
Match the cert to the IPsec authentication list. In the Windows Firewall with Advanced Security console, go to Connection Security Rules, pick the rule, and inspect the Authentication tab. If it says "Computer certificate from these CAs" with a specific CA list, the peer's issuing CA must be in that list. If it says "Computer certificate" (any), Windows still enforces chain trust. The subtle trap: an intermediate CA that's valid on the wire is still rejected if it's not in the local machine store. Add it via certlm.msc, not certmgr.msc — user-store certs are ignored by IKE.
Restart the IKEEXT service and re-key. After any cert change, existing SAs keep the old cert cached. Run:
```
Restart-Service IKEEXT -Force
Get-NetIPsecMainModeSA | Remove-NetIPsecMainModeSA
```
Then re-initiate the connection. If you skip this, you'll keep seeing 0x3633 in the log from stale SAs even after the cert issue is fixed.

If it still fails

Turn on IKE tracing. The default event log truncates the useful part. Run:

netsh trace start scenario=NetConnection capture=yes tracefile=C:\ike.etl
:: reproduce the failure, then:
netsh trace stop

Open the ETL in Message Analyzer or convert with tracerpt C:\ike.etl -o C:\ike.xml -of XML. Look for the IKE_AUTH packet where the signature validation fails. The trace will tell you whether the signature bytes themselves failed (rare — usually means a mismatched private key on the sender) or whether the chain build failed (common — means a missing root or intermediate).

Two other things worth checking: NTP. If the clock on either side is off by more than the certificate's validity skew (usually a few minutes), Windows treats a valid cert as not-yet-valid or expired, and you'll get 0x3633 with no other clues. And check whether the peer is behind a load balancer that swaps certs mid-session — some F5 and Citrix NetScaler configs will do this, and it produces 0x3633 intermittently under load. If your failures only happen at high connection counts, that's your smoking gun.

If you're using a third-party CA (DigiCert, GlobalSign, Sectigo) for machine certs on a private IPsec mesh — stop. Public CAs don't issue certs with the IPsec EKU to non-publicly-routable names, and even the ones that do will fail chain checks because your machines can't reach the OCSP responder. The real fix is an internal two-tier PKI. Anything else is duct tape.