ESX 3.5 - 2 node cluster managed by VCenter 2.5
Working fine up until last week when I saw the error appear on the second node "HA agent on xxx in cluster yyy in zzz has an error".
I clicked "Reconfigure for VMware HA" and that worked for about 30 seconds then errored again.
The detailed events for that host in VCenter say sufficient resources when enabling then change to Insufficient resources to satisfy HA failover level on cluster.
We haven't added any new machines on either host nor has the configuration changed. Resource Distribution is 0-10% for CPU and 20-30% RAM on one host and 30-40% RAM on second host.
I checked the vmware_hostname.log file on the problematic host and the only thing that seems wrong is
Error FT Mon Nov 5 14:22:14 2012
By: FullTime/Process Monitor on Node: hostname
MESSAGE: Invalid Failure Detection IP Address 10.99.10.152, please fix.
followed by
Warning SEC Mon Nov 5 14:22:14 2012
By: FT/Agent on Node: msvottsanhost1
MESSAGE: Rejected Message. msgid 98 from (1/3:24716.0)
Then it continues with "Node is running" and both hosts are receiving heartbeats from each other.
We've tried disabling HA and re-enabling but that didn't work.
Under DNS and Routing, both hosts match domain, preferred/alternate DNS, search domains, default gateways for service console and VMKernel (all lowercase too).
Running out of things to check/try! Any direction would be appreciated.