Understanding a Traffic Manager’s Fault Tolerance Checks
The Traffic Managers in a cluster periodically check that they can communicate with the network using ICMP pings through each active network interface. They then broadcast a message describing their health as good or failed.
If the Traffic Manager hosting the Traffic IP address in a cluster fails, one of the other Traffic Managers takes over the address.
To configure fault tolerance checks, use the settings in System > Fault Tolerance.
Local Health Checks
Each Traffic Manager checks that its network interfaces are operating correctly. It does this by periodically pinging each of the back-end nodes in the pools that are in use, to ensure that its back-end network interfaces are functioning. The Traffic Manager concludes that it has failed if it cannot ping any of the nodes in any of the currently used pools.
If your pools contain nodes that are hosted outside the EC2 network, you might see warnings about node failures when Traffic IP addresses fail over. These warnings occur because an EC2 instance loses all external connectivity while its public IP address is changed, which causes back-end connectivity checks to fail. These warnings can safely be ignored and are cleared when the instance regains external connectivity. Backend nodes hosted inside the EC2 network are not affected by this issue.
Health Broadcasts
Each Traffic Manager machine regularly broadcasts the results of its local health checks, whether it is healthy or not.
By default, each machine broadcasts these heartbeat messages every 2 seconds. You can configure this behavior with the flipper!monitor_interval setting.
The tcpdump program is a useful debugging tool. You can capture heartbeat messages in your Traffic Manager clusters by running the following commands on each Traffic Manager:
# tcpdump -i eth0 udp and port 9090Determining the Health of a Traffic Manager Cluster
Each Traffic Manager listens for the health messages from all of the other Traffic Managers in the cluster. A Traffic Manager concludes that one of its peers has failed if:
- It receives an “I have failed” health message from the peer.
- It does not receive any messages from the peer within the value set in flipper!monitor_timeout.
- The peer reports that one of its child process is no longer servicing traffic. This occurs if any child process fails to respond within the value set in flipper!child_timeout.
The Traffic Manager concludes that a peer has recovered when it starts receiving “I am healthy” messages from the peer.