Phi Accrual Failure Detector

Rather than using configured constant timeouts, systems can continually measure response times and their variability, and automatically adjust timeouts according to the observed response time distribution.

This can be done with a Phi Accrual failure detector, which is used by Cassandra and Akka.

http://ternarysearch.blogspot.ca/2013/08/phi-accrual-failure-detector.html