Currently, we do 3 retries when a benchmark fails. This is really useful for racy implementations. Though, it's just a time waster for most predictable sequential benchmarks. The number of retries be an option in the configuration file, and as a command line parameter.