Step down if local_put fails in leader worker

[andrewjstone]

If we fail to write locally as a leader, we cannot get a valid quorum since those quorums include the leader. Therefore we need to step down to prevent committing unsafely.

In riak_ensemble_peer:put_obj/4 when local_put/4 fails, step down by
sending a request_failed message to the leader.

[andrewjstone]

Fixes #2

[jtuple]

/cc basho/riak#536

[lordnull]

Had to rebase to get ee to compile. ensemble develop introduced a type that does not exist in this branch.

riak_test run:

./riak_test -c ee -t ensemble_basic -t ensemble_basic2 -t ensemble_basic3 -t ensemble_basic4 -t ensemble_interleave -t ensemble_remove_node -t ensemble_remove_node2 -t ensemble_start_without_aae -t ensemble_sync -t ensemble_util -t ensemble_vnode_crash

First run had sync, remove_node2, and remove_node all fail. Running them individually had them pass, and 2nd run with all ensemble tests passed.

Code is nicely contained.

I'll run the ensemble suite a couple more times to see if the failure can be replicated.

[lordnull]

👍 02ee10c

At least one test will timeout when running the suite, but each test will pass individually, so I don't think the changes in this pr are responsible for that. Since that was the only concern, this is ready for merging.

[andrewjstone]

@borshop merge

[andrewjstone]

👍 8f6efad

[andrewjstone]

@borshop merge