Removing WaitToWrite in dist kvstore

[eric-haibin-lin]

As reported in #8097 WaitToWrite blocks push until all gradients are computed, which was introduced in #7082. After discussing it again with @mli we decided to remove it and updated the document accordingly.

@solin319

[piiswrong]

I thought this was already removed before we release 0.11?

[eric-haibin-lin]

That one was WaitToRead if I recall correctly.

[piiswrong]

why did you add this?

[rahul003]

Removing this from master is breaking the test dist_sync_kvstore.py
The simplest case of check_default_keys for small array of size 'shape' fails.
If I understand this right, accurate serialization of pushes might not matter for actual neural network training, so we removed this Wait. So we need to change the tests?

Traceback (most recent call last):
  File "dist_sync_kvstore.py", line 176, in <module>
    test_sync_push_pull()
  File "dist_sync_kvstore.py", line 169, in test_sync_push_pull
    check_default_keys(kv, my_rank, nworker)
  File "dist_sync_kvstore.py", line 67, in check_default_keys
    check_diff_to_scalar(val, num)
  File "dist_sync_kvstore.py", line 30, in check_diff_to_scalar
    assert(np.sum(np.abs((A - x).asnumpy())) == 0),(rank, A.asnumpy(), x)
AssertionError: (None, array([[ 21.,  21.,  21.],
       [ 21.,  21.,  21.]], dtype=float32), 37)

[eric-haibin-lin]

We should update the test and document for push