Flaky test: TestUpdaterRollbackAndPauseNoMonitor

[aaronlehmann]

--- FAIL: TestUpdaterRollbackAndPauseNoMonitor (1.05s)
        Error Trace:    testutils.go:26
                        update_test.go:210
                        update_test.go:23
        Error:          no task creation

Seen here: https://circleci.com/gh/docker/swarmkit/7016

[ijc]

I saw this a bunch of times when testing #2221/#2224. Twice on master (out of 20 iterations, labelled before in the logs below), once with the fix for #2224 and an updated grpc (out of 20 iterations, labelled fix in the logs below) and three times with #2222+#2223+#2224 and an updated grpc (out of 20 iterations, labelled fix+ in the logs below).

In every case the failure was preceded by at least one (sometimes two) messages of the general form:

time="2017-06-07T12:47:46+01:00" level=error msg="update failed" error="task df5dljt99xglmvul3gyqgu8bs was already shut down when reached by updater" task.id=m988bw1mxq8e1u34rvlrauro8 

• test-before.17.log
• test-before.01.log
• test-fix.19.log
• test-fix+.18.log
• test-fix+.16.log
• test-fix+.14.log

[aaronlehmann]

The test is occasionally exposing a race condition. A task failure separately triggers the update rollback and makes the orchestrator change the desired state of the task to "shutdown". Generally the desired state change happens first, but if the failure happens very soon after the task was started, it's possible for the rollback to begin first. In this case, the rollback process will try to shut down a task which has already been shut down, and to avoid the possibility of creating an extra replica, it won't start a replacement task for something it hasn't shut down itself.

#2242 has more information on the problem and possible solutions (this is case (2) in that issue).

[aaronlehmann]

(Note that this test sets the restart condition to "none", otherwise the orchestrator would restart the failed task, and we'd still have the expected number of replicas.)