Fix manager/state/store.timedMutex

[Adirio]

- What I did
Fix for #2602

- How I did it
Move the conflicting operation inside the critical section (before the Unlock() call).

- How to test it

- Description for the changelog
Fix #2602: timedMutex critical operation outside of critical section

[GordonTheTurtle]

Please sign your commits following these rules:
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#sign-your-work
The easiest way to do this is to amend the last commit:

$ git clone -b "patch-1" git@github.com:Adirio/swarmkit.git somewhere
$ cd somewhere
$ git commit --amend -s --no-edit
$ git push -f

Amending updates the existing PR. You DO NOT need to open a new one.

[codecov]

Codecov Report

Merging #2603 into master will increase coverage by 0.58%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2603      +/-   ##
==========================================
+ Coverage   61.58%   62.16%   +0.58%     
==========================================
  Files         134      134              
  Lines       21763    21763              
==========================================
+ Hits        13403    13530     +127     
+ Misses       6918     6778     -140     
- Partials     1442     1455      +13

[Adirio]

ci/circleci reporting:

time="2018-04-11T12:33:11Z" level=error msg="task allocation failure" error="failed to retrieve network testID3 while allocating task testTaskID3"
time="2018-04-11T12:33:11Z" level=error msg="Failed allocation for network testID5" error="failed while allocating driver state for network testID5: could not obtain vxlan id for pool 10.0.4.0/24: requested bit is already allocated"
time="2018-04-11T12:33:11Z" level=error msg="task allocation failure" error="network testID5 attached to task testTaskID6 not allocated yet"
time="2018-04-11T12:33:12Z" level=error msg="Failed allocation for service testServiceID4" error="requested bit is already allocated"
time="2018-04-11T12:33:12Z" level=error msg="task allocation failure" error="service testServiceID4 to which this task testTaskID7 belongs has pending allocations"
--- FAIL: TestNodeAllocator (0.03s)
	allocator_test.go:1427: timed out before watchNode found expected node state
FAIL
coverage: 71.8% of statements
FAIL	github.com/docker/swarmkit/manager/allocator	4.328s
make: *** [coverage] Error 1

Is this change really making the difference?

[cyli]

@Adirio Thanks for finding and submitting this fix. The TestNodeAllocator is a flakey test, and the failure is probably unrelated. I've kicked off a rebuild.

LGTM pending CI - this seems like if there is a context switch at an inopportune time, it could potentially cause swarm to occasionally fail to detect a wedge, since LockedAt().IsZero() would return true and hence Wedged() would return false. I can't think of a good way to unit test this change, though.

[Adirio]

Thank you two for the revission. I changed the PR description to incorporate the just merged template.

[cyli]

Thanks @aaronlehmann and @Adirio!