Concurrent_Enqueue_TryPeek_TryDequeue_AllItemsSeen failing sporadically

[stephentoub]

Since dotnet/corefx#14254 was merged, the Concurrent_Enqueue_TryPeek_TryDequeue_AllItemsSeen test has failed a few times, but only on Unix. The test is simple and has been very stable, so there's likely a product issue. It looks like something is amiss with the fences, as we appear to be overwriting a value that shouldn't be.

https://ci.dot.net/job/dotnet_corefx/job/master/job/centos7.1_release/330/consoleText
https://ci.dot.net/job/dotnet_corefx/job/master/job/outerloop_ubuntu14.04_debug/40/consoleText

        System.AggregateException : One or more errors occurred. (Assert.Equal() Failure
        Expected: 1
        Actual:   33)
        ---- Assert.Equal() Failure
        Expected: 1
        Actual:   33
        Stack Trace:
              at System.ThrowHelper.ThrowAggregateException(List`1 exceptions)
              at System.Threading.Tasks.Task.WaitAll(Task[] tasks, Int32 millisecondsTimeout, CancellationToken cancellationToken)
           /mnt/resource/j/workspace/dotnet_corefx/master/centos7.1_release/src/System.Collections.Concurrent/tests/ConcurrentQueueTests.cs(91,0): at System.Collections.Concurrent.Tests.ConcurrentQueueTests.Concurrent_Enqueue_TryPeek_TryDequeue_AllItemsSeen()
           ----- Inner Stack Trace -----
           /mnt/resource/j/workspace/dotnet_corefx/master/centos7.1_release/src/System.Collections.Concurrent/tests/ConcurrentQueueTests.cs(76,0): at System.Collections.Concurrent.Tests.ConcurrentQueueTests.<>c__DisplayClass10_0.<Concurrent_Enqueue_TryPeek_TryDequeue_AllItemsSeen>b__0()
              at System.Threading.Tasks.Task.Execute()

[stephentoub]

The problem appears to be in ConcurrentQueue<T>.TryPeek. We incorrectly loop through the segments looking for the first that's not empty, under the idea that if there's more than one segment, all segments before the last are frozen, thus if we go through each segment in turn we can return from the first that's not empty. The problem is that we could check segment s, see that it's empty, but before we're able to check its next segment, so many enqueues could come in that they not only fill this segment, they spill into the next, causing the s._nextSegment to be non-null, at which point our loop will continue and we'll peek from s._nextSegment instead of from s.