Unstable unit tests that need to be disabled and fixed

[LeafShi1]

Disable failing tests and open tracking bugs

1. When test fails in the CI or PR build, open a bug for each test fail to investigate the test further and add a disabled-test label to the bug.
2. Disable test by adding a [skip] attribute, If only a specific theory data causes the failure, comment that data out. (In order to prevent some problems from being caused by changes in the PR itself or machine problems, if it fails only once in a month, we will not disable it. If it fails a second time, we will disable it.)
3. Add a link to the tracking bug to the SkipAttribute of the comment line.
4. Collect all data relevant to the failure into the bug.
5. Try to root-cause the flakiness, and fix these tests:
   • Replacing SendInput or cursor-positioning steps with direct method invocations, or by sending WM_COMMAND
   • If UI element is easily identifiable, for example by and accessible name, or we have access to the AccessibleObject, we can use UIA
     APIs to invoke methods
   • Add logging if possible.
   • If the test must use SendInput, then make sure it is moved to the UI Integration tests assembly

The following unit tests exist flaky issues:

• System.Windows.Forms.Tests.ImageCollectionTests.ImageCollection_Item_Get32bppColorDepth_Success(pixelFormat: Format1bppIndexed, pixel00Color: Color [Empty], givenPixel10Color: Color [Empty], expectedPixel10Color: Color [A=255, R=0, G=0, B=0])
• System.Windows.Forms.Tests.ToolTipTests.ToolTip_SetToolTip_TabControl_DoesNotAddToolForTabControlItself
• System.Windows.Forms.Tests.ToolTipTests.ToolTip_WmShow_Invokes_AnnounceText_WithExpectedText_ForTabControlTabs
• System.Windows.Forms.Tests.DataGridViewTests.DataGridView_OnColumnHeadersHeightChanged_InvokeWithHandle_CallsColumnHeadersHeightChanged(columnHeadersWidthSizeMode: EnableResizing, columnHeadersVisible: True, eventArgs: null)
• System.Windows.Forms.UITests.ButtonTests.Button_CancelButton_EscapeClicksCancelButtonAsync
• System.Windows.Forms.UITests.ButtonTests.Button_DialogResult_SpaceToClickFocusedButtonAsync
• System.Windows.Forms.UITests.ButtonTests.Button_Hotkey_Fires_OnClickAsync
• System.Windows.Forms.UITests.ButtonTests.Button_Press_Enter_Fires_OnClickAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Group_NavigateKeyboard_SucceedsAsync
• System.Windows.Forms.UITests.NumericUpDownTests.NumericUpDownAccessibleObject_Focused_ReturnsCorrectValueAsync
• System.Windows.Forms.Tests.CheckBoxRendererTests.CheckBoxRenderer_DrawCheckBox_OverloadWithSizeAndText
• System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_ClearUndo_CanUndo_Success
• System.Windows.Forms.Tests.ToolStripTests.ToolStrip_WndProc_InvokeMouseActivate_Success
• System.Windows.Forms.Tests.ToolStripTests.ToolStrip_WndProc_InvokeMouseActivateWithHandle_Success
• System.Windows.Forms.Tests.DataGridViewTests.DataGridView_ColumnHeadersHeight_SetWithHandle_GetReturnsExpected(columnHeadersWidthSizeMode: AutoSize, columnHeadersVisible: True, autoSize: True, value: 4, expectedValue: 18, expectedInvalidatedCallCount: 0)
• System.Windows.Forms.Tests.DataGridViewTests.DataGridView_ColumnHeadersHeight_SetWithParentWithHandle_GetReturnsExpected(columnHeadersWidthSizeMode: AutoSize, columnHeadersVisible: True, autoSize: True, value: 4, expectedValue: 18, expectedInvalidatedCallC
• System.Windows.Forms.Tests.MenuStripTests.MenuStrip_WndProc_InvokeMouseActivate_Success
• System.Windows.Forms.Tests.MenuStripTests.MenuStrip_WndProc_InvokeMouseActivateWithHandle_Success
• System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_Copy_PasteNotEmptyWithHandle_Success
• System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_Paste_InvokeNotEmpty_Success
• System.Windows.Forms.Tests.ListViewItem_IKeyboardToolTipTests.ListViewItemKeyboardToolTip_InvokeIsHoveredWithMouse_ReturnsExpected(insideListView: True, virtualMode: True, isHovered: True, expected: True)
• System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_Undo_CanUndo_Success
• System.Windows.Forms.Tests.TextBoxBaseTests.TextBoxBase_Copy_PasteNotEmpty_Success
• System.Windows.Forms.Tests.BindingContextTests.BindingContext_Item_GetIListSourceDataSourceWithDataMemberReturningIListNull_ThrowsArgumentNullException
• System.Windows.Forms.Tests.ClipboardTests.Clipboard_SetText_InvokeString_GetReturnsExpected
• System.Windows.Forms.Tests.TreeNode_TreeNodeIKeyboardToolTipTests.TreeNodeIKeyboardToolTip_InvokeIsHoveredWithMouse_ReturnsExpected(insideTreeView: True, isHovered: True, expected: True)
• System.Windows.Forms.Tests.TextBoxBaseTests+ClipboardTests.TextBoxBase_Copy_PasteNotEmptyWithHandle_Success
• System.Windows.Forms.Tests.TreeViewAccessibilityObjectTests.TreeViewAccessibilityObject_Role_IsOutline_ByDefault
• System.Windows.Forms.Tests.TextBoxBaseTests+ClipboardTests.TextBoxBase_Undo_CanUndo_Success
• Microsoft.VisualBasic.Devices.Tests.NetworkTests.PingUri_ShortTimeout_Success
• System.Windows.Forms.Tests.AccessibleObjects.TextBoxAccessibleObjectTests.TextBoxAccessibilityObject_Role_IsText_ByDefault
• System.Windows.Forms.Tests.CheckBoxRendererTests.CheckBoxRenderer_DrawCheckBox_VisualStyleOn_OverloadWithTextFormat
• System.Windows.Forms.UITests.MonthCalendarTests.MonthCalendar_Click_Date_InvokeEventsAsync(delta: -1)
• Microsoft.VisualBasic.Forms.Tests.WindowsFormsApplicationBaseTests.Run_SingleInstanceNoStartupFormException
• System.Windows.Forms.Tests.ClipboardTests.Clipboard_SetText_InvokeStringTextDataFormat_GetReturnsExpected(format: Text)
• System.Private.Windows.Core.BinaryFormat.Tests.WinFormsBinaryFormattedObjectTests.BinaryFormattedObject_ImageListStreamer_FromBinaryFormatter
• System.Windows.Forms.Tests.RadioButtonRendererTests.RadioButtonRenderer_DrawRadioButton_OverloadWithHandle
• System.Windows.Forms.Tests.ClipboardTests.Clipboard_RoundTrip_ManagedDataObject_SupportsTypedInterface
• System.Windows.Forms.Tests.ClipboardTests.Clipboard_SetDataObject_InvokeObjectIComDataObject_GetReturnsExpected(data: 1)
• System.Windows.Forms.Tests.ClipboardTests.Clipboard_SetDataObject_WithJson_ReturnsExpected
• System.Windows.Forms.Tests.ClipboardTests.RoundTrip_DataObject_SupportsTypedInterface
• System.Windows.Forms.Tests.ClipboardTests.SetDataObject_InvokeObjectBoolIComDataObject_GetReturnsExpected
• System.Windows.Forms.Tests.ClipboardTests.SetDataObject_InvokeObjectBoolNotIComDataObject_GetReturnsExpected
• System.Windows.Forms.Design.Tests.DesignerAttributeTests.EditorAttribute_TypeExists
• System.Windows.Forms.Tests.ClipboardTests.SetDataObject_WithJson_ReturnsExpected
• System.Windows.Forms.UITests.ListViewTests.ListView_Click_On_Second_Column_Does_Not_Alter_CheckBoxesAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_FragmentNavigate_ReturnsNullAsync(columnCount: 1, subItemsCount: 0, width: 150, height: 150, direction: 3)
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_FragmentNavigate_WorksExpectedAsync(columnCount: 2, subItemsCount: 2, width: 150, height: 150, itemIndex: 0, subItemIndex: 1, direction: 3)
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChild_ReturnsExpected_For_BigSizeAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChild_ReturnsExpectedAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChild_ReturnsNullAsync(columnCount: 4, subItemsCount: 0, width: 150, height: 150)
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildAsync(columnCount: 4, subItemsCount: 1)
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildCount_ReturnsExpectedAsync(columnCount: 4, subItemsCount: 3, width: 150, height: 150, expected: 3)
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildIndex_ForDifferentSize_ReturnsExpectedAsync(columnCount: 4, subItemsCount: 1, width: 150, height: 150, expected1: -1, expected2: 1)
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildIndex_ReturnsExpected_For_BigSizeAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildIndex_ReturnsExpected_SubItemsMoreThanColumnsAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildIndex_ReturnsExpectedAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_SubItem_FragmentNavigate_Child_ReturnsNullAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_SubItem_FragmentNavigate_NextSibling_ReturnsExpectedAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_SubItem_FragmentNavigate_Sibling_ReturnsNull_For_SmallSizeAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_SubItem_FragmentNavigate_Sibling_ReturnsNullAsync(columnCount: 4, subItemsCount: 1)
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_SubItem_HitTest_ReturnExpectedAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_ColumnProperty_ReturnsMinusOneAsync
• System.Windows.Forms.UITests.ListViewTests.ListView_Tile_GetChildIndex_ReturnsMinusOne_For_Single_ColumnAsync
• System.Windows.Forms.Tests.ClipboardTests.SetDataObject_WithMultipleData_ReturnsExpected
• System.Windows.Forms.Tests.HtmlDocumentTests.HtmlDocument_LinkColor_GetCustomValueSet_ReturnsExpected(value: null, expected: Color [A=255, R=0, G=0, B=255])
• System.Windows.Forms.Analyzers.Tests.AvoidPassingTaskWithoutCancellationTokenTests.CS_AvoidPassingTaskWithoutCancellationAnalyzer
• System.Windows.Forms.Tests.ClipboardTests.RoundTrip_TypedAndRuntimeDataObject_SupportsTypedInterface
• System.Windows.Forms.Tests.ClipboardTests.RoundTrip_UntypedDataObject_SupportsTypedInterface
• Microsoft.VisualBasic.Forms.Tests.WindowsFormsApplicationBaseTests.ShowHideSplashScreenSuccess

[paul1956]

Tests involving shared resources like Clipboard are also flaky when run in parallel.

[LeafShi1]

Tests involving shared resources like Clipboard are also flaky when run in parallel.

The current list is Flaky problems that have appeared in PR builds in the past two weeks. Regarding the Clipboard-related errors you mentioned, if we encounter them in subsequent pipelines, I will update them in the issue list.

[weltkante]

Tests involving shared resources like Clipboard are also flaky when run in parallel.

In the past we tried to ensure those aren't run in parallel by adding annotation attributes to the test classes preventing that.

[paul1956]

@weltkante
Do you have example of required annotations?

One issue with Clipboard is there are tests in different test classes using it is that possible or do they all need to be in same class?

[weltkante]

@weltkante Do you have example of required annotations?

One issue with Clipboard is there are tests in different test classes using it is that possible or do they all need to be in same class?

Couldn't quickly find it regarding clipboard, but WebBrowser tests had the same issue (WebBrowser control didn't like being run from multiple UI threads and caused memory corruption) - see #3429

You basically put a [Collection("...")] attribute on the class with a string token and tests from all classes using the same collection string are run sequentially (tests within a class a run sequentially anyways).

[paul1956]

@weltkante thanks I found this when looking at Collection and it seems to address the whole issue even with tests writing in C# AND Visual Basic

<Collection("Clipboard Tests")>
<CollectionDefinition("Clipboard Tests", DisableParallelization:=True)>