Skip flaky ToolTaskThatTimeoutAndRetry test on CI

[jankratochvilcz]

Disables the ToolTaskThatTimeoutAndRetry test on CI using [SkipOnCI]. This test has been consistently flaky despite multiple stabilization attempts.

[Copilot]

Pull request overview

Disables the flaky ToolTaskThatTimeoutAndRetry unit test on CI by adding a CI-only skip attribute, aligning with the ongoing effort to reduce spurious CI failures (Fixes #13667).

Changes:

• Added a [SkipOnCI(...)] attribute to skip ToolTaskThatTimeoutAndRetry when running in CI.

[Copilot]

SkipOnCI is only referenced here and isn’t defined anywhere in this repo (no SkipOnCIAttribute implementation under src/). If this attribute isn’t provided by the referenced xUnit packages, this will fail to compile or won’t actually skip on CI. Please either (1) add a SkipOnCIAttribute to the shared test infrastructure (e.g., src/UnitTests.Shared) that checks common CI env vars, or (2) switch to an existing conditional-skip mechanism already used in this repo.

[github-actions]

Review Summary

✅ Attribute validity

[SkipOnCI] is provided by Microsoft.DotNet.XUnitV3Extensions (from dotnet/arcade), which is automatically referenced for all test projects. The attribute detects CI via AGENT_OS / DOTNET_CI / HELIX_WORKITEM_ROOT environment variables, which are set on Azure DevOps. Usage is correct.

✅ Attribute placement

[SkipOnCI] targets AttributeTargets.Method and works correctly alongside [Theory] + [InlineData] — all test cases will be skipped on CI.

✅ Pragmatic approach given history

This test has a well-documented history of flakiness with at least 3 prior stabilization attempts:

• PR NoProfile in ToolTaskThatTimeoutAndRetry test #8659 (Apr 2023): Added -NoProfile to PowerShell
• PR Enable ToolTaskThatTimeoutAndRetry test after change powershell.exe to cmd.exe #8800 (May 2023): Switched from PowerShell to cmd.exe
• PR Stabilize ToolTaskThatTimeoutAndRetry test #13489 (Apr 2026): Reduced fastDelayMilliseconds to flat 100ms

The test is fundamentally timing-dependent (it expects a 100ms sleep/ping to complete within a 2000ms timeout, while a 5000ms sleep should NOT). On loaded CI machines, even "fast" operations can exceed expected durations, making this inherently flaky.

📋 Suggestions (non-blocking)

1. Add a tracking issue reference in the skip message — this makes it clear the skip is intended to be temporary and provides a breadcrumb for future cleanup.
2. Consider whether increasing the timeout gap could help — e.g., using a 500ms fast delay with a 10s timeout gives much more headroom. Though given the history, [SkipOnCI] is reasonable if further effort isn't justified.

🔍 Test coverage impact

The test verifies that ToolTask correctly handles timeout + retry by re-executing the same task instance. With this change, this scenario is only validated locally. This is acceptable given:

• The underlying ToolTask retry logic is also exercised by other tests
• The test has been causing false-negative CI signal, reducing overall CI trust

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• #13668 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #8544 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Expert Code Review (on open) for issue #13668 · ● 6.7M · ◷

[github-actions]

LGTM with minor suggestion. The use of [SkipOnCI] is correct, the attribute is available from Microsoft.DotNet.XUnitV3Extensions, and placement on the [Theory] method is valid. Given the extensive history of stabilization attempts (3+ PRs), skipping this inherently timing-sensitive test on CI is a pragmatic choice.

Only suggestion: reference a tracking issue in the skip message to ensure the test doesn't stay permanently disabled without visibility.

Note

🔒 Integrity filter blocked 2 items

The following items were blocked because they don't meet the GitHub integrity level.

• #13668 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
• #8544 search_pull_requests: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".

To allow these resources, lower min-integrity in your GitHub frontmatter:

tools:
  github:
    min-integrity: approved  # merged | approved | unapproved | none

Generated by Expert Code Review (on open) for issue #13668 · ● 6.7M

[github-actions]

Suggestion (non-blocking): Consider adding a tracking issue URL to the skip message so it's clear this is intended to be temporary and there's a path to re-enablement. For example:

[SkipOnCI("https://github.com/dotnet/msbuild/issues/XXXX - consistently flaky due to timing sensitivity")]

This follows the pattern used elsewhere in the repo (e.g., [WindowsOnlyFact(Skip = "https://github.com/dotnet/msbuild/issues/1250")]).