Skip to content

Conversation

@mkhamoyan
Copy link
Contributor

@mkhamoyan mkhamoyan commented May 17, 2024

OuterLoop tests that are disabled still are running when /p:TestScope=outerloop.
Xharness ignores -notrait category=failing and runs tests cases marked with SkipOnPlatform,ActiveIssue

Fixes #102157

@mkhamoyan mkhamoyan self-assigned this May 17, 2024
@mkhamoyan mkhamoyan added arch-wasm WebAssembly architecture and removed area-Infrastructure-libraries labels May 17, 2024
@mkhamoyan
Copy link
Contributor Author

/azp run runtime-libraries-mono outerloop

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@build-analysis build-analysis bot mentioned this pull request May 17, 2024
@mkhamoyan mkhamoyan marked this pull request as ready for review May 17, 2024 11:17
@mkhamoyan mkhamoyan requested review from lewing, maraf and pavelsavara May 17, 2024 11:17
@mkhamoyan
Copy link
Contributor Author

/azp run runtime-libraries-mono outerloop

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@pavelsavara
Copy link
Member

Could you please also fix all the PlatformNotSupportedException in the outer loop ?

@mkhamoyan
Copy link
Contributor Author

/azp run runtime-libraries-mono outerloop

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@mkhamoyan mkhamoyan marked this pull request as draft May 21, 2024 07:33
@mkhamoyan
Copy link
Contributor Author

/azp run runtime-libraries-mono outerloop

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@mkhamoyan
Copy link
Contributor Author

/azp run runtime-libraries-mono outerloop

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@mkhamoyan
Copy link
Contributor Author

mkhamoyan commented May 21, 2024

I find out that -trait and -notrait are respected when passing with outerloop attribute

[OuterLoop("Resource-intensive due to parallel processing", ~TestPlatforms.Browser)]
[SkipOnPlatform(TestPlatforms.Browser, "Not supported on browser")]

but ignored when passed like

[OuterLoop]
[SkipOnPlatform(TestPlatforms.Browser, "Not supported on browser")]

There are 2 ways to go

  1. One solution is to update these test cases to skip on wasm from outerloop attribute. (This is implemented in this PR).
  2. Fix it in dotnet/arcade repo, I opened a ticket for it Tests marked with OuterLoop attribute ignore -notrait category=failing arcade#14779

@lewing @pavelsavara what do you think ?

@mkhamoyan
Copy link
Contributor Author

/azp run runtime-libraries-mono outerloop

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@mkhamoyan
Copy link
Contributor Author

All test cases throwing PlatformNotSupportedException are now disabled.
System.Linq.Expressions.Tests, System.Linq.Parallel.Tests, System.Threading.Tasks.Parallel.Tests, System.Threading.Tasks.Tests are failing because of timeout

Tests timed out after 1800secs
XHarness exit code: 70 (TIMED_OUT)

@mkhamoyan mkhamoyan marked this pull request as ready for review May 22, 2024 09:00
@mkhamoyan mkhamoyan requested a review from pavelsavara May 23, 2024 07:12
@akoeplinger
Copy link
Member

The underlying problem is this xharness issue: dotnet/xharness#908

The PR looks good to me, though maybe we should also spend some time to actually fix xharness instead?

@mkhamoyan
Copy link
Contributor Author

mkhamoyan commented May 24, 2024

I'll merge this to reduce pipeline noise until we fix xharness.

@mkhamoyan mkhamoyan merged commit f4c9264 into dotnet:main May 24, 2024
Ruihan-Yin pushed a commit to Ruihan-Yin/runtime that referenced this pull request May 30, 2024
Update test cases to skip on wasm from outerloop attribute.
@github-actions github-actions bot locked and limited conversation to collaborators Jun 23, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

arch-wasm WebAssembly architecture

Projects

None yet

Development

Successfully merging this pull request may close these issues.

WebSocket failures on wasm test runs

3 participants