support Junit5 parallel execution #1472 by fladdimir · Pull Request #1519 · microsoft/vscode-java-test

fladdimir · 2023-01-11T22:03:23Z

resolves #1472

src/runners/junitRunner/JUnitRunnerResultAnalyzer.ts

fladdimir · 2023-01-11T22:23:13Z

(eclipse-jdt/eclipse.jdt.ui#378 will need to be resolved before this feature will work correctly)

jdneo · 2023-01-12T01:20:56Z

@fladdimir Thank you for taking look into this and deal with the upstream issue.

Will it be fine if I review this PR after the upstream changes are merged?

fladdimir · 2023-01-12T01:43:54Z

Will it be fine if I review this PR after the upstream changes are merged?

@jdneo of course thats fine, thanks for the quick reply

fladdimir · 2023-01-12T01:50:02Z

Manual tests (including the locally built upstream fix for the RemoteTestRunner):

before - test2 appears not to be run at all, swallowing its failure:
before.webm

fixed - both test failures are shown, comparison failures and traces are placed correctly:
fixed.webm

fladdimir · 2023-01-12T22:11:22Z

@jdneo the upstream PR has been merged.

Unfortunately I am not familiar with the workflow. I assume the upstream artifact (org.eclipse.jdt.junit.runtime_3.7.x.jar) needs to be published somewhere to become part of the plugins of a eclipse.jdt.ls snapshot build, which could then be included into an upcoming vscode-java extension (preview) release?
(In case that is the way it works, do you have an idea of what could be a timeline for that?)

jdneo · 2023-01-13T02:51:57Z

Cool! I saw the upstream PR has been merged, great job!

The workflow looks like in this way:

The upstream JDT will first release an I-build which includes your changes. I noticed the latest I-build should have already included your changes.
Wait for the vscode-java update the dependency to include the org.eclipse.jdt.junit.runtime which contains your change.
Review your PR, merge it.
After your PR is merged, it will be available very soon in the extension pre-release channel.

I can help update the vscode-java side next week

test/test-projects/junit/src/test/java/junit5/ParallelExecutionTest.java

fladdimir · 2023-01-14T17:25:45Z

The upstream JDT will first release an I-build which includes your changes. I noticed the latest I-build should have already included your changes.

Wait for the vscode-java update the dependency to include the org.eclipse.jdt.junit.runtime which contains your change.
I can help update the vscode-java side next week

That sounds great, thanks a lot!

Kropie · 2023-01-15T23:56:40Z

Awesome work @fladdimir! When I was looking into this I was noticing the issues regarding the interleaving of messages from the eclipse provided unit test runner - great job finding the cause of the issue and working to resolve it!

jdneo · 2023-01-16T05:27:22Z

I raised a PR to update the dependency in VS Code Java: eclipse-jdtls/eclipse.jdt.ls#2403.

since it's about to release a new version this week, that PR should be able to be merged after the release (late this week)

jdneo

I looked the PR today and overall it looks good! 👍

After the latest pre-release of vscode-java is available, I'll play with it manually and merge this PR if everything works fine.

jdneo · 2023-01-18T02:24:20Z

src/runners/junitRunner/JUnitRunnerResultAnalyzer.ts

+    private currentStates: Map<TestItem, CurrentItemState> = new Map();
+
+    // failure info for a test is received consecutively:
    private currentItem: TestItem | undefined;


nit: how about give it a more specific name, something like: tracingItem?

just changed the name 👍🏼

jdneo · 2023-01-20T05:03:03Z

I found that there will be two assertion failure message when running the added sample tests

Can we deduplicate then to single one?

fladdimir · 2023-01-20T19:40:49Z

I found that there will be two assertion failure message when running the added sample tests
Can we deduplicate then to single one?

While debugging I realized that this duplication of diff-messages also occurs during normal test execution on the current main branch, without being related to parallel test execution.

Apparently its a problem of the general stack trace processing, occuring when a failing equality assertion stack trace contains multiple entries from the test class(?).

(Non-concurrently executed) sample based on the current main branch:

#1521

There is also a unit-test (part of JUnitAnalyzer.test.ts) to further debug this issue, which reveals that the following is called with a new diff-message each time a matching stack-trace entry is processed:

vscode-java-test/src/runners/baseRunner/RunnerResultAnalyzer.ts

Line 60 in a3848d4

    
           setTestState(this.testContext.testRun, currentItem, TestResultState.Failed, assertionFailure);

I'd like to continue investigating, however, would it be ok to open a separate issue for this?

jdneo · 2023-01-24T03:25:09Z

Sorry for the late reply. I'm on vacation these days :)

Yes, I can confirm that this bug can also be observed in the marketplace version. And it's totally fine to deal with it in a separate issue.

Thank you for the contribution!

fladdimir · 2023-01-24T23:27:47Z

Sorry for the late reply. I'm on vacation these days :)

Yes, I can confirm that this bug can also be observed in the marketplace version. And it's totally fine to deal with it in a separate issue.

Thank you for the contribution!

no worries at all - thanks for the review and merge!
enjoy your holidays!

fladdimir commented Jan 11, 2023

View reviewed changes

src/runners/junitRunner/JUnitRunnerResultAnalyzer.ts Show resolved Hide resolved

jdneo self-requested a review January 12, 2023 01:46

jdneo reviewed Jan 13, 2023

View reviewed changes

test/test-projects/junit/src/test/java/junit5/ParallelExecutionTest.java Outdated Show resolved Hide resolved

fladdimir force-pushed the feat/junit5-parallel-tests branch from 48a5910 to ff57ce2 Compare January 14, 2023 17:35

jdneo reviewed Jan 18, 2023

View reviewed changes

support Junit5 parallel execution microsoft#1472

2b6530c

fladdimir force-pushed the feat/junit5-parallel-tests branch from ff57ce2 to 2b6530c Compare January 19, 2023 23:11

Merge branch 'main' into feat/junit5-parallel-tests

f8d614b

fladdimir mentioned this pull request Jan 23, 2023

deduplicate diff test messages #1521

Merged

jdneo approved these changes Jan 24, 2023

View reviewed changes

jdneo merged commit c868001 into microsoft:main Jan 24, 2023

jdneo added this to the 0.38.0 milestone Jan 24, 2023

jdneo added the enhancement label Jan 24, 2023

Conversation

fladdimir commented Jan 11, 2023

Uh oh!

Uh oh!

fladdimir commented Jan 11, 2023

Uh oh!

jdneo commented Jan 12, 2023

Uh oh!

fladdimir commented Jan 12, 2023

Uh oh!

fladdimir commented Jan 12, 2023

Uh oh!

fladdimir commented Jan 12, 2023

Uh oh!

jdneo commented Jan 13, 2023

Uh oh!

Uh oh!

fladdimir commented Jan 14, 2023

Uh oh!

Kropie commented Jan 15, 2023

Uh oh!

jdneo commented Jan 16, 2023

Uh oh!

jdneo left a comment

Choose a reason for hiding this comment

Uh oh!

jdneo Jan 18, 2023

Choose a reason for hiding this comment

Uh oh!

fladdimir Jan 19, 2023

Choose a reason for hiding this comment

Uh oh!

jdneo commented Jan 20, 2023

Uh oh!

fladdimir commented Jan 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jdneo commented Jan 24, 2023

Uh oh!

fladdimir commented Jan 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fladdimir commented Jan 20, 2023 •

edited

Loading