[JAVA][BUG] Flaky test detected - JavaClientCodegenTest#testMicroprofileRestClientIncorrectVersion

[Suraj-Vashista-BK]

Bug Report Checklist

• Have you provided a full/minimal spec to reproduce the issue?
• Have you validated the input using an OpenAPI validator (example)?
• Have you tested with the latest master to confirm the issue still exists?
• Have you searched for related issues/PRs?
• What's the actual output vs expected output?
• [Optional] Sponsorship to speed up the bug fix or feature request (example)

Description

The test case was showing non deterministic behavior which was recognized by using the NonDex tool.

openapi-generator version

v7.0.1 - Not a regression

OpenAPI declaration file content or url

Non-Dex detected flakiness in the message given when this test throws an exception. More precisely as shown below:

The exception was thrown with the wrong message: expected "Version incorrectVersion of MicroProfile Rest Client is not supported or incorrect. Supported versions are 1.4.1, 2.0, 3.0" but got "Version incorrectVersion of MicroProfile Rest Client is not supported or incorrect. Supported versions are 2.0, 3.0, 1.4.1"

The error message code causing the flakiness -

openapi-generator/modules/openapi-generator/src/test/java/org/openapitools/codegen/java/JavaClientCodegenTest.java

Lines 1704 to 1709 in 0aff1a6

	@Test(
	expectedExceptions = IllegalArgumentException.class,
	expectedExceptionsMessageRegExp =
	"Version incorrectVersion of MicroProfile Rest Client is not supported or incorrect."
	+ " Supported versions are 1.4.1, 2.0, 3.0")
	public void testMicroprofileRestClientIncorrectVersion() throws Exception {

Generation Details

Command to run the tool -

mvn  -pl modules/openapi-generator  edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.openapitools.codegen.DefaultCodegenTest#testVarsAndRequiredVarsPresent

Test Environment:

openjdk version "11.0.20.1"
Apache Maven 3.6.3
Ubuntu 20.04.6 LTS
Linux version 5.4.0-156-generic

The NonDex tool fails if the test case is flaky.

Steps to reproduce

Run the NonDex tool as shown above to reproduce this issue.

Details

The flakiness in the error message is happening because we have hard defined an expected message string in the test as shown:

openapi-generator/modules/openapi-generator/src/test/java/org/openapitools/codegen/java/JavaClientCodegenTest.java

Lines 1706 to 1708 in 0aff1a6

	expectedExceptionsMessageRegExp =
	"Version incorrectVersion of MicroProfile Rest Client is not supported or incorrect."
	+ " Supported versions are 1.4.1, 2.0, 3.0")

This exception string is built in the file JavaClientCodegen.java in org.openapitools.codegen.languages
and while we are building the exception string, we use a hashmap called mpRestClientVersions to do so as shown below:

openapi-generator/modules/openapi-generator/src/main/java/org/openapitools/codegen/languages/JavaClientCodegen.java

Lines 364 to 369 in 0aff1a6

	throw new IllegalArgumentException(
	String.format(Locale.ROOT,
	"Version %s of MicroProfile Rest Client is not supported or incorrect. Supported versions are %s",
	mpRestClientVersion,
	String.join(", ", mpRestClientVersions.keySet())
	)

The main reason for flakiness is mpRestClientVersions.keySet() because a map's keySet function returns a Set and the set's iterator method says in its documentation that the elements are returned in no particular order (unless this set is an instance of some class that provides a guarantee). This introduces non determinism in the output of the keySet resulting in flakiness in the generated string.

Suggest a fix

To address this issue and maintain the order , I have converted mpRestClientVersions from HashMap to linkedHashMap during its declaration:

 protected Map<String, MpRestClientVersion> mpRestClientVersions = new LinkedHashMap<>(); 

This ensures that we get the versions in the expected order. After this fix, nondex did not detect any flakiness.