SteeringTest fails when LetsEncryptDnsChallengeWatcher hangs

[zrhoffman]

I'm submitting a ...

• bug report

Traffic Control components affected ...

• Traffic Router

Current behavior:

When running the Traffic Router tests in ExternalTestSuite, SteeringTest fails with

Expected: a value greater than <986>
     but: <986> was equal to <986>
	at com.comcast.cdn.traffic_control.traffic_router.core.external.SteeringTest.z_itemsMigrateFromSmallerToLargerBucket(SteeringTest.java:356)

Expected behavior:

The tests in SteeringTest should pass

Minimal reproduction of the problem with instructions:

Run the external tests. For convenience, I have made a branch that runs SteeringTest only in CDN-in-a-Box
https://github.com/zrhoffman/trafficcontrol/commits/steering-test

You can use it to run the Traffic Router tests, including external tests, as follows:

docker-compose up -d
docker-compose -f docker-compose.readiness.yml up readiness # Waits for Traffic Monitor to be usable by Traffic Routers
docker-compose -f docker-compose.traffic-router-test.yml up tr-integration-test

Anything else:

SteeringTest has failed since 18fe13a/#3534 (and snuck into 4.1.0). The only 2 assertions that fail are these:

		assertThat(rehashedPaths.get(smallerTarget).size(), greaterThan(hashedPaths.get(smallerTarget).size()));
		assertThat(rehashedPaths.get(largerTarget).size(), lessThan(hashedPaths.get(largerTarget).size()));

I think this is because SteeringWatcher doesn't get to update after the POST request to the mock TO server to change the steering attributes.

The reason I don't think SteeringWatcher updates after that point is that LetsEncryptDnsChallengeWatcher's ScheduledExecutorService hangs when trying to update LetsEncryptDnsChallengeWatcher's database from https://${toHostname}/api/2.0/letsencrypt/dnsrecords/.

For whatever reason, Fetcher's request for LetsEncryptDnsChallengeWatcher never makes it to HttpDataServer (the mock TO server) even though the host and port seem to line up. But, if you make Fetcher return for LetsEncryptDnsChallengeWatcher requests right before it sends the request:

			if (url.contains("letsencrypt")) {
				return http;
			}

then SteeringTest succeeds. If you don't, the request doesn't seem to ever complete.

--

Other notes:

• Changing DEFAULT_LE_DNS_CHALLENGE_URL to "https://${toHostname}/api/2.0/letsencrypt/dnsrecords" so it doesn't end in a slash, then adding a mock response for it does not keep LetsEncryptDnsChallengeWatcher's Fetcher request from hanging.
• One AbstractResourceWatcher hanging should not make another AbstractResourceWatcher hang. If Traffic Router does this outside of tests, it's a serious bug that we should fix.

[zrhoffman]

Since I haven't PRed it yet: The zrhoffman/tr-int-test branch can be used to run the Traffic Router External Tests.

1. Start CDN-in-a-Box
2. Wait for Traffic Monitor to start, since it is used by some Integration Tests (like AbstractResourceWatcherTest)
3. Run the Traffic Router External Tests and Integration Tests:

   docker-compose -f docker-compose.traffic-router-test.yml up --build tr-integration-test

You can also debug the tr-integration-test service by setting the TR_TEST_DEBUG_ENABLE to true in variables.env.

Debugging settings to use in Intellij Idea:

• Listen to remote JVM instead of Attach to remote JVM
• Enable Auto restart, since otherwise the debugger would stop listening after each test
• Port 8000
• Make sure the IDE is listening for debug connections before starting the tests or they will all fail.

Edit: And if you're on Linux (no need if using Docker Desktop on Mac or Windows), you'll need to use your own resolver so you can set the IP address for host.docker.internal to 172.17.0.1. Alternatively, change host.docker.internal to 172.17.0.1 on the following lines:
https://github.com/apache/trafficcontrol/blob/56a80d381a4856fbfcf866c019aeefa864c292ab/infrastructure/cdn-in-a-box/traffic_router_integration_test/run.sh#L44-L45

[rawlinp]

Reopening per @zrhoffman's comment on #5342.

[zrhoffman]

#5069 no longer occurs for me.