Traffic Monitor Integration Test Framework by rob05c · Pull Request #3707 · apache/trafficcontrol

rob05c · 2019-07-01T14:38:58Z

Adds a Traffic Monitor Integration Test Framework.

WIP, but making a draft for input. It needs some cleanup, but it works, and has a few tests. Instructions are in the README, let me know if you have trouble getting it to work.

Also adds a Go client for the Monitor (because it needed it).

Comments welcome.

What does this PR (Pull Request) do?

This PR fixes #REPLACE_ME OR is not related to any Issue

Which Traffic Control components are affected by this PR?

CDN in a Box
Documentation
Grove
Traffic Control Client
Traffic Monitor
Traffic Ops
Traffic Ops ORT
Traffic Portal
Traffic Router
Traffic Stats
Traffic Vault

What is the best way to verify this PR?

If this is a bug fix, what versions of Traffic Ops are affected?

The following criteria are ALL met by this PR

This PR includes tests OR I have explained why tests are unnecessary
This PR includes documentation OR I have explained why documentation is unnecessary
This PR includes an update to CHANGELOG.md OR such an update is not necessary
This PR includes any and all required license headers
This PR ensures that database migration sequence is correct OR this PR does not include a database migration
This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

asfgit · 2019-07-01T15:01:15Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/3918/
Test FAILed.

This is with the intention of adding an integration test framework for Traffic Monitor, which can use this tool and send commands to change the fake caches to test various scenarios.

rob05c · 2020-09-25T20:31:51Z

Rebased, removed draft/WIP. I haven't had time to do more with this, but it basically worked. I don't think there's a reason not to merge it, if anyone wants to use and extend it for Traffic Monitor Integration Testing.

zrhoffman · 2020-09-25T20:38:52Z

@@ -0,0 +1,22 @@
+# environment variables for the Traffic Monitor Docker Compose files


This file needs an Apache license header

zrhoffman · 2020-09-25T20:57:55Z

+```
+(cd tools/testto && go build)
+(cd tools/testcaches && go build)
+(cd tests/integration && go test -c -o traffic_monitor_integration_test)


Instead of tests/integration, could you make the directory name tests/_integration so that go testing ./... does not test this directory?

zrhoffman · 2020-09-25T21:02:49Z

+	if _, err := os.Stat(confPath); !os.IsNotExist(err) {
+		confBytes, err := ioutil.ReadFile(confPath)
+		if err != nil {
+			return Config{}, fmt.Errorf("Reading CDN conf '%s': %v", confPath, err)


Reading should not be capitalized

zrhoffman · 2020-09-25T21:03:37Z

+	}
+	err := envconfig.Process("traffic-ops-client-tests", &cfg)
+	if err != nil {
+		fmt.Errorf("cannot parse config: %v\n", err)


Is this error meant to be printed?

zrhoffman · 2020-09-25T21:06:14Z

+	// //Load the test data
+	// LoadFixtures(*tcFixturesFileName)
+
+	// var db *sql.DB
+	// db, err = OpenConnection()
+	// if err != nil {
+	// 	fmt.Printf("\nError opening connection to %s - %s, %v\n", Config.TrafficOps.URL, Config.TrafficOpsDB.User, err)
+	// 	os.Exit(1)
+	// }
+	// defer db.Close()
+
+	// err = Teardown(db)
+	// if err != nil {
+	// 	fmt.Printf("\nError tearingdown data %s - %s, %v\n", Config.TrafficOps.URL, Config.TrafficOpsDB.User, err)
+	// 	os.Exit(1)
+	// }
+
+	// err = SetupTestData(db)
+	// if err != nil {
+	// 	fmt.Printf("\nError setting up data %s - %s, %v\n", Config.TrafficOps.URL, Config.TrafficOpsDB.User, err)
+	// 	os.Exit(1)
+	// }


This is all commented out because it's still todo, right? Could we get a TODO comment about this code block?

zrhoffman · 2020-09-25T21:12:34Z

+		if maxDelayMS != 0 {
+			delayMS := minDelayMS
+			if minDelayMS != maxDelayMS {
+				delayMS += uint64(rand.Int63n(int64((maxDelayMS - minDelayMS))))


Redundant parenthesis here

zrhoffman · 2020-09-25T21:18:12Z

+    "interfaceMtu": null,
+    "interfaceName": "bond0",
+    "ip6Address": null,
+    "ip6Gateway": null,
+    "ipAddress": "trafficmonitor",
+    "ipGateway": "192.0.0.1",
+    "ipNetmask": "255.255.255.0",


After #4700 (for Dual Homing), this part looks like

"interfaces": [ { "ipAddresses": [ { "address": "172.16.239.6", "gateway": "172.16.239.1", "serviceAddress": true }, { "address": "fc01:9400:1000:8::6", "gateway": "fc01:9400:1000:8::1", "serviceAddress": true } ], "maxBandwidth": null, "monitor": true, "mtu": 1500, "name": "eth0" } ],

zrhoffman · 2020-09-25T21:22:02Z

+}
+'
+
+	curl -Lvsk ${TESTTO_URI}/api/1.2/cdns/fake/configs/monitoring.json -X POST -d '


As with the rest of the API v2.0 routes, this has become cdns/fake/configs/monitoring, json suffix is no longer supported

zrhoffman · 2020-09-25T21:37:58Z

+(cd tools/testto && go build)
+(cd tools/testcaches && go build)
+(cd tests/integration && go test -c -o traffic_monitor_integration_test)


When I run the docker-compose command to start the test,

docker-compose -p tmi --project-directory . -f tests/integration/docker-compose.yml run tmintegrationtest

I indefinitely get

Waiting for Traffic Ops to return a 200 OK

Looking at the testto container, it says

[user@computer traffic_monitor]$ 3 tmi --project-directory . -f tests/integration/docker-compose.yml logs -f testto Attaching to tmi_testto_1 testto_1 | testto: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by testto) tmi_testto_1 exited with code 1

So the instructions should have you build with CGO_ENABLED=0 (and GOOS=linux).

zrhoffman · 2020-09-25T21:53:16Z

+(cd tools/testto && go build)
+(cd tools/testcaches && go build)
+(cd tests/integration && go test -c -o traffic_monitor_integration_test)
+sudo docker-compose -p tmi --project-directory . -f tests/integration/docker-compose.yml run tmintegrationtest


After resolving the above linking error, rebuilding the images, and re-running, I get unmarshalling errors like

{"error": "unmarshalling posted body: json: cannot unmarshal bool into Go struct field CRConfigDeliveryService.deliveryServices.anonymousBlockingEnabled of type string"}

It eventually says

Error communicating with Monitor 'http://trafficmonitor' - didn't return a 200 OK in 30s

Full tmintegrationtest logs are here (click to expand)

< HTTP/1.1 200 OK * About to connect() to testto port 80 (#0) * Trying 172.23.0.3... * Connected to testto (172.23.0.3) port 80 (#0) > POST /api/1.2/cdns/fake/snapshot HTTP/1.1 > User-Agent: curl/7.29.0 > Host: testto > Accept: */* > Content-Length: 3756 > Content-Type: application/x-www-form-urlencoded > Expect: 100-continue > < HTTP/1.1 100 Continue < HTTP/1.1 400 Bad Request < Date: Fri, 25 Sep 2020 21:32:19 GMT < Content-Length: 169 < Content-Type: text/plain; charset=utf-8 * HTTP error before end of send, stop sending < * Closing connection 0 {"error": "unmarshalling posted body: json: cannot unmarshal bool into Go struct field CRConfigDeliveryService.deliveryServices.anonymousBlockingEnabled of type string"}* About to connect() to testto port 80 (#0) * Trying 172.23.0.3... * Connected to testto (172.23.0.3) port 80 (#0) > POST /api/1.2/cdns/fake/configs/monitoring.json HTTP/1.1 > User-Agent: curl/7.29.0 > Host: testto > Accept: */* > Content-Length: 2308 > Content-Type: application/x-www-form-urlencoded > Expect: 100-continue > < HTTP/1.1 100 Continue < HTTP/1.1 400 Bad Request < Date: Fri, 25 Sep 2020 21:32:19 GMT < Content-Length: 167 < Content-Type: text/plain; charset=utf-8 * HTTP error before end of send, stop sending < * Closing connection 0 {"error": "unmarshalling posted body: json: cannot unmarshal object into Go struct field TrafficServer.trafficServers.deliveryServices of type []tc.tsdeliveryService"}* About to connect() to testto port 80 (#0) * Trying 172.23.0.3... * Connected to testto (172.23.0.3) port 80 (#0) > POST /api/1.2/servers HTTP/1.1 > User-Agent: curl/7.29.0 > Host: testto > Accept: */* > Content-Length: 1135 > Content-Type: application/x-www-form-urlencoded > Expect: 100-continue > < HTTP/1.1 100 Continue < HTTP/1.1 204 No Content < Date: Fri, 25 Sep 2020 21:32:19 GMT < * Connection #0 to host testto left intact testto: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3877 0 3877 0 0 657k 0 --:--:-- --:--:-- --:--:-- 757k {"response":{ "config": { "api.cache-control.max_age": "30", "consistent.dns.routing": "true", "coveragezone.polling.interval": "30", testcaches: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed { 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 1 "ats": { 0 "plugin.remap_stats.num7.example.net.in_bytes": 2, 0 "plugin.remap_stats.num7.example.net.out_bytes": 2, "plugin.remap_stats.num7.example.net.status_2xx": 1, 34955 0 34955 0 0 4648k 0 --:--:-- --:--:-- --:--:-- 4876k (23) Failed writing body traffic_monitor: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6522<!DOCTYPE html>0 0 0 0 --:--:-- --:--:-- --:--:-- 0 <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file 0 6522 0 0 1159k 0 --:--:-- --:--:-- --:--:-- 1273k (23) Failed writing body DEBUG traffic_monitor_integration starting Error communicating with Monitor 'http://trafficmonitor' - didn't return a 200 OK in 30s

Looking at the logs of the trafficmonitor container, it doen't seem to get past

ERROR: datareq.go:152: 2020-09-25T21:32:38.143874758Z: Request Error: /api/version: service still starting, some caches unpolled: map[]

although there are other errors in there too, like

trafficmonitor_1 | ERROR: monitorconfig.go:346: 2020-09-25T21:32:33.530687712Z: Failed to parse polling strings for cache server 'server0': no service addresses found trafficmonitor_1 | ERROR: monitorconfig.go:346: 2020-09-25T21:32:33.530798803Z: Failed to parse polling strings for cache server 'server1': no service addresses found

Full trafficmonitor container logs are here (click to expand)

Attaching to tmi_trafficmonitor_1 trafficmonitor_1 | Failed to get D-Bus connection: Operation not permitted trafficmonitor_1 | /etc/init.d/traffic_monitor: line 41: /etc/sysconfig/network: No such file or directory trafficmonitor_1 | Starting traffic_monitor: trafficmonitor_1 | ERROR: opsconfig.go:77: 2020-09-25T21:32:18.530467624Z: OpsConfigManager: getting CDN name from Traffic Ops, using config CDN 'nocdn': getting monitor CDN: no server 'trafficmonitor' found in Traffic Ops trafficmonitor_1 | trafficmonitor_1 | ERROR: opsconfig.go:77: 2020-09-25T21:32:18.538113399Z: OpsConfigManager: Error getting Traffic Ops data: Error getting CRconfig from Traffic Ops: invalid CRConfig: CRConfig.Stats.CDN missing trafficmonitor_1 | trafficmonitor_1 | ERROR: opsconfig.go:205: 2020-09-25T21:32:18.538157142Z: retrying in 100ms trafficmonitor_1 | ERROR: opsconfig.go:77: 2020-09-25T21:32:18.638750564Z: OpsConfigManager: Error getting Traffic Ops data: Error getting CRconfig from Traffic Ops: invalid CRConfig: CRConfig.Stats.CDN missing trafficmonitor_1 | trafficmonitor_1 | ERROR: opsconfig.go:205: 2020-09-25T21:32:18.638781915Z: retrying in 248.612746ms trafficmonitor_1 | ERROR: opsconfig.go:77: 2020-09-25T21:32:18.888066827Z: OpsConfigManager: Error getting Traffic Ops data: Error getting CRconfig from Traffic Ops: invalid CRConfig: CRConfig.Stats.CDN missing trafficmonitor_1 | trafficmonitor_1 | ERROR: opsconfig.go:205: 2020-09-25T21:32:18.888119514Z: retrying in 642.263625ms trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:20.102734056Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:21.1049107Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:22.107247084Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:23.109295834Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:24.111672602Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:25.113719758Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:26.115813436Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:27.118294393Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:28.120237324Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:29.122398443Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:30.123841491Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:31.125970238Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:32.128329372Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:33.130950123Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: monitorconfig.go:346: 2020-09-25T21:32:33.530687712Z: Failed to parse polling strings for cache server 'server0': no service addresses found trafficmonitor_1 | ERROR: monitorconfig.go:346: 2020-09-25T21:32:33.530798803Z: Failed to parse polling strings for cache server 'server1': no service addresses found trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:34.135402114Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:35.136748588Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:36.139031306Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:37.141564084Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:38.143874758Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:39.146134866Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:40.14928317Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:41.151503741Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:42.154148808Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:43.156421926Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:44.158672414Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:45.159938856Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:46.161990084Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:47.163412929Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:48.164121248Z: Request Error: /api/version: service still starting, some caches unpolled: map[] trafficmonitor_1 | ERROR: datareq.go:152: 2020-09-25T21:32:49.165983499Z: Request Error: /api/version: service still starting, some caches unpolled: map[]

mitchell852 · 2021-05-10T19:16:53Z

@rob05c can this be closed in favor of #5817

rob05c · 2021-05-10T19:35:16Z

@rob05c can this be closed in favor of #5817

Once it's merged, yeah. I'd prefer to keep it open until then so we don't lose track of it, just in case something happens to 5817 and it's closed or doesn't get merged or something

zrhoffman · 2021-06-01T16:05:47Z

#5817 is based off of #3707 and is merged.

rob05c added Traffic Monitor related to Traffic Monitor tests related to tests and/or testing infrastructure WIP "Work-in-Progress" - do not merge! (use 'draft' pull requests from now on) labels Jul 1, 2019

rob05c added 2 commits September 25, 2020 14:24

Add tm testcaches http commands to change caches

658e96a

This is with the intention of adding an integration test framework for Traffic Monitor, which can use this tool and send commands to change the fake caches to test various scenarios.

Add TM client, integration test framework

bdfd9f3

rob05c force-pushed the tm-integration-test-framework branch from b7d2d9b to bdfd9f3 Compare September 25, 2020 20:29

rob05c removed the WIP "Work-in-Progress" - do not merge! (use 'draft' pull requests from now on) label Sep 25, 2020

rob05c marked this pull request as ready for review September 25, 2020 20:30

zrhoffman suggested changes Sep 25, 2020

View reviewed changes

shamrickus mentioned this pull request May 6, 2021

Traffic Monitor Integration Test Framework #5817

Merged

6 tasks

zrhoffman closed this Jun 1, 2021

		@@ -0,0 +1,22 @@
		# environment variables for the Traffic Monitor Docker Compose files

Conversation

rob05c commented Jul 1, 2019

What does this PR (Pull Request) do?

Which Traffic Control components are affected by this PR?

What is the best way to verify this PR?

If this is a bug fix, what versions of Traffic Ops are affected?

The following criteria are ALL met by this PR

Uh oh!

asfgit commented Jul 1, 2019

Uh oh!

rob05c commented Sep 25, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zrhoffman Sep 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zrhoffman Sep 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mitchell852 commented May 10, 2021

Uh oh!

rob05c commented May 10, 2021

Uh oh!

zrhoffman commented Jun 1, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zrhoffman Sep 25, 2020 •

edited

Loading

zrhoffman Sep 25, 2020 •

edited

Loading