Run tests in Gitlab #8656

randomanderson · 2025-04-01T18:58:03Z

What Does This Do

Runs the test suite in Gitlab.

The main bulk of the changes are in .gitlab-ci.yml which creates the test jobs, applies the appropriate caches, uploads the results to Datadog CI Observability, saves artifacts, etc.

Changes to tests

Use specific container versions for test containers
Replace many instances where localhost was assumed with the appropriate getHost()-like methods
Use forward-headers-strategy: NONE for Spring Boot tests. Spring Boot strips x-forwarded-for headers which breaks forwarding tests
Many minor changes to decrease flakiness

Motivation

Eventually, all CircleCI jobs will be migrated to Gitlab

Future Work

Further work needs to be done on "right-sizing" the test jobs. There are multiple knobs to tweak between:
- total memory
- CPU cores
- Gradle workers
- Gradle memory per work
- Number of parallel jobs
  where there is an optimization between job duration, number of jobs, size of cache, "stability".
This PR doesn't support using PR labels to enable specific jdks
CircleCI scripts for reporting are used mostly unmodified. When CircleCI is no longer being used, these scripts can be updated to be more Gitlab specific
Gating release jobs and publishing on tests passing
system-tests are not run in Gitlab. They will temporarily run in Github Actions

Flaky Tests

InstrumenterUnloadTest
GradleLauncherSmokeTest
ApmTracingDisabledMatrixSmokeTest

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any useful labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

This reverts commit 26a7adb.

…ecks-migration

# Conflicts: # .gitlab-ci.yml

…ests

…-tests # Conflicts: # dd-java-agent/instrumentation/kafka-clients-3.8/src/test/groovy/KafkaClientCustomPropagationConfigTest.groovy # dd-java-agent/instrumentation/kafka-clients-3.8/src/test/groovy/KafkaClientTestBase.groovy

…ests

.gitlab-ci.yml

.gitlab/check_test_agent_results.sh

sarahchen6

Excited to see this go in!

...test/java/datadog/trace/bootstrap/instrumentation/jfr/exceptions/ExceptionHistogramTest.java

...instrumentation/spring-webmvc-3.1/src/latestDepTest/groovy/test/boot/SpringBootServer.groovy

dd-java-agent/testing/src/main/groovy/datadog/trace/agent/test/base/HttpServerTest.groovy

dd-smoke-tests/log-injection/src/test/resources/SpockConfig.groovy

…ests

dd-smoke-tests/log-injection/src/test/resources/SpockConfig.groovy

…ests # Conflicts: # dd-smoke-tests/spring-boot-3.0-native/src/test/groovy/SpringBootNativeInstrumentationTest.groovy

* first pass checks * add some debug * Revert "add some debug" This reverts commit 26a7adb. * Normalize node index and node total * fix normalization * actually use the normalized values * first pass: migrate tests to Gitlab * less scenarios for now * bump memory and workers * actually use task partitions * Add section start/end * Allow individual tests to control gradle memory * remove a little of the reportspam * use concrete docker image tags * configure test containers to use mirror * fix limit * fix missing container versions * Better gitlab naming with smarter split * update caches with test dep cache, save test results * Cleanup repositories {} blocks * update proxy url to use internal routing * no "s" * allow insecure everywhere * include apache with confluence * remove intentional fail * use gradlePluginCache, mavenCentral after pluginportal * proxy mule as well * change BUILD_CACHE_TYPE to CACHE_TYPE since its used for both * split cache like with circleci. Cache became too big * Some caches are too big to be saved * test fixes * more test fixes * disable some tests for now * more fixes * Use maven proxy with openliberty smoketests * missed one more mongodb case * enable some more of the matrix * Fix lettuce tests * Fix lettuce tests * cleanup imports * create asyncCommands * codenarc * Cleanup aggregator tests to try to reduce flakiness * debug maven logs for now * try non-ssh url * Revert "debug maven logs for now" This reverts commit c5c158f. * add some debug logging * codenarc * try fixed value of parallelism * enable debug logs for smoketest building only * limit number of reserved threads * some debug for why the smoketest is not running * more debug for gradle launcher failure * try fixing graal test * try to force the logging * codenarc * set available processors for jetty * codenarc * bluntly bump max threads * some more debug * remove verbose logging * try memcached test with new location * opt out of split by parallel * Try with 3 gradle workers * try decreasing reporting interval * Refractor CrashTrackingSmokeTest to use OutputThreads * Simple forked memory test * better ergonomics * try increasing timeout * extend agent test runner * don't override forkedMinHeapSize and forkedMaxHeapSize * print out requests * remove logging statement * Better crash tracking * remove intentionally failing test * use a test UDP server to record oom events * Make crash tracking smoke test more resiliant * loop through messages until the correct event * some debug * configurable dogstatsd port * enable everything * use sleep instead of park nanos * save body data ahead of time * retry once for system failures * spotless * cleanup * spotless/merge issues * Skip ExceptionHistogramTest on J9 * add a print statement around the body for debugging * comment out all but 8, 17, and 21 for now * fix spring 6 forwarding headers * revert logging statements * needs to be set in 3.1 for latestDepTest * faster OOM * property was set in incorrect location * ssi_smoke, flaky, and esoteric jvms on master only * use normal runner when TestContainers isn't needed * skip forwarding header filter here too * tweaks * limit executor size and fix IPV4 assumptions * try more workers with less memory * set max memory for all crashtracking tests * double retry * scale up memory, less jobs * revert memory bump, increase forked tests * tweaks * add flag for nondefault jvms and flaky tests * correct description * Externalize runtime processors override * ExceptionHistogramTest exclusions already handled by master * increase spring native build memory * bump gradle build timeout * bump graalvm compile memory --------- Co-authored-by: Santiago Mola <santiago.mola@datadoghq.com>

randomanderson added 30 commits February 28, 2025 10:39

first pass checks

10ecbb9

add some debug

26a7adb

Revert "add some debug"

f8448c9

This reverts commit 26a7adb.

Normalize node index and node total

0b8643a

fix normalization

65d41ac

actually use the normalized values

661749d

Merge remote-tracking branch 'origin/master' into landerson/gitlab-ch…

7e21e7b

…ecks-migration

first pass: migrate tests to Gitlab

70c494d

Merge branch 'master' into landerson/migrate-tests

f3ca3e1

# Conflicts: # .gitlab-ci.yml

less scenarios for now

2ae3cf3

bump memory and workers

c3ebdc0

actually use task partitions

2d78199

Add section start/end

aba4f20

Allow individual tests to control gradle memory

87bffbf

remove a little of the reportspam

6b6d4e5

use concrete docker image tags

b2340e0

configure test containers to use mirror

42784ee

Merge remote-tracking branch 'origin/master' into landerson/migrate-t…

3bed067

…ests

fix limit

e781c3b

fix missing container versions

ffb90db

Better gitlab naming with smarter split

c18b491

update caches with test dep cache, save test results

a09c626

Cleanup repositories {} blocks

97d5bc6

update proxy url to use internal routing

2b37e3b

no "s"

7092cd0

allow insecure everywhere

391d0c3

include apache with confluence

e225757

remove intentional fail

7019854

use gradlePluginCache, mavenCentral after pluginportal

08a85b0

randomanderson requested review from a team as code owners April 18, 2025 15:38

randomanderson requested review from amarziali and tonyredondo April 18, 2025 15:38

amarziali requested a review from PerfectSlayer April 18, 2025 15:41

Merge remote-tracking branch 'origin/master' into landerson/migrate-t…

3fc0b97

…ests

sarahchen6 reviewed May 5, 2025

View reviewed changes

.gitlab-ci.yml Outdated Show resolved Hide resolved

sarahchen6 reviewed May 5, 2025

View reviewed changes

.gitlab/check_test_agent_results.sh Show resolved Hide resolved

sarahchen6 approved these changes May 5, 2025

View reviewed changes

bric3 reviewed May 6, 2025

View reviewed changes

randomanderson added 4 commits May 6, 2025 12:47

correct description

c3fd666

Externalize runtime processors override

6e5f1a9

Merge remote-tracking branch 'origin/master' into landerson/migrate-t…

d0b313d

…ests

ExceptionHistogramTest exclusions already handled by master

2b5d9ac

randomanderson force-pushed the landerson/migrate-tests branch from 528ce5a to 2b5d9ac Compare May 6, 2025 19:50

randomanderson added 3 commits May 7, 2025 12:39

increase spring native build memory

0df2344

bump gradle build timeout

8d74d0e

Merge remote-tracking branch 'origin/master' into landerson/migrate-t…

e77ec82

…ests

bric3 reviewed May 7, 2025

View reviewed changes

dd-smoke-tests/log-injection/src/test/resources/SpockConfig.groovy Show resolved Hide resolved

Merge remote-tracking branch 'origin/master' into landerson/migrate-t…

545cf9a

…ests # Conflicts: # dd-smoke-tests/spring-boot-3.0-native/src/test/groovy/SpringBootNativeInstrumentationTest.groovy

randomanderson enabled auto-merge (squash) May 9, 2025 15:17

randomanderson disabled auto-merge May 9, 2025 15:17

bump graalvm compile memory

99c67a3

randomanderson enabled auto-merge (squash) May 9, 2025 21:06

randomanderson merged commit f494c33 into master May 9, 2025
584 of 589 checks passed

randomanderson deleted the landerson/migrate-tests branch May 9, 2025 21:23

github-actions bot added this to the 1.50.0 milestone May 9, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Run tests in Gitlab #8656

Run tests in Gitlab #8656

Uh oh!

randomanderson commented Apr 1, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

sarahchen6 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Run tests in Gitlab #8656

Run tests in Gitlab #8656

Uh oh!

Conversation

randomanderson commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Changes to tests

Motivation

Future Work

Flaky Tests

Contributor Checklist

Uh oh!

Uh oh!

Uh oh!

sarahchen6 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

randomanderson commented Apr 1, 2025 •

edited

Loading