Update system-tests to 9659a5e409cf8bacabd739b6af07c5f03c9bcbd6 by smola · Pull Request #5738 · DataDog/dd-trace-java

smola · 2023-08-18T09:46:20Z

What Does This Do

Fix the update script for the new dynamic config.
Comparison:
https://github.com/datadog/system-tests/compare/68da674b83d7a1a1730387a3b27a1d4328392f3c..9659a5e409cf8bacabd739b6af07c5f03c9bcbd6

Motivation

Additional Notes

Previous update: #5703

Relevant changes:

pr-commenter · 2023-08-18T10:27:26Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
commit	1.20.0-SNAPSHOT~aed58b8056	1.20.0-SNAPSHOT~dcce6e941f
config	baseline	candidate

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
module	Agent	Agent
parent	None	None
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 cases.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.20.0-SNAPSHOT~dcce6e941f, baseline=1.20.0-SNAPSHOT~aed58b8056

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (985.502 ms) : 0, 985502
Total [baseline] (9.173 s) : 0, 9172977
Agent [candidate] (994.612 ms) : 0, 994612
Total [candidate] (9.195 s) : 0, 9194741
section appsec
Agent [baseline] (1.078 s) : 0, 1077724
Total [baseline] (9.319 s) : 0, 9319326
Agent [candidate] (1.084 s) : 0, 1083621
Total [candidate] (9.314 s) : 0, 9314414
section iast
Agent [baseline] (1.085 s) : 0, 1084743
Total [baseline] (9.424 s) : 0, 9424390
Agent [candidate] (1.085 s) : 0, 1084573
Total [candidate] (9.411 s) : 0, 9411111
section profiling
Agent [baseline] (1.151 s) : 0, 1151399
Total [baseline] (9.43 s) : 0, 9430459
Agent [candidate] (1.157 s) : 0, 1157423
Total [candidate] (9.475 s) : 0, 9474895

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	985.502 ms	-
Agent	appsec	1.078 s	92.222 ms (9.4%)
Agent	iast	1.085 s	99.242 ms (10.1%)
Agent	profiling	1.151 s	165.898 ms (16.8%)
Total	tracing	9.173 s	-
Total	appsec	9.319 s	146.348 ms (1.6%)
Total	iast	9.424 s	251.412 ms (2.7%)
Total	profiling	9.43 s	257.481 ms (2.8%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	994.612 ms	-
Agent	appsec	1.084 s	89.009 ms (8.9%)
Agent	iast	1.085 s	89.961 ms (9.0%)
Agent	profiling	1.157 s	162.811 ms (16.4%)
Total	tracing	9.195 s	-
Total	appsec	9.314 s	119.674 ms (1.3%)
Total	iast	9.411 s	216.37 ms (2.4%)
Total	profiling	9.475 s	280.154 ms (3.0%)

gantt
    title petclinic - break down per module: candidate=1.20.0-SNAPSHOT~dcce6e941f, baseline=1.20.0-SNAPSHOT~aed58b8056

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (610.404 ms) : 0, 610404
BytebuddyAgent [candidate] (615.493 ms) : 0, 615493
GlobalTracer [baseline] (283.82 ms) : 0, 283820
GlobalTracer [candidate] (287.155 ms) : 0, 287155
AppSec [baseline] (50.292 ms) : 0, 50292
AppSec [candidate] (50.652 ms) : 0, 50652
Remote Config [baseline] (657.394 µs) : 0, 657
Remote Config [candidate] (667.337 µs) : 0, 667
Telemetry [baseline] (5.989 ms) : 0, 5989
Telemetry [candidate] (6.067 ms) : 0, 6067
section appsec
BytebuddyAgent [baseline] (614.886 ms) : 0, 614886
BytebuddyAgent [candidate] (618.51 ms) : 0, 618510
GlobalTracer [baseline] (285.847 ms) : 0, 285847
GlobalTracer [candidate] (288.337 ms) : 0, 288337
AppSec [baseline] (135.883 ms) : 0, 135883
AppSec [candidate] (135.642 ms) : 0, 135642
Remote Config [baseline] (727.32 µs) : 0, 727
Remote Config [candidate] (723.365 µs) : 0, 723
Telemetry [baseline] (5.759 ms) : 0, 5759
Telemetry [candidate] (5.78 ms) : 0, 5780
section iast
BytebuddyAgent [baseline] (712.868 ms) : 0, 712868
BytebuddyAgent [candidate] (712.826 ms) : 0, 712826
GlobalTracer [baseline] (270.637 ms) : 0, 270637
GlobalTracer [candidate] (270.808 ms) : 0, 270808
AppSec [baseline] (46.44 ms) : 0, 46440
AppSec [candidate] (46.288 ms) : 0, 46288
IAST [baseline] (14.191 ms) : 0, 14191
IAST [candidate] (14.102 ms) : 0, 14102
Remote Config [baseline] (596.606 µs) : 0, 597
Remote Config [candidate] (595.307 µs) : 0, 595
Telemetry [baseline] (5.727 ms) : 0, 5727
Telemetry [candidate] (5.695 ms) : 0, 5695
section profiling
BytebuddyAgent [baseline] (620.791 ms) : 0, 620791
BytebuddyAgent [candidate] (622.487 ms) : 0, 622487
GlobalTracer [baseline] (347.804 ms) : 0, 347804
GlobalTracer [candidate] (348.876 ms) : 0, 348876
AppSec [baseline] (48.99 ms) : 0, 48990
AppSec [candidate] (49.128 ms) : 0, 49128
Remote Config [baseline] (650.457 µs) : 0, 650
Remote Config [candidate] (655.905 µs) : 0, 656
Telemetry [baseline] (5.841 ms) : 0, 5841
Telemetry [candidate] (5.917 ms) : 0, 5917
ProfilingAgent [baseline] (74.659 ms) : 0, 74659
ProfilingAgent [candidate] (77.539 ms) : 0, 77539
Profiling [baseline] (74.683 ms) : 0, 74683
Profiling [candidate] (77.563 ms) : 0, 77563

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.20.0-SNAPSHOT~dcce6e941f, baseline=1.20.0-SNAPSHOT~aed58b8056

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (983.709 ms) : 0, 983709
Total [baseline] (8.623 s) : 0, 8623066
Agent [candidate] (984.142 ms) : 0, 984142
Total [candidate] (8.634 s) : 0, 8633765
section appsec
Agent [baseline] (1.086 s) : 0, 1086295
Total [baseline] (8.801 s) : 0, 8801090
Agent [candidate] (1.085 s) : 0, 1085281
Total [candidate] (8.773 s) : 0, 8772875
section iast
Agent [baseline] (1.085 s) : 0, 1085156
Total [baseline] (9.2 s) : 0, 9200163
Agent [candidate] (1.086 s) : 0, 1086054
Total [candidate] (9.185 s) : 0, 9184674
section profiling
Agent [baseline] (1.151 s) : 0, 1150855
Total [baseline] (8.859 s) : 0, 8859287
Agent [candidate] (1.162 s) : 0, 1162321
Total [candidate] (8.871 s) : 0, 8870751

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	983.709 ms	-
Agent	appsec	1.086 s	102.586 ms (10.4%)
Agent	iast	1.085 s	101.446 ms (10.3%)
Agent	profiling	1.151 s	167.146 ms (17.0%)
Total	tracing	8.623 s	-
Total	appsec	8.801 s	178.024 ms (2.1%)
Total	iast	9.2 s	577.097 ms (6.7%)
Total	profiling	8.859 s	236.222 ms (2.7%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	984.142 ms	-
Agent	appsec	1.085 s	101.139 ms (10.3%)
Agent	iast	1.086 s	101.912 ms (10.4%)
Agent	profiling	1.162 s	178.179 ms (18.1%)
Total	tracing	8.634 s	-
Total	appsec	8.773 s	139.11 ms (1.6%)
Total	iast	9.185 s	550.909 ms (6.4%)
Total	profiling	8.871 s	236.986 ms (2.7%)

gantt
    title insecure-bank - break down per module: candidate=1.20.0-SNAPSHOT~dcce6e941f, baseline=1.20.0-SNAPSHOT~aed58b8056

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (609.088 ms) : 0, 609088
BytebuddyAgent [candidate] (609.36 ms) : 0, 609360
GlobalTracer [baseline] (283.829 ms) : 0, 283829
GlobalTracer [candidate] (283.699 ms) : 0, 283699
AppSec [baseline] (49.818 ms) : 0, 49818
AppSec [candidate] (50.107 ms) : 0, 50107
Remote Config [baseline] (657.229 µs) : 0, 657
Remote Config [candidate] (653.594 µs) : 0, 654
Telemetry [baseline] (5.969 ms) : 0, 5969
Telemetry [candidate] (5.988 ms) : 0, 5988
section appsec
BytebuddyAgent [baseline] (618.35 ms) : 0, 618350
BytebuddyAgent [candidate] (618.832 ms) : 0, 618832
GlobalTracer [baseline] (289.778 ms) : 0, 289778
GlobalTracer [candidate] (288.698 ms) : 0, 288698
AppSec [baseline] (136.716 ms) : 0, 136716
AppSec [candidate] (136.518 ms) : 0, 136518
Remote Config [baseline] (739.844 µs) : 0, 740
Remote Config [candidate] (735.813 µs) : 0, 736
Telemetry [baseline] (5.851 ms) : 0, 5851
Telemetry [candidate] (5.798 ms) : 0, 5798
section iast
BytebuddyAgent [baseline] (712.834 ms) : 0, 712834
BytebuddyAgent [candidate] (713.331 ms) : 0, 713331
GlobalTracer [baseline] (270.639 ms) : 0, 270639
GlobalTracer [candidate] (271.49 ms) : 0, 271490
AppSec [baseline] (46.68 ms) : 0, 46680
AppSec [candidate] (46.492 ms) : 0, 46492
IAST [baseline] (14.196 ms) : 0, 14196
IAST [candidate] (14.179 ms) : 0, 14179
Remote Config [baseline] (601.336 µs) : 0, 601
Remote Config [candidate] (596.239 µs) : 0, 596
Telemetry [baseline] (5.754 ms) : 0, 5754
Telemetry [candidate] (5.687 ms) : 0, 5687
section profiling
BytebuddyAgent [baseline] (620.686 ms) : 0, 620686
BytebuddyAgent [candidate] (626.86 ms) : 0, 626860
GlobalTracer [baseline] (347.653 ms) : 0, 347653
GlobalTracer [candidate] (350.589 ms) : 0, 350589
AppSec [baseline] (49.176 ms) : 0, 49176
AppSec [candidate] (49.497 ms) : 0, 49497
Remote Config [baseline] (653.098 µs) : 0, 653
Remote Config [candidate] (667.779 µs) : 0, 668
Telemetry [baseline] (5.854 ms) : 0, 5854
Telemetry [candidate] (5.955 ms) : 0, 5955
ProfilingAgent [baseline] (74.18 ms) : 0, 74180
ProfilingAgent [candidate] (75.576 ms) : 0, 75576
Profiling [baseline] (74.204 ms) : 0, 74204
Profiling [candidate] (75.6 ms) : 0, 75600

Load

Parameters

	Baseline	Candidate
commit	1.20.0-SNAPSHOT~aed58b8056	1.20.0-SNAPSHOT~dcce6e941f
config	baseline	candidate
end_time	2023-08-18T10:04:05	2023-08-18T10:22:13
start_time	2023-08-18T10:03:47	2023-08-18T10:21:56

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~dcce6e941f, baseline=1.20.0-SNAPSHOT~aed58b8056
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.36 ms) : 1341, 1379
.   : milestone, 1360,
appsec (1.69 ms) : 1666, 1714
.   : milestone, 1690,
iast (1.475 ms) : 1451, 1500
.   : milestone, 1475,
profiling (1.471 ms) : 1446, 1496
.   : milestone, 1471,
tracing (1.473 ms) : 1448, 1498
.   : milestone, 1473,
section candidate
no_agent (1.327 ms) : 1307, 1346
.   : milestone, 1327,
appsec (1.685 ms) : 1661, 1709
.   : milestone, 1685,
iast (1.471 ms) : 1448, 1495
.   : milestone, 1471,
profiling (1.47 ms) : 1445, 1494
.   : milestone, 1470,
tracing (1.44 ms) : 1415, 1464
.   : milestone, 1440,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.36 ms [1.341 ms, 1.379 ms]	-
appsec	1.69 ms [1.666 ms, 1.714 ms]	330.373 µs (24.3%)
iast	1.475 ms [1.451 ms, 1.5 ms]	115.267 µs (8.5%)
profiling	1.471 ms [1.446 ms, 1.496 ms]	111.007 µs (8.2%)
tracing	1.473 ms [1.448 ms, 1.498 ms]	113.28 µs (8.3%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.327 ms [1.307 ms, 1.346 ms]	-
appsec	1.685 ms [1.661 ms, 1.709 ms]	358.511 µs (27.0%)
iast	1.471 ms [1.448 ms, 1.495 ms]	144.53 µs (10.9%)
profiling	1.47 ms [1.445 ms, 1.494 ms]	142.96 µs (10.8%)
tracing	1.44 ms [1.415 ms, 1.464 ms]	113.164 µs (8.5%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~dcce6e941f, baseline=1.20.0-SNAPSHOT~aed58b8056
    dateFormat X
    axisFormat %s
section baseline
no_agent (367.777 µs) : 347, 389
.   : milestone, 368,
appsec (662.142 µs) : 642, 683
.   : milestone, 662,
iast (467.851 µs) : 447, 489
.   : milestone, 468,
iast_FULL (522.015 µs) : 501, 543
.   : milestone, 522,
iast_INACTIVE (437.446 µs) : 416, 459
.   : milestone, 437,
profiling (442.986 µs) : 422, 464
.   : milestone, 443,
tracing (437.198 µs) : 416, 458
.   : milestone, 437,
section candidate
no_agent (361.854 µs) : 342, 382
.   : milestone, 362,
appsec (659.523 µs) : 640, 680
.   : milestone, 660,
iast (456.261 µs) : 436, 477
.   : milestone, 456,
iast_FULL (524.836 µs) : 504, 545
.   : milestone, 525,
iast_INACTIVE (427.051 µs) : 407, 447
.   : milestone, 427,
profiling (452.791 µs) : 431, 475
.   : milestone, 453,
tracing (440.964 µs) : 420, 462
.   : milestone, 441,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	367.777 µs [346.648 µs, 388.906 µs]	-
appsec	662.142 µs [641.668 µs, 682.615 µs]	294.365 µs (80.0%)
iast	467.851 µs [446.927 µs, 488.775 µs]	100.074 µs (27.2%)
iast_FULL	522.015 µs [501.193 µs, 542.836 µs]	154.238 µs (41.9%)
iast_INACTIVE	437.446 µs [416.127 µs, 458.765 µs]	69.669 µs (18.9%)
profiling	442.986 µs [421.899 µs, 464.074 µs]	75.21 µs (20.4%)
tracing	437.198 µs [416.238 µs, 458.158 µs]	69.421 µs (18.9%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	361.854 µs [341.845 µs, 381.864 µs]	-
appsec	659.523 µs [639.542 µs, 679.504 µs]	297.669 µs (82.3%)
iast	456.261 µs [435.655 µs, 476.868 µs]	94.407 µs (26.1%)
iast_FULL	524.836 µs [504.359 µs, 545.313 µs]	162.982 µs (45.0%)
iast_INACTIVE	427.051 µs [406.643 µs, 447.458 µs]	65.197 µs (18.0%)
profiling	452.791 µs [430.752 µs, 474.83 µs]	90.936 µs (25.1%)
tracing	440.964 µs [420.426 µs, 461.501 µs]	79.109 µs (21.9%)

smola added 2 commits August 18, 2023 11:37

fix update_pinned_system_tests.sh

90916f8

Update system-tests to 9659a5e409cf8bacabd739b6af07c5f03c9bcbd6

dcce6e9

smola added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Aug 18, 2023

smola requested a review from a team as a code owner August 18, 2023 09:46

smola requested review from a team and jandro996 August 18, 2023 10:50

smola enabled auto-merge (squash) August 18, 2023 11:12

jandro996 approved these changes Aug 21, 2023

View reviewed changes

smola merged commit dbf3422 into master Aug 21, 2023

smola deleted the smola/update-system-tests-20230818 branch August 21, 2023 06:10

github-actions Bot added this to the 1.20.0 milestone Aug 21, 2023

smola mentioned this pull request Aug 28, 2023

Update system-tests to 3749ab47dba7ceb1ac48bc9de6c89cb6427b0daa #5794

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update system-tests to 9659a5e409cf8bacabd739b6af07c5f03c9bcbd6#5738

Update system-tests to 9659a5e409cf8bacabd739b6af07c5f03c9bcbd6#5738
smola merged 2 commits intomasterfrom
smola/update-system-tests-20230818

smola commented Aug 18, 2023

Uh oh!

pr-commenter Bot commented Aug 18, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

smola commented Aug 18, 2023

What Does This Do

Motivation

Additional Notes

Uh oh!

pr-commenter Bot commented Aug 18, 2023

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants