Skip to content

Update system-tests to a5c7ef0fa0f988b4c09e3734b99a37dd436a3eef#5915

Merged
smola merged 2 commits intomasterfrom
smola/system-tests-20230919
Sep 20, 2023
Merged

Update system-tests to a5c7ef0fa0f988b4c09e3734b99a37dd436a3eef#5915
smola merged 2 commits intomasterfrom
smola/system-tests-20230919

Conversation

@smola
Copy link
Copy Markdown
Member

@smola smola commented Sep 19, 2023

What Does This Do

Comparison:
https://github.com/datadog/system-tests/compare/3598f2583cd5c25326cd7a5c01097374d94c1993..a5c7ef0fa0f988b4c09e3734b99a37dd436a3eef

Motivation

Additional Notes

Previous update: #5848

Relevant changes:

@smola smola added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Sep 19, 2023
@smola smola requested a review from a team as a code owner September 19, 2023 16:05
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Sep 19, 2023 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
commit 1.21.0-SNAPSHOT~7ab7c4f068 1.21.0-SNAPSHOT~33e58f3d7c
config baseline candidate
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
module Agent Agent
parent None None
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 cases.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.21.0-SNAPSHOT~33e58f3d7c, baseline=1.21.0-SNAPSHOT~7ab7c4f068

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.006 s) : 0, 1005942
Total [baseline] (9.37 s) : 0, 9370171
Agent [candidate] (1.011 s) : 0, 1011492
Total [candidate] (9.249 s) : 0, 9249095
section appsec
Agent [baseline] (1.099 s) : 0, 1099274
Total [baseline] (9.292 s) : 0, 9291636
Agent [candidate] (1.092 s) : 0, 1092478
Total [candidate] (9.282 s) : 0, 9281585
section iast
Agent [baseline] (1.113 s) : 0, 1113373
Total [baseline] (9.436 s) : 0, 9435702
Agent [candidate] (1.123 s) : 0, 1122577
Total [candidate] (9.525 s) : 0, 9524931
section profiling
Agent [baseline] (1.175 s) : 0, 1175432
Total [baseline] (9.418 s) : 0, 9418451
Agent [candidate] (1.185 s) : 0, 1184812
Total [candidate] (9.403 s) : 0, 9403397
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.006 s -
Agent appsec 1.099 s 93.333 ms (9.3%)
Agent iast 1.113 s 107.432 ms (10.7%)
Agent profiling 1.175 s 169.49 ms (16.8%)
Total tracing 9.37 s -
Total appsec 9.292 s -78.535 ms (-0.8%)
Total iast 9.436 s 65.531 ms (0.7%)
Total profiling 9.418 s 48.28 ms (0.5%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.011 s -
Agent appsec 1.092 s 80.986 ms (8.0%)
Agent iast 1.123 s 111.085 ms (11.0%)
Agent profiling 1.185 s 173.32 ms (17.1%)
Total tracing 9.249 s -
Total appsec 9.282 s 32.49 ms (0.4%)
Total iast 9.525 s 275.836 ms (3.0%)
Total profiling 9.403 s 154.302 ms (1.7%) 
gantt
    title petclinic - break down per module: candidate=1.21.0-SNAPSHOT~33e58f3d7c, baseline=1.21.0-SNAPSHOT~7ab7c4f068

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (622.697 ms) : 0, 622697
BytebuddyAgent [candidate] (626.429 ms) : 0, 626429
GlobalTracer [baseline] (293.409 ms) : 0, 293409
GlobalTracer [candidate] (295.134 ms) : 0, 295134
AppSec [baseline] (48.845 ms) : 0, 48845
AppSec [candidate] (48.7 ms) : 0, 48700
Remote Config [baseline] (669.304 µs) : 0, 669
Remote Config [candidate] (682.185 µs) : 0, 682
Telemetry [baseline] (5.978 ms) : 0, 5978
Telemetry [candidate] (6.005 ms) : 0, 6005
section appsec
BytebuddyAgent [baseline] (627.166 ms) : 0, 627166
BytebuddyAgent [candidate] (622.93 ms) : 0, 622930
GlobalTracer [baseline] (294.84 ms) : 0, 294840
GlobalTracer [candidate] (293.364 ms) : 0, 293364
AppSec [baseline] (136.098 ms) : 0, 136098
AppSec [candidate] (135.401 ms) : 0, 135401
Remote Config [baseline] (651.719 µs) : 0, 652
Remote Config [candidate] (635.917 µs) : 0, 636
Telemetry [baseline] (5.904 ms) : 0, 5904
Telemetry [candidate] (5.852 ms) : 0, 5852
section iast
BytebuddyAgent [baseline] (735.196 ms) : 0, 735196
BytebuddyAgent [candidate] (741.586 ms) : 0, 741586
GlobalTracer [baseline] (277.301 ms) : 0, 277301
GlobalTracer [candidate] (280.047 ms) : 0, 280047
AppSec [baseline] (45.571 ms) : 0, 45571
AppSec [candidate] (45.861 ms) : 0, 45861
IAST [baseline] (14.196 ms) : 0, 14196
IAST [candidate] (14.326 ms) : 0, 14326
Remote Config [baseline] (541.144 µs) : 0, 541
Remote Config [candidate] (544.973 µs) : 0, 545
Telemetry [baseline] (6.335 ms) : 0, 6335
Telemetry [candidate] (5.748 ms) : 0, 5748
section profiling
BytebuddyAgent [baseline] (633.787 ms) : 0, 633787
BytebuddyAgent [candidate] (640.028 ms) : 0, 640028
GlobalTracer [baseline] (352.958 ms) : 0, 352958
GlobalTracer [candidate] (354.44 ms) : 0, 354440
AppSec [baseline] (48.617 ms) : 0, 48617
AppSec [candidate] (48.89 ms) : 0, 48890
Remote Config [baseline] (652.012 µs) : 0, 652
Remote Config [candidate] (657.901 µs) : 0, 658
Telemetry [baseline] (6.035 ms) : 0, 6035
Telemetry [candidate] (6.056 ms) : 0, 6056
ProfilingAgent [baseline] (80.573 ms) : 0, 80573
ProfilingAgent [candidate] (81.195 ms) : 0, 81195
Profiling [baseline] (80.597 ms) : 0, 80597
Profiling [candidate] (81.219 ms) : 0, 81219
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.21.0-SNAPSHOT~33e58f3d7c, baseline=1.21.0-SNAPSHOT~7ab7c4f068

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.018 s) : 0, 1017647
Total [baseline] (8.698 s) : 0, 8698309
Agent [candidate] (1.006 s) : 0, 1006140
Total [candidate] (8.655 s) : 0, 8655381
section appsec
Agent [baseline] (1.091 s) : 0, 1091030
Total [baseline] (8.696 s) : 0, 8695737
Agent [candidate] (1.1 s) : 0, 1100108
Total [candidate] (8.751 s) : 0, 8751135
section iast
Agent [baseline] (1.115 s) : 0, 1115126
Total [baseline] (9.189 s) : 0, 9188805
Agent [candidate] (1.125 s) : 0, 1124602
Total [candidate] (9.255 s) : 0, 9254651
section profiling
Agent [baseline] (1.177 s) : 0, 1177238
Total [baseline] (8.866 s) : 0, 8865809
Agent [candidate] (1.184 s) : 0, 1184295
Total [candidate] (8.866 s) : 0, 8865975
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.018 s -
Agent appsec 1.091 s 73.383 ms (7.2%)
Agent iast 1.115 s 97.479 ms (9.6%)
Agent profiling 1.177 s 159.591 ms (15.7%)
Total tracing 8.698 s -
Total appsec 8.696 s -2.573 ms (-0.0%)
Total iast 9.189 s 490.496 ms (5.6%)
Total profiling 8.866 s 167.499 ms (1.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.006 s -
Agent appsec 1.1 s 93.968 ms (9.3%)
Agent iast 1.125 s 118.462 ms (11.8%)
Agent profiling 1.184 s 178.155 ms (17.7%)
Total tracing 8.655 s -
Total appsec 8.751 s 95.754 ms (1.1%)
Total iast 9.255 s 599.269 ms (6.9%)
Total profiling 8.866 s 210.594 ms (2.4%) 
gantt
    title insecure-bank - break down per module: candidate=1.21.0-SNAPSHOT~33e58f3d7c, baseline=1.21.0-SNAPSHOT~7ab7c4f068

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (630.206 ms) : 0, 630206
BytebuddyAgent [candidate] (622.596 ms) : 0, 622596
GlobalTracer [baseline] (296.934 ms) : 0, 296934
GlobalTracer [candidate] (294.074 ms) : 0, 294074
AppSec [baseline] (48.953 ms) : 0, 48953
AppSec [candidate] (48.48 ms) : 0, 48480
Remote Config [baseline] (685.843 µs) : 0, 686
Remote Config [candidate] (674.676 µs) : 0, 675
Telemetry [baseline] (6.011 ms) : 0, 6011
Telemetry [candidate] (5.97 ms) : 0, 5970
section appsec
BytebuddyAgent [baseline] (622.419 ms) : 0, 622419
BytebuddyAgent [candidate] (627.822 ms) : 0, 627822
GlobalTracer [baseline] (292.185 ms) : 0, 292185
GlobalTracer [candidate] (295.746 ms) : 0, 295746
AppSec [baseline] (135.71 ms) : 0, 135710
AppSec [candidate] (135.517 ms) : 0, 135517
Remote Config [baseline] (637.748 µs) : 0, 638
Remote Config [candidate] (645.462 µs) : 0, 645
Telemetry [baseline] (5.832 ms) : 0, 5832
Telemetry [candidate] (5.89 ms) : 0, 5890
section iast
BytebuddyAgent [baseline] (736.713 ms) : 0, 736713
BytebuddyAgent [candidate] (742.517 ms) : 0, 742517
GlobalTracer [baseline] (277.758 ms) : 0, 277758
GlobalTracer [candidate] (280.645 ms) : 0, 280645
AppSec [baseline] (45.718 ms) : 0, 45718
AppSec [candidate] (45.965 ms) : 0, 45965
IAST [baseline] (14.35 ms) : 0, 14350
IAST [candidate] (14.61 ms) : 0, 14610
Remote Config [baseline] (551.774 µs) : 0, 552
Remote Config [candidate] (555.061 µs) : 0, 555
Telemetry [baseline] (5.772 ms) : 0, 5772
Telemetry [candidate] (5.814 ms) : 0, 5814
section profiling
BytebuddyAgent [baseline] (634.357 ms) : 0, 634357
BytebuddyAgent [candidate] (639.371 ms) : 0, 639371
GlobalTracer [baseline] (354.304 ms) : 0, 354304
GlobalTracer [candidate] (355.479 ms) : 0, 355479
AppSec [baseline] (48.85 ms) : 0, 48850
AppSec [candidate] (49.209 ms) : 0, 49209
Remote Config [baseline] (643.808 µs) : 0, 644
Remote Config [candidate] (655.431 µs) : 0, 655
Telemetry [baseline] (6.071 ms) : 0, 6071
Telemetry [candidate] (6.071 ms) : 0, 6071
ProfilingAgent [baseline] (79.975 ms) : 0, 79975
ProfilingAgent [candidate] (80.195 ms) : 0, 80195
Profiling [baseline] (79.999 ms) : 0, 79999
Profiling [candidate] (80.22 ms) : 0, 80220
Loading

Load

Parameters

Baseline Candidate
commit 1.21.0-SNAPSHOT~7ab7c4f068 1.21.0-SNAPSHOT~33e58f3d7c
config baseline candidate
end_time 2023-09-20T09:36:33 2023-09-20T09:54:40
start_time 2023-09-20T09:36:16 2023-09-20T09:54:22
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.21.0-SNAPSHOT~33e58f3d7c, baseline=1.21.0-SNAPSHOT~7ab7c4f068
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.331 ms) : 1312, 1349
.   : milestone, 1331,
appsec (1.687 ms) : 1662, 1711
.   : milestone, 1687,
iast (1.462 ms) : 1437, 1487
.   : milestone, 1462,
profiling (1.475 ms) : 1451, 1499
.   : milestone, 1475,
tracing (1.428 ms) : 1404, 1452
.   : milestone, 1428,
section candidate
no_agent (1.324 ms) : 1304, 1343
.   : milestone, 1324,
appsec (1.694 ms) : 1670, 1718
.   : milestone, 1694,
iast (1.446 ms) : 1422, 1470
.   : milestone, 1446,
profiling (1.449 ms) : 1425, 1474
.   : milestone, 1449,
tracing (1.457 ms) : 1433, 1481
.   : milestone, 1457,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.331 ms [1.312 ms, 1.349 ms] -
appsec 1.687 ms [1.662 ms, 1.711 ms] 356.137 µs (26.8%)
iast 1.462 ms [1.437 ms, 1.487 ms] 131.659 µs (9.9%)
profiling 1.475 ms [1.451 ms, 1.499 ms] 144.635 µs (10.9%)
tracing 1.428 ms [1.404 ms, 1.452 ms] 97.967 µs (7.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.324 ms [1.304 ms, 1.343 ms] -
appsec 1.694 ms [1.67 ms, 1.718 ms] 370.245 µs (28.0%)
iast 1.446 ms [1.422 ms, 1.47 ms] 122.328 µs (9.2%)
profiling 1.449 ms [1.425 ms, 1.474 ms] 125.512 µs (9.5%)
tracing 1.457 ms [1.433 ms, 1.481 ms] 133.401 µs (10.1%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.21.0-SNAPSHOT~33e58f3d7c, baseline=1.21.0-SNAPSHOT~7ab7c4f068
    dateFormat X
    axisFormat %s
section baseline
no_agent (362.702 µs) : 342, 383
.   : milestone, 363,
appsec (664.905 µs) : 644, 685
.   : milestone, 665,
iast (462.476 µs) : 441, 484
.   : milestone, 462,
iast_FULL (519.963 µs) : 500, 540
.   : milestone, 520,
iast_INACTIVE (421.688 µs) : 401, 442
.   : milestone, 422,
profiling (432.27 µs) : 411, 453
.   : milestone, 432,
tracing (429.787 µs) : 409, 451
.   : milestone, 430,
section candidate
no_agent (362.804 µs) : 343, 383
.   : milestone, 363,
appsec (667.655 µs) : 647, 688
.   : milestone, 668,
iast (453.126 µs) : 433, 474
.   : milestone, 453,
iast_FULL (514.859 µs) : 494, 536
.   : milestone, 515,
iast_INACTIVE (425.639 µs) : 405, 446
.   : milestone, 426,
profiling (438.561 µs) : 417, 460
.   : milestone, 439,
tracing (439.94 µs) : 419, 461
.   : milestone, 440,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 362.702 µs [342.241 µs, 383.163 µs] -
appsec 664.905 µs [644.477 µs, 685.332 µs] 302.203 µs (83.3%)
iast 462.476 µs [441.349 µs, 483.604 µs] 99.774 µs (27.5%)
iast_FULL 519.963 µs [499.61 µs, 540.316 µs] 157.261 µs (43.4%)
iast_INACTIVE 421.688 µs [401.245 µs, 442.13 µs] 58.986 µs (16.3%)
profiling 432.27 µs [411.163 µs, 453.377 µs] 69.568 µs (19.2%)
tracing 429.787 µs [408.997 µs, 450.577 µs] 67.085 µs (18.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 362.804 µs [342.609 µs, 382.999 µs] -
appsec 667.655 µs [647.418 µs, 687.892 µs] 304.851 µs (84.0%)
iast 453.126 µs [432.668 µs, 473.585 µs] 90.323 µs (24.9%)
iast_FULL 514.859 µs [494.157 µs, 535.56 µs] 152.055 µs (41.9%)
iast_INACTIVE 425.639 µs [404.949 µs, 446.33 µs] 62.835 µs (17.3%)
profiling 438.561 µs [417.218 µs, 459.904 µs] 75.757 µs (20.9%)
tracing 439.94 µs [418.78 µs, 461.099 µs] 77.136 µs (21.3%)

@smola smola merged commit fbf7efb into master Sep 20, 2023
@smola smola deleted the smola/system-tests-20230919 branch September 20, 2023 13:44
@github-actions github-actions Bot added this to the 1.22.0 milestone Sep 20, 2023
@smola smola mentioned this pull request Sep 29, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bantonsson bantonsson bantonsson approved these changes

@nayeem-kamal nayeem-kamal Awaiting requested review from nayeem-kamal nayeem-kamal is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: testing Testing tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.22.0

Development

Successfully merging this pull request may close these issues.

2 participants

@smola @bantonsson