Skip to content

Update system-tests to 3749ab47dba7ceb1ac48bc9de6c89cb6427b0daa#5794

Merged
smola merged 1 commit intomasterfrom
smola/update-system-tests-20230828
Aug 29, 2023
Merged

Update system-tests to 3749ab47dba7ceb1ac48bc9de6c89cb6427b0daa#5794
smola merged 1 commit intomasterfrom
smola/update-system-tests-20230828

Conversation

@smola
Copy link
Copy Markdown
Member

@smola smola commented Aug 28, 2023

@smola smola added comp: testing Testing tag: no release notes Changes to exclude from release notes labels Aug 28, 2023
@smola smola requested a review from a team as a code owner August 28, 2023 15:36
@smola smola requested a review from mcculls August 28, 2023 15:36
@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Aug 28, 2023

Benchmarks

Startup

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~995e9c41d8 1.20.0-SNAPSHOT~a7de0be339
config baseline candidate
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
module Agent Agent
parent None None
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 cases.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.20.0-SNAPSHOT~a7de0be339, baseline=1.20.0-SNAPSHOT~995e9c41d8

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (995.978 ms) : 0, 995978
Total [baseline] (8.664 s) : 0, 8664338
Agent [candidate] (989.42 ms) : 0, 989420
Total [candidate] (8.641 s) : 0, 8641454
section appsec
Agent [baseline] (1.073 s) : 0, 1073400
Total [baseline] (8.726 s) : 0, 8726047
Agent [candidate] (1.078 s) : 0, 1078422
Total [candidate] (8.755 s) : 0, 8754809
section iast
Agent [baseline] (1.091 s) : 0, 1091360
Total [baseline] (9.169 s) : 0, 9169326
Agent [candidate] (1.089 s) : 0, 1089355
Total [candidate] (9.154 s) : 0, 9154008
section profiling
Agent [baseline] (1.159 s) : 0, 1159470
Total [baseline] (8.892 s) : 0, 8891913
Agent [candidate] (1.173 s) : 0, 1172764
Total [candidate] (8.894 s) : 0, 8894489
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 995.978 ms -
Agent appsec 1.073 s 77.422 ms (7.8%)
Agent iast 1.091 s 95.382 ms (9.6%)
Agent profiling 1.159 s 163.492 ms (16.4%)
Total tracing 8.664 s -
Total appsec 8.726 s 61.709 ms (0.7%)
Total iast 9.169 s 504.988 ms (5.8%)
Total profiling 8.892 s 227.575 ms (2.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 989.42 ms -
Agent appsec 1.078 s 89.002 ms (9.0%)
Agent iast 1.089 s 99.935 ms (10.1%)
Agent profiling 1.173 s 183.345 ms (18.5%)
Total tracing 8.641 s -
Total appsec 8.755 s 113.355 ms (1.3%)
Total iast 9.154 s 512.555 ms (5.9%)
Total profiling 8.894 s 253.036 ms (2.9%) 
gantt
    title insecure-bank - break down per module: candidate=1.20.0-SNAPSHOT~a7de0be339, baseline=1.20.0-SNAPSHOT~995e9c41d8

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (618.11 ms) : 0, 618110
BytebuddyAgent [candidate] (613.227 ms) : 0, 613227
GlobalTracer [baseline] (286.243 ms) : 0, 286243
GlobalTracer [candidate] (285.194 ms) : 0, 285194
AppSec [baseline] (50.235 ms) : 0, 50235
AppSec [candidate] (50.002 ms) : 0, 50002
Remote Config [baseline] (666.078 µs) : 0, 666
Remote Config [candidate] (658.005 µs) : 0, 658
Telemetry [baseline] (6.02 ms) : 0, 6020
Telemetry [candidate] (5.977 ms) : 0, 5977
section appsec
BytebuddyAgent [baseline] (613.277 ms) : 0, 613277
BytebuddyAgent [candidate] (615.538 ms) : 0, 615538
GlobalTracer [baseline] (284.095 ms) : 0, 284095
GlobalTracer [candidate] (285.772 ms) : 0, 285772
AppSec [baseline] (135.215 ms) : 0, 135215
AppSec [candidate] (136.135 ms) : 0, 136135
Remote Config [baseline] (726.882 µs) : 0, 727
Remote Config [candidate] (723.552 µs) : 0, 724
Telemetry [baseline] (5.76 ms) : 0, 5760
Telemetry [candidate] (5.761 ms) : 0, 5761
section iast
BytebuddyAgent [baseline] (718.991 ms) : 0, 718991
BytebuddyAgent [candidate] (717.698 ms) : 0, 717698
GlobalTracer [baseline] (270.325 ms) : 0, 270325
GlobalTracer [candidate] (270.113 ms) : 0, 270113
AppSec [baseline] (47.329 ms) : 0, 47329
AppSec [candidate] (47.008 ms) : 0, 47008
Remote Config [baseline] (552.266 µs) : 0, 552
Remote Config [candidate] (542.295 µs) : 0, 542
Telemetry [baseline] (5.726 ms) : 0, 5726
Telemetry [candidate] (5.688 ms) : 0, 5688
IAST [baseline] (14.042 ms) : 0, 14042
IAST [candidate] (14.013 ms) : 0, 14013
section profiling
BytebuddyAgent [baseline] (625.636 ms) : 0, 625636
BytebuddyAgent [candidate] (632.643 ms) : 0, 632643
GlobalTracer [baseline] (348.188 ms) : 0, 348188
GlobalTracer [candidate] (351.752 ms) : 0, 351752
AppSec [baseline] (48.99 ms) : 0, 48990
AppSec [candidate] (49.638 ms) : 0, 49638
Remote Config [baseline] (646.591 µs) : 0, 647
Remote Config [candidate] (653.922 µs) : 0, 654
Telemetry [baseline] (5.98 ms) : 0, 5980
Telemetry [candidate] (5.997 ms) : 0, 5997
ProfilingAgent [baseline] (77.142 ms) : 0, 77142
ProfilingAgent [candidate] (78.658 ms) : 0, 78658
Profiling [baseline] (77.166 ms) : 0, 77166
Profiling [candidate] (78.683 ms) : 0, 78683
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.20.0-SNAPSHOT~a7de0be339, baseline=1.20.0-SNAPSHOT~995e9c41d8

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (987.839 ms) : 0, 987839
Total [baseline] (9.224 s) : 0, 9224012
Agent [candidate] (1.001 s) : 0, 1001076
Total [candidate] (9.204 s) : 0, 9204113
section appsec
Agent [baseline] (1.072 s) : 0, 1072107
Total [baseline] (9.266 s) : 0, 9266018
Agent [candidate] (1.081 s) : 0, 1081481
Total [candidate] (9.28 s) : 0, 9280449
section iast
Agent [baseline] (1.091 s) : 0, 1090859
Total [baseline] (9.432 s) : 0, 9432064
Agent [candidate] (1.097 s) : 0, 1097189
Total [candidate] (9.446 s) : 0, 9446051
section profiling
Agent [baseline] (1.174 s) : 0, 1173694
Total [baseline] (9.58 s) : 0, 9579676
Agent [candidate] (1.158 s) : 0, 1158185
Total [candidate] (9.448 s) : 0, 9447567
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 987.839 ms -
Agent appsec 1.072 s 84.268 ms (8.5%)
Agent iast 1.091 s 103.02 ms (10.4%)
Agent profiling 1.174 s 185.855 ms (18.8%)
Total tracing 9.224 s -
Total appsec 9.266 s 42.006 ms (0.5%)
Total iast 9.432 s 208.053 ms (2.3%)
Total profiling 9.58 s 355.664 ms (3.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.001 s -
Agent appsec 1.081 s 80.405 ms (8.0%)
Agent iast 1.097 s 96.113 ms (9.6%)
Agent profiling 1.158 s 157.109 ms (15.7%)
Total tracing 9.204 s -
Total appsec 9.28 s 76.336 ms (0.8%)
Total iast 9.446 s 241.938 ms (2.6%)
Total profiling 9.448 s 243.454 ms (2.6%) 
gantt
    title petclinic - break down per module: candidate=1.20.0-SNAPSHOT~a7de0be339, baseline=1.20.0-SNAPSHOT~995e9c41d8

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (612.7 ms) : 0, 612700
BytebuddyAgent [candidate] (620.143 ms) : 0, 620143
GlobalTracer [baseline] (284.177 ms) : 0, 284177
GlobalTracer [candidate] (288.505 ms) : 0, 288505
AppSec [baseline] (50.069 ms) : 0, 50069
AppSec [candidate] (50.988 ms) : 0, 50988
Remote Config [baseline] (662.241 µs) : 0, 662
Remote Config [candidate] (674.496 µs) : 0, 674
Telemetry [baseline] (5.989 ms) : 0, 5989
Telemetry [candidate] (6.077 ms) : 0, 6077
section appsec
BytebuddyAgent [baseline] (612.567 ms) : 0, 612567
BytebuddyAgent [candidate] (617.217 ms) : 0, 617217
GlobalTracer [baseline] (283.843 ms) : 0, 283843
GlobalTracer [candidate] (287.423 ms) : 0, 287423
AppSec [baseline] (134.909 ms) : 0, 134909
AppSec [candidate] (135.726 ms) : 0, 135726
Remote Config [baseline] (727.242 µs) : 0, 727
Remote Config [candidate] (733.295 µs) : 0, 733
Telemetry [baseline] (5.748 ms) : 0, 5748
Telemetry [candidate] (5.815 ms) : 0, 5815
section iast
BytebuddyAgent [baseline] (718.892 ms) : 0, 718892
BytebuddyAgent [candidate] (722.434 ms) : 0, 722434
GlobalTracer [baseline] (270.031 ms) : 0, 270031
GlobalTracer [candidate] (272.236 ms) : 0, 272236
AppSec [baseline] (47.394 ms) : 0, 47394
AppSec [candidate] (47.568 ms) : 0, 47568
Remote Config [baseline] (542.27 µs) : 0, 542
Remote Config [candidate] (557.175 µs) : 0, 557
Telemetry [baseline] (5.697 ms) : 0, 5697
Telemetry [candidate] (5.737 ms) : 0, 5737
IAST [baseline] (13.997 ms) : 0, 13997
IAST [candidate] (14.188 ms) : 0, 14188
section profiling
BytebuddyAgent [baseline] (631.945 ms) : 0, 631945
BytebuddyAgent [candidate] (625.088 ms) : 0, 625088
GlobalTracer [baseline] (352.362 ms) : 0, 352362
GlobalTracer [candidate] (348.434 ms) : 0, 348434
AppSec [baseline] (49.66 ms) : 0, 49660
AppSec [candidate] (49.164 ms) : 0, 49164
Remote Config [baseline] (654.253 µs) : 0, 654
Remote Config [candidate] (640.883 µs) : 0, 641
Telemetry [baseline] (6.073 ms) : 0, 6073
Telemetry [candidate] (5.916 ms) : 0, 5916
ProfilingAgent [baseline] (79.652 ms) : 0, 79652
ProfilingAgent [candidate] (76.09 ms) : 0, 76090
Profiling [baseline] (79.677 ms) : 0, 79677
Profiling [candidate] (76.114 ms) : 0, 76114
Loading

Load

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~995e9c41d8 1.20.0-SNAPSHOT~a7de0be339
config baseline candidate
end_time 2023-08-28T15:54:10 2023-08-28T16:12:16
start_time 2023-08-28T15:53:53 2023-08-28T16:11:59
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~a7de0be339, baseline=1.20.0-SNAPSHOT~995e9c41d8
    dateFormat X
    axisFormat %s
section baseline
no_agent (364.604 µs) : 344, 386
.   : milestone, 365,
appsec (653.777 µs) : 633, 674
.   : milestone, 654,
iast (454.966 µs) : 435, 475
.   : milestone, 455,
iast_FULL (519.587 µs) : 499, 540
.   : milestone, 520,
iast_INACTIVE (427.819 µs) : 407, 449
.   : milestone, 428,
profiling (434.567 µs) : 413, 456
.   : milestone, 435,
tracing (434.103 µs) : 413, 456
.   : milestone, 434,
section candidate
no_agent (361.522 µs) : 341, 382
.   : milestone, 362,
appsec (661.473 µs) : 641, 682
.   : milestone, 661,
iast (465.82 µs) : 445, 487
.   : milestone, 466,
iast_FULL (520.472 µs) : 499, 541
.   : milestone, 520,
iast_INACTIVE (433.034 µs) : 412, 454
.   : milestone, 433,
profiling (432.64 µs) : 411, 454
.   : milestone, 433,
tracing (433.123 µs) : 412, 454
.   : milestone, 433,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 364.604 µs [343.621 µs, 385.587 µs] -
appsec 653.777 µs [633.312 µs, 674.241 µs] 289.173 µs (79.3%)
iast 454.966 µs [434.596 µs, 475.335 µs] 90.362 µs (24.8%)
iast_FULL 519.587 µs [499.031 µs, 540.142 µs] 154.983 µs (42.5%)
iast_INACTIVE 427.819 µs [407.06 µs, 448.577 µs] 63.215 µs (17.3%)
profiling 434.567 µs [413.345 µs, 455.788 µs] 69.963 µs (19.2%)
tracing 434.103 µs [412.652 µs, 455.554 µs] 69.499 µs (19.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 361.522 µs [341.288 µs, 381.757 µs] -
appsec 661.473 µs [640.783 µs, 682.163 µs] 299.951 µs (83.0%)
iast 465.82 µs [444.568 µs, 487.073 µs] 104.298 µs (28.8%)
iast_FULL 520.472 µs [499.469 µs, 541.475 µs] 158.949 µs (44.0%)
iast_INACTIVE 433.034 µs [411.577 µs, 454.49 µs] 71.511 µs (19.8%)
profiling 432.64 µs [411.296 µs, 453.985 µs] 71.118 µs (19.7%)
tracing 433.123 µs [412.155 µs, 454.091 µs] 71.601 µs (19.8%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~a7de0be339, baseline=1.20.0-SNAPSHOT~995e9c41d8
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.342 ms) : 1323, 1361
.   : milestone, 1342,
appsec (1.682 ms) : 1657, 1707
.   : milestone, 1682,
iast (1.472 ms) : 1448, 1497
.   : milestone, 1472,
profiling (1.485 ms) : 1458, 1511
.   : milestone, 1485,
tracing (1.438 ms) : 1413, 1462
.   : milestone, 1438,
section candidate
no_agent (1.322 ms) : 1303, 1341
.   : milestone, 1322,
appsec (1.675 ms) : 1651, 1699
.   : milestone, 1675,
iast (1.456 ms) : 1432, 1480
.   : milestone, 1456,
profiling (1.463 ms) : 1437, 1488
.   : milestone, 1463,
tracing (1.459 ms) : 1435, 1483
.   : milestone, 1459,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.342 ms [1.323 ms, 1.361 ms] -
appsec 1.682 ms [1.657 ms, 1.707 ms] 339.942 µs (25.3%)
iast 1.472 ms [1.448 ms, 1.497 ms] 130.306 µs (9.7%)
profiling 1.485 ms [1.458 ms, 1.511 ms] 142.614 µs (10.6%)
tracing 1.438 ms [1.413 ms, 1.462 ms] 95.66 µs (7.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.322 ms [1.303 ms, 1.341 ms] -
appsec 1.675 ms [1.651 ms, 1.699 ms] 352.651 µs (26.7%)
iast 1.456 ms [1.432 ms, 1.48 ms] 133.61 µs (10.1%)
profiling 1.463 ms [1.437 ms, 1.488 ms] 140.525 µs (10.6%)
tracing 1.459 ms [1.435 ms, 1.483 ms] 136.663 µs (10.3%)

@smola smola merged commit df23c00 into master Aug 29, 2023
@smola smola deleted the smola/update-system-tests-20230828 branch August 29, 2023 11:58
@github-actions github-actions Bot added this to the 1.20.0 milestone Aug 29, 2023
@smola smola mentioned this pull request Sep 6, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mcculls mcculls Awaiting requested review from mcculls mcculls is a code owner automatically assigned from DataDog/apm-java

1 more reviewer

@devinsba devinsba devinsba approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

comp: testing Testing tag: no release notes Changes to exclude from release notes

Projects

None yet

Milestone

1.20.0

Development

Successfully merging this pull request may close these issues.

2 participants

@smola @devinsba