Skip to content

R&D week: blackhole span #6326

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
amarziali merged 7 commits into from
Jan 9, 2024
Merged

R&D week: blackhole span #6326

amarziali merged 7 commits into from
Jan 9, 2024

Conversation

@amarziali
Copy link
Contributor

@amarziali amarziali commented Dec 6, 2023 •
edited
Loading

What Does This Do

Introduces the blackhole span who's muting the tracing during its lifetime (as soon as a blackhole scope is opened)
As analogy with the @Trace annotation, a @DoNotTrace annotation is also provided in order to decorate a method with a blackhole activated span.

It also provides the possibility to obtain a blackhole scope using the DD public API.

Finally showcases how it can be used in an instrumentation. In this case the example is in decoupling the apache http client instrumentation from the aws sdk one

Motivation

Additional Notes

Jira ticket: [PROJ-IDENT]

@pr-commenter
Copy link

pr-commenter bot commented Dec 6, 2023 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master andrea.marziali/blackhole
git_commit_date 1704706030 1704739464
git_commit_sha 5bf1b2f 1105e7f
release_version 1.28.0-SNAPSHOT~5bf1b2f68e 1.28.0-SNAPSHOT~1105e7f502
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1704742062 1704742062
ci_job_id 403145314 403145314
ci_pipeline_id 26067995 26067995
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 9 unstable metrics.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.049 s) : 0, 1049263
Total [baseline] (8.71 s) : 0, 8710407
Agent [candidate] (1.054 s) : 0, 1053972
Total [candidate] (8.712 s) : 0, 8711754
section iast
Agent [baseline] (1.168 s) : 0, 1167713
Total [baseline] (9.234 s) : 0, 9234058
Agent [candidate] (1.18 s) : 0, 1180195
Total [candidate] (9.278 s) : 0, 9278168
section iast_TELEMETRY_OFF
Agent [baseline] (1.172 s) : 0, 1171895
Total [baseline] (9.248 s) : 0, 9248266
Agent [candidate] (1.171 s) : 0, 1170553
Total [candidate] (9.247 s) : 0, 9247283
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.049 s -
Agent iast 1.168 s 118.451 ms (11.3%)
Agent iast_TELEMETRY_OFF 1.172 s 122.633 ms (11.7%)
Total tracing 8.71 s -
Total iast 9.234 s 523.651 ms (6.0%)
Total iast_TELEMETRY_OFF 9.248 s 537.859 ms (6.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.054 s -
Agent iast 1.18 s 126.223 ms (12.0%)
Agent iast_TELEMETRY_OFF 1.171 s 116.581 ms (11.1%)
Total tracing 8.712 s -
Total iast 9.278 s 566.414 ms (6.5%)
Total iast_TELEMETRY_OFF 9.247 s 535.529 ms (6.1%) 
gantt
    title insecure-bank - break down per module: candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (649.595 ms) : 0, 649595
BytebuddyAgent [candidate] (652.112 ms) : 0, 652112
GlobalTracer [baseline] (306.709 ms) : 0, 306709
GlobalTracer [candidate] (308.864 ms) : 0, 308864
AppSec [baseline] (50.865 ms) : 0, 50865
AppSec [candidate] (50.697 ms) : 0, 50697
Remote Config [baseline] (672.161 µs) : 0, 672
Remote Config [candidate] (674.925 µs) : 0, 675
Telemetry [baseline] (7.221 ms) : 0, 7221
Telemetry [candidate] (7.239 ms) : 0, 7239
section iast
BytebuddyAgent [baseline] (769.717 ms) : 0, 769717
BytebuddyAgent [candidate] (777.459 ms) : 0, 777459
GlobalTracer [baseline] (284.642 ms) : 0, 284642
GlobalTracer [candidate] (288.102 ms) : 0, 288102
AppSec [baseline] (51.439 ms) : 0, 51439
AppSec [candidate] (53.726 ms) : 0, 53726
IAST [baseline] (20.65 ms) : 0, 20650
IAST [candidate] (19.241 ms) : 0, 19241
Remote Config [baseline] (567.423 µs) : 0, 567
Remote Config [candidate] (567.665 µs) : 0, 568
Telemetry [baseline] (6.441 ms) : 0, 6441
Telemetry [candidate] (6.467 ms) : 0, 6467
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (769.395 ms) : 0, 769395
BytebuddyAgent [candidate] (768.745 ms) : 0, 768745
GlobalTracer [baseline] (288.078 ms) : 0, 288078
GlobalTracer [candidate] (287.579 ms) : 0, 287579
AppSec [baseline] (49.81 ms) : 0, 49810
AppSec [candidate] (49.462 ms) : 0, 49462
IAST [baseline] (22.96 ms) : 0, 22960
IAST [candidate] (23.34 ms) : 0, 23340
Remote Config [baseline] (631.682 µs) : 0, 632
Remote Config [candidate] (603.368 µs) : 0, 603
Telemetry [baseline] (6.463 ms) : 0, 6463
Telemetry [candidate] (6.392 ms) : 0, 6392
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.053 s) : 0, 1053330
Total [baseline] (9.374 s) : 0, 9373900
Agent [candidate] (1.059 s) : 0, 1059420
Total [candidate] (9.36 s) : 0, 9360284
section appsec
Agent [baseline] (1.148 s) : 0, 1148194
Total [baseline] (9.447 s) : 0, 9447453
Agent [candidate] (1.15 s) : 0, 1149684
Total [candidate] (9.442 s) : 0, 9442355
section iast
Agent [baseline] (1.178 s) : 0, 1178055
Total [baseline] (9.583 s) : 0, 9583103
Agent [candidate] (1.179 s) : 0, 1179318
Total [candidate] (9.586 s) : 0, 9585596
section profiling
Agent [baseline] (1.273 s) : 0, 1273418
Total [baseline] (9.631 s) : 0, 9630822
Agent [candidate] (1.281 s) : 0, 1280666
Total [candidate] (9.689 s) : 0, 9689076
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.053 s -
Agent appsec 1.148 s 94.863 ms (9.0%)
Agent iast 1.178 s 124.725 ms (11.8%)
Agent profiling 1.273 s 220.088 ms (20.9%)
Total tracing 9.374 s -
Total appsec 9.447 s 73.553 ms (0.8%)
Total iast 9.583 s 209.203 ms (2.2%)
Total profiling 9.631 s 256.922 ms (2.7%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.059 s -
Agent appsec 1.15 s 90.264 ms (8.5%)
Agent iast 1.179 s 119.898 ms (11.3%)
Agent profiling 1.281 s 221.246 ms (20.9%)
Total tracing 9.36 s -
Total appsec 9.442 s 82.071 ms (0.9%)
Total iast 9.586 s 225.312 ms (2.4%)
Total profiling 9.689 s 328.792 ms (3.5%) 
gantt
    title petclinic - break down per module: candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (653.172 ms) : 0, 653172
BytebuddyAgent [candidate] (656.389 ms) : 0, 656389
GlobalTracer [baseline] (307.108 ms) : 0, 307108
GlobalTracer [candidate] (309.631 ms) : 0, 309631
AppSec [baseline] (50.658 ms) : 0, 50658
AppSec [candidate] (50.761 ms) : 0, 50761
Remote Config [baseline] (674.566 µs) : 0, 675
Remote Config [candidate] (689.576 µs) : 0, 690
Telemetry [baseline] (7.237 ms) : 0, 7237
Telemetry [candidate] (7.208 ms) : 0, 7208
section appsec
BytebuddyAgent [baseline] (650.373 ms) : 0, 650373
BytebuddyAgent [candidate] (650.871 ms) : 0, 650871
GlobalTracer [baseline] (307.526 ms) : 0, 307526
GlobalTracer [candidate] (308.422 ms) : 0, 308422
AppSec [baseline] (148.523 ms) : 0, 148523
AppSec [candidate] (148.579 ms) : 0, 148579
Remote Config [baseline] (641.231 µs) : 0, 641
Remote Config [candidate] (645.457 µs) : 0, 645
Telemetry [baseline] (6.891 ms) : 0, 6891
Telemetry [candidate] (6.852 ms) : 0, 6852
section iast
BytebuddyAgent [baseline] (777.722 ms) : 0, 777722
BytebuddyAgent [candidate] (777.723 ms) : 0, 777723
GlobalTracer [baseline] (286.026 ms) : 0, 286026
GlobalTracer [candidate] (288.337 ms) : 0, 288337
AppSec [baseline] (53.497 ms) : 0, 53497
AppSec [candidate] (53.4 ms) : 0, 53400
IAST [baseline] (18.349 ms) : 0, 18349
IAST [candidate] (17.892 ms) : 0, 17892
Remote Config [baseline] (586.205 µs) : 0, 586
Remote Config [candidate] (578.672 µs) : 0, 579
Telemetry [baseline] (7.275 ms) : 0, 7275
Telemetry [candidate] (6.589 ms) : 0, 6589
section profiling
BytebuddyAgent [baseline] (660.359 ms) : 0, 660359
BytebuddyAgent [candidate] (665.683 ms) : 0, 665683
GlobalTracer [baseline] (374.906 ms) : 0, 374906
GlobalTracer [candidate] (376.609 ms) : 0, 376609
AppSec [baseline] (51.236 ms) : 0, 51236
AppSec [candidate] (51.475 ms) : 0, 51475
Remote Config [baseline] (1.014 ms) : 0, 1014
Remote Config [candidate] (1.002 ms) : 0, 1002
Telemetry [baseline] (7.248 ms) : 0, 7248
Telemetry [candidate] (7.361 ms) : 0, 7361
ProfilingAgent [baseline] (124.319 ms) : 0, 124319
ProfilingAgent [candidate] (123.84 ms) : 0, 123840
Profiling [baseline] (124.344 ms) : 0, 124344
Profiling [candidate] (123.865 ms) : 0, 123865
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-01-08T19:06:54 2024-01-08T19:23:28
git_branch master andrea.marziali/blackhole
git_commit_date 1704706030 1704739464
git_commit_sha 5bf1b2f 1105e7f
release_version 1.28.0-SNAPSHOT~5bf1b2f68e 1.28.0-SNAPSHOT~1105e7f502
start_time 2024-01-08T19:06:41 2024-01-08T19:23:15
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1704742062 1704742062
ci_job_id 403145314 403145314
ci_pipeline_id 26067995 26067995
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 13 unstable metrics.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e
    dateFormat X
    axisFormat %s
section baseline
no_agent (359.35 µs) : 340, 379
.   : milestone, 359,
iast (476.785 µs) : 456, 497
.   : milestone, 477,
iast_FULL (545.864 µs) : 525, 566
.   : milestone, 546,
iast_INACTIVE (448.723 µs) : 428, 470
.   : milestone, 449,
iast_TELEMETRY_OFF (464.861 µs) : 444, 485
.   : milestone, 465,
tracing (437.77 µs) : 417, 459
.   : milestone, 438,
section candidate
no_agent (363.367 µs) : 343, 384
.   : milestone, 363,
iast (470.084 µs) : 450, 491
.   : milestone, 470,
iast_FULL (536.006 µs) : 516, 556
.   : milestone, 536,
iast_INACTIVE (440.044 µs) : 420, 460
.   : milestone, 440,
iast_TELEMETRY_OFF (468.398 µs) : 447, 490
.   : milestone, 468,
tracing (442.466 µs) : 422, 463
.   : milestone, 442,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 359.35 µs [339.706 µs, 378.994 µs] -
iast 476.785 µs [456.164 µs, 497.405 µs] 117.435 µs (32.7%)
iast_FULL 545.864 µs [525.486 µs, 566.241 µs] 186.514 µs (51.9%)
iast_INACTIVE 448.723 µs [427.839 µs, 469.607 µs] 89.373 µs (24.9%)
iast_TELEMETRY_OFF 464.861 µs [444.389 µs, 485.333 µs] 105.511 µs (29.4%)
tracing 437.77 µs [416.606 µs, 458.934 µs] 78.42 µs (21.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 363.367 µs [342.854 µs, 383.879 µs] -
iast 470.084 µs [449.609 µs, 490.56 µs] 106.718 µs (29.4%)
iast_FULL 536.006 µs [515.693 µs, 556.318 µs] 172.639 µs (47.5%)
iast_INACTIVE 440.044 µs [419.808 µs, 460.281 µs] 76.677 µs (21.1%)
iast_TELEMETRY_OFF 468.398 µs [447.204 µs, 489.593 µs] 105.032 µs (28.9%)
tracing 442.466 µs [421.506 µs, 463.426 µs] 79.099 µs (21.8%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.28.0-SNAPSHOT~1105e7f502, baseline=1.28.0-SNAPSHOT~5bf1b2f68e
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.344 ms) : 1325, 1363
.   : milestone, 1344,
appsec (1.758 ms) : 1732, 1783
.   : milestone, 1758,
iast (1.525 ms) : 1501, 1549
.   : milestone, 1525,
profiling (1.552 ms) : 1526, 1578
.   : milestone, 1552,
tracing (1.484 ms) : 1459, 1509
.   : milestone, 1484,
section candidate
no_agent (1.342 ms) : 1323, 1361
.   : milestone, 1342,
appsec (1.772 ms) : 1747, 1797
.   : milestone, 1772,
iast (1.526 ms) : 1502, 1550
.   : milestone, 1526,
profiling (1.519 ms) : 1494, 1544
.   : milestone, 1519,
tracing (1.501 ms) : 1476, 1526
.   : milestone, 1501,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.344 ms [1.325 ms, 1.363 ms] -
appsec 1.758 ms [1.732 ms, 1.783 ms] 413.788 µs (30.8%)
iast 1.525 ms [1.501 ms, 1.549 ms] 181.316 µs (13.5%)
profiling 1.552 ms [1.526 ms, 1.578 ms] 207.912 µs (15.5%)
tracing 1.484 ms [1.459 ms, 1.509 ms] 140.244 µs (10.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.342 ms [1.323 ms, 1.361 ms] -
appsec 1.772 ms [1.747 ms, 1.797 ms] 430.09 µs (32.0%)
iast 1.526 ms [1.502 ms, 1.55 ms] 184.002 µs (13.7%)
profiling 1.519 ms [1.494 ms, 1.544 ms] 177.092 µs (13.2%)
tracing 1.501 ms [1.476 ms, 1.526 ms] 159.164 µs (11.9%)

@amarziali amarziali force-pushed the andrea.marziali/blackhole branch 2 times, most recently from 1b07497 to 6628dbb Compare December 8, 2023 13:23
@amarziali amarziali marked this pull request as ready for review December 13, 2023 14:13
@amarziali amarziali requested a review from a team as a code owner December 13, 2023 14:13
@amarziali amarziali requested review from mcculls and ygree December 13, 2023 14:13
@amarziali amarziali added comp: core Tracer core tag: experimental Experimental changes comp: api Tracer public API inst: trace annotation Trace annotations instrumentation comp: opentracing OpenTracing labels Dec 13, 2023
@amarziali amarziali force-pushed the andrea.marziali/blackhole branch from 4964cb5 to 5508303 Compare January 5, 2024 10:52
@amarziali amarziali requested a review from ygree January 5, 2024 10:54
ygree
ygree approved these changes Jan 5, 2024
View reviewed changes
Copy link
Contributor

@ygree ygree left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@amarziali amarziali enabled auto-merge (squash) January 8, 2024 18:44
@amarziali amarziali merged commit f46fcc6 into master Jan 9, 2024
@amarziali amarziali deleted the andrea.marziali/blackhole branch January 9, 2024 11:06
@github-actions github-actions bot added this to the 1.28.0 milestone Jan 9, 2024
smola added a commit that referenced this pull request Jan 30, 2024
@smola
Mute tracing for subprocesses executed in by the tracer
d438c2e 
The tracer sometimes runs external processes in some cases. Currently, `hostname`
and `getenforce`. The subprocess instrumentation creates spans for
these, leading to visible `command_execution` spans for `hostname`
and/or `enforce` which may be confusing to users.

This commit uses the bew blackhole span API (#6326) to mute tracing
during when `hostname` and `getenforce` are run, preventing the creation
of these spans.
@smola smola mentioned this pull request Jan 30, 2024
Merged
smola added a commit that referenced this pull request Jan 30, 2024
@smola
Mute tracing for subprocesses executed in by the tracer (#6584)
4c70643 
The tracer sometimes runs external processes in some cases. Currently, `hostname`
and `getenforce`. The subprocess instrumentation creates spans for
these, leading to visible `command_execution` spans for `hostname`
and/or `enforce` which may be confusing to users.

This commit uses the bew blackhole span API (#6326) to mute tracing
during when `hostname` and `getenforce` are run, preventing the creation
of these spans.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ygree ygree ygree approved these changes

@mcculls mcculls Awaiting requested review from mcculls mcculls is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: api Tracer public API comp: core Tracer core comp: opentracing OpenTracing inst: trace annotation Trace annotations instrumentation tag: experimental Experimental changes

Projects

None yet

Milestone

1.28.0

Development

Successfully merging this pull request may close these issues.

3 participants

@amarziali @ygree