Tracez Backend Benchmarking

[jajanet]

Code for benchmarking backend components that are used in Tracez. Friend classes are used to access private functions and/or modify parts of the benchmarked classes, reducing noise in the tests. Data aggregator periodically updates are stopped, and we do benchmarking with and without a different background thread to call GetAggregatedTracezData.

Processor results:

Load Average: 0.27, 0.21, 0.19
-------------------------------------------------------------------------------------------------------------------------
Benchmark                                                                               Time             CPU   Iterations
-------------------------------------------------------------------------------------------------------------------------
TracezProcessor/BM_MakeRunning/10                                                    1474 ns         1473 ns        94675
TracezProcessor/BM_MakeRunning/1000                                                 54392 ns        54367 ns         2578
TracezProcessor/BM_GetSpans/10                                                        473 ns          473 ns       296279
TracezProcessor/BM_GetSpans/1000                                                    46988 ns        46986 ns         2986
TracezProcessor/BM_MakeRunningMakeComplete/10/process_time/real_time                37332 ns        48653 ns         3641
TracezProcessor/BM_MakeRunningMakeComplete/1000/process_time/real_time             514363 ns       904488 ns          224
TracezProcessor/BM_MakeRunningGetSpans/10/process_time/real_time                    38095 ns        48823 ns         3756
TracezProcessor/BM_MakeRunningGetSpans/1000/process_time/real_time               32439828 ns     33694986 ns            4
TracezProcessor/BM_GetSpansMakeComplete/10/process_time/real_time                   47408 ns        68470 ns         3106
TracezProcessor/BM_GetSpansMakeComplete/1000/process_time/real_time                770039 ns      1041735 ns          186
TracezProcessor/BM_MakeRunningGetSpansMakeComplete/10/process_time/real_time        59006 ns        91096 ns         2444
TracezProcessor/BM_MakeRunningGetSpansMakeComplete/1000/process_time/real_time   71411371 ns     73876469 ns            2

Aggregator results:

Load Average: 0.40, 0.16, 0.11
-----------------------------------------------------------------------------------------------
Benchmark                                                     Time             CPU   Iterations
-----------------------------------------------------------------------------------------------
TracezAggregator/BM_SingleBucketSingleName/10              3669 ns         3674 ns        38418
TracezAggregator/BM_SingleBucketSingleName/1000          279355 ns       279482 ns          501
TracezAggregatorFetch/BM_SingleBucketSingleName/10         4107 ns         4088 ns        35268
TracezAggregatorFetch/BM_SingleBucketSingleName/1000     288939 ns       288882 ns          489
TracezAggregator/BM_SingleBucketManyNames/10               3597 ns         3610 ns        39089
TracezAggregator/BM_SingleBucketManyNames/1000           279545 ns       279641 ns          501
TracezAggregatorFetch/BM_SingleBucketManyNames/10          4119 ns         4099 ns        34889
TracezAggregatorFetch/BM_SingleBucketManyNames/1000      280687 ns       280694 ns          499
TracezAggregator/BM_ManyBucketsSingleName/10             826830 ns       827120 ns         1000
TracezAggregator/BM_ManyBucketsSingleName/1000         13750523 ns     13750447 ns           79
TracezAggregatorFetch/BM_ManyBucketsSingleName/10        943028 ns       942948 ns         1000
TracezAggregatorFetch/BM_ManyBucketsSingleName/1000    11751015 ns     11749654 ns           67
TracezAggregator/BM_ManyBucketsManyNames/10             1077521 ns      1077855 ns          872
TracezAggregator/BM_ManyBucketsManyNames/1000          14186373 ns     14185956 ns           53
TracezAggregatorFetch/BM_ManyBucketsManyNames/10        1090309 ns      1089959 ns         1000
TracezAggregatorFetch/BM_ManyBucketsManyNames/1000     14621460 ns     14131165 ns           52

Related Links:

• OTLP benchmark tests
• Sampler benchmark tests
• Google Benchmark

[liadavid]

Overall looks good, added few comments.

[kmanghat]

Nice job solving the periodic update problem using friend classes!

[liadavid]

Nice!

[kmanghat]

I think we should also test the performance for sparse vs dense spans, I think the amount of information in a span would affect the performance.

[kmanghat]

Could you explain what you mean by sorting of latencies?

[jajanet]

i reclarified in the docstring, and it's supposed to be explaining the aggregator performance may be affected by needing to have computer whether a span is in error, running, and latency (including within bands)

[kmanghat]

Why do you say that the aggregator performance might be affected by needing to compute the type of span? Running spans are received in a separate container altogether and distinguishing between error and latency span is done with a single if statement. Are you seeing a significant variance in performance when you test with the same number of running, completed and error spans? I am just curious and I am sure you understand these tests better so feel free to push this through.

[jajanet]

thanks for pointing that out! yes there's that minimal effect, and there's also memory considerations which i forgot to mention. all 11 buckets filled for each span name means there's more overhead compared to only 1 latency bucket

[kmanghat]

Thanks for the clarification, I forgot the tests measured memory as well.