Skip to content

[improve][java-client] Replace ScheduledExecutor to improve performance of message consumption #16236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
codelipenghui merged 8 commits into from
Jun 28, 2022
Merged

[improve][java-client] Replace ScheduledExecutor to improve performance of message consumption #16236

codelipenghui merged 8 commits into from
Jun 28, 2022

Conversation

@codelipenghui
Copy link
Contributor

@codelipenghui codelipenghui commented Jun 27, 2022 •
edited
Loading

Motivation

The Scheduled Executor doesn't work very efficiently because each task will add to a DelayedQueue(A priority queue) first even if using the .execute() method without any schedule delay.

image

image

Profile result:
perf_consumer_0.html.txt

Running a performance test for single topic max message read rate test:

bin/pulsar-perf consume test -q 1000000 -p 100000000
bin/pulsar-perf produce test -r 1000000 -s 1 -mk random -o 10000 -threads 2

Without this PR (2.10.1):

Profiling started
2022-06-27T13:44:01,183+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 23919664 msg --- 265702.851  msg/s --- 2.027 Mbit/s  --- Latency: mean: 49430.572 ms - med: 49406 - 95pct: 52853 - 99pct: 53024 - 99.9pct: 53053 - 99.99pct: 53056 - Max: 53057
2022-06-27T13:44:11,196+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 26690802 msg --- 276759.125  msg/s --- 2.112 Mbit/s  --- Latency: mean: 56106.186 ms - med: 56000 - 95pct: 59289 - 99pct: 59985 - 99.9pct: 60037 - 99.99pct: 60042 - Max: 60042
2022-06-27T13:44:21,216+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 28788693 msg --- 209467.861  msg/s --- 1.598 Mbit/s  --- Latency: mean: 63523.143 ms - med: 63580 - 95pct: 67202 - 99pct: 67523 - 99.9pct: 67547 - 99.99pct: 67548 - Max: 67548
2022-06-27T13:44:31,233+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 31255365 msg --- 246190.932  msg/s --- 1.878 Mbit/s  --- Latency: mean: 71152.370 ms - med: 71058 - 95pct: 74555 - 99pct: 74806 - 99.9pct: 74842 - 99.99pct: 74847 - Max: 74847
2022-06-27T13:44:41,247+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 33606630 msg --- 234769.313  msg/s --- 1.791 Mbit/s  --- Latency: mean: 78636.478 ms - med: 78724 - 95pct: 81694 - 99pct: 82090 - 99.9pct: 82279 - 99.99pct: 82285 - Max: 82286

With this PR:

Profiling started
2022-06-27T13:56:20,426+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 431272207 msg --- 1079360.516  msg/s --- 8.235 Mbit/s  --- Latency: mean: 272.645 ms - med: 334 - 95pct: 470 - 99pct: 510 - 99.9pct: 522 - 99.99pct: 523 - Max: 524
2022-06-27T13:56:30,438+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 441292346 msg --- 1000645.852  msg/s --- 7.634 Mbit/s  --- Latency: mean: 15.512 ms - med: 14 - 95pct: 29 - 99pct: 39 - 99.9pct: 54 - 99.99pct: 55 - Max: 55
2022-06-27T13:56:40,450+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 451303308 msg --- 999973.040  msg/s --- 7.629 Mbit/s  --- Latency: mean: 18.265 ms - med: 14 - 95pct: 53 - 99pct: 98 - 99.9pct: 174 - 99.99pct: 176 - Max: 177
2022-06-27T13:56:50,462+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 461308082 msg --- 999309.458  msg/s --- 7.624 Mbit/s  --- Latency: mean: 14.728 ms - med: 14 - 95pct: 18 - 99pct: 41 - 99.9pct: 50 - 99.99pct: 51 - Max: 52
2022-06-27T13:57:00,475+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 471327606 msg --- 1000738.584  msg/s --- 7.635 Mbit/s  --- Latency: mean: 21.291 ms - med: 16 - 95pct: 52 - 99pct: 61 - 99.9pct: 65 - 99.99pct: 66 - Max: 66

Profile result with this PR:

perf_consumer_1.html.txt

Modification

  • Change internal executor and external executor to normal executor service
  • Added a new ScheduledExecutorProvider to handle the scheduled tasks.

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API: (no)
  • The schema: (no)
  • The default values of configurations: (no)
  • The wire protocol: (no)
  • The rest endpoints: (no)
  • The admin cli options: (no)
  • Anything that affects deployment: (no)

Documentation

Check the box below or label this PR directly.

Need to update docs?

  • doc-required
    (Your PR needs to update docs and you will update later)

  • doc-not-needed
    (Please explain why)

  • doc
    (Your PR contains doc changes)

  • doc-complete
    (Docs have been already added)

@codelipenghui
[improve][java-client] Replace ScheduledExecutor to improve performan…
aff6bca 
…ce of message consumption

### Motivation

The Scheduled Executor doesn't work very efficiently because each task will add to a DelayedQueue(A priority queue) first
even if using the `.execute()` method with any schedule delay.

Running a performance test for single topic max message read rate test:

```
bin/pulsar-perf consume test -q 1000000 -p 100000000
bin/pulsar-perf produce test -r 1000000 -s 1 -mk random -o 10000 -threads 2
```

Without this PR (2.10.1):

Profiling started
2022-06-27T13:44:01,183+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 23919664 msg --- 265702.851  msg/s --- 2.027 Mbit/s  --- Latency: mean: 49430.572 ms - med: 49406 - 95pct: 52853 - 99pct: 53024 - 99.9pct: 53053 - 99.99pct: 53056 - Max: 53057
2022-06-27T13:44:11,196+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 26690802 msg --- 276759.125  msg/s --- 2.112 Mbit/s  --- Latency: mean: 56106.186 ms - med: 56000 - 95pct: 59289 - 99pct: 59985 - 99.9pct: 60037 - 99.99pct: 60042 - Max: 60042
2022-06-27T13:44:21,216+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 28788693 msg --- 209467.861  msg/s --- 1.598 Mbit/s  --- Latency: mean: 63523.143 ms - med: 63580 - 95pct: 67202 - 99pct: 67523 - 99.9pct: 67547 - 99.99pct: 67548 - Max: 67548
2022-06-27T13:44:31,233+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 31255365 msg --- 246190.932  msg/s --- 1.878 Mbit/s  --- Latency: mean: 71152.370 ms - med: 71058 - 95pct: 74555 - 99pct: 74806 - 99.9pct: 74842 - 99.99pct: 74847 - Max: 74847
2022-06-27T13:44:41,247+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 33606630 msg --- 234769.313  msg/s --- 1.791 Mbit/s  --- Latency: mean: 78636.478 ms - med: 78724 - 95pct: 81694 - 99pct: 82090 - 99.9pct: 82279 - 99.99pct: 82285 - Max: 82286

With this PR:

Profiling started
2022-06-27T13:56:20,426+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 431272207 msg --- 1079360.516  msg/s --- 8.235 Mbit/s  --- Latency: mean: 272.645 ms - med: 334 - 95pct: 470 - 99pct: 510 - 99.9pct: 522 - 99.99pct: 523 - Max: 524
2022-06-27T13:56:30,438+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 441292346 msg --- 1000645.852  msg/s --- 7.634 Mbit/s  --- Latency: mean: 15.512 ms - med: 14 - 95pct: 29 - 99pct: 39 - 99.9pct: 54 - 99.99pct: 55 - Max: 55
2022-06-27T13:56:40,450+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 451303308 msg --- 999973.040  msg/s --- 7.629 Mbit/s  --- Latency: mean: 18.265 ms - med: 14 - 95pct: 53 - 99pct: 98 - 99.9pct: 174 - 99.99pct: 176 - Max: 177
2022-06-27T13:56:50,462+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 461308082 msg --- 999309.458  msg/s --- 7.624 Mbit/s  --- Latency: mean: 14.728 ms - med: 14 - 95pct: 18 - 99pct: 41 - 99.9pct: 50 - 99.99pct: 51 - Max: 52
2022-06-27T13:57:00,475+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 471327606 msg --- 1000738.584  msg/s --- 7.635 Mbit/s  --- Latency: mean: 21.291 ms - med: 16 - 95pct: 52 - 99pct: 61 - 99.9pct: 65 - 99.99pct: 66 - Max: 66

### Modification

- Change internal executor and external executor to normal executor service
- Added a new ScheduledExecutorProvider to handle the scheduled tasks.
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jun 27, 2022
@codelipenghui codelipenghui self-assigned this Jun 27, 2022
@codelipenghui codelipenghui added type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages component/client-java labels Jun 27, 2022
@codelipenghui codelipenghui added this to the 2.11.0 milestone Jun 27, 2022
315157973
315157973 approved these changes Jun 27, 2022
View reviewed changes
Copy link
Contributor

@315157973 315157973 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work

hangc0276
hangc0276 approved these changes Jun 28, 2022
View reviewed changes
Copy link
Contributor

@hangc0276 hangc0276 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job!

@codelipenghui codelipenghui merged commit 96237a9 into apache:master Jun 28, 2022
@codelipenghui codelipenghui deleted the penghui/optimize_internal_external_executor branch June 28, 2022 03:30
michaeljmarshall
michaeljmarshall reviewed Jun 28, 2022
View reviewed changes
Copy link
Member

@michaeljmarshall michaeljmarshall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your thorough analysis @codelipenghui! I just left a minor comment for optimizing brokers with many internal Pulsar clients.

codelipenghui added a commit that referenced this pull request Jun 28, 2022
@codelipenghui
[improve][java-client] Replace ScheduledExecutor to improve performan…
63f5289 
…ce of message consumption (#16236)

### Motivation

The Scheduled Executor doesn't work very efficiently because each task will add to a DelayedQueue(A priority queue) first even if using the `.execute()` method without any schedule delay.

<img width="1845" alt="image" src="https://user-images.githubusercontent.com/12592133/175871343-ecda138f-43a2-472e-ac42-8efdefb58810.png">

<img width="1848" alt="image" src="https://user-images.githubusercontent.com/12592133/175871415-3d8d9fbd-f140-4a4b-a78d-306c1ec9673c.png">

Profile result:
[perf_consumer_0.html.txt](https://github.com/apache/pulsar/files/8989093/perf_consumer_0.html.txt)

Running a performance test for single topic max message read rate test:

```
bin/pulsar-perf consume test -q 1000000 -p 100000000
bin/pulsar-perf produce test -r 1000000 -s 1 -mk random -o 10000 -threads 2
```

Without this PR (2.10.1):

```
Profiling started
2022-06-27T13:44:01,183+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 23919664 msg --- 265702.851  msg/s --- 2.027 Mbit/s  --- Latency: mean: 49430.572 ms - med: 49406 - 95pct: 52853 - 99pct: 53024 - 99.9pct: 53053 - 99.99pct: 53056 - Max: 53057
2022-06-27T13:44:11,196+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 26690802 msg --- 276759.125  msg/s --- 2.112 Mbit/s  --- Latency: mean: 56106.186 ms - med: 56000 - 95pct: 59289 - 99pct: 59985 - 99.9pct: 60037 - 99.99pct: 60042 - Max: 60042
2022-06-27T13:44:21,216+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 28788693 msg --- 209467.861  msg/s --- 1.598 Mbit/s  --- Latency: mean: 63523.143 ms - med: 63580 - 95pct: 67202 - 99pct: 67523 - 99.9pct: 67547 - 99.99pct: 67548 - Max: 67548
2022-06-27T13:44:31,233+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 31255365 msg --- 246190.932  msg/s --- 1.878 Mbit/s  --- Latency: mean: 71152.370 ms - med: 71058 - 95pct: 74555 - 99pct: 74806 - 99.9pct: 74842 - 99.99pct: 74847 - Max: 74847
2022-06-27T13:44:41,247+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 33606630 msg --- 234769.313  msg/s --- 1.791 Mbit/s  --- Latency: mean: 78636.478 ms - med: 78724 - 95pct: 81694 - 99pct: 82090 - 99.9pct: 82279 - 99.99pct: 82285 - Max: 82286
```

With this PR:

```
Profiling started
2022-06-27T13:56:20,426+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 431272207 msg --- 1079360.516  msg/s --- 8.235 Mbit/s  --- Latency: mean: 272.645 ms - med: 334 - 95pct: 470 - 99pct: 510 - 99.9pct: 522 - 99.99pct: 523 - Max: 524
2022-06-27T13:56:30,438+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 441292346 msg --- 1000645.852  msg/s --- 7.634 Mbit/s  --- Latency: mean: 15.512 ms - med: 14 - 95pct: 29 - 99pct: 39 - 99.9pct: 54 - 99.99pct: 55 - Max: 55
2022-06-27T13:56:40,450+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 451303308 msg --- 999973.040  msg/s --- 7.629 Mbit/s  --- Latency: mean: 18.265 ms - med: 14 - 95pct: 53 - 99pct: 98 - 99.9pct: 174 - 99.99pct: 176 - Max: 177
2022-06-27T13:56:50,462+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 461308082 msg --- 999309.458  msg/s --- 7.624 Mbit/s  --- Latency: mean: 14.728 ms - med: 14 - 95pct: 18 - 99pct: 41 - 99.9pct: 50 - 99.99pct: 51 - Max: 52
2022-06-27T13:57:00,475+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 471327606 msg --- 1000738.584  msg/s --- 7.635 Mbit/s  --- Latency: mean: 21.291 ms - med: 16 - 95pct: 52 - 99pct: 61 - 99.9pct: 65 - 99.99pct: 66 - Max: 66
```

Profile result with this PR:

[perf_consumer_1.html.txt](https://github.com/apache/pulsar/files/8989095/perf_consumer_1.html.txt)

### Modification

- Change internal executor and external executor to normal executor service
- Added a new ScheduledExecutorProvider to handle the scheduled tasks.

(cherry picked from commit 96237a9)
codelipenghui added a commit to codelipenghui/incubator-pulsar that referenced this pull request Jul 1, 2022
@codelipenghui
[improve][java-client] Support passing existing scheduled executor pr…
0495207 
…oviders to the client

### Motivation

apache#16236 introduced a new scheduled executor but does not support passing the existing scheduled executor providers like apache#12037.
@codelipenghui codelipenghui mentioned this pull request Jul 1, 2022
Merged
4 tasks
merlimat pushed a commit that referenced this pull request Jul 1, 2022
@codelipenghui
[improve][java-client] Support passing existing scheduled executor pr…
f159f62 
…oviders to the client (#16334)

* [improve][java-client] Support passing existing scheduled executor providers to the client

### Motivation

#16236 introduced a new scheduled executor but does not support passing the existing scheduled executor providers like #12037.

* Apply comment.
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jul 4, 2022
@codelipenghui @nicoloboschi
[improve][java-client] Replace ScheduledExecutor to improve performan…
da72a32 
…ce of message consumption (apache#16236)

### Motivation

The Scheduled Executor doesn't work very efficiently because each task will add to a DelayedQueue(A priority queue) first even if using the `.execute()` method without any schedule delay.

<img width="1845" alt="image" src="https://user-images.githubusercontent.com/12592133/175871343-ecda138f-43a2-472e-ac42-8efdefb58810.png">

<img width="1848" alt="image" src="https://user-images.githubusercontent.com/12592133/175871415-3d8d9fbd-f140-4a4b-a78d-306c1ec9673c.png">

Profile result:
[perf_consumer_0.html.txt](https://github.com/apache/pulsar/files/8989093/perf_consumer_0.html.txt)

Running a performance test for single topic max message read rate test:

```
bin/pulsar-perf consume test -q 1000000 -p 100000000
bin/pulsar-perf produce test -r 1000000 -s 1 -mk random -o 10000 -threads 2
```

Without this PR (2.10.1):

```
Profiling started
2022-06-27T13:44:01,183+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 23919664 msg --- 265702.851  msg/s --- 2.027 Mbit/s  --- Latency: mean: 49430.572 ms - med: 49406 - 95pct: 52853 - 99pct: 53024 - 99.9pct: 53053 - 99.99pct: 53056 - Max: 53057
2022-06-27T13:44:11,196+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 26690802 msg --- 276759.125  msg/s --- 2.112 Mbit/s  --- Latency: mean: 56106.186 ms - med: 56000 - 95pct: 59289 - 99pct: 59985 - 99.9pct: 60037 - 99.99pct: 60042 - Max: 60042
2022-06-27T13:44:21,216+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 28788693 msg --- 209467.861  msg/s --- 1.598 Mbit/s  --- Latency: mean: 63523.143 ms - med: 63580 - 95pct: 67202 - 99pct: 67523 - 99.9pct: 67547 - 99.99pct: 67548 - Max: 67548
2022-06-27T13:44:31,233+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 31255365 msg --- 246190.932  msg/s --- 1.878 Mbit/s  --- Latency: mean: 71152.370 ms - med: 71058 - 95pct: 74555 - 99pct: 74806 - 99.9pct: 74842 - 99.99pct: 74847 - Max: 74847
2022-06-27T13:44:41,247+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 33606630 msg --- 234769.313  msg/s --- 1.791 Mbit/s  --- Latency: mean: 78636.478 ms - med: 78724 - 95pct: 81694 - 99pct: 82090 - 99.9pct: 82279 - 99.99pct: 82285 - Max: 82286
```

With this PR:

```
Profiling started
2022-06-27T13:56:20,426+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 431272207 msg --- 1079360.516  msg/s --- 8.235 Mbit/s  --- Latency: mean: 272.645 ms - med: 334 - 95pct: 470 - 99pct: 510 - 99.9pct: 522 - 99.99pct: 523 - Max: 524
2022-06-27T13:56:30,438+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 441292346 msg --- 1000645.852  msg/s --- 7.634 Mbit/s  --- Latency: mean: 15.512 ms - med: 14 - 95pct: 29 - 99pct: 39 - 99.9pct: 54 - 99.99pct: 55 - Max: 55
2022-06-27T13:56:40,450+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 451303308 msg --- 999973.040  msg/s --- 7.629 Mbit/s  --- Latency: mean: 18.265 ms - med: 14 - 95pct: 53 - 99pct: 98 - 99.9pct: 174 - 99.99pct: 176 - Max: 177
2022-06-27T13:56:50,462+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 461308082 msg --- 999309.458  msg/s --- 7.624 Mbit/s  --- Latency: mean: 14.728 ms - med: 14 - 95pct: 18 - 99pct: 41 - 99.9pct: 50 - 99.99pct: 51 - Max: 52
2022-06-27T13:57:00,475+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 471327606 msg --- 1000738.584  msg/s --- 7.635 Mbit/s  --- Latency: mean: 21.291 ms - med: 16 - 95pct: 52 - 99pct: 61 - 99.9pct: 65 - 99.99pct: 66 - Max: 66
```

Profile result with this PR:

[perf_consumer_1.html.txt](https://github.com/apache/pulsar/files/8989095/perf_consumer_1.html.txt)

### Modification

- Change internal executor and external executor to normal executor service
- Added a new ScheduledExecutorProvider to handle the scheduled tasks.

(cherry picked from commit 96237a9)
(cherry picked from commit 63f5289)
Nicklee007 pushed a commit to Nicklee007/pulsar that referenced this pull request Jul 5, 2022
@codelipenghui
[improve][java-client] Support passing existing scheduled executor pr…
dc328d8 
…oviders to the client (apache#16334)

* [improve][java-client] Support passing existing scheduled executor providers to the client

### Motivation

apache#16236 introduced a new scheduled executor but does not support passing the existing scheduled executor providers like apache#12037.

* Apply comment.
codelipenghui added a commit that referenced this pull request Jul 10, 2022
@codelipenghui
[improve][java-client] Support passing existing scheduled executor pr…
4562d25 
…oviders to the client (#16334)

* [improve][java-client] Support passing existing scheduled executor providers to the client

### Motivation

#16236 introduced a new scheduled executor but does not support passing the existing scheduled executor providers like #12037.

* Apply comment.

(cherry picked from commit f159f62)
nicoloboschi pushed a commit to datastax/pulsar that referenced this pull request Jul 11, 2022
@codelipenghui @nicoloboschi
[improve][java-client] Support passing existing scheduled executor pr…
3f06415 
…oviders to the client (apache#16334)

* [improve][java-client] Support passing existing scheduled executor providers to the client

### Motivation

apache#16236 introduced a new scheduled executor but does not support passing the existing scheduled executor providers like apache#12037.

* Apply comment.

(cherry picked from commit f159f62)
(cherry picked from commit 4562d25)
wuxuanqicn pushed a commit to wuxuanqicn/pulsar that referenced this pull request Jul 14, 2022
@codelipenghui @weimob-wuxuanqi
[improve][java-client] Support passing existing scheduled executor pr…
dc850cc 
…oviders to the client (apache#16334)

* [improve][java-client] Support passing existing scheduled executor providers to the client

### Motivation

apache#16236 introduced a new scheduled executor but does not support passing the existing scheduled executor providers like apache#12037.

* Apply comment.
congbobo184 pushed a commit that referenced this pull request Nov 10, 2022
@codelipenghui
[improve][java-client] Replace ScheduledExecutor to improve performan…
ca76f45 
…ce of message consumption (#16236)

The Scheduled Executor doesn't work very efficiently because each task will add to a DelayedQueue(A priority queue) first even if using the `.execute()` method without any schedule delay.

<img width="1845" alt="image" src="https://user-images.githubusercontent.com/12592133/175871343-ecda138f-43a2-472e-ac42-8efdefb58810.png">

<img width="1848" alt="image" src="https://user-images.githubusercontent.com/12592133/175871415-3d8d9fbd-f140-4a4b-a78d-306c1ec9673c.png">

Profile result:
[perf_consumer_0.html.txt](https://github.com/apache/pulsar/files/8989093/perf_consumer_0.html.txt)

Running a performance test for single topic max message read rate test:

```
bin/pulsar-perf consume test -q 1000000 -p 100000000
bin/pulsar-perf produce test -r 1000000 -s 1 -mk random -o 10000 -threads 2
```

Without this PR (2.10.1):

```
Profiling started
2022-06-27T13:44:01,183+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 23919664 msg --- 265702.851  msg/s --- 2.027 Mbit/s  --- Latency: mean: 49430.572 ms - med: 49406 - 95pct: 52853 - 99pct: 53024 - 99.9pct: 53053 - 99.99pct: 53056 - Max: 53057
2022-06-27T13:44:11,196+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 26690802 msg --- 276759.125  msg/s --- 2.112 Mbit/s  --- Latency: mean: 56106.186 ms - med: 56000 - 95pct: 59289 - 99pct: 59985 - 99.9pct: 60037 - 99.99pct: 60042 - Max: 60042
2022-06-27T13:44:21,216+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 28788693 msg --- 209467.861  msg/s --- 1.598 Mbit/s  --- Latency: mean: 63523.143 ms - med: 63580 - 95pct: 67202 - 99pct: 67523 - 99.9pct: 67547 - 99.99pct: 67548 - Max: 67548
2022-06-27T13:44:31,233+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 31255365 msg --- 246190.932  msg/s --- 1.878 Mbit/s  --- Latency: mean: 71152.370 ms - med: 71058 - 95pct: 74555 - 99pct: 74806 - 99.9pct: 74842 - 99.99pct: 74847 - Max: 74847
2022-06-27T13:44:41,247+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 33606630 msg --- 234769.313  msg/s --- 1.791 Mbit/s  --- Latency: mean: 78636.478 ms - med: 78724 - 95pct: 81694 - 99pct: 82090 - 99.9pct: 82279 - 99.99pct: 82285 - Max: 82286
```

With this PR:

```
Profiling started
2022-06-27T13:56:20,426+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 431272207 msg --- 1079360.516  msg/s --- 8.235 Mbit/s  --- Latency: mean: 272.645 ms - med: 334 - 95pct: 470 - 99pct: 510 - 99.9pct: 522 - 99.99pct: 523 - Max: 524
2022-06-27T13:56:30,438+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 441292346 msg --- 1000645.852  msg/s --- 7.634 Mbit/s  --- Latency: mean: 15.512 ms - med: 14 - 95pct: 29 - 99pct: 39 - 99.9pct: 54 - 99.99pct: 55 - Max: 55
2022-06-27T13:56:40,450+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 451303308 msg --- 999973.040  msg/s --- 7.629 Mbit/s  --- Latency: mean: 18.265 ms - med: 14 - 95pct: 53 - 99pct: 98 - 99.9pct: 174 - 99.99pct: 176 - Max: 177
2022-06-27T13:56:50,462+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 461308082 msg --- 999309.458  msg/s --- 7.624 Mbit/s  --- Latency: mean: 14.728 ms - med: 14 - 95pct: 18 - 99pct: 41 - 99.9pct: 50 - 99.99pct: 51 - Max: 52
2022-06-27T13:57:00,475+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 471327606 msg --- 1000738.584  msg/s --- 7.635 Mbit/s  --- Latency: mean: 21.291 ms - med: 16 - 95pct: 52 - 99pct: 61 - 99.9pct: 65 - 99.99pct: 66 - Max: 66
```

Profile result with this PR:

[perf_consumer_1.html.txt](https://github.com/apache/pulsar/files/8989095/perf_consumer_1.html.txt)

- Change internal executor and external executor to normal executor service
- Added a new ScheduledExecutorProvider to handle the scheduled tasks.

(cherry picked from commit 96237a9)
@congbobo184 congbobo184 added the cherry-picked/branch-2.9 Archived: 2.9 is end of life label Nov 10, 2022
congbobo184 pushed a commit that referenced this pull request Nov 10, 2022
@codelipenghui
[improve][java-client] Support passing existing scheduled executor pr…
a58894a 
…oviders to the client (#16334)

* [improve][java-client] Support passing existing scheduled executor providers to the client

### Motivation

#16236 introduced a new scheduled executor but does not support passing the existing scheduled executor providers like #12037.

* Apply comment.

(cherry picked from commit f159f62)
congbobo184 pushed a commit that referenced this pull request Nov 26, 2022
@codelipenghui
[improve][java-client] Replace ScheduledExecutor to improve performan…
d9211c9 
…ce of message consumption (#16236)

The Scheduled Executor doesn't work very efficiently because each task will add to a DelayedQueue(A priority queue) first even if using the `.execute()` method without any schedule delay.

<img width="1845" alt="image" src="https://user-images.githubusercontent.com/12592133/175871343-ecda138f-43a2-472e-ac42-8efdefb58810.png">

<img width="1848" alt="image" src="https://user-images.githubusercontent.com/12592133/175871415-3d8d9fbd-f140-4a4b-a78d-306c1ec9673c.png">

Profile result:
[perf_consumer_0.html.txt](https://github.com/apache/pulsar/files/8989093/perf_consumer_0.html.txt)

Running a performance test for single topic max message read rate test:

```
bin/pulsar-perf consume test -q 1000000 -p 100000000
bin/pulsar-perf produce test -r 1000000 -s 1 -mk random -o 10000 -threads 2
```

Without this PR (2.10.1):

```
Profiling started
2022-06-27T13:44:01,183+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 23919664 msg --- 265702.851  msg/s --- 2.027 Mbit/s  --- Latency: mean: 49430.572 ms - med: 49406 - 95pct: 52853 - 99pct: 53024 - 99.9pct: 53053 - 99.99pct: 53056 - Max: 53057
2022-06-27T13:44:11,196+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 26690802 msg --- 276759.125  msg/s --- 2.112 Mbit/s  --- Latency: mean: 56106.186 ms - med: 56000 - 95pct: 59289 - 99pct: 59985 - 99.9pct: 60037 - 99.99pct: 60042 - Max: 60042
2022-06-27T13:44:21,216+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 28788693 msg --- 209467.861  msg/s --- 1.598 Mbit/s  --- Latency: mean: 63523.143 ms - med: 63580 - 95pct: 67202 - 99pct: 67523 - 99.9pct: 67547 - 99.99pct: 67548 - Max: 67548
2022-06-27T13:44:31,233+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 31255365 msg --- 246190.932  msg/s --- 1.878 Mbit/s  --- Latency: mean: 71152.370 ms - med: 71058 - 95pct: 74555 - 99pct: 74806 - 99.9pct: 74842 - 99.99pct: 74847 - Max: 74847
2022-06-27T13:44:41,247+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 33606630 msg --- 234769.313  msg/s --- 1.791 Mbit/s  --- Latency: mean: 78636.478 ms - med: 78724 - 95pct: 81694 - 99pct: 82090 - 99.9pct: 82279 - 99.99pct: 82285 - Max: 82286
```

With this PR:

```
Profiling started
2022-06-27T13:56:20,426+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 431272207 msg --- 1079360.516  msg/s --- 8.235 Mbit/s  --- Latency: mean: 272.645 ms - med: 334 - 95pct: 470 - 99pct: 510 - 99.9pct: 522 - 99.99pct: 523 - Max: 524
2022-06-27T13:56:30,438+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 441292346 msg --- 1000645.852  msg/s --- 7.634 Mbit/s  --- Latency: mean: 15.512 ms - med: 14 - 95pct: 29 - 99pct: 39 - 99.9pct: 54 - 99.99pct: 55 - Max: 55
2022-06-27T13:56:40,450+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 451303308 msg --- 999973.040  msg/s --- 7.629 Mbit/s  --- Latency: mean: 18.265 ms - med: 14 - 95pct: 53 - 99pct: 98 - 99.9pct: 174 - 99.99pct: 176 - Max: 177
2022-06-27T13:56:50,462+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 461308082 msg --- 999309.458  msg/s --- 7.624 Mbit/s  --- Latency: mean: 14.728 ms - med: 14 - 95pct: 18 - 99pct: 41 - 99.9pct: 50 - 99.99pct: 51 - Max: 52
2022-06-27T13:57:00,475+0800 [main] INFO  org.apache.pulsar.testclient.PerformanceConsumer - Throughput received: 471327606 msg --- 1000738.584  msg/s --- 7.635 Mbit/s  --- Latency: mean: 21.291 ms - med: 16 - 95pct: 52 - 99pct: 61 - 99.9pct: 65 - 99.99pct: 66 - Max: 66
```

Profile result with this PR:

[perf_consumer_1.html.txt](https://github.com/apache/pulsar/files/8989095/perf_consumer_1.html.txt)

- Change internal executor and external executor to normal executor service
- Added a new ScheduledExecutorProvider to handle the scheduled tasks.

(cherry picked from commit 96237a9)
congbobo184 pushed a commit that referenced this pull request Dec 1, 2022
@codelipenghui
[improve][java-client] Support passing existing scheduled executor pr…
bb4f1e0 
…oviders to the client (#16334)

* [improve][java-client] Support passing existing scheduled executor providers to the client

### Motivation

#16236 introduced a new scheduled executor but does not support passing the existing scheduled executor providers like #12037.

* Apply comment.

(cherry picked from commit f159f62)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@michaeljmarshall michaeljmarshall michaeljmarshall left review comments

@Jason918 Jason918 Jason918 approved these changes

@hangc0276 hangc0276 hangc0276 approved these changes

@Technoboy- Technoboy- Technoboy- approved these changes

@315157973 315157973 315157973 approved these changes

@hezhangjian hezhangjian hezhangjian approved these changes

@RobertIndie RobertIndie RobertIndie approved these changes

@Demogorgon314 Demogorgon314 Demogorgon314 approved these changes

@coderzc coderzc coderzc approved these changes

@shibd shibd shibd approved these changes

@congbobo184 congbobo184 congbobo184 approved these changes

@mattisonchao mattisonchao mattisonchao approved these changes

@eolivelli eolivelli Awaiting requested review from eolivelli

@lhotari lhotari Awaiting requested review from lhotari

@merlimat merlimat Awaiting requested review from merlimat

@BewareMyPower BewareMyPower Awaiting requested review from BewareMyPower

@gaoran10 gaoran10 Awaiting requested review from gaoran10

Assignees

@codelipenghui codelipenghui

Labels

cherry-picked/branch-2.9 Archived: 2.9 is end of life cherry-picked/branch-2.10 doc-not-needed Your PR changes do not impact docs release/2.9.4 release/2.10.2 type/enhancement The enhancements for the existing features or docs. e.g. reduce memory usage of the delayed messages

Projects

None yet

Milestone

2.11.0

Development

Successfully merging this pull request may close these issues.