[Feature] [Java] Support collect thread pool metrics

[xu1009]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

thread pool monitor is important in web service, server container, like tomcat, undertow etc, monitor its worker pool is very import, every project needs implement it

Use case

when use skywalking, we can config target class extends ThreadPoolExecutor, get thread pool metric, like coresize, maxsize, poolsize, queue size..., and send this metric to oap service every second, from ui, can see the metric

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[wu-sheng]

Then we need a way to build a labels(pool name) metric for connection pool. What do you want to build?
Could you provide more details?

[xu1009]

now, in my projeject, agent core has a model contains pool name and thead threadPoolExecutor instance, the agent plugin set the thread pool name and instance to it on constructor, and the core get message from the model, and send metrics to oap service

[wu-sheng]

Could you share what do the OAL script, the new part of proto, and UI look like?

[xu1009]

instance_undertow_thread_pool_core_size = from(ServiceInstanceJVMThreadPool.corePoolSize).filter(threadPoolNameType == ThreadPoolNameType.UNDERTOW).longAvg();
instance_undertow_thread_pool_max_size = from(ServiceInstanceJVMThreadPool.maximumPoolSize).filter(threadPoolNameType == ThreadPoolNameType.UNDERTOW).longAvg();
instance_undertow_thread_pool_size = from(ServiceInstanceJVMThreadPool.poolSize).filter(threadPoolNameType == ThreadPoolNameType.UNDERTOW).longAvg();
instance_undertow_thread_pool_queue_size = from(ServiceInstanceJVMThreadPool.queueSize).filter(threadPoolNameType == ThreadPoolNameType.UNDERTOW).longAvg();
instance_undertow_thread_pool_active_count = from(ServiceInstanceJVMThreadPool.activeCount).filter(threadPoolNameType == ThreadPoolNameType.UNDERTOW).longAvg();

message ThreadPool {
ThreadPoolNameType type = 1;
int64 corePoolSize = 2;
int64 maximumPoolSize = 3;
int64 poolSize = 4;
int64 queueSize = 5;
int64 activeCount = 6;

}

enum ThreadPoolNameType {
UNDERTOW_WORKER_POOL = 0;
TOMCAT_WORKER_POOL = 1;
}

[wu-sheng]

This kind of filter function is very hard coded, and it has to set for all kinds of pools, which is not flexible, and have to set a lot of UI components to list all possibilities.
I am thinking whether we should treat this as a logic endpoint? And metrics reporting through meter API rather than JVM and OAL analysis.

[xu1009]

emm, the agent plugin and oap server is easy to add new thread pool, the problem is how to display it in ui, i am not familiar with what you say, this has some demo in skywalking project，or, we can make basic plugin to collect metirc and save it, every one can customizing their dashboard ui

[wu-sheng]

For agent side, it is very similar to agent tracing plugin. Just calling different core APIs. Check this.

https://skywalking.apache.org/docs/skywalking-java/latest/en/setup/service-agent/java-agent/java-plugin-development-guide/#meter-plugin

[wu-sheng]

At the backend, you just need to learn meter receiver and MAL

1. https://skywalking.apache.org/docs/main/latest/en/setup/backend/backend-meter/#manual-meter-api
2. Meter design, https://skywalking.apache.org/docs/main/latest/en/concepts-and-designs/meter/
3. MAL, https://skywalking.apache.org/docs/main/latest/en/concepts-and-designs/mal/

And learn fron slueth MAL setup

1. https://skywalking.apache.org/docs/main/latest/en/setup/backend/spring-sleuth-setup/
2. Configuration file example, https://github.com/apache/skywalking/blob/fc27f766c5da0f4165152025a6ba4080d034d1fc/oap-server/server-starter/src/main/resources/meter-analyzer-config/spring-sleuth.yaml

[xu1009]

ok, i have a look