feat(bottlecap): generate file descriptor and threads enhanced metrics#453
Conversation
alexgallotta
left a comment
alexgallotta
left a comment
There was a problem hiding this comment.
Good work!
Left some comments on style and suggestions, but the only thing to double check is the i32 bits for pids I think
| debug!("File descriptor max data not found in file {}", limits_path); | ||
| break; | ||
| }; | ||
| fd_max = fd_max.min(fd_max_pid); |
There was a problem hiding this comment.
not sure I understand here: the fd_max is the minimum between current and fd_max_pid? Shouldn't it be the max?
There was a problem hiding this comment.
My thought on this was to get the min of all the limits to find the most constraining limit, I'll add a comment about that I see how its confusing
| let threads_max = proc::get_threads_max_data(&pids); | ||
| let mut threads_use = -1_f64; | ||
|
|
||
| let mut interval = interval(Duration::from_millis(1)); |
There was a problem hiding this comment.
why are we doing this operation every millisecond? Isn't it too much? Could you explain why this low threshold?
There was a problem hiding this comment.
It's to catch the fd count more accurately in case a large number of files are opened and closed very quickly, 2ms also gets pretty close
duncanista
merged commit 72181b1
into
jordan.gonzalez/bottlecap/universal-instrumentation
#453) * add fd and threads enhanced metrics * clippy fixes * fixes * rename var
#453) * add fd and threads enhanced metrics * clippy fixes * fixes * rename var
3 participants
What does this PR do?
This PR introduces four new enhanced lambda metrics. These metrics are emitted once per invocation and represent file descriptor and threads data across all running processes.
The two file descriptor metrics are:
aws.lambda.enhanced.fd_max- maximum number of file descriptors that can be opened by the functionaws.lambda.enhanced.fd_use- maximum number of file descriptors open at once over the duration of the functionThe two threads metrics are:
aws.lambda.enhanced.threads_max- maximum number of threads that can be used by the functionaws.lambda.enhanced.threads_use- maximum number of threads in use at once over the duration of the functionDescribe how to test/QA changes
Additional Notes
fd_maxandthreads_maxboth default to send 1024, even if we are not able to read this data, because that is the AWS Lambda limit of max open files and max processesfd_useandthreads_useconsiders all running processes, whereas Lambda Insights only considers the process of the lambda itself, so these metrics will always be higher than the numbers reported by Lambda Insightsfd_usereflects a peak over the course of the request. Suppose 200 file descriptors are opened, then deleted, then 300 are opened, then deleted. The metric will report 300.threads_usealso reflects a peak over the course of the request. Suppose 20 threads are created, then closed, then 30 are created, then closed. The metric will report 30.