-
Notifications
You must be signed in to change notification settings - Fork 357
Add /proc/net/udp parsing especially for the tx_queue and rx_queue lengths #235
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
pgier
merged 3 commits into
prometheus:master
from
LogMeIn:peterbueschel/add-udp-rx-tx-queue-lengths
Dec 9, 2019
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,227 @@ | ||
| // Copyright 2019 The Prometheus Authors | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // Unless required by applicable law or agreed to in writing, software | ||
| // distributed under the License is distributed on an "AS IS" BASIS, | ||
| // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| // See the License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
| package procfs | ||
|
|
||
| import ( | ||
| "bufio" | ||
| "fmt" | ||
| "io" | ||
| "os" | ||
| "strconv" | ||
| "strings" | ||
| ) | ||
|
|
||
| const ( | ||
| // readLimit is used by io.LimitReader while reading the content of the | ||
| // /proc/net/udp{,6} files. The number of lines inside such a file is dynamic | ||
| // as each line represents a single used socket. | ||
| // In theory, the number of available sockets is 65535 (2^16 - 1) per IP. | ||
| // With e.g. 150 Byte per line and the maximum number of 65535, | ||
| // the reader needs to handle 150 Byte * 65535 =~ 10 MB for a single IP. | ||
| readLimit = 4294967296 // Byte -> 4 GiB | ||
| ) | ||
|
|
||
| type ( | ||
| // NetUDP represents the contents of /proc/net/udp{,6} file without the header. | ||
| NetUDP []*netUDPLine | ||
|
|
||
| // NetUDPSummary provides already computed values like the total queue lengths or | ||
| // the total number of used sockets. In contrast to NetUDP it does not collect | ||
| // the parsed lines into a slice. | ||
| NetUDPSummary struct { | ||
| // TxQueueLength shows the total queue length of all parsed tx_queue lengths. | ||
| TxQueueLength uint64 | ||
| // RxQueueLength shows the total queue length of all parsed rx_queue lengths. | ||
| RxQueueLength uint64 | ||
| // UsedSockets shows the total number of parsed lines representing the | ||
| // number of used sockets. | ||
| UsedSockets uint64 | ||
| } | ||
|
|
||
| // netUDPLine represents the fields parsed from a single line | ||
| // in /proc/net/udp{,6}. Fields which are not used by UDP are skipped. | ||
| // For the proc file format details, see https://linux.die.net/man/5/proc. | ||
| netUDPLine struct { | ||
| Sl uint64 | ||
| LocalAddr uint64 | ||
| LocalPort uint64 | ||
| RemAddr uint64 | ||
| RemPort uint64 | ||
| St uint64 | ||
| TxQueue uint64 | ||
| RxQueue uint64 | ||
| UID uint64 | ||
| } | ||
| ) | ||
|
|
||
| // NetUDP returns the IPv4 kernel/networking statistics for UDP datagrams | ||
| // read from /proc/net/udp. | ||
| func (fs FS) NetUDP() (NetUDP, error) { | ||
| return newNetUDP(fs.proc.Path("net/udp")) | ||
| } | ||
|
|
||
| // NetUDP6 returns the IPv6 kernel/networking statistics for UDP datagrams | ||
| // read from /proc/net/udp6. | ||
| func (fs FS) NetUDP6() (NetUDP, error) { | ||
| return newNetUDP(fs.proc.Path("net/udp6")) | ||
| } | ||
|
|
||
| // NetUDPSummary returns already computed statistics like the total queue lengths | ||
| // for UDP datagrams read from /proc/net/udp. | ||
| func (fs FS) NetUDPSummary() (*NetUDPSummary, error) { | ||
| return newNetUDPSummary(fs.proc.Path("net/udp")) | ||
| } | ||
|
|
||
| // NetUDP6Summary returns already computed statistics like the total queue lengths | ||
| // for UDP datagrams read from /proc/net/udp6. | ||
| func (fs FS) NetUDP6Summary() (*NetUDPSummary, error) { | ||
| return newNetUDPSummary(fs.proc.Path("net/udp6")) | ||
| } | ||
|
|
||
| // newNetUDP creates a new NetUDP{,6} from the contents of the given file. | ||
| func newNetUDP(file string) (NetUDP, error) { | ||
| f, err := os.Open(file) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| defer f.Close() | ||
|
|
||
| netUDP := NetUDP{} | ||
|
|
||
| lr := io.LimitReader(f, readLimit) | ||
| s := bufio.NewScanner(lr) | ||
| s.Scan() // skip first line with headers | ||
| for s.Scan() { | ||
| fields := strings.Fields(s.Text()) | ||
| line, err := parseNetUDPLine(fields) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| netUDP = append(netUDP, line) | ||
| } | ||
| if err := s.Err(); err != nil { | ||
| return nil, err | ||
| } | ||
| return netUDP, nil | ||
| } | ||
|
|
||
| // newNetUDPSummary creates a new NetUDP{,6} from the contents of the given file. | ||
| func newNetUDPSummary(file string) (*NetUDPSummary, error) { | ||
| f, err := os.Open(file) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| defer f.Close() | ||
|
|
||
| netUDPSummary := &NetUDPSummary{} | ||
|
|
||
| lr := io.LimitReader(f, readLimit) | ||
| s := bufio.NewScanner(lr) | ||
| s.Scan() // skip first line with headers | ||
| for s.Scan() { | ||
| fields := strings.Fields(s.Text()) | ||
| line, err := parseNetUDPLine(fields) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| netUDPSummary.TxQueueLength += line.TxQueue | ||
| netUDPSummary.RxQueueLength += line.RxQueue | ||
| netUDPSummary.UsedSockets++ | ||
| } | ||
| if err := s.Err(); err != nil { | ||
| return nil, err | ||
| } | ||
| return netUDPSummary, nil | ||
| } | ||
|
|
||
| // parseNetUDPLine parses a single line, represented by a list of fields. | ||
| func parseNetUDPLine(fields []string) (*netUDPLine, error) { | ||
| line := &netUDPLine{} | ||
| if len(fields) < 8 { | ||
| return nil, fmt.Errorf( | ||
| "cannot parse net udp socket line as it has less then 8 columns: %s", | ||
| strings.Join(fields, " "), | ||
| ) | ||
| } | ||
| var err error // parse error | ||
|
|
||
| // sl | ||
| s := strings.Split(fields[0], ":") | ||
| if len(s) != 2 { | ||
| return nil, fmt.Errorf( | ||
| "cannot parse sl field in udp socket line: %s", fields[0]) | ||
| } | ||
|
|
||
| if line.Sl, err = strconv.ParseUint(s[0], 0, 64); err != nil { | ||
| return nil, fmt.Errorf("cannot parse sl value in udp socket line: %s", err) | ||
| } | ||
| // local_address | ||
| l := strings.Split(fields[1], ":") | ||
| if len(l) != 2 { | ||
| return nil, fmt.Errorf( | ||
| "cannot parse local_address field in udp socket line: %s", fields[1]) | ||
| } | ||
| if line.LocalAddr, err = strconv.ParseUint(l[0], 16, 64); err != nil { | ||
| return nil, fmt.Errorf( | ||
| "cannot parse local_address value in udp socket line: %s", err) | ||
| } | ||
| if line.LocalPort, err = strconv.ParseUint(l[1], 16, 64); err != nil { | ||
| return nil, fmt.Errorf( | ||
| "cannot parse local_address port value in udp socket line: %s", err) | ||
| } | ||
|
|
||
| // remote_address | ||
| r := strings.Split(fields[2], ":") | ||
| if len(r) != 2 { | ||
| return nil, fmt.Errorf( | ||
| "cannot parse rem_address field in udp socket line: %s", fields[1]) | ||
| } | ||
| if line.RemAddr, err = strconv.ParseUint(r[0], 16, 64); err != nil { | ||
| return nil, fmt.Errorf( | ||
| "cannot parse rem_address value in udp socket line: %s", err) | ||
| } | ||
| if line.RemPort, err = strconv.ParseUint(r[1], 16, 64); err != nil { | ||
| return nil, fmt.Errorf( | ||
| "cannot parse rem_address port value in udp socket line: %s", err) | ||
| } | ||
|
|
||
| // st | ||
| if line.St, err = strconv.ParseUint(fields[3], 16, 64); err != nil { | ||
| return nil, fmt.Errorf( | ||
| "cannot parse st value in udp socket line: %s", err) | ||
| } | ||
|
|
||
| // tx_queue and rx_queue | ||
| q := strings.Split(fields[4], ":") | ||
| if len(q) != 2 { | ||
| return nil, fmt.Errorf( | ||
| "cannot parse tx/rx queues in udp socket line as it has a missing colon: %s", | ||
| fields[4], | ||
| ) | ||
| } | ||
| if line.TxQueue, err = strconv.ParseUint(q[0], 16, 64); err != nil { | ||
| return nil, fmt.Errorf("cannot parse tx_queue value in udp socket line: %s", err) | ||
| } | ||
| if line.RxQueue, err = strconv.ParseUint(q[1], 16, 64); err != nil { | ||
| return nil, fmt.Errorf("cannot parse rx_queue value in udp socket line: %s", err) | ||
| } | ||
|
|
||
| // uid | ||
| if line.UID, err = strconv.ParseUint(fields[7], 0, 64); err != nil { | ||
| return nil, fmt.Errorf( | ||
| "cannot parse uid value in udp socket line: %s", err) | ||
| } | ||
|
|
||
| return line, nil | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We're trying to make use of the procfs
utli.ReadFileNoStat(), in order to read this data more atomically from the kernel.But, look at this file, it seems like it could be very big in a real-world environment. Do you have any data on what size this file can grow to?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have a hard limit on our hosts of 1300 concurrent connections. I measured locally that one line is about 150 Byte. So, 1300x150 Byte = 19.5 KByte =~ 19 Kibibyte.
If we play a bit with the numbers and say we will use the number of possible ports (and forget about other limits like open files), which is limited to 16 bits for the source port field in a UDPv4 header, we could have 2^16 = 65536 lines for a server listening on a single IP.
--> So, with 150 Byte per line we theoretically have a filesize of 65536 * 150 Byte ~ 10 MByte
But lets have a look at some numbers available at the internet:
Means, we have a filesize range from more realistic KBytes to unusual GBytes. As an engineer I would choose the middle and estimate a maximum of multiple MByte per IP.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the followup, good stuff.
I guess we should stick to this streaming reader for large file safety. What do you think @mdlayher?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's most reasonable to stick to a streaming read but would also encourage using an io.LimitReader when initializing the scanner and set some sane upper bound like 4GiB of file.