METRON-854 create dhcp dump parser #531
Conversation
…ta to pass integration test
basvdl
force-pushed
the
METRON-854_Create_DHCPDump_Parser
branch
from
d7e9dd7 to
16a7c96
Compare
Cleaned up formatting, fixed test and added a timestamp parameter
…t / structure changes
|
Please note that the original DHCPDump format is multi line (http://www.mavetju.org/unix/dhcpdump-man.php). Since this is inconvenient to collect, ship and parse we have modified the DHCPDump to generate single line output. The compatible version of DHCPDump for this parser, is available through my github (https://github.com/basvdl/dhcpdump). This note is also enclosed as comment in the code itself. |
|
Can you edit the title to start with METRON-854? If it doesn't the scripts won't work with jira |
|
Please keep an eye on METRON-777. If that hits first, then I'll help you re-do your parser as a parser extension |
basvdl
changed the title
|
Thnx for the heads-up. METRON-777 is a great improvement! |
Is there no way to handle the original, multi-line format? I think we are limiting the usefulness of this by relying on a modified version of the original source program. |
|
I agree that using the original format is the preferred. Are we able to ship the original multi line format and put the separate lines back together before or during the Metron parsing stage? To give you an idea of the output format of the original DHCPDump available on Ubuntu (https://launchpad.net/ubuntu/+source/dhcpdump/1.8-2). |
|
@basvdl I should have first said, thanks for all your hard work! You probably have already thought through many of these options, so please educate me on their pros/cons. I am going to lay out all of the possibilities that I can think of just so that we don't leave any stone unturned. (1) Alter the Source of Telemetry - This is the option that you chose. This has the downside that we now have a forked, external dependency that we are relying on you to maintain going forward. That's a lot of work for you! (2) Use an Alternative Source of Telemetry - Is there an alternative to (3) Reunite lines at the parser - Each line would come in as a separate Kafka message. We cannot rely on ordering of those messages, so we would need some kind of unique identifier contained within each message to then reunite them all into a single DHCPDump record. This would be very different from our current architecture and is not currently supported. (4) Transport Mechanism - How do you envision getting the DHCPDump records into Kafka? aka the transport mechanism? No matter what, we need some kind of transport mechanism. You could potentially create a solution that takes the separate record lines, stitches them into a single line, then pushes them to Kafka as a complete DHCPDump record. |
|
@nickwallen, these are indeed the options we have discussed...
Another option that just came as a brainwave, maybe we can develop a kind of yaf / yafscii solution. Where you pipe the output of DHCPDump into the stdin of a |
|
As an alternative method for getting DHCP data out of pcap, you might consider the existing Bro sensor, which essentially does what dhcpdump does, but for a wider range of protocols, in a more sophisticated way. We also already have a built in parser. That said it would great to have this parser too for people not looking for the full range of bro. The multi-line aspect may not be an issue. The boundary for Metron is the Kafka message, not really the line, so if you can split the log into multi-line chunks prior to kafka, potentially with something like NiFi based on a delimiter. The way to do this is to use nifi to insert a true delimiter (not end of line) and then use the SplitContent to send individual log entries via kafka. It's a little heavy, but solves the multi-line problem as long as you're not going to crazy levels of throughput e.g. hundreds of thousands of EPS. |
The current Bro parser only supports HTTP and DNS records coming out of Bro. Bro does a lot more indeed, but we don't have parsers for all of that yet. I think enhancing the existing Bro parser to support DHCP records would be an excellent approach. The Bro plugin that we have can be configured to land any of the record types in Kafka, so that would not be a problem. |
|
We also have a I cannot think of any reason it wouldn't work off the top of my head, but I am not sure. Then you wouldn't even have to write any code at all. And also with PR #545 we can now send different Bro record types to their own Kafka topics. I have included an example in the README on how you can do that. I only mention that if that becomes a problem for the |
|
unless of course someone can't use bro for some reason |
|
The Bro parsers is actually pretty generic, and will take whatever json bro dumps out. From a quick inspection you should just need to configure the bro instance to send out dhcp, and in theory it should just work. (I've not tested that though) |
|
I would love to see Metron have a solution for both approaches - ingesting DHCP server logs, as well as DHCP observations based on network traffic. Like @ottobackwards mentioned, not everyone can get the right infrastructure/viewpoint on their network to run something like Bro and get the DHCP traffic to their sensors to be processed. I have definitely sent more than just DNS and HTTP from Bro to Metron and it has been properly ingested, but to date I haven't done DHCP. Like @simonellistonball and @nickwallen mentioned, both the parser and the kafka plugin are setup to handle new bro logs quite well, and a while back I worked on updating Metron's support for more Bro sources via METRON-508. I just never had a chance to test it, so I haven't yet opened a PR. Once #545 and #547 get merged into master, and I'm able to finish METRON-813, I would be happy to work on anything related to Bro and DHCP logs at scale, including finishing up METRON-508. I have two hardware bro environments and my larger one currently sees about 7 million DHCP observations/day and sends ~30,000 messages per second into Metron. |
|
@JonZeolla does this relate to your latest PR (#586)? What is the status of this pr? |
|
I am a -1 on merging this PR. It relies on a forked version of the original sensor which limits its general usefulness. I have concerns about maintaining and supporting that fork long-term. I think there are other more attractive ways to go about this. I would prefer to support both (1) parsing of server-side DHCP logs or (2) using Bro to sniff out the DHCP records. I believe this agrees with the point that @JonZeolla was making earlier in this thread. I could be convinced otherwise should others like to offer counterpoints to the issues I have described. I do hope to see many more contributions from @basvdl . I think PRs, even ones that don't make master, are useful for the community to help define the direction of Metron. I have had my own contributions not make the cut before. |
|
@nickwallen I agree that relying on a modified source is not ideal. However with bro I'm not sure if you have all the functionality people wish for. If i'm correctly informed by the docs, bro will give you the IP and MAC relation, which differs from DHCPDump which captures IP and Hostname relations. Giving context to an IP by adding the Hostname looks more promising... So I would still like to discuss the opportunities of getting the original DHCPDump log format into Metron via NiFi. |
Sure, I think that sounds like another reasonable approach. |
I am a little confused by what you are looking for though. But maybe I am just misunderstanding and need more coffee. The purpose of DHCP is to hand out an IP that typically gets associated with a MAC address. That's the pairing I would be interested in from DHCP. Bro can help with this. If you are looking for IP to hostname, then I would think you would be interested in DNS. Bro can also help with this. |
|
With bro there's also an option to do a lookup and add it into the DHCP log, although we shouldn't assume that will exist by default. That would also require a DNS lookup for each DHCP handshake. That said, I agree that we should probably pursue getting DHCPDump logs into Metron more natively. |
|
dhcp also carries a client-id that is often (but not always and not reliably) the hostname. While not reliable, this is intersting information, especially since you don't have to perform (expensive) DNS lookups, but can consume whats on the wire or a dhcp server reports. |
|
It looks like that's actually considered in bro, but not written to the log. Perhaps there is a trivial way to expose it, I'm not sure offhand. |
|
@nickwallen sometimes we are not able to grep DNS events from the customer server. In these cases we use DHCPDump. I've to admit, Bro is new to me, but it looks promising. If this can fulfill our requirement, It would be a good replacement since it can also grep other protocols as well. @JonZeolla lookups are also not always an option for us. |
|
Huh, look at that. It looks like it is pretty trivial. |
|
Is there enough interest for me to pursue support of this in #586? I could probably throw that together today. |
|
I'd love to see your bro PR expand for this @JonZeolla DHCP is a pretty key source, and Bro is a great way to extract it from taps. Let me know if there is anything I can do to help. |
|
@simonellistonball after some testing we concluded that Bro is not giving the output we want (source: https://bro-tracker.atlassian.net/browse/BIT-1630). The output doesn't contain hostnames, so the relation IP / Hostname can't be made. I still agree on modifying the source, DHCPDump, is not the preferred way to go. Can you assist in how to ship and parse multi-line log events, so I can adjust the parser accordingly without messing with the source? Thanks |
|
So, what is the status of this? Can we at least sum up? |
|
@basvdl Please close this PR unless you believe it is still needed. This PR will be closed per the Metron Development Guidelines on Inactive PRs, unless you provide some reasoning as to why it is stil needed. Thanks! |
|
@nickwallen based on the following pull request (https://github.com/bro/bro/pull/121) Bro has extended their DHCP protocol analyzer. Based on their writing it seems that the same information in DHCPDump (IP / Hostname combination) is also available using the new Bro DHCP analyzer. If that is the case, I think we should go for Bro since we do not need to modify the source of DHCPDump and Bro is already supported within the Metron project. |
|
Agreed @basvdl . I think you are right on with the approach of relying on Bro. Let's close this PR as it definitely is no longer needed then. |
|
@nickwallen agreed! |
|
So we are going to close this? |
|
Just to be clear @basvdl, you are the only one who can close this PR. |
|
Closing PR, since we don't want to change the source code of DHCPdump. Bro should be able to deliver the same functionality. |
7 participants
I've written a DHCPDump parser for Metron. I would like to submit my current version and receive feedback to get it finalized.
The code includes basic Unit and Integration test and covers most of the fields within the DHCPDump log events.