IPFIXcol Quick Start Guide

What is IPFIXcol

IPFIXcol is a flexible IPFIX and NetFlow data collector designed to be easily extensible by plugins. The collector uses input, intermediate and output plugins to receive, modify and store data. For more information see the IPFIXcol Github page.

For installation steps see SW prerequisities.

Tip: If you are not going to use SCTP protocol (alternative for TCP and UDP) for receiving IPFIX packets, add switch --disable-sctp to ./configure during build of "IPFIXcol base".

##Configuration

IPFIXcol stores its configurations in the /etc/ipfixcol/ directory (or elsewhere depending on build parameters).

• ipfix-elements.xml contains a description of the known IPFIX Information Elements assigned by IANA and enterprise-specific Information Elements. Feel free to add here your elements.

• internalcfg.xml contains configuration of plugins (name, path, etc.) used in startup.xml. Can be viewed/edited with ipfixconf tool.

• startup.xml (the most important) describes how IPFIXcol is configured at startup, which plugins are used and, for example, where flows will be stored. The XML is self-documented, so read the elements description carefully. Several configuration examples are given in the "First start" section.

Every "startup" configuration consists at least of one collecting and one exporting process.

• <collectingProcess> specifies an input plugin (TCP/UDP/...).
• <exportingProcess> specifies output plugin(s).

Section with intermediate plugins (<intermediatePlugins>) is optional. The names and pages of all supported plugins are available in IPFIXcol Plugins.

Most of the IPFIXcol plugins have manual pages that closely describe their available parameters and include example configuration for plugins themselves. Usual name of a manual page is: ipfixcol-<name of plugin>-<type of plugin>. For example (manual page of JSON storage plugin): man ipfixcol-json-output

If manual page is not available you can use README file in plugin's source code directory that should include same info.

Recommendation

One of the most common causes of lost flow records transmitted over networks using UDP protocol is an undersized receive buffer of the socket. The actual size of the buffer the operating system provides is limited by operating system-level maximums.

We strongly recommend to increase the limits before first start of the collector. For the most Linux distributions, you can use for example following commands (values are in bytes):

sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.rmem_default=26214400

First start

In the file "/etc/ipficol/startup.xml" is already prepared a complex configuration, but let's start with something really simple. For example, say you want to store all flows received by the collector over UDP or TCP protocol into JSON format. This can be very useful for analysis what is in the records (an identification and a content of IPFIX elements) from your exporters.

You have to create a new configuration with one input plugin (TCP/UDP/SCTP) and one storage plugin (json). Example configuration is available here for UDP and TCP.

<ipfix xmlns="...">
	<!--## Collecting process configuration -->
	<collectingProcess>
		<name>TCP collector</name>
		<tcpCollector>
			...
		</tcpCollector>
		<exportingProcess>File writer</exportingProcess>
	</collectingProcess>

    <!--## Exporting process configuration -->
	<exportingProcess>
		<name>File writer</name>
		<destination>
			<name>JSON storage plugin</name>
			<fileWriter>
				<fileFormat>json</fileFormat>
				...
			</fileWriter>
		</destination>

		<singleManager>yes</singleManager>
	</exportingProcess>
</ipfix>

As you can see, both files have the node <collectingProcess> with a configuration of TCP input <tcpCollector> or UDP input <udpCollector>. On the same level as the <collectingProcess> is also <exportingProcess> with the configuration of JSON output plugin (type of the plugin is defined by the node <fileFormat> inside <fileWriter>). Various options define the transformation process of IPFIX records to an output format. JSON records will be stored into files to the directory /tmp/ipfixcol/flow/%Y/%m/%d/ where /%Y/%m/%d/ is replaced with the current system time. The output file is automatically rotated and renamed every N minutes, in this case 5 minutes.

Note that the node <exportingProcess> inside the <collectingProcess> MUST match with the <name> inside the node <exportingProcess> which is on the same level as <collectingProcess>.

To start collector with one of the configurations above use:

ipfixcol -c <file> # replace <file> with tcp2json.xml or udp2json.xml

Tip: By default, IPFIXcol shows only "error" messages. To show also warnings add switch -v 1. The higher the number, the more information is printed. Maximum verbosity level is 3 (debug).

When you are satisfied with the configuration, you can optionally replace the content of the file "/etc/ipfixcol/startup.xml" and just start the collector:

ipfixcol

Warning: Keep in mind, when using UDP input plugin, IPFIX templates (description of the content from an exporter) are distributed by an exporter (i.e. probe) at regular intervals. The collector may not be able to interpret the data immediately after start for a certain amount of time (sometimes more than 10 minutes, depending on a configuration of the exporter). During this time no JSON files are created. This issue does not apply to communications over TCP and SCTP.

We prepared a file with few anonymized IPFIX flows, so you can try the IPFIXcol configurations, even without running your flow exporter. Just download the file and use ipfixsend tool (installed together with IPFIXcol). For example, to send flow records over UDP protocol with realtime simulation use:

ipfixsend -d 127.0.0.1 -p 4739 -t UDP -i example_flows.ipfix -n 1 -R 1.0

where:

• -d 127.0.0.1 - destination IP address of the collector
• -p 4739 - destination port
• -t UDP - transmission protocol
• -i example_flows.ipfix - input file
• -n 1 - how many times the file should be sent
• -R 1.0 - real-time sending mode

When all records are sent, the ipfixsend will terminate (for all options see ipfixsend -h). In this case, the problem with UDP templates does not apply, because the tool sends templates immediately after the start.

After a while, it is time to analyze JSON files. To transform any file into more human readable form you can use any JSON formatter or this simple script:

cat <json_file> | python json_pretty.py > <output_file>   # (Python 2.7)

If you can see one or more elements with name "eXXidYY" where XX is an Enterprise number and YY is an Element ID, it means the collector doesn't known name and semantic of these elements. Because this information is not part of IPFIX templates, you have to add descriptions into the IPFIXcol configuration file ipfix-elements.xml. The elements with the Enterprise number "0" are defined by IANA here and others are defined by private organizations (list of Enterprise IDs and maintainers here)

---

Maybe you have noticed that <singleManager>yes</singleManager> is always used in all configurations listed above. It prevents the collector from creating more instances of output plugins. If the option is set to no, then for each Observation Domain ID (ODID) are created separated instances of output plugins. In this project it can cause unsuitable collisions and for this reason, all configurations are using "single manager" enabled.