The utility is designed to receive data stream from devices using the IPFIX protocol and save the data as a file for subsequent processing by other means.
rpm --import http://vasexperts.ru/centos/RPM-GPG-KEY-vasexperts.ru rpm -Uvh http://vasexperts.ru/centos/6/x86_64/vasexperts-repo-1-0.noarch.rpm
yum install -y ipfixreceiver
rpm --import http://vasexperts.ru/centos/RPM-GPG-KEY-vasexperts.ru rpm -Uvh http://vasexperts.ru/centos/6/x86_64/vasexperts-repo-1-0.noarch.rpm
yum -y install epel-release
rpm --import https://forensics.cert.org/forensics.asc rpm -Uvh https://forensics.cert.org/cert-forensics-tools-release-el7.rpm
yum -y install libfixbuf --disablerepo=forensics yum -y install netsa-python netsa_silk yum -y install ipfixreceiver --disablerepo=forensics
decodeipv4
, decodeipv6 in the export model, for example:source_ip4, ''decodeipv4''
destination_ip4, decodeipv4
buffer_size
parameter is added; it specifies the size of the i/o buffer between the process of receiving and writing to a file, it is used in the [dump] section, the default value of the parameter is 100000 records (it is focused on 20 Gbit traffic or 25 000 sessions per second). If the number of sessions per second is considerably less than the mentioned value, then you should to change this parameter proportionally./etc/dpiui/ipfixreceiver.conf is the clickstream configuration sample (http requests) /etc/dpiui/ipfixreceiverflow.conf is the sample configuration for information on sessions (netflow counterpart) /etc/dpiui/ipfixreceiversip.conf is the sample configuration for information on sip connections
/usr/local/lib/ipfixreceiver.d/
directory
/etc/dpiui/port_proto.txt contains the information on the translation of protocol identifier to its string representation, it is used by the utility to get the protocol text-based name by its identifier
/usr/local/bin/ipfixreceiver -> link to the /usr/local/lib/ipfixreceiver.d/ipfixreceiver
[connect] protocol=tcp host=212.12.11.10 port=1500
To receive an IPFIX stream, you should have the following rule in the /etc/sysconfig/iptables:
-A INPUT -p tcp -m state --state NEW -m tcp --dport 1500 -j ACCEPT
Do not forget that after the adding rule to iptables a restart is required:
service iptables restart
/var/log/dpiui*.log { rotate 5 missingok notifempty compress size 10M daily copytruncate nocreate postrotate endscript }
Please pay attention that the copytruncate method is used, otherwise the file will be recreated and writing the log by the process will stop.
Respectively, in the ipfixreceiver configuration file, you have the following settings in the [handler_ipfixreceiverlogger] section:
args=('/var/log/dpiuiflow.log', 'a+')
15 4 * * * /bin/find /var/dump/dpiui/ -name url_\*.dump.gz -cmin +44640 -delete > /dev/null 2>&1
Change the line according to your requirements and add to the /var/spool/cron/root file.
The ipfixreceiver utility has the following startup options:
usage: ipfixreceiver start|stop|restart|status|-v [-f <config file>] где start - start as a service stop - service stop state - get the service state restart - service restart -v - show version info -f <config file> - specify the configuration file for the service to start Example: ipfixreceiver start -f /etc/dpiui/ipfixreceiverflow.conf
The default configuration file is /etc/dpiui/ipfixreceiver.conf.
:!:More information on configuring logging can be found here: Logging
CRITICAL - only critical errors, minimum message level ERROR - including errors WARNING - including warnings INFO - including information DEBUG - including debug messages NOTSET - all, the maximum level of messages (including all of the above)
Example:
level=DEBUG
handlers=ipfixreceiverlogger
class=FileHandler
level=DEBUG
formatter=ipfixreceiverlogger
args=('/var/log/dpiuiflow.log', 'a+')
format=%(asctime)s - %(name)s - %(levelname)s - %(message)s here %(name)s - log name %(levelname)s - message level ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'). %(asctime)s - date, the default format is “2003-07-08 16:49:45,896” (the comma field corresponds to milliseconds). %(message)s - message
datefmt='%m-%d %H:%M'
protocol=udp
host=localhost
port=9996
rotate_minutes=10
processcmd=gzip %%s
dumpfiledir=/var/dump/dpiui/ipfixflow/
The block specifies the data received via the IPFIX protocol.
InfoElements = octetDeltaCount, 0, 1, UINT64, True packetDeltaCount, 0, 2, UINT64, True protocolIdentifier, 0, 3, UINT8 session_id, 43823, 2000, UINT64, True here, session_id - is the name of the field from the IPFIX description, for more detail see corresponding sections 43823 - unique organization number (enterprise number) 1 - unique field number UINT64 - field type True - use reverse byte order (endian). Possible values are: True or empty.
Field types:
Type | Length | Type IPFIX |
---|---|---|
OCTET_ARRAY | VARLEN | octetArray |
UINT8 | 1 | unsigned8 |
UINT16 | 2 | unsigned16 |
UINT32 | 4 | unsigned32 |
UINT64 | 8 | unsigned64 |
INT8 | 1 | signed8 |
INT16 | 2 | signed16 |
INT32 | 4 | signed32 |
INT64 | 8 | signed64 |
FLOAT32 | 4 | float32 |
FLOAT64 | 8 | float64 |
BOOL | 1 | boolean |
MAC_ADDR | 6 | macAddress |
STRING | VARLEN | string |
SECONDS | 4 | dateTimeSeconds |
MILLISECONDS | 8 | dateTimeMilliseconds |
MICROSECONDS | 8 | dateTimeMicroseconds |
NANOSECONDS | 8 | dateTimeNanoseconds |
IP4ADDR | 4 | ipv4Address |
IP6ADDR | 16 | ipv6Address |
The field names and their description can be accessed from the following links:
Additional information:
Information Model for IP Flow Information Export
specifies the model parameters used for export, is reserved for future use.
Mode = File
Description of the File export model.
Delimiter = \t
ExportElements = timestamp, seconds, %%Y-%%m-%%d %%H:%%M:%%S.000+03 login source_ip4 destination_ip4 host, decodehost path, decodepath referal, decodereferer session_id where the fields in each row are the following: name - the field name from the information model [InfoModel] (login, session_id and etc.) handler - field processing procedure before output seconds - field in seconds, format is expected milliseconds - field in milliseconds, microseconds, nanoseconds format is expected decodehost - recode from punycode to UTF-8 decodepath - recode from urlencoding to UTF-8 decodereferer - recode from (punycode,urlencoding) to UTF-8 decodeproto - recode the protocol identifier to the string format - format description for seconds, milliseconds. Example: %%Y-%%m-%%d %%H:%%M:%%S.%%f+0300 Result: 2016-05-25 13:13:35.621000+0300
Step-by-step creation of service in Centos 7, here the service name is ipfix1 , its configuration is in the /etc/dpiui/ipfixreceiver.conf file, listening port is 1500.
Create the /etc/systemd/system/ipfix1.service file as follows:
[Unit] Description=ipfix test restart After=network.target After=syslog.target [Service] Type=forking PIDFile=/tmp/ipfixreceiver.1500.pid ExecStart=/usr/local/bin/ipfixreceiver start -f /etc/dpiui/ipfixreceiver.conf ExecStop=/usr/local/bin/ipfixreceiver stop -f /etc/dpiui/ipfixreceiver.conf ExecReload=/usr/local/bin/ipfixreceiver restart -f /etc/dpiui/ipfixreceiver.conf Restart=always RestartSec=10s [Install] WantedBy=multi-user.target
Issue the following commands:
systemctl enable ipfix1.service systemctl start ipfix1.service systemctl daemon-reload
Check whether the service is running:
systemctl status ipfix1.service -l
Do not forget to check the service status after rebooting!
ipfixreceiver -v
yum info ipfixreceiver
netstat -nlp | grep 1500
b) check the log for errors
c) check that the writing to the temporary file occurs, for example for port 9996 (directory for dump files: /var/dump/dpiui/ipfixurl):
tail -f /var/dump/dpiui/ipfixurl/9996.url.dump
buffer_size
when number of sessions per second is more than 30k along with the following item d)