====== Receiving IPFIX data by ipfixreceiver ====== ===== Introduction ===== The utility is designed to receive data stream from devices using the IPFIX protocol and save the data as a file for subsequent processing by other means. ===== Installation and Update ===== ==== CentOS6 ==== - add the VAS Experts repository rpm --import http://vasexperts.ru/centos/RPM-GPG-KEY-vasexperts.ru rpm -Uvh http://vasexperts.ru/centos/6/x86_64/vasexperts-repo-1-0.noarch.rpm - instal the ipfixreceiver:\\ yum install -y ipfixreceiver - check for the changes in the configuration files so they to be consistent with ipfixreceiver current version, see the "Important changes" section. ==== CentOS7 ==== - add the VAS Experts repository rpm --import http://vasexperts.ru/centos/RPM-GPG-KEY-vasexperts.ru rpm -Uvh http://vasexperts.ru/centos/6/x86_64/vasexperts-repo-1-0.noarch.rpm - install the epel repository yum -y install epel-release - install the forencis repository: rpm --import https://forensics.cert.org/forensics.asc rpm -Uvh https://forensics.cert.org/cert-forensics-tools-release-el7.rpm - install the ipfixreceiver:\\ yum -y install libfixbuf --disablerepo=forensics yum -y install netsa-python netsa_silk yum -y install ipfixreceiver --disablerepo=forensics - check for the changes in the configuration files so they to be consistent with ipfixreceiver current version, see the "Important changes" section. ===== Important changes in version 1.0.3 relative to 1.0.2 one ===== - the configuration file has been changed with respect to IP address translation, starting from the 1.0.3 version you should specify ''decodeipv4'', decodeipv6 in the export model, for example: source_ip4, ''decodeipv4'' destination_ip4, decodeipv4 - the process of information saving has been allocated to a separate process; remember that when dealing with a large number of sessions (> 25k sessions per second) the process will completely load 2 processor cores. In order to check that the process has time to process the entire data stream the following messages are added in the DEBUG mode:\\ (a)cnt=NNNNN - the buffer has been sent with the given number\\ (b)cnt=YYYYY - the buffer with the given number is saved. - a new ''buffer_size'' parameter is added; it specifies the size of the i/o buffer between the process of receiving and writing to a file, it is used in the [dump] section, the default value of the parameter is 100000 records (it is focused on 20 Gbit traffic or 25 000 sessions per second). If the number of sessions per second is considerably less than the mentioned value, then you should to change this parameter proportionally. ===== The files supplied with the ipfixreceiver ===== - configuration examples:\\ /etc/dpiui/ipfixreceiver.conf is the clickstream configuration sample (http requests) /etc/dpiui/ipfixreceiverflow.conf is the sample configuration for information on sessions (netflow counterpart) /etc/dpiui/ipfixreceiversip.conf is the sample configuration for information on sip connections - program files are located under the:\\ /usr/local/lib/ipfixreceiver.d/ directory - auxiliary files:\\ /etc/dpiui/port_proto.txt contains the information on the translation of protocol identifier to its string representation, it is used by the utility to get the protocol text-based name by its identifier - links to the executables:\\ /usr/local/bin/ipfixreceiver -> link to the /usr/local/lib/ipfixreceiver.d/ipfixreceiver ===== Additional OS settings ===== - configure iptables to accept external data\\ For ipfixreceiver to work properly, you should open the ports that will also be used in the [connect] section of the configuration. For example, you use the TCP protocol, port 1500 and IP=212.12.11.10\\ [connect] protocol=tcp host=212.12.11.10 port=1500\\ To receive an IPFIX stream, you should have the following rule in the /etc/sysconfig/iptables:\\ -A INPUT -p tcp -m state --state NEW -m tcp --dport 1500 -j ACCEPT Do not forget that after the adding rule to iptables a restart is required:\\ service iptables restart - configure log rotation\\ An example of rotation for the /var/log/dpiuiflow.log log file: you should create within the /etc/logrotate.d/ directory the flowlog file of the following content: /var/log/dpiui*.log { rotate 5 missingok notifempty compress size 10M daily copytruncate nocreate postrotate endscript } \\ Please pay attention that the copytruncate method is used, otherwise the file will be recreated and writing the log by the process will stop.\\ Respectively, in the ipfixreceiver configuration file, you have the following settings in the [handler_ipfixreceiverlogger] section: \\ args=('/var/log/dpiuiflow.log', 'a+') - Configure the deleting of old files. For example, deleting old archives (more than 31 days) containing sessions records packed with gzip:\\ 15 4 * * * /bin/find /var/dump/dpiui/ -name url_\*.dump.gz -cmin +44640 -delete > /dev/null 2>&1\\ Change the line according to your requirements and add to the /var/spool/cron/root file. ===== Ipfixreceiver startup options ===== The ipfixreceiver utility has the following startup options: usage: ipfixreceiver start|stop|restart|status|-v [-f ] где start - start as a service stop - service stop state - get the service state restart - service restart -v - show version info -f - specify the configuration file for the service to start Example: ipfixreceiver start -f /etc/dpiui/ipfixreceiverflow.conf ===== Configuration ===== The default configuration file is /etc/dpiui/ipfixreceiver.conf.\\ :!:More information on configuring logging can be found here: [[https://docs.python.org/2.6/library/logging.html | Logging ]] ==== Service sections ==== - loggers - specifies the log identifiers used - handlers - specifies the handlers used to save the log - formatters - specifies the formats used for the log ==== logger_root ==== - level - specifies the logging level (upper level)\\ Possible values are: CRITICAL - only critical errors, minimum message level ERROR - including errors WARNING - including warnings INFO - including information DEBUG - including debug messages NOTSET - all, the maximum level of messages (including all of the above) \\ Example:\\ level=DEBUG - handlers - message handlers used\\ Example: handlers=ipfixreceiverlogger ==== handler_ipfixreceiverlogger ==== - class - handler class\\ Example: class=FileHandler - level - message level level=DEBUG - formatter - message format nameformatter=ipfixreceiverlogger - args - handler parameters args=('/var/log/dpiuiflow.log', 'a+') ==== formatter_ipfixreceiverlogger ==== - format - message format description\\ Example: format=%(asctime)s - %(name)s - %(levelname)s - %(message)s here %(name)s - log name %(levelname)s - message level ('DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'). %(asctime)s - date, the default format is “2003-07-08 16:49:45,896” (the comma field corresponds to milliseconds). %(message)s - message - datefmt - date format description\\ Example: datefmt='%m-%d %H:%M' ==== connect ==== - protocol - protocl (tcp or udp). protocol=udp - host - server IP or its name. host=localhost - port - port number. port=9996 ==== dump ==== - rotate_minutes is the period in minutes, after which the temporary file in dumpfiledir/.url.dump will be moved to the archive (mv) and a new temporary file will be created. rotate_minutes=10 - processcmd is the command that will be launched at the end of the file rotation, the file name parameter with the path to it.processcmd=gzip %%s - dumpfiledir is a directory to store the files with data received. dumpfiledir=/var/dump/dpiui/ipfixflow/ - buffer_size is the size of the i/o buffer between the process of receiving and writing to a file, it is used in the [dump] section, the default value of the parameter is 100000 records (it is focused on 20 Gbit traffic or 25 000 sessions per second). If the number of sessions per second is considerably less than the mentioned value, then you should to change this parameter proportionally. ==== InfoModel ==== The block specifies the data received via the IPFIX protocol. - InfoElements - parameter describing the information model elements for IPFIXInfoElements = octetDeltaCount, 0, 1, UINT64, True packetDeltaCount, 0, 2, UINT64, True protocolIdentifier, 0, 3, UINT8 session_id, 43823, 2000, UINT64, True here, session_id - is the name of the field from the IPFIX description, for more detail see corresponding sections 43823 - unique organization number (enterprise number) 1 - unique field number UINT64 - field type True - use reverse byte order (endian). Possible values are: True or empty. Field types:\\ ^ Type ^ Length ^ Type IPFIX ^ | OCTET_ARRAY | VARLEN | octetArray | | UINT8 | 1 | unsigned8 | | UINT16 | 2 | unsigned16 | | UINT32 | 4 | unsigned32 | | UINT64 | 8 | unsigned64 | | INT8 | 1 | signed8 | | INT16 | 2 | signed16 | | INT32 | 4 | signed32 | | INT64 | 8 | signed64 | | FLOAT32 | 4 | float32 | | FLOAT64 | 8 | float64 | | BOOL | 1 | boolean | | MAC_ADDR | 6 | macAddress | | STRING | VARLEN | string | | SECONDS | 4 | dateTimeSeconds | | MILLISECONDS | 8 | dateTimeMilliseconds | | MICROSECONDS | 8 | dateTimeMicroseconds | | NANOSECONDS | 8 | dateTimeNanoseconds | | IP4ADDR | 4 | ipv4Address | | IP6ADDR | 16 | ipv6Address | The field names and their description can be accessed from the following links:\\ - [[en:dpi:dpi_options:opt_statistics:statistics_ipfix|Netflow export template using the IPFIX format]] - [[en:dpi:dpi_options:opt_li:li_ipfix|Clickstream and SIP export templates]] - [[en:dpi:dpi_components:radius:radmon_acct_ipfix|AAA export template using the IPFIX format]] Additional information:\\ [[https://tools.ietf.org/html/rfc5102.html | Information Model for IP Flow Information Export]] ==== ExportModel ==== specifies the model parameters used for export, is reserved for future use. - Mode - the type of export usedMode = File ==== ExportModelFile ==== Description of the File export model. - Delimiter ( \t - tabulation, more examples - |,;) Delimiter = \t - ExportElements - description of the fields that will be saved to the file.ExportElements = timestamp, seconds, %%Y-%%m-%%d %%H:%%M:%%S.000+03 login source_ip4 destination_ip4 host, decodehost path, decodepath referal, decodereferer session_id where the fields in each row are the following: name - the field name from the information model [InfoModel] (login, session_id and etc.) handler - field processing procedure before output seconds - field in seconds, format is expected milliseconds - field in milliseconds, microseconds, nanoseconds format is expected decodehost - recode from punycode to UTF-8 decodepath - recode from urlencoding to UTF-8 decodereferer - recode from (punycode,urlencoding) to UTF-8 decodeproto - recode the protocol identifier to the string format - format description for seconds, milliseconds. Example: %%Y-%%m-%%d %%H:%%M:%%S.%%f+0300 Result: 2016-05-25 13:13:35.621000+0300 ==== Creation of systemd ipfixreceiver service in Centos7 ==== Step-by-step creation of service in Centos 7, here the service name is ** ipfix1 **, its configuration is in the **/etc/dpiui/ipfixreceiver.conf** file, listening port is **1500**. \\ Create the /etc/systemd/system/ipfix1.service file as follows: [Unit] Description=ipfix test restart After=network.target After=syslog.target [Service] Type=forking PIDFile=/tmp/ipfixreceiver.1500.pid ExecStart=/usr/local/bin/ipfixreceiver start -f /etc/dpiui/ipfixreceiver.conf ExecStop=/usr/local/bin/ipfixreceiver stop -f /etc/dpiui/ipfixreceiver.conf ExecReload=/usr/local/bin/ipfixreceiver restart -f /etc/dpiui/ipfixreceiver.conf Restart=always RestartSec=10s [Install] WantedBy=multi-user.target Issue the following commands: systemctl enable ipfix1.service systemctl start ipfix1.service systemctl daemon-reload Check whether the service is running: systemctl status ipfix1.service -l :!: **Do not forget to check the service status after rebooting!** ===== Troubleshooting ===== - how to get utility version?\\ You should use the following commands:\\ ipfixreceiver -vyum info ipfixreceiver - Is it allowed to send IPFIX flows from different DPI to one port?\\ Yes, it is. The only thing is that they can not be distinguished in the recorded flow. - How can I understand that the utility works properly?\\ a) check that the port specified in the configuration file is listened on by the utility, for example 1500:netstat -nlp | grep 1500 b) check the log for errors\\ c) check that the writing to the temporary file occurs, for example for port 9996 (directory for dump files: /var/dump/dpiui/ipfixurl): tail -f /var/dump/dpiui/ipfixurl/9996.url.dump - everything is checked, but the messages are not received?\\ a) it seems you have forgotten to open port in iptables\\ b) it seems you have initialized ipfixreceiver with the wrong server IP. - a huge number of sessions (more than 2 million sessions/min) is going from the DPI, it can be seen with the DEBUG mode on that the buffer exchange counter does not have time to write before receiving the next block of records, what can be done in this case?\\ a) remove the date-to-line conversion, this will reduce the time needed for processing and in addition you will receive reduction the size of resulting file\\ b) remove the decodeipv4 conversion; it will not affect significantly, but you can get the higher speed of writing the file\\ c) configure the ''buffer_size'' when number of sessions per second is more than 30k along with the following item d)\\ d) increase the processor frequency and RAM