Full NetFlow export format

General Settings

Enabling statistics collection and export:

netflow=1

0 or not set - the option is disabled

1 - export statistics by protocols (port numbers)

2 - export statistics by directions (autonomous system numbers)

4 - export statistics for billing

8 - export full session statistics

3 = 1 + 2 — simultaneous export of protocol-based and direction-based statistics (the same logic applies to other combinations) 12 = 8 + 4 — simultaneous export of full NetFlow and billing statistics. In particular, this is used for RADIUS Accounting

Network interface name through which NetFlow statistics will be sent:

netflow_dev=eth2

Data export interval (in seconds):

netflow_timeout=10

Session timeout:

netflow_passive_timeout — inactivity timeout after which, if there is no activity, the session is considered finished and its data is exported

netflow_active_timeout — interval after which information about long sessions is reported (i.e., long sessions are effectively split into fragments of this duration)

To smooth peaks and distribute the load on the collector more evenly, set the tuning parameter

netflow_rate_limit=60

, where 60 is the maximum NetFlow rate in Mbit/s.

The parameter value should be set based on the following calculation: 6 Mbit/s per each 1G of external link bandwidth. Setting an insufficient value will result in data being dropped on the DPI side. Information about this event will be recorded in the log /var/log/dpi/fastdpi_alert.log.
A separate collector must be allocated for each type so that data is not mixed!
IPFIX/NetFlow parameters can be changed without restarting fastDPI. The configuration parameter ipfix_reserved allows reserving the required amount of memory to enable/change IPFIX/NetFlow parameters. If IPFIX/NetFlow parameters are specified in the configuration file, memory reservation for IPFIX/NetFlow is enabled automatically, and parameters as well as new IPFIX/NetFlow exporter types can be changed without restarting fastDPI.

Full NetFlow

IP address and port number of the NetFlow collector with full statistics; a separate collector must be allocated so that data is not mixed with other statistics:

netflow_full_collector=192.168.0.1:9996

In NetFlow5 format, the full statistics retain the original port numbers, and information about detected protocols is transmitted in the usually unused bytes 46–47. If it is necessary to analyze the protocols in use, you can enable a setting in which protocol information is transmitted in the port number:

netflow_full_port_swap=1

For compatibility with older collectors, this setting also applies to the IPFIX format, but it is not recommended to use it together with IPFIX, since protocol information in IPFIX is transmitted in a separate dedicated field.

It is also necessary to define the full NetFlow export format:

netflow_full_collector_type=2

Possible values:

0 - export in NetFlow5 format (default)

1 - export IPFIX to a UDP collector

2 - export IPFIX to a TCP collector

We recommend using full NetFlow export in IPFIX format over TCP (parameter value 2). The NetFlow protocol does not guarantee packet delivery (as it runs over UDP), and if the collector cannot handle the incoming data rate, some packets will simply be lost. Exporting full NetFlow statistics for a 10G link requires the collector to be able to receive data at a rate of at least 60 Mbit/s. Check the capabilities of your collector before sending NetFlow traffic to it. At the same time, short-term bursts of up to 100 Mbit/s may occur when exporting NetFlow from DPI. Only a few collectors can handle such a data stream without losses, for example, nfsen/nfdump.

The netflow_tos_format parameter defines the data format of the TOS field in IPFIX. Possible values:

0 - 3 bits are transmitted (default)

1 - 6 bits are transmitted (full DSCP)

The netflow_plc_stat parameter defines the set of transmitted statistics data for dropped packets according to policing or drop rules. The parameter is a bit mask.
By default, the mask value is 0x07 — statistics are transmitted for dropped data due to session + subscriber + virtual channel policing.
Affects the formation of the DROPPED_BYTES and DROPPED_PACKETS counters.
Mask values:

0xff - count any drop

0 - do not count

1 - count for session policing

2 - count for subscriber policing

4 - count for virtual channel policing

8 - count drops by protocol

16 - count in all other cases

The ipfix_mtu_limit parameter sets the maximum UDP packet size when sending IPFIX. By default, it is equal to the minimum MTU size of the interfaces used for transmission.

The tethering_ttl_allowed = 128:64 parameter specifies the list of allowed TTL values for subscriber traffic that are not considered tethering. Values are separated by ':'. Up to 256 values (0–255) are supported.

For receiving, processing, and storing IPFIX data, it is recommended to use the QoE Store statistics collection software and the DPIUI2 graphical interface. For collecting data in IPFIX format, any universal IPFIX collector that supports templates, or the IPFIX Receiver utility, can be used.

Export Template in IPFIX Format (Netflow v10) for IPv4 Protocol

Export Template for IPv4
№	Bytes	Data Type	IANA	Description	Notes	Used in QoEStor
1	8	int64	0	OCTET_DELTA_COUNT	Analog in NetFlow v9 IN_BYTES	Used
2	8	int64	0	PACKET_DELTA_COUNT	Analog in NetFlow v9 IN_PKTS	Used
4	1	int8	0	PROTOCOL_IDENTIFIER	Analog in NetFlow v9 PROTOCOL	Used
5	1	int8	0	IP_CLASS_OF_SERVICE	Analog in NetFlow v9 TOS	Used
7	2	int16	0	SOURCE_TRANSPORT_PORT	Analog in NetFlow v9 L4_SRC_PORT	Used
8	4	int32	0	SOURCE_IPV4_ADDRESS	Analog in NetFlow v9 IPV4_SRC_ADDR	Used
11	2	int16	0	DESTINATION_TRANSPORT_PORT	Analog in NetFlow v9 L4_DST_PORT	Used
12	4	int32	0	DESTINATION_IPV4_ADDRESS	Analog in NetFlow v9 IPV4_DST_ADDR	Used
16	4	int32	0	BGP_SOURCE_AS_NUMBER	Analog in NetFlow v9 SRC_AS	Used
17	4	int32	0	BGP_DESTINATION_AS_NUMBER	Analog in NetFlow v9 DST_AS	Used
152	8	int64	0	FLOW_START_MILLISECOND	Used
153	8	int64	0	FLOW_END_MILLISECOND	Used
10	2	int16	0	INPUT_SNMP	Analog in NetFlow v9 IngressInterface	Used
14	2	int16	0	OUTPUT_SNMP	Analog in NetFlow v9 EgressInterface	Used
60	1	int8	0	IP_VERSION	Analog in NetFlow v9 IP_PROTOCOL_VERSION	Used
2000	8	int64	43823	SESSION_ID	Used
2001	-	string	43823	HTTP_HOST or CN_HTTPS	Used
2002	2	int16	43823	DPI_PROTOCOL	Used
2003	-	string	43823	LOGIN	Analog in Radius User-Name	Used
225	4	int32	0	POST_NAT_SOURCE_IPV4_ADDRESS	Used
227	2	int16	0	POST_NAPT_SOURCE_TRANSPORT_PORT	Used
2010	2	int16	43823	FRGMT_DELTA_PACKS	Delta of fragmented packets.	Used
2011	2	int16	43823	REPEAT_DELTA_PACK	Delta of retransmissions.	Used
2012	4	int32	43823	PACKET_DELIVER_TIME	Delay (RTT/2) in ms (RTT=round-trip time).	Used
2016	2	int16	43823	BRIDGE_CHANNEL_NUM	Channel number (vchannel) or bridge. If vchannels are configured in DPI, the channel number will be transmitted, otherwise the bridge number.	Used
6	2	int16	0	TCP_FLAGS	TCP control bits	Used
58	2	int16	0	SRC_VLAN	VLAN ID	Used
59	2	int16	0	DST_VLAN	Post VLAN ID	Used
56	6	mac_address	0	SRC_MAC	Source MAC address	Used
57	6	mac_address	0	DST_MAC	Destination MAC address	Used
2017	-	raw	43823	MPLS Lables	Used
132	8	int64	0	DROPPED_BYTES	Delta count of dropped octets. For example: data is dumped at minute T1 and T2. The delta will show the difference in the number of octets between minute T1 and T2.	Used
133	8	int64	0	DROPPED_PACKETS	Delta count of dropped packets. For example: data is dumped at minute T1 and T2. The delta will show the difference in the number of packets between minute T1 and T2.	Used
2019	1	int8	43823	originalTOS	Original TOS value from IP header	Used
192	1	int8	0	IP_TTL	TTL packets
2020	2	int16	43823	RATING_GROUP	Rating group number

Export Template in IPFIX Format (Netflow v10) for IPv6 Protocol

The template is similar to IPv4 except that the following fields are absent: SOURCE_IPV4_ADDRESS, DESTINATION_IPV4_ADDRESSs, POST_NAT_SOURCE_IPV4_ADDRESS, POST_NAT_SOURCE_TRANSPORT_PORT, – and the following are present:

Export Template for IPv6
№	Bytes	Data Type	IANA	Description	Notes
27	16	int128	0	SOURCE_IPV6_ADDRESS	Analog in NetFlow v9 IPV6_SRC_ADDR
28	16	int128	0	DESTINATION_IPV6_ADDRESS	Analog in NetFlow v9 IPV6_DST_ADDR