Differences
This shows you the differences between two versions of the page.
| en:dpi:dpi_components:platform:dpi_config [2018/03/18 11:33] – created lexx26 | en:dpi:dpi_components:platform:dpi_config [2024/09/26 15:29] (current) – external edit 127.0.0.1 | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | ====== | + | ====== |
| + | {{indexmenu_n> | ||
| + | [[https:// | ||
| + | |||
| + | ===== System Preparation ===== | ||
| + | |||
| + | The initial installation of DPI is done by VAS Experts technical support. Please do not try to do the initial installation yourself, as we may need to check all the steps you have done later, which increases the workload of tech support. | ||
| + | |||
| + | Later on you will be able to add or remove network ports and change the configuration yourself. | ||
| + | |||
| + | ===== Ports configuration ===== | ||
| + | |||
| + | The network cards that Stingray will work with are removed from the control of the operating system and therefore are not visible as Ethernet devices to the operating system. | ||
| + | The DPDK addresses Ethernet devices by their PCI identifiers, | ||
| + | |||
| + | < | ||
| + | lspci -D|grep Eth | ||
| + | |||
| + | 0000: | ||
| + | 0000: | ||
| + | </ | ||
| + | |||
| + | This command outputs a list of all ethernet-type PCI devices. Each line starts with a PCI device system identifier – these PCI identifiers are the unique identifiers of the network card in the DPDK. | ||
| + | |||
| + | The list of cards in DPDK mode can be checked with the command '' | ||
| + | |||
| + | < | ||
| + | 0000: | ||
| + | 0000: | ||
| + | </ | ||
| + | |||
| + | If necessary, the cards can be taken out of DPDK mode with a [[en: | ||
| + | |||
| + | You will need to stop the Fastdpi process beforehand. | ||
| + | |||
| + | < | ||
| + | service fastdpi stop | ||
| + | </ | ||
| + | |||
| + | < | ||
| + | driverctl unset-override 0000: | ||
| + | driverctl unset-override 0000: | ||
| + | </ | ||
| + | |||
| + | After working with the regular driver, do not forget to put them back under DPDK control with the [[en: | ||
| + | |||
| + | < | ||
| + | driverctl -v set-override 0000: | ||
| + | driverctl -v set-override 0000: | ||
| + | </ | ||
| + | |||
| + | <note important> | ||
| + | <note important> | ||
| + | driverctl list-overrides | ||
| + | 0000: | ||
| + | In this case it is recommended to switch to the vfio-pci driver. | ||
| + | To do this, run these commands for all devices in the list of list-overrides: | ||
| + | < | ||
| + | driverctl -v set-override 0000: | ||
| + | Setting '' | ||
| + | </ | ||
| + | |||
| + | ===== Stingray SG Configuration ===== | ||
| + | When the system is configured to work with DPDK, you can start configuring the Stingray SG. | ||
| + | The interfaces are configured with «in»-«out» pairs (for the future convenience, | ||
| + | < | ||
| + | # In - port 41:00.0 | ||
| + | in_dev=41-00.0 | ||
| + | # Out - port 41:00.1 | ||
| + | out_dev=41-00.1 | ||
| + | </ | ||
| + | This configuration sets a single bridge 41-00.0 ←→ 41-00.1 \\ | ||
| + | You can specify a group of interfaces with ':' | ||
| + | < | ||
| + | in_dev=41-00.0: | ||
| + | out_dev=41-00.1: | ||
| + | </ | ||
| + | This group forms the following pairs (bridges): \\ | ||
| + | 41-00.0 ←→ 41-00.1 \\ | ||
| + | 01-00.0 ←→ 01-00.1 \\ | ||
| + | 05-00.0 ←→ 05-00.1 \\ | ||
| + | The pairs must have devices of the same speed; it is unacceptable to pair 10G and 40G cards. However, the group can have interfaces of different speeds, for example, one pair is 10G, the other is 40G. | ||
| + | |||
| + | The maximum ethernet packet size on the devices is set by the '' | ||
| + | ===== Setting device aliases ===== | ||
| + | Starting from version 9.5.3, the SSG now allows you to specify aliases for devices. This is due to the fact that DPDK supports numerous devices, not only PCI devices, but also, for example, vmbus devices (Hyper-V) or virtual (vdev) devices. Additionally, | ||
| + | |||
| + | The essence of the alias is very simple: you describe the desired device in a separate parameter and give this description a name. Then in the '' | ||
| + | |||
| + | Each alias is specified by a separate '' | ||
| + | < | ||
| + | dpdk_device=alias: | ||
| + | </ | ||
| + | Here: | ||
| + | * '' | ||
| + | * '' | ||
| + | * '' | ||
| + | For example: | ||
| + | < | ||
| + | # eth1 is the alias of PCI device 41:00.0 | ||
| + | dpdk_device=eth1: | ||
| + | # eth2 eth2 is the alias of PCI device 41:00.1 | ||
| + | dpdk_device=eth2: | ||
| + | |||
| + | in_dev=eth1 | ||
| + | out_dev=eth2 | ||
| + | </ | ||
| + | This description is equivalent to the following: | ||
| + | < | ||
| + | in_dev=41-00.0 | ||
| + | out_dev=41-00.1 | ||
| + | </ | ||
| + | Note that in '' | ||
| + | |||
| + | <note tip>For PCI devices, assigning to '' | ||
| + | |||
| + | If you want to connect Hyper-V devices (and these are not PCI devices, but VMbus devices), then the use of aliases is mandatory. Example: | ||
| + | < | ||
| + | dpdk_device=subs1: | ||
| + | dpdk_device=subs2: | ||
| + | dpdk_device=inet1: | ||
| + | dpdk_device=inet2: | ||
| + | in_dev=subs1: | ||
| + | out_dev=inet1: | ||
| + | </ | ||
| + | Here we not only set the alias, but also specify the '' | ||
| + | |||
| + | ===== Configuration in Hyper-V ===== | ||
| + | Starting with version 9.5.3, SSG supports running in a Hyper-V virtual machine. | ||
| + | On guest [[en: | ||
| + | < | ||
| + | # Multi-queue support - required for SSG | ||
| + | dnf install kernel-modules-extra | ||
| + | </ | ||
| + | <note important> | ||
| + | |||
| + | Devices on Hyper-V are VMBus, and not PCI devices, so they require a special conversion to DPDK mode. | ||
| + | Each device (interface) is identified by its unique UUID, so first you need to know the UUIDs of all interfaces that SSG will work with. Then you have to put the device into DPDK mode: | ||
| + | < | ||
| + | # switch the interfaces eth0 and eth2 into DPDK mode | ||
| + | for DEV in eth0 eth2 | ||
| + | do | ||
| + | # get the UUID for the device | ||
| + | DEV_UUID=$(basename $(readlink / | ||
| + | # switch to DPDK compatible mode | ||
| + | driverctl -b vmbus set-override $DEV_UUID uio_hv_generic | ||
| + | |||
| + | # Device appears in | ||
| + | # / | ||
| + | |||
| + | echo "$DEV uuid=$DEV_UUID" | ||
| + | done | ||
| + | </ | ||
| + | If necessary, the interface can be switched back to kernel-mode like this: | ||
| + | < | ||
| + | ETH0_UUID=< | ||
| + | driverctl -b vmbus unset-override $ETH0_UUID | ||
| + | </ | ||
| + | |||
| + | Next, configure the SSG – set the devices in '' | ||
| + | < | ||
| + | # eth0 UUID=392b7b0f-dbd7-4225-a43f-4c926fc87e39 | ||
| + | dpdk_device=eth0: | ||
| + | # eth2 UUID=34f1cc16-4b3f-4d8a-b567-a0eb61dc2b78 | ||
| + | dpdk_device=eth2: | ||
| + | |||
| + | # then use the aliases eth0 and eth2 everywhere when specifying the devices | ||
| + | in_dev=eth0 | ||
| + | out_dev=eth2 | ||
| + | </ | ||
| + | |||
| + | ===== Clusters ===== | ||
| + | The DPDK version of Stingray SG supports clustering: you can specify which interfaces are included in each cluster. The clusters are separated with the ' | ||
| + | < | ||
| + | in_dev=41-00.0|01-00.0: | ||
| + | out_dev=41-00.1|01-00.1: | ||
| + | </ | ||
| + | This example creates two clusters: | ||
| + | * cluster with bridge 41-00.0 ←→ 41-00.1 | ||
| + | * cluster with bridges | ||
| + | Clusters are a kind of a legacy of the Stingray SG pf_ring-version: | ||
| + | |||
| + | In DPDK, clusters are also isolated from each other, but unlike pf_ring, here a cluster is a more logical concept inherited from pf_ring. DPDK is much more flexible than pf_ring and allows you to build complex multi-bridge configurations with many dispatchers without using clusters. In fact, the only " | ||
| + | <note tip>Tip: instead of using clusters, consider switching to a different '' | ||
| + | The following descriptions of configurations assume that there is only one cluster (no clustering). | ||
| + | |||
| + | ===== Number of Cores (Threads) ===== | ||
| + | CPU cores are perhaps the most critical resource for the Stingray SG. The more physical cores there are in the system, the more traffic can be processed by the SSG. | ||
| + | <note important> | ||
| + | Stingray SG needs the following threads to operate: | ||
| + | * processing threads - process incoming packets and write to the TX-queue of the card; | ||
| + | * dispatcher threads - read the card's RX queues and distribute incoming packets among processing threads; | ||
| + | * service threads - perform deferred (time-consuming) actions, receive and process fdpi_ctrl and CLI, connection with PCRF, sending netflow | ||
| + | * system kernel - dedicated to the operating system. | ||
| + | Processing and dispatcher threads cannot be located on the same core. At start, Stingray SG binds threads to cores. | ||
| + | Stingray SG by default selects the number of handler threads depending on the interface speed:\\ | ||
| + | 10G - 4 threads\\ | ||
| + | 25G - 8 threads\\ | ||
| + | 40G, 50G, 56G - 16 threads\\ | ||
| + | 100G - 32 threads\\ | ||
| + | For a group, the number of threads is equal to the sum of threads number for each pair; e.g., for the cards: | ||
| + | < | ||
| + | # 41-00.x - 25G NIC | ||
| + | # 01-00.x - 10G NIC | ||
| + | in_dev=41-00.0: | ||
| + | out_dev=41-00.1: | ||
| + | </ | ||
| + | 12 processing threads will be created (8 for 25G card and 4 for 10G card) | ||
| + | |||
| + | In fastdpi.conf, | ||
| + | < | ||
| + | # 41-00.x - 25G NIC | ||
| + | # 01-00.x - 10G NIC | ||
| + | in_dev=41-00.0: | ||
| + | out_dev=41-00.1: | ||
| + | |||
| + | num_threads=8 | ||
| + | </ | ||
| + | |||
| + | This configuration will create 8 processing threads. | ||
| + | |||
| + | <note important> | ||
| + | |||
| + | In addition to the handler threads, for operating you also need at least one dispatcher thread (and therefore at least one more core) that reads the rx-queues of the interfaces. The dispatcher' | ||
| + | |||
| + | The internal architecture of working with one or many dispatchers is strikingly different, therefore Stingray provides several engines configured by the '' | ||
| + | * '' | ||
| + | * '' | ||
| + | * '' | ||
| + | * '' | ||
| + | |||
| + | Further, all these engines are described in detail, their configuration features and areas of application, | ||
| + | ==== Explicit Binding to Cores ==== | ||
| + | You can explicitly bind threads to cores in fastdpi.conf. The parameters: | ||
| + | * '' | ||
| + | * '' | ||
| + | |||
| + | The format for specifying these lists is the same: | ||
| + | < | ||
| + | # 10G cards - 4 processor threads, 1 dispatcher per cluster | ||
| + | in_dev=01-00.0|02-00.0 | ||
| + | out_dev=01-00.1|02-00.1 | ||
| + | |||
| + | # Bind processing threads for cluster #1 to cores 2-5, dispatcher to core 1 | ||
| + | # for cluster #2 - to cores 7-10, dispatcher to core 6 | ||
| + | engine_bind_cores=2: | ||
| + | rx_bind_core=1|6 | ||
| + | </ | ||
| + | Without clustering: | ||
| + | < | ||
| + | # 10G cards - 4 processing threads per card | ||
| + | in_dev=01-00.0: | ||
| + | out_dev=01-00.1: | ||
| + | # 2 dispatchers (by directions) | ||
| + | dpdk_engine=1 | ||
| + | |||
| + | # Bind processing threads and dispatcher threads | ||
| + | engine_bind_cores=3: | ||
| + | rx_bind_core=1: | ||
| + | </ | ||
| + | |||
| + | As noted, the handler and dispatcher threads must have dedicated cores; it is not allowed to bind several threads to one core - the Stingray SG will display an error in fastdpi_alert.log and will not start. | ||
| + | <note tip> | ||
| + | <note important> | ||
| + | |||
| + | ===== The Dispatcher Thread Load ===== | ||
| + | If the load of the dispatcher thread is close to 100%, it does not mean that the dispatcher cannot cope: DPDK assumes that data from the card is read by the consumer (this is the dispatcher) without any interruptions, | ||
| + | <note tip>The load of SSG threads can be viewed with the following command: | ||
| + | < | ||
| + | top -H -p `pidof fastdpi` | ||
| + | </ | ||
| + | </ | ||
| + | The real state of each dispatcher can be seen in fastdpi_stat.log, | ||
| + | < | ||
| + | [STAT ][2020/ | ||
| + | drop (worker queue full) | ||
| + | Cluster #0: | ||
| + | </ | ||
| + | here '' | ||
| + | |||
| + | There is another good indicator that the current engine cannot cope: a non-zero delta value for the '' | ||
| + | * either there are too few processing threads, you need to increase the '' | ||
| + | * or the traffic is heavily skewed and most of the packets go to one or two handlers, while the rest are free. In this situation, you need to analyze the traffic structure. You can try to increase or decrease the number of handler threads by one, so that the dispatcher hash function would distribute packets more evenly (the number of the processing thread is '' | ||
| + | |||
| + | ==== dpdk_engine=0: | ||
| + | In this mode, Stingray SG creates one dispatcher thread per cluster. | ||
| + | The dispatcher reads incoming packets from all in_dev and out_dev devices and distributes the packets to the handler threads. | ||
| + | Suitable for 10G cards, withstands loads up to 20G or more (depends on the CPU model and the [[en: | ||
| + | |||
| + | <note tip>The total number of cores required is equal to the number of handlers plus one core per dispatcher.</ | ||
| + | |||
| + | Stingray SG configures cards as follows: | ||
| + | * RX queue count = 1 | ||
| + | * TX queue count = number of processing threads. Processing threads record data directly each to their TX-card queue. | ||
| + | |||
| + | <note tip>For read-only mode (without '' | ||
| + | ==== dpdk_engine=1: | ||
| + | In this mode, two dispatcher threads are created: one for directing from subscribers to inet (for in_dev), the other for directing from inet to subscribers (for out_dev). | ||
| + | Suitable for loads over 20G (25G, 40G cards). | ||
| + | <note tip>The total number of cores required is equal to the number of handlers plus two cores per dispatcher.</ | ||
| + | |||
| + | Stingray SG configures cards as follows: | ||
| + | * RX queue count = 1 | ||
| + | * TX queue count = number of processing threads. Processing threads record data directly each to their TX-card queue. | ||
| + | |||
| + | ==== dpdk_engine=2: | ||
| + | In this mode, RSS (receive side scaling) cards are used. | ||
| + | The RSS value is set in fastdpi.conf with the parameter: | ||
| + | < | ||
| + | dpdk_rss=2 | ||
| + | </ | ||
| + | The '' | ||
| + | For each direction, dispatcher '' | ||
| + | <note tip>The total number of cores required is equal to the number of handlers plus '' | ||
| + | Suitable for powerful 50G+ cards (for SSG-100+). If you have a grouping of 50G from several cards, this mode is hardly suitable, since for each card from the group it requires at least 2 additional cores (with '' | ||
| + | |||
| + | Stingray SG configures cards as follows: | ||
| + | * RX queue count = '' | ||
| + | * TX queue count = number of processing threads. Processing threads record data directly each to their TX-card queue. | ||
| + | |||
| + | ==== dpdk_engine=3: | ||
| + | A separate dispatcher thread is created for each bridge. | ||
| + | Designed for configurations with multiple input and output devices: | ||
| + | < | ||
| + | in_dev=01-00.0: | ||
| + | out_dev=01-00.1: | ||
| + | dpdk_engine=3 | ||
| + | </ | ||
| + | In this example, three dispatcher threads are created: | ||
| + | * for bridge 01-00.0 <--> 01-00.1 | ||
| + | * for bridge 02-00.0 <--> 02-00.1 | ||
| + | * for bridge 03-00.0 <--> 03-00.1 | ||
| + | |||
| + | <note tip>The total number of cores required is equal to the number of handlers plus the number of bridges.</ | ||
| + | |||
| + | This engine is designed for several 25G/40G/50G cards in a group (that is, for SSG-100+). | ||
| + | |||
| + | Stingray SG configures cards as follows: | ||
| + | * RX queue count = 1 | ||
| + | * TX queue count = number of processing threads. Processing threads record data directly each to their TX-card queue. | ||
| + | |||
| + | ==== dpdk_engine=4: | ||
| + | A separate dispatcher thread is created for each port (device). | ||
| + | Designed for configurations with a range of devices on input and output: | ||
| + | < | ||
| + | in_dev=01-00.0: | ||
| + | out_dev=01-00.1: | ||
| + | dpdk_engine=4 | ||
| + | </ | ||
| + | For this example, six dispatcher threads are created – one dispatcher for each device. Obviously, if we have only one bridge, this engine is equivalent to '' | ||
| + | <note tip>The total number of cores required is equal to the number of handlers plus the number of ports</ | ||
| + | |||
| + | This engine is designed for multiple 25G/40G/50G cards in a group (i.e. for SSG-100+) | ||
| + | |||
| + | The SSG configures the cards as follows: | ||
| + | * RX queue count = 1 | ||
| + | * TX queue count = The processing threads write directly each to its own TX queue card. | ||