| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| en:dpi:dpi_components:platform:dpi_config [2024/09/26 15:29] – external edit 127.0.0.1 | en:dpi:dpi_components:platform:dpi_config [2026/10/02 09:31] (current) – [Number of Cores (Threads)] grader7934 |
|---|
| |
| ===== Ports configuration ===== | ===== Ports configuration ===== |
| | |
| | <note important> |
| | Please note that Mellanox network interface cards cannot be switched to DPDK using the driverctl utility — their driver is installed in a different way. They also remain under operating system control, therefore the interfaces will still appear in the output of ip/ipconfig utilities. |
| | If it is necessary to install a driver to support DPDK on Mellanox network cards, please [[en:dpi:techsupport_info|contact technical support]]. |
| | </note> |
| |
| The network cards that Stingray will work with are removed from the control of the operating system and therefore are not visible as Ethernet devices to the operating system. | The network cards that Stingray will work with are removed from the control of the operating system and therefore are not visible as Ethernet devices to the operating system. |
| * cluster with bridge 41-00.0 ←→ 41-00.1 | * cluster with bridge 41-00.0 ←→ 41-00.1 |
| * cluster with bridges 01-00.0 ←→ 01-00.1 and 05-00.0 ←→ 05-00.1 | * cluster with bridges 01-00.0 ←→ 01-00.1 and 05-00.0 ←→ 05-00.1 |
| Clusters are a kind of a legacy of the Stingray SG pf_ring-version: in pf_ring, cluster is the basic concept of "one dispatcher thread + RSS handler threads" and is almost the only way to scale. The disadvantage of the cluster approach is that the clusters are physically isolated from each other: it is impossible to forward a packet from the X-interface of cluster #1 to the Y-interface of cluster #2. This can be a significant obstacle in the SKAT L2 BRAS mode. | Clusters are a kind of a legacy of the Stingray SG pf_ring-version: in pf_ring, cluster is the basic concept of "one dispatcher thread + RSS handler threads" and is almost the only way to scale. The disadvantage of the cluster approach is that the clusters are physically isolated from each other: it is impossible to forward a packet from the X-interface of cluster #1 to the Y-interface of cluster #2. This can be a significant obstacle in the SSG L2 BRAS mode. |
| |
| In DPDK, clusters are also isolated from each other, but unlike pf_ring, here a cluster is a more logical concept inherited from pf_ring. DPDK is much more flexible than pf_ring and allows you to build complex multi-bridge configurations with many dispatchers without using clusters. In fact, the only "pro" argument for clustering in the Stingray-DPDK version is the case when you have two independent networks A and B connected to the Stingray SG, which should not interact with each other in any way. | In DPDK, clusters are also isolated from each other, but unlike pf_ring, here a cluster is a more logical concept inherited from pf_ring. DPDK is much more flexible than pf_ring and allows you to build complex multi-bridge configurations with many dispatchers without using clusters. In fact, the only "pro" argument for clustering in the Stingray-DPDK version is the case when you have two independent networks A and B connected to the Stingray SG, which should not interact with each other in any way. |
| * ''dpdk_engine=2'' - read/write engine with RSS support: for each direction ''dpdk_rss'' dispatchers are created (''dpdk_rss=2'' by default). Thus, the total number of dispatchers = ''2 * dpdk_rss''; | * ''dpdk_engine=2'' - read/write engine with RSS support: for each direction ''dpdk_rss'' dispatchers are created (''dpdk_rss=2'' by default). Thus, the total number of dispatchers = ''2 * dpdk_rss''; |
| * ''dpdk_engine=3'' - read/write engine with a separate dispatcher for each bridge. | * ''dpdk_engine=3'' - read/write engine with a separate dispatcher for each bridge. |
| | * ''dpdk_engine=4'' - read/write engine with a separate dispatcher for each port (device); |
| | * ''dpdk_engine=6'' - read/write engine with RSS support for each bridge: ''dpdk_rss'' dispatchers are created for each bridge, so the total number of dispatchers = ''dpdk_rss * number of bridges'' (32 at most); |
| | * ''dpdk_engine=7'' - read/write engine with explicitly assigned dispatchers: the ''dpdk_dispatch'' parameter sets which ports each dispatcher serves. |
| |
| Further, all these engines are described in detail, their configuration features and areas of application, and the dispatcher threads in gneral. | Further, all these engines are described in detail, their configuration features and areas of application, and the dispatcher threads in gneral. |
| * RX queue count = 1 | * RX queue count = 1 |
| * TX queue count = The processing threads write directly each to its own TX queue card. | * TX queue count = The processing threads write directly each to its own TX queue card. |
| | |
| | ==== dpdk_engine=6: RSS dispatchers per bridge ==== |
| | <note important>This ''dpdk_engine'' is available starting from version 14.0!</note> |
| | This engine is intended for configurations with multiple bridges (dev1:dev2:dev3:...) for 100G+ cards. |
| | |
| | <code> |
| | in_dev=41-00.0:02-00.0:c3-00.0:c1-00.0:04-00.0:04-00.1 |
| | out_dev=41-00.1:41-00.1:02-00.1:02-00.1:c3-00.1:c3-00.1 |
| | |
| | dpdk_engine=6 |
| | dpdk_rss=4 |
| | num_threads=64 |
| | |
| | dpdk_mempool_size=256000 |
| | mem_tracking_flow=40000000 |
| | mem_tracking_ip=40000000 |
| | dpdk_emit_mempool_size=256000 |
| | mem_ssl_parsers=18000000 |
| | mem_http_parsers=512000 |
| | </code> |
| | This example creates 24 dispatcher threads — 4 dispatchers per bridge. |
| | |
| | <note tip> |
| | Total number of dispatchers = ''dpdk_rss'' * number of bridges.\\ |
| | For 100G+ NICs, with a ratio of 1 dispatcher per 10G, the minimum number of dispatchers is 10.\\ |
| | Starting from version 14.0, the maximum number of dispatchers is 32. |
| | </note> |
| | |
| | |
| | On-stick devices are supported.\\ |
| | SSG configures the cards as follows: |
| | * RX queue count = ''dpdk_rss'' |
| | * TX queue count = number of processing threads. Processing threads write directly to their own TX queue on the card. |
| | |
| | ==== dpdk_engine=7: Explicit Dispatcher Assignment ==== |
| | <note important>This ''dpdk_engine'' is available starting from version 14.2!</note> |
| | |
| | This engine allows explicitly specifying which ports are served by each dispatcher. It is intended for complex and heterogeneous configurations where standard dispatcher distribution schemes are not suitable. For example, when ports with different performance levels (100G, 40G, 10G) are used in the same cluster, or when individual processing parameters need to be configured for specific groups of ports. |
| | |
| | The number of dispatchers is determined by the number of ''dpdk_dispatch'' parameters. If the ''rss=N'' parameter is specified for a dispatcher, N dispatchers are created — one for each RX queue. |
| | |
| | Each dispatcher must have a ''mempool'' specified. The size and number of ''mempool'' instances are configured independently and may differ between dispatchers. |
| | |
| | The engine is universal and allows all dispatcher distribution schemes supported by other ''dpdk_engine'' values to be implemented. |
| | |
| | Dispatchers are specified using the ''dpdk_dispatch'' parameters. |
| | |
| | The following configuration cases are considered fatal errors: |
| | * a cluster port is not included in any ''<port-list>'' of a ''dpdk_dispatch'' parameter |
| | * a cluster port is included in the ''<port-list>'' of multiple different ''dpdk_dispatch'' parameters |
| | * the ''<port-list>'' of a ''dpdk_dispatch'' parameter contains ports from different clusters — each dispatcher must serve ports belonging to only one cluster |
| | |
| | This engine is universal — all other engines can be expressed through it, for example: |
| | <code> |
| | dpdk_engine=0: one dispatcher for all ports |
| | in_dev=port1:port2 |
| | out_dev=port3:port4 |
| | dpdk_dispatch=port1,port2,port3,port4 |
| | |
| | dpdk_engine=1: one dispatcher per direction |
| | in_dev=port1:port2 |
| | out_dev=port3:port4 |
| | dpdk_dispatch=port1,port2 |
| | dpdk_dispatch=port3,port4 |
| | |
| | dpdk_engine=2: one dispatcher per direction with RSS support |
| | dpdk_rss=4 |
| | in_dev=port1:port2 |
| | out_dev=port3:port4 |
| | dpdk_dispatch=port1,port2;rss=4 |
| | dpdk_dispatch=port3,port4;rss=4 |
| | |
| | dpdk_engine=3: one dispatcher per bridge |
| | in_dev=port1:port2 |
| | out_dev=port3:port4 |
| | dpdk_dispatch=port1,port3 |
| | dpdk_dispatch=port2,port4 |
| | |
| | dpdk_engine=4: one dispatcher per port |
| | in_dev=port1:port2 |
| | out_dev=port3:port4 |
| | dpdk_dispatch=port1 |
| | dpdk_dispatch=port2 |
| | dpdk_dispatch=port3 |
| | dpdk_dispatch=port4 |
| | |
| | dpdk_engine=6: one dispatcher per bridge with RSS support |
| | dpdk_rss=4 |
| | in_dev=port1:port2 |
| | out_dev=port3:port4 |
| | dpdk_dispatch=port1,port3;rss=4 |
| | dpdk_dispatch=port2,port4;rss=4 |
| | </code> |
| | |
| | The ''mempool'' used by the dispatcher must be specified. |
| | |
| | * **''mempool'' descriptor** — only for ''dpdk_engine=7''\\ Format: <code>dpdk_mempool=name=<name>;size=N</code> |
| | * ''name'' specifies the name of the ''mempool'' (maximum 15 characters) |
| | * ''size'' specifies the size of the ''mempool'' in packets |
| | * both parameters are mandatory\\ Multiple ''dpdk_mempool'' options can be specified, each describing a separate ''mempool'': <code>dpdk_mempool=...</code> |
| | |
| | The ''mempool'' size determines the number of packets that can simultaneously reside in the corresponding pool. The required size depends on the traffic processing configuration and, in particular, on the following parameters: |
| | * the network cards used; |
| | * the size of RX and TX queues on the network cards; |
| | * the number of processing threads (slave threads); |
| | * the size of the queues between the dispatcher and slave threads. |
| | |
| | The ''mempool'' size is determined experimentally based on the specific configuration and is specified in packets. The maximum packet size in bytes is determined by the ''snaplen'' parameter. Accordingly, the approximate amount of memory required to store packets in the ''mempool'' can be estimated as follows: |
| | |
| | <code> |
| | memory usage = number of packets in mempool × maximum packet size (snaplen) |
| | </code> |
| | |
| | For example, for a ''mempool'' containing 1,600,000 packets and a ''snaplen'' of 2048 bytes, the memory required for packet data is approximately 3.28 GB. The actual memory usage may differ, since the estimate must also account for DPDK internal structures and other overhead. |
| | |
| | When using multiple ''mempool'' instances, each one can be configured with its own size. This allows different amounts of memory to be allocated for groups of ports with different performance levels or loads. |
| | |
| | * **''dpdk_max_memzone''** [cold] — sets the maximum number of ''memzone'' instances in DPDK. By default, DPDK uses a limit on the number of ''memzone'' instances (usually 5120, depending on the DPDK version). Increasing this value may be required for large configurations with a high number of network cards and ''mempool'' instances. |
| | * ''0'' — use the default value defined in DPDK |
| | * if the ''Number of requested memzone segments exceeds maximum 5120'' error occurs when starting fastDPI, increase the ''dpdk_max_memzone'' parameter value |
| | |
| | There is no universal exact value for ''dpdk_max_memzone'' that can be determined in advance for a given configuration. The parameter must be increased if the number of ''memzone'' instances allocated by DPDK for the specific configuration is insufficient. In practice, the error displayed when starting fastDPI serves as an indicator that the value needs to be increased: ''Number of requested memzone segments exceeds maximum 5120''. |
| | |
| | * **Dispatcher descriptor** — only for ''dpdk_engine=7''\\ Format: <code>dpdk_dispatch=<port-list>;mempool=<name>[;params]*</code> |
| | * ''<port-list>'' specifies which ports are served by this dispatcher |
| | * ''mempool=<name>'' specifies the name of the ''mempool'' used by this dispatcher (mandatory parameter) |
| | * ''rss=N'' enables RSS on all ports served by this dispatcher; N dispatchers are created — one for each RX queue\\ Multiple ''dpdk_dispatch'' options can be specified, each describing a separate dispatcher (or a group of dispatchers if ''rss'' is specified): <code>dpdk_dispatch=...</code> |
| | |
| | === Example of Use in a Heterogeneous Configuration === |
| | |
| | ''dpdk_engine=7'' is useful in configurations where ports of different types and performance levels need to be processed independently. For example, a test stand may include 16 10G ports on the incoming traffic side and 2 100G ports on the outgoing traffic side. Such a scheme can be used to connect a BRAS with different interface bandwidths on the LAN and WAN sides. |
| | |
| | In this case, separate dispatchers with one ''mempool'' can be created for groups of 10G ports, while a separate dispatcher with RSS support and another, larger ''mempool'' can be used for 100G ports: |
| | |
| | <code> |
| | dpdk_engine=7 |
| | |
| | dpdk_dispatch=30,31;mempool=main10G |
| | dpdk_dispatch=32,33;mempool=main10G |
| | dpdk_dispatch=34,35;mempool=main10G |
| | dpdk_dispatch=36,37;mempool=main10G |
| | dpdk_dispatch=40,41;mempool=main10G |
| | dpdk_dispatch=42,43;mempool=main10G |
| | dpdk_dispatch=50,51;mempool=main10G |
| | dpdk_dispatch=52,53;mempool=main10G |
| | |
| | dpdk_dispatch=10,20;rss=16;mempool=main100G |
| | |
| | dpdk_mempool=name=main10G;size=1600000 |
| | dpdk_mempool=name=main100G;size=8000000 </code> |
| | |
| | In this example: |
| | * ports ''30–37'', ''40–43'', and ''50–53'' are grouped into pairs and served by separate dispatchers; |
| | * all dispatchers serving 10G ports use the shared ''mempool'' ''main10G'' with a size of 1,600,000 packets; |
| | * ports ''10'' and ''20'' are served by a dispatcher with RSS; |
| | * the ''rss=16'' parameter creates 16 dispatchers — one for each RX queue; |
| | * a separate ''mempool'' ''main100G'' with a size of 8,000,000 packets is used for 100G ports. |
| | |
| | Thus, ''dpdk_engine=7'' allows the load to be distributed independently between groups of ports with different performance levels and different ''mempool'' instances to be allocated for them. The ''mempool'' sizes in this example were selected based on the characteristics of the equipment and the expected load; for a specific configuration, the values should be selected based on the RX and TX queue sizes, the number of processing threads, and the sizes of the queues between dispatchers and slave threads. |
| | |
| | In the configuration described above, ''dpdk_engine=7'' is used as part of a test stand with simultaneous support for IPv4 and IPv6 (dual-stack), VRF isolation, DHCPv4/v6, PPPoE, and QinQ. The configuration is also designed to work with large routing tables: up to 5 million IPv4 routes and up to 800,000 IPv6 routes, including scenarios with BGP and OSPF and full-view BGP tables. |
| | |
| | <note important>The ''size=1600000'' and ''size=8000000'' values in the example are parameters of a specific test stand and are not recommended values for all configurations. The ''mempool'' size must be selected for the specific configuration based on the equipment characteristics and load.</note> |
