fastdpi_stat.log [Документация VAS Experts]
Документация VAS Experts Документация VAS Experts-mobile
in

Settings

  • Show page
  • Old revisions
  • Export to PDF
  • Fold/unfold all
  • Backlinks
  • Show page
  • Old revisions
  • Export to PDF
  • Fold/unfold all
  • Backlinks

    • Differences

    This shows you the differences between two versions of the page.

    Link to this comparison view

    Both sides previous revisionPrevious revision
    Next revision    		Previous revision
    en:dpi:dpi_components:platform:mon_stat:stat_log [2020/09/10 13:03] – edrudichgmailcomen:dpi:dpi_components:platform:mon_stat:stat_log [2025/01/20 10:18] (current) – elena.krasnobryzh
    Line 1: Line 1:
     ====== fastdpi_stat.log ====== ====== fastdpi_stat.log ======
     +{{indexmenu_n>3}}
     +It is located in the folder: ''/var/log/dpi/fastdpi_stat.log''\\.
     +This log contains statistics on the traffic processed by SSG DPI
     +<code bash>
     +[STAT    ][2022/04/07-13:36:24:608426] Memory usage : 'Virtual'/'Resident' 177805156352/3238039552
     +[STAT    ][2022/04/07-13:36:24:608441] CPU statistics :
     +        cpu_total : 18.8%us  1.7%sy  0.0%ni 79.5%id  0.0%wa
     +                cpu0 :  4.6%us  2.7%sy  0.0%ni 92.8%id  0.0%wa
     +                cpu1 : 20.1%us  0.4%sy  0.0%ni 79.5%id  0.0%wa
     +                cpu2 : 19.5%us  0.4%sy  0.0%ni 80.1%id  0.0%wa
     +                cpu3 : 23.4%us  2.1%sy  0.0%ni 74.5%id  0.0%wa
     +</code>
      
    -The file is placed in the directory: ///var/log/dpi/fastdpi_stat.log//\\+This site has a stat log:\\ 
     +''[2022/04/07-13:36:24:608426]'' — date and time of data capture
      
    -This log contains statistics on traffic processed and blocked by VAS Experts DPI, on memory and processor load (file **stat.log**).\\+'' 'Virtual'/'Resident' ''  — memory type.\\ 
     +''Virtual'' — virtual process size. Shows the total amount of memory that the program is able to address at a given time\\ 
     +''Resident'' — shows how much physical memory the process uses.\\ 
     +This parameter is followed by specific values
      
    -{{ :dpi:dpi_components:platform:mon_stat:stat_log1.png?600 |}}\\ +''CPU statistics'' — CPU utilization statistics\\ 
    -              Image 1+''cpu_total'' — total CPU utilization\\ 
     +''cpu0'', ''cpu1'', ''cpu2'', ''cpu3'' — kernel-specific loading\\ 
     +''us'' — how much is currently in use\\ 
     +''sy'' — how much the system is currently using\\ 
     +''id'' — how much is currently available\\ 
     +''wa'' — shows the percentage of operations ready to be executed by the processor but pending from the disk
      
    -Information is presented as follows (see Image 1, Image 2): +**Interface information:** 
    -  * Memory used:\\ **1** – date and time of data accessing, \\ **2** – memory type, \\ **3** – information volume. +<code bash> 
    -  * CPU load: \\ **4** – general load, \\ **5** – load by cores. +[STAT    ][2022/04/08-16:25:25:309501] Interface statistics : 
    -  * Statistics on VAS Experts DPI interfaces:\\ **6** – full statistics on received packets/bytes, blocked packets on all interfaces, \\ **7** - full statistics on received packets/bytes, blocked packets on the dna0 interface, here: +        Cluster #0 Absolute Stats Rcvd: [5000 pkts][527701 bytes][1+2=3 pkts dropped] 
    -    * Rcvd: [2372621 pkts][381635326 bytes][0 pkts dropped] – received packets/bytes +        Cluster #0 : IF 01-00.0 (01:00.0): 
    -    * Send: [4457954 pkts][4526709192 bytes] – transmitted packets/bytes +                Absolute Stats Rcvd: [4873 pkts][507823 bytes][0 pkts dropped] 
    -    * Esnd: [0 err_pkts][0.00 %] - errors occurred while sending packets +                               Send: [127 pkts][19878 bytes] 
    -    * Drop: [0 pkts][0 bytes] – blocked packets/bytes +                               Esnd: [0 err_pkts][0.00 %] 
    -    * Pthr: [0 pkts][0 bytes] – the number of packets/bytes passing without analysis and processing +                               Drop: [0 pkts][0 bytes] 
    -    * Emit: [0 pkts][0 bytes] – packets formed by VAS Experts DPI +                               Pthr: [0 pkts][0 bytes] 
    -    * Eemt: [0 err_pkts][0.00 %] - errors that occurred when sending packets generated by VAS Experts DPI\\ +                               Emit: [0 pkts][0 bytes] 
    -**8** – actual statistics on received packets/bytes, blocked packets on dna0 interface, \\ +                               Eemt: [0 err_pkts][0.00 %] 
    -**9** – full statistics on the number of captured, processed, sent packets/sec (see Image 2), e.g. [Captured 1.47 pkt/sec][Processed 1.47 pkt/sec][Send 0.00 pkt/sec].\\ IPv4_thread_slave=#1 or 0 – flow statistics (0 or 1) – flow number.\\ +                Actual   Stats Rcvd: [0 bytes][0.00 Mbit/sec] 
    -{{ :dpi:dpi_components:platform:mon_stat:stat_log2.png?600 |}}\\ +                                     [0 pkts ][0.00 pkt/sec] 
    -           Image 2+                               Send: [0 bytes][0.00 Mbit/sec] 
     +                                     [0 pkts ][0.00 pkt/sec] 
     +                               Esnd: [0 err_pkts][0.00 %] 
     +                               Drop: [0 bytes][0.00 %] 
     +                                     [0 pkts ][0.00 %] 
     +                               Pthr: [0 bytes][0.00 %] 
     +                                     [0 pkts ][0.00 %] 
     +                               Emit: [0 bytes][0.00 Mbit/sec] 
     +                                     [0 pkts ][0.00 pkt/sec] 
     +                               Eemt: [0 err_pkts][0.00 %] 
     +        Cluster #0 : IF 01-00.1 (01:00.1): 
     +                Absolute Stats Rcvd: [127 pkts][19878 bytes][0 pkts dropped] 
     +                               Send: [4873 pkts][507823 bytes] 
     +                               Esnd: [0 err_pkts][0.00 %] 
     +                               Drop: [0 pkts][0 bytes] 
     +                               Pthr: [0 pkts][0 bytes] 
     +                               Emit: [0 pkts][0 bytes] 
     +                               Eemt: [0 err_pkts][0.00 %] 
     +                Actual   Stats Rcvd: [0 bytes][0.00 Mbit/sec] 
     +                                     [0 pkts ][0.00 pkt/sec] 
     +                               Send: [0 bytes][0.00 Mbit/sec] 
     +                                     [0 pkts ][0.00 pkt/sec] 
     +                               Esnd: [0 err_pkts][0.00 %] 
     +                               Drop: [0 bytes][0.00 %] 
     +                                     [0 pkts ][0.00 %] 
     +                               Pthr: [0 bytes][0.00 %] 
     +                                     [0 pkts ][0.00 %] 
     +                               Emit: [0 bytes][0.00 Mbit/sec] 
     +                                     [0 pkts ][0.00 pkt/sec] 
     +                               Eemt: [0 err_pkts][0.00 %] 
     +          Cluster #0 : Aggregated Actual stats: [Captured 0.00 pkt/sec][Processed 0.00     pkt/sec][Send 0.00 pkt/sec] 
     +</code>
      
    -  * Protocol statistics: +''Absolute Stats Rcvd'' — total statistics of received packets/bytes, blocked packets on all interfaces, since the last restart of the fastDPI process\\ 
    -    * Statistics by IP:\\ **10** – current flows number, here\\ IPv4_total : allocate=1708/3008000 – parameter is set in ///etc/dpi/fastdpi.conf://\\ mem_tracking_flow (e.g.=3008000)\\ 3008000 – total / 1708 – taken +[1+2=3 pkts dropped]\\ 
    -    * Blocking counters:\\ url/lock=341/5 ( 0,0 )( 1,1,0,98879 )\\ ssl/lock=47/0 ( 21,457 )( 0,69,69,196647 )\\ chnprc=0\\ //ccheck/ip_check/lock//=2954/503/76\\ url/lock – URL checked/blocked\\  +1 — losses on the port (not even read, buffer overflowed)\\ 
    -    * ( 0,0 ) :\\ first 0 – number of URLs that could not be parsed\\ second 0 – number of packets with partial URLs (URL in several packets)\\ ( 1,1,0,98879 ) :\\ 1 – parsers used\\ 1 – parsers were used in total\\ 0 – how many parsers are not involved after use\\ 98879 –  +2 — SSG couldn't process 
    -how many parsers can be created\\ +
    -    * ssl/lock – similarly to URL, but for cname\\ chnprc=0 – parser change http ←→ htpps\\ //ccheck/ip_check/lock// – 2954/503/76 statistics on check by IP/port\\  +
    -    * 2954 – were to check by IP \\ 503 – how many times the check was actually performed\\ 76 – packets blocked +
    -  * Firewall statistics – **11**. +
    -  * Netflow statistics – **12**, +
      
    -In version 9.4.1 statistics on packet sizes have been expanded, Jumbo Frames have been added\\ +This is followed by information on each specific interface 
    -[STAT    ][2020/09/09-13:44:33:322801] Packet size (abs/delta, in %):+ 
     +''Cluster #0 : IF 01-00.0 (01:00.0)'':\\ 
     +''Absolute Stats'' — full statistics of received packets/bytes, blocked packets on interface ''01-00.0'' 
     +  * ''Rcvd: [4873 pkts][507823 bytes][0 pkts dropped]'' — received packets/bytes 
     +  * ''Send: [127 pkts][19878 bytes]'' — packets/bytes transmitted 
     +  * ''Esnd: [0 err_pkts][0.00 %]'' — errors that occurred when sending packets 
     +  * ''Drop: [0 pkts][0 bytes]'' — dropped packets/bytes, as a result of filtering/policing, etc. (“good” drops) 
     +  * ''Pthr: [0 pkts][0 bytes]'' — number of packets/bytes passing without analysis and processing 
     +  * ''Emit: [0 pkts][0 bytes]'' — the packets that the SSG generated 
     +  * ''Eemt: [0 err_pkts][0.00 %]'' — errors occurred when sending SSG-generated packets 
     +''Actual   Stats'' —  actual statistics of received packets/bytes, blocked packets on interface ''01-00.0''\\ 
     +''Aggregated Actual stats'' — aggregate statistics per cluster: how many packets captured, processed, sent/sec. 
     + 
     +<code bash> 
     +[STAT    ][2022/04/08-16:25:25:309514] [HAL] DPDK device statistics: 
     +        dev 01-00.0 (01:00.0) 
     +             RX pkt/bytes abs (delta):            4873/390871          (0/0) 
     +             TX pkt/bytes abs (delta):             127/16830           (0/0) 
     +                Error pkts, abs/delta: rx_queue_full=0/0, bad_pkt=0/0, tx_fail=0/0, rx_nombuf=0/0 
     +        dev 01-00.1 (01:00.1) 
     +             RX pkt/bytes abs (delta):             127/16830           (0/0) 
     +             TX pkt/bytes abs (delta):            4873/390871          (0/0) 
     +                Error pkts, abs/delta: rx_queue_full=0/0, bad_pkt=0/0, tx_fail=0/0, rx_nombuf=0/0 
     + 
     + 
     +[STAT    ][2022/04/08-16:25:25:309514] [HAL] DPDK device statistics: 
     +        dev 01-00.0 (01:00.0) 
     +             RX pkt/bytes abs (delta):            4873/390871          (0/0) 
     +             TX pkt/bytes abs (delta):             127/16830           (0/0) 
     +                Error pkts, abs/delta: rx_queue_full=0/0, bad_pkt=0/0, tx_fail=0/0, rx_nombuf=0/0 
     +        dev 01-00.1 (01:00.1) 
     +             RX pkt/bytes abs (delta):             127/16830           (0/0) 
     +             TX pkt/bytes abs (delta):            4873/390871          (0/0) 
     +                Error pkts, abs/delta: rx_queue_full=0/0, bad_pkt=0/0, tx_fail=0/0, rx_nombuf=0/0 
     +[STAT    ][2022/04/08-16:25:25:309644]  [HAL][DPDK] Dispatcher statistics abs/delta: 
     +                         drop (worker queue full)           |  empty NIC RX |       RX packets 
     +        Cluster #0:               0/0           0.0%/  0.0% | 100.0%/100.0% |            5000/0 
     +</code> 
     + 
     +Above are the statistics for the interfaces:\\ 
     +''RX pkt/bytes abs (delta):            4873/390871          (0/0)'' — packets/byte received\\ 
     +''4873/390871'' — from the start\\ 
     +''(0/0)'' — for the last 15 sec (since the last stat log output) 
     + 
     +''TX pkt/bytes abs (delta):'' — packets sent/byte 
     + 
     +<code bash>Error pkts, abs/delta: rx_queue_full=0/0, bad_pkt=0/0, tx_fail=0/0, rx_nombuf=0/0</code> 
     + 
     +''rx_queue_full=0/0'' — dispatcher queue overflow\\ 
     +''bad_pkt=0/0'' — bad packages\\ 
     +''tx_fail=0/0'' — sending errors\\ 
     +''rx_nombuf=0/0'' — there wasn't enough buffer for reception 
     + 
     +''drop (worker queue full)'' — illegitimate drops (handler overflow) 
     + 
     +''empty NIC RX'' — percentage of idle polling of rx queues of cards - absolute percentage (since the beginning of SSG operation) and relative (delta since the last output to stat-log). 100% - means that there are no input packets, the dispatcher is idle 
     + 
     +<code bash> 
     +[STAT    ][2022/04/08-16:25:25:309647] [HAL][DPDK/SQRX] Mempool state: 
     +        cluster #0: avail_count=24448, in-use_count=8319 
     +</code> 
     + 
     +Memory Pool Utilization ''dpdk_mempool_size'':\\ 
     +''avail_count'' — available for use\\ 
     +''in-use_count'' — currently in use 
     + 
     +The plot below shows statistics on packet sizes, and Jumbo Frames ranges have also been added  
     +<code bash> 
     +[STAT    ][2022/04/08-16:25:25:309650] Packet size (abs/delta, in %):
                              <=64       <=128       <=256       <=512      <=1024      <=2048      <=4096      <=8192     >8192                          <=64       <=128       <=256       <=512      <=1024      <=2048      <=4096      <=8192     >8192
    -        subs->inet:   0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0 +        subs->inet:   0.5/0.0    98.7/0.0     0.6/0.0     0.2/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0 
    -        inet->subs:   0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0+        inet->subs:  17.3/0.0    51.2/0.0    25.2/0.0     6.3/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0     0.0/0.0 
     +</code> 
     + 
     +The following are statistics on protocols\\ 
     +IP statistics:\\ 
     +This shows the number of streams (flow) and information about them 
     +<code bash> 
     +[STAT    ][2022/04/08-16:25:25:309664] IPv4_Statistics 'flow nodes' : 
     +        IPv4_thread_slave=#0 : 0/0/505/0/0 ( 180/0/325 ) ( 0-0/0-0/0-0/0-0 ) 
     +                0/0/66666/66666 ( 0/0 0/0 0/0/0 ) 
     +        IPv4_thread_slave=#1 : 0/0/1796/0/0 ( 436/0/1360 ) ( 0-0/0-0/0-0/0-0 ) 
     +                0/0/66666/66666 ( 0/0 0/0 0/0/0 ) 
     + IPv4_total : allocate=616/4896000 ( 0/0/2301/0/0/0 ) ( 616/0/1685 ) ( 0-0/0-0/0-0/0-0 ) 
     +                0/0/133332/133332 ( 0/0 0/0 0/0/0 ) 
     +        IPv4_actual: new=0 [0 flw/sec] close=0 [0 flw/sec] rei=0 [0 flw/sec] 
     +</code> 
     + 
     +''IPv4_total : allocate=616/4896000'' — shows allocated memory occupancy for IPv4 flow\\ 
     +''616'' — busy, ''4896000'' — total. This parameter is set in the total file. This parameter is set in the ''/etc/dpi/fastdpi.conf'' file (''[[en:dpi:dpi_components:platform:dpi_admin:mem_problems#mem_tracking_flow|mem_tracking_flow]]'') 
     + 
     +<code bash>IPv4_actual: new=0 [0 flw/sec] close=0 [0 flw/sec] rei=0 [0 flw/sec]</code> 
     +''new'' — number of new flows\\ 
     +''close'' — flow rate\\ 
     +''rei'' — ready for reuse 
     + 
     +<code bash> 
     +[STAT    ][2022/04/12-11:15:31:688997] IPv4_Statistics_error  : 
     +        IPv4_ste_flow   : 0/0/0 
     +        IPv4_ste_invlen : 0/0/0 
     +</code> 
     + 
     +''IPv4_ste_flow'' — processing errors. This is a critical counter. It should be zero (everything is fine here)\\ 
     +''IPv4_ste_invlen'' — errors of read lengths from the input frame (when the actual length diverges from the length specified in the header). I.e. the reason is in the package 
     + 
     +''0/0/0'' — ip/tcp/udp 
     + 
     + 
     +Blocking statistics:\\ 
     +These parameters are scheduled for each specific thread (''thread_slave''), as well as the total value (''Total'') 
     + 
     +<code bash> 
     +[STAT    ][2022/04/12-11:15:31:688999] Detailed statistics on HTTP : 
     +        thread_slave=0 : 
     +                url/lock=28/0 ( 12,0,0 )( 1,1,0 ) 
     +                ssl/lock=191/0 ( 0,54,0 )( 1,17,16 ) 
     +                        cna/lock=4/0 ( 0,37 ) 
     +                        sni/lock=187/0 ( 0,17 ) 
     +                quic/lock=0/0 ( 1,0,0 )( 0,0,0 ) 
     +                chnprc=0 
     +                ccheck/ip_check/lock=2203/579/0 0/0/0 
     +        thread_slave=1 : 
     +                url/lock=187/0 ( 1287,0,0 )( 1,1,0 ) 
     +                ssl/lock=268/2 ( 0,313,0 )( 2,36,34 ) 
     +                        cna/lock=1/0 ( 0,171 ) 
     +                        sni/lock=267/2 ( 0,142 ) 
     +                quic/lock=9/0 ( 0,0,0 )( 0,0,0 ) 
     +                chnprc=0 
     +                ccheck/ip_check/lock=9404/747/0 0/0/0 
     +        Total : 
     +                url/lock=215/0 ( 1299,0,0 )( 2,2,0,98879 ) 
     +                ssl/lock=459/2 ( 0,367,0 )( 3,53,50,392183 ) 
     +                        cna/lock=5/0 ( 0,208 ) 
     +                        sni/lock=454/2 ( 0,159 ) 
     +                quic/lock=9/0 ( 1,0,0 )( 0,0,0,0 ) 
     +                chnprc=0 
     +                ccheck/ip_check/lock=11607/1326/0 0/0/0 
     +</code> 
     + 
     +''url/lock'' — URL checked / blocked (similar for ssl, cna, sni, quic)\\ 
     +''chnprc=0'' — change parser http ←→ https\\ 
     +''ccheck/ip_check/lock=11607/1326/0'' — IP/port check statistics 
     + 
     +''11607'' — should have run an IP check\\ 
     +''1326'' — how many times the test was actually performed\\ 
     +''0'' — blocked packets 
     + 
     +Below are the statistics on firewall and syn packages: 
     + 
     +<code bash> 
     +[STAT    ][2022/04/12-11:15:31:689052] FRWL statistics : 0/0/0 
     +[STAT    ][2022/04/12-11:15:31:689054] Statistics SYN : 
     +        total : syn=1, syn_ack=1 (0/0/0/0 0/0) 
     +        actual: syn=0 [0 syn/sec] syn_ack=0 [0 syn_ack/sec] [prcnt=0%] (0/0/0/0 0/0) 
     +[STAT    ][2022/04/12-11:15:31:689052] FRWL statistics : <wrap hi>0/0/0</wrap> 
     +[STAT    ][2022/04/12-11:15:31:689054] Statistics SYN : 
     +        total : syn=1, syn_ack=1 (0/0/0/0 0/0) 
     +        actual: syn=0 [0 syn/sec] syn_ack=0 [0 syn_ack/sec] [prcnt=0%] (0/0/0/0 0/0) 
     +</code> 
     + 
     +''total : syn=1'' — total SYN packets\\ 
     +''syn_ack'' — total SYN-ACK packets\\ 
     +''actual:'' — the same, only for the last 15 seconds (since the last output to stat log) + number of SYN/SYN-ACK per second