QoE Stor (Statistics collection module) [Документация VAS Experts]
Документация VAS Experts Документация VAS Experts-mobile
in

Settings

  • Show page
  • Old revisions
  • Export to PDF
  • Fold/unfold all
  • Backlinks
  • Show page
  • Old revisions
  • Export to PDF
  • Fold/unfold all
  • Backlinks

    • Differences

    This shows you the differences between two versions of the page.

    Link to this comparison view

    Both sides previous revisionPrevious revision
    Next revision    		Previous revision
    en:dpi:dpi_components:qoestor [2019/04/04 13:33] – kvazikraven:dpi:dpi_components:qoestor [2024/09/26 15:29] (current) – created - external edit 127.0.0.1
    Line 1: Line 1:
    -====== 7 QoE Stor Module ====== +====== QoE Stor (Statistics collection module) ====== 
    -== QoE analytics data collection and storage module ==+{{indexmenu_n>6}}
      
    - +  -[[en:dpi:dpi_components:qoestor:intro]] 
    -===== Introduction ===== +  -[[en:dpi:dpi_components:qoestor:install_and_update]] 
    - +  -[[en:dpi:dpi_components:qoestor:configuration]] 
    -The module is designed to collect and store Neflow and Clickstream data. Data is used to analyze QoE in DPIUI2. +  -[[en:dpi:dpi_components:qoestor:dictionaries]] 
    - +  -[[en:dpi:dpi_components:qoestor:data_export]] 
    - +  -[[en:dpi:dpi_components:qoestor:commands]]
    -===== Architecture ===== +
    - +
    -Data from the VAS Experts DPI is received on several sockets (tcp or udp) using [[en:dpi:dpi_components:utilities:ipfixreceiver2| utility designed to collect IPFIX stream data]].   +
    - +
    -The collected data is stored in the ClickHouse database. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_arc.png?400 |}} +
    - +
    -===== Installation and Update ===== +
    - +
    -==== Recommendations on device to be used for installing QoE Stor Module  ==== +
    - +
    -<note important>Do not install the module on the same server with the DPI platform!</note> +
    - +
    -=== Minimum requirements === +
    - +
    -For the subsystem, you can use hardware or virtual machines with the characteristics listed below: +
    -  - Processor (CPU) 2.5 ГГц - 1 pce +
    -  - RAM - at least 16 GB +
    -  - Hard disk drive (SSD is highly desirable) - at least 500 GB +
    -  - Operating system - Cent OS 6.4 - 7.x+ +
    -  - Network interface (NIC) - at least 1 Gbps +
    -**10 Gbps average daily traffic generates approximately 25 GB of data per hour in the QoE Store** +
    -== Processor == +
    - +
    -SSE 4.2 instruction set support is required. +
    - +
    -Choose processors with a large number of cores. Clock frequency is less important. For example, 16 cores with 2,600 MHz are better than 8 cores at 3,600 MHz. +
    - +
    -<note important>Do not disable Hyper-threading and Turbo-Boost.</note> +
    - +
    -== RAM == +
    - +
    -RAM should be no less than the amount of data requested. +
    - +
    -The more RAM server has, the better performance will be achieved when building reports. +
    - +
    -The more memory, the less disk system is stressing. +
    - +
    -Minimum prerequisites is 16 GB. +
    - +
    -Always disable the swap file. +
    - +
    -== Disk == +
    - +
    -Required disk space is at least 16 GB for every storage day, it's actual value depends on daily traffic. +
    - +
    -If your budget makes you possible to use SSD, use it. Otherwise use HDD. SATA HDDs 7200 RPM will be suitable. +
    - +
    -When using HDD, you can combine them into RAID-10, RAID-5, RAID-6 or RAID-50. +
    - +
    -Most of the data is stored in the ''/var/lib/clickhouse/'' directory. You can mount a drive/partition on this directory. +
    - +
    -Temporary data (ipfix dumps) is stored in the ''/var/qoestor/backend/dump/'' directory. 50 GB be sufficient for this. +
    - +
    -=== Maintenance advice from Yandex ClickHouse === +
    - +
    -You could familiarize yourself with the contents of maintenance advice from Yandex ClickHouse by following the [[https://clickhouse.yandex/docs/en/operations/tips/|https://clickhouse.yandex/docs/ru/operations/tips/]] link. +
    - +
    - +
    -==== Version Information ==== +
    - +
    -=== Version v.1.0.9 (21.02.2019) === +
    - +
    -  * Bug concerning incorrect recognition of trunk switches is fixed +
    -  * ''protocols_dic'' dictionary is updated +
    - +
    -=== Version v.1.0.7 hot fixes (24.12.2018) === +
    - +
    -  * ipfix re-export feature is added and is available through the ipfixreceiver configuration file: IPFIX_FULLFLOW_EXPORT and IPFIX_CLICKSTREAM_EXPORT +
    - +
    -=== Version v.1.0.6 hot fixes (04.12.2018) === +
    - +
    -  * Bugs concerning the work of the subnets_local_dic dictionary (such as "A call to function range would produce 12884901882 array elements") +
    -  * ipfixreceiver2 receiver configuration has been fixed (FileWriter queue is full. Records dropped.) +
    - +
    -=== Version v.1.0.5 (03.12.2018) === +
    - +
    -  * Host Category Dictionaries +
    -  * Transition to the ipfixreceiver2 +
    - +
    -<note important> +
    -  - Do not forget to update the installation script (in the Installation section) before upgrading. To upgrade, use the installation script. +
    -  - After the upgrade has been finished, check whether the receivers are running: <code>netstat -nlp | grep 1500</code> and <code>netstat -nlp | grep 1501</code> ipfixreceiver2 should listen corresponding sockets.  +
    -  - If receivers are not running, execute the /var/qoestor/backend/qoestor-config.sh script +
    -</note> +
    - +
    -=== Version v.1.0.4 (02.11.2018) === +
    - +
    -  * Preaggregation is implemented resulting in reducing the netflow by 6-7 times and clickstream by 3 times +
    -  * The following dictionaries are implemented: subscribers, switches, autonomous systems (AS), crc +
    -  * The following features are added: identification the traffic direction and subscribers filtering  (separation of IP hosts and IP subscribers) by AS and CIDR. This option  is meaningful only if the VAS Experts DPI is implemented using the mirror connection scheme. +
    - +
    -This QoE Stor version works with the version of DPIUI2-2.1.5 and higher +
    - +
    -<note important>If you have already installed version 1.0.0, then you should to delete the database due to complete version incompatibility before installing the new version. +
    -To do so issue the command:  <code>clickhouse-client --query="drop database qoestor"</code></note> +
    - +
    - +
    -=== Version v.1.0.0 (20.09.2018) === +
    - +
    -  * A new QoE Stor module is implemented +
    - +
    - +
    -==== Installation ==== +
    - +
    -<note warning>Before installing or upgrading, check your Internet connection. Make shure you run scripts under the root or using sudo.</note> +
    - +
    -For automatically installation or upgrading follow these steps: +
    - +
    -  - Execute [[https://vasexperts.ru/install/clickhouse-repos_config.sh.gz|clickhouse-repos_config.sh.gz]] script. The repository will be prepared for the subsequent installation of the clickhouse database as a result. +
    -  - Execute [[https://vasexperts.ru/install/fastor-rpm_install.sh.gz|fastor-rpm_install.sh.gz]] script. It will cause the installation of the following packages: ipfixreceiver, clickhouse, fastor. All of them will be automatically configured according to the defaults. +
    -  - Check whether the qoestor database in clickhouse is available. To do this, issue the command<code>clickhouse-client --query="show databases" | grep qoestor</code> +
    -  - If there is no database (probably the database server is not running), you should to create it by issuing the following command<code>clickhouse-client -n < /var/qoestor/backend/etc/db/qoestor.sql</code> +
    - +
    -==== Upgrading ==== +
    - +
    -Upgrading is performed using the same scripts as in the [[en:dpi:dpi_components:qoestor#инсталляция|installation]] section.  +
    - +
    -If receivers stopped after you have executed the <code>yum -y update</code> command, pleas refer to the troubleshooting section following the [[en:dpi:dpi_components:qoestor#выполнили_yum_y_update_не_запускаются_ресиверы|link]]. +
    - +
    -==== Configuration ==== +
    - +
    -=== ipfix receivers configuration === +
    - +
    - ipfix receivers configuration is implemented through the .env file: \\ <code>/var/qoestor/backend/.env</code> +
    - +
    -Default configuration looks like:\\ <code> +
    -#Ipfix form DPI 0 +
    -IPFIX_FULLFLOW_PORT_TYPE[0]=tcp +
    -IPFIX_FULLFLOW_PORT[0]=1500 +
    -#IPFIX_FULLFLOW_EXPORT[0]=10.0.0.2/9920/tcp,10.0.0.3/3440/udp +
    - +
    -IPFIX_CLICKSTREAM_PORT_TYPE[0]=tcp +
    -IPFIX_CLICKSTREAM_PORT[0]=1501 +
    -#IPFIX_CLICKSTREAM_EXPORT[0]=10.0.0.2/9921/tcp,10.0.0.3/3441/udp +
    - +
    -#Traffic direction definition +
    -# 0 - as is +
    -# 1 - by AS (for fullflow only) +
    -# 2 - by CIDR (for fullflow and clickstream) +
    -# 3 - by both: AS and CIDR +
    -# 4 - any: AS or CIDR +
    -TRAFFIC_DIR_DEF_MODE=0 +
    - +
    -#Subscriber filter +
    -# 0 - no filter +
    -# 1 - by AS (for fullflow only) +
    -# 2 - by CIDR (for fullflow and clickstream) +
    -# 3 - by both: AS and CIDR +
    -# 4 - any: AS or CIDR +
    -SUBSCRIBER_FILTER_MODE=0 +
    -</code> +
    - +
    -In the configuration above fullflow and clickstream receivers are listening 1500 and 1501 socket respectively. <<0>> in array subscript means that the receiver get the data from DPI number 0. +
    - +
    -<note tip>It is better to prefer tcp over udp because udp packets can be lost when the MTU is exceeded.</note> +
    - +
    -The values TRAFFIC_DIR_DEF_MODE = 0 and SUBSCRIBER_FILTER_MODE = 0 mean that there is no need to calculate the traffic direction and apply filters to subscribers. +
    - +
    -The IPFIX_FULLFLOW_EXPORT and IPFIX_CLICKSTREAM_EXPORT parameters allow you to configure export to third-party receivers. Format for use: ip/port/proto[,ip/port/proto. +
    - +
    -<note important>If the configuration has changed, you should run the /var/qoestor/backend/qoestor-config.sh script</note> +
    - +
    -**The following example shows how to configure data reception from several DPIs** \\ <code> +
    -#Ipfix form DPI 0 +
    -IPFIX_FULLFLOW_PORT_TYPE[0]=tcp +
    -IPFIX_FULLFLOW_PORT[0]=1500 +
    - +
    -IPFIX_CLICKSTREAM_PORT_TYPE[0]=tcp +
    -IPFIX_CLICKSTREAM_PORT[0]=1501 +
    - +
    -#Ipfix form DPI 1 +
    -IPFIX_FULLFLOW_PORT_TYPE[1]=tcp +
    -IPFIX_FULLFLOW_PORT[1]=1510 +
    - +
    -IPFIX_CLICKSTREAM_PORT_TYPE[1]=tcp +
    -IPFIX_CLICKSTREAM_PORT[1]=1511 +
    - +
    -#Ipfix form DPI 2 +
    -IPFIX_FULLFLOW_PORT_TYPE[2]=tcp +
    -IPFIX_FULLFLOW_PORT[2]=1520 +
    - +
    -IPFIX_CLICKSTREAM_PORT_TYPE[2]=tcp +
    -IPFIX_CLICKSTREAM_PORT[2]=1521 +
    -</code> +
    - +
    -**The following example corresponds to the situation when you need to identify subscribers by CIDR** +
    - +
    -This configuration makes sense only when the VAS Experts DPI is installed using port mirroring. +
    - +
    -<code> +
    -TRAFFIC_DIR_DEF_MODE=2 +
    -SUBSCRIBER_FILTER_MODE=2 +
    -</code> +
    - +
    -Be sure to configure subnets_local_dic dictionary for this configuration example! +
    - +
    -**The following example corresponds to the situation when export to third-party receivers is configured** +
    - +
    -<code> +
    -IPFIX_FULLFLOW_PORT_TYPE[0]=tcp +
    -IPFIX_FULLFLOW_PORT[0]=1500 +
    -IPFIX_FULLFLOW_EXPORT[0]=10.0.0.2/1600/tcp +
    - +
    -IPFIX_CLICKSTREAM_PORT_TYPE[0]=tcp +
    -IPFIX_CLICKSTREAM_PORT[0]=1501 +
    -IPFIX_CLICKSTREAM_EXPORT[0]=10.0.0.2/1601/tcp +
    -</code> +
    - +
    -== Restarting the receivers == +
    - +
    -All receivers can be restarted using the command: <code>/var/qoestor/backend/qoestor-config.sh</code> +
    - +
    -If you need to restart the receivers one by one, you can do this by restarting corresponding systemd service units, for example +
    - +
    -  * For CentOS 7 <code>systemctl restart qoestor_fullflow_0.service +
    -systemctl restart qoestor_clickstream_0.service</code> +
    -  * For CentOS 6 <code>service qoestor_fullflow_0 stop +
    -service qoestor_clickstream_0 stop +
    -service qoestor_fullflow_0 start +
    -service qoestor_clickstream_0 start</code> +
    - +
    -== Stoping the receivers == +
    -  * For CentOS 7 <code>systemctl stop qoestor_fullflow_0.service +
    -systemctl stop qoestor_clickstream_0.service</code> +
    -  * For CentOS 6 <code>service qoestor_clickstream_0 stop +
    -service qoestor_fullflow_0 stop</code> +
    - +
    -== Сlickhouse DB stop and start == +
    -  * Stop <code>sudo /etc/init.d/clickhouse-server stop</code> +
    -  * Start <code>sudo /etc/init.d/clickhouse-server restart</code> +
    - +
    -=== DPI configuration === +
    - +
    -== Export configuration == +
    - +
    -The DPI version must be at least 8.1. +
    - +
    -You can configure ipfix export by editing the fastdpi.conf configuration file on your DPI device. +
    -<code> +
    -netflow=8 +
    -netflow_dev=em1 +
    -netflow_timeout=10 +
    -netflow_as_direction=3 +
    -netflow_full_collector_type=2 +
    -netflow_full_port_swap=0 +
    -netflow_full_collector=YOUR_QOESTOR_IP:1500 +
    -netflow_passive_timeout=20 +
    -netflow_active_timeout=60 +
    -netflow_rate_limit=120 +
    -ipfix_dev=em1 +
    -ipfix_tcp_collectors=YOUR_QOESTOR_IP:1501 +
    -</code> +
    - +
    -<note important>fastdpi restart is needed for the changes to take effect:\\ +
    -**service fastdpi restart**</note> +
    - +
    - +
    -This can be achieved also using [[en:dpi:dpi_components:dpiui:dpiui2|DPIUI2]].The dpiui2 version must be at least 2.1.0. +
    - +
    -To perform configuration using DPIUI2 you should open the section DPI CONTROL -> CONFIGURATION. Open the tab **Collection and analysis of statistics on protocols and directions**. +
    - +
    -Set neflow option to **Export of complete statistics for sessions**. See figure below. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_конфигурация_dpi.png?400 |}} +
    - +
    -Then specify socket for fullflow receiver within "netflow_full_collector" parameter: "IP address of the netflow collector with full statistics (netflow_full_collector)". "netflow_full_collector_type" should be set to "Export ipfix to udp header", whereas "netflow_full_port_swap" should be empty or equals to "Keep original port numbers". See the figure below. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_конфигурация_dpi_1.png?400 |}} +
    - +
    -Type in the clickstream receiver socket in "ipfix_udp_collectors" parameter under the "DPI CONTROL -> CONFIGURATION -> Operator's SORM -> IP or the domain name (: port) of the ipfix collector clickstream (ipfix_udp_collectors)". See the figure below. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_конфигурация_dpi_2.png?400 |}} +
    - +
    -Press Save and then restart fast_dpi. See the figure below. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_конфигурация_dpi_save_restart.png?400 |}} +
    - +
    - +
    -== ID assignment to ipfix collector == +
    - +
    -Open the section Hardware "Management -> Equipment -> Hardware settings". Type in an identifier within the "Id on ipfix collector" section (under the "Ipfix settings") for the ipfix collector. See the figure below. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_конфигурация_dpi_set_dpi_num.png?400 |}} +
    - +
    -=== Setting up DPIUI2 connection to the QoE Stor module === +
    - +
    -In order to browse QoE reports, you should configure the DPIUI2 connection to the QoE Stor. See the [[en:dpi:dpi_components:dpiui:dpiui2#настройка_подключения_к_qoe_stor|Setting up a connection to the QoE Stor]] section. +
    - +
    -==== Configuring the dictionaries ==== +
    - +
    -All dictionaries are in the /var/qoestor/backend/etc/db/ directory and have .txt extension +
    - +
    -Each dictionary has it's own sample.txt which can be used as a pattern. +
    - +
    -All columns within the dictionaries are separated by a tab character (\t). The number of \t should be one less than the number of columns in dictionary. Please, pay close attention to this circumstance. +
    - +
    -When files are changed, the data is loaded into the database automatically. +
    - +
    -Some useful commands when working with dictionaries: +
    - +
    -  * Speed up data updates in directories <code>clickhouse-client --database=qoestor --query="system reload dictionaries"</code> +
    -  * Check dictionaries for errors <code>clickhouse-client --database=qoestor --query="select * from system.dictionaries"</code> +
    -  * Check if there is data in the directory, for example for the ''subnets_local_dic'' <code>clickhouse-client --database=qoestor --query="select * from subnets_local_dic"</code> +
    - +
    - +
    - +
    -==== asnum_local_dic and subnets_local_dic dictionaries ==== +
    - +
    -The lists of your local AS and local subnets are specified in these dictionaries. +
    -Dictionaries are used to identify the traffic direction (true in case the DPI is installed using mirroring) and to filter subscribers (so that the hosts IP addresses do not appear in the subscriber reports) +
    - +
    -Example of **asnum_local_dic** dictionary +
    -<code> +
    -12345 LOCAL +
    -65535 UNKNOWN +
    -</code> +
    - +
    -The first column is AS number, the second one is it's name (it is displayed in reports). +
    - +
    -Example of **subnets_local_dic** dictionary +
    -<code> +
    -192.168.1.0/24 LOCAL +
    -10.64.66.0/24 LOCAL +
    -172.16.0.0 LOCAL +
    -2a02:2168:aaa:bbbb::2 LOCAL +
    -</code> +
    - +
    -The first column is IP address or CIDR, the second one is the name (it is NOT displayed in reports, but it is required by format). +
    - +
    -<note important>Do not add too large subnet. Break into small ones. Highest value (limit) is 100000000</note> +
    - +
    -==== subscribers_dic, switches_dic, crc_dic dictionaries ==== +
    - +
    -=== subscribers_dic === +
    - +
    -Dictionary of subscribers. +
    - +
    -Dictionary example +
    -<code> +
    -10.64.66.100 login 5 port1 unit_vendor cabel contract services mac +
    -10.64.66.101 login 2 port1 unit_vendor cabel contract services mac +
    -10.64.66.102 login 3 port1 unit_vendor cabel contract services mac +
    -10.64.66.103 login 4 port1 unit_vendor cabel contract services mac +
    -10.64.66.104 login 5 port1 unit_vendor cabel contract services mac +
    -10.64.66.105 login 5 port2 unit_vendor cabel contract services mac +
    -10.64.66.106 login 5 port3 unit_vendor cabel contract services mac +
    -</code> +
    - +
    -Columns: +
    -  - IP address  +
    -  - Login +
    -  - Switch ID (access switch) +
    -  - Switch port +
    -  - Subscriber device vendor +
    -  - Cable +
    -  - Contract +
    -  - Services +
    -  - Subscriber device MAC address (is reserved for future purposes) +
    - +
    -=== switches_dic === +
    - +
    -Hierarchical dictionary of devices (access switches and trunk switches) +
    - +
    -Dictionary example +
    -<code> +
    -1 Switch_1 Ethernet Region_1 Address_1 10.140.1.18 ISP_1 0 0  +
    -2 Switch_2 Ethernet Region_2 Address_2 10.140.2.18 ISP_1 0 0  +
    -3 Switch_3 Ethernet Region_3 Address_3 10.140.3.18 ISP_1 0 1 port1  +
    -4 Switch_4 Ethernet Region_4 Address_4 10.140.4.18 ISP_1 0 3 port1  +
    -5 Switch_5 Ethernet Region_5 Address_5 10.140.5.18 ISP_1 0 4 port1  +
    -</code> +
    - +
    -Columns: +
    -  - Device ID UInt64 +
    -  - Device name +
    -  - Device type +
    -  - Region +
    -  - Address +
    -  - Switch IP address +
    -  - Internet service provider +
    -  - Indicator: trunk switch indicator (1 if so). Is not currently used, you can set 0 everywhere +
    -  - Upstream Switch ID UInt64 +
    -  - Upstream Switch port +
    -  - The owner +
    - +
    -=== crc_dic === +
    - +
    -Справочник ошибок (CRC) на портах коммутаторов +
    - +
    -Пример справочника +
    -<code> +
    -2 port1 450 +
    -5 port1 550 +
    -5 port2 500 +
    -4 port1 780 +
    -</code> +
    - +
    -Столбцы +
    -  - Идентификатор коммутатора +
    -  - Порт коммутатора +
    -  - Значение CRC  +
    - +
    -==== Справочники urlcats_dic и urlcats_host_dic ==== +
    - +
    -Справочники Категорий хостов. Предназначены для определения принадлежности хоста определённой категории.  +
    - +
    -Справочники подкачиваются автоматически с ресурсов vasexperts.ru. +
    - +
    -Для ускорения начальной загрузки выполните +
    -  - <code>sh /var/qoestor/backend/etc/cron_daily.sh</code> +
    -  - <code>clickhouse-client --database=qoestor --query="system reload dictionaries"</code> +
    - +
    - +
    -===== Проблемы и решения ===== +
    - +
    -==== Не работает, хотя все установили по инструкции ==== +
    - +
    -Если вы все установили и настроили по инструкции, а в разделе DPIUI2 “QoE Аналитика” пусто, то вот перечень шагов, которые стоит выполнить, прежде чем обращаться в тех. поддержку. +
    - +
    -  - В GUI попробовать  изменить период (из-за таймзон может не показывать) +
    -  - На сервере с QoE Stor проверить, создана ли БД <code>clickhouse-client --query="show databases" | grep qoestor</code> Если БД не создана, создать ее командой <code>clickhouse-client -n < /var/qoestor/backend/etc/db/qoestor.sql</code> +
    -  - На сервере с QoE Stor проверить, есть ли данные в БД <code>clickhouse-client --query="select count(), min(flow_start_time), max(flow_start_time) from qoestor.fullflow"</code> и <code>clickhouse-client --query="select count(), min(time), max(time) from qoestor.clickstream"</code> +
    -  - На сервере с QoE Stor проверить, есть ли что-либо в дампах ресиверов <code>/var/qoestor/backend/dump/fullflow</code> и <code>/var/qoestor/backend/dump/clickstream</code> +
    -  - На сервере с QoE Stor проверить логи ресиверов в папке <code>/var/qoestor/backend/logs</code> Eсть ли там что-то типа “oops!”? Обращайтесь в тех. поддержку, т.к скорее всего установлены компоненты не из нужных репо. Если там “Illegal IPFIX Message Version 0x0005”, то еще раз проверьте настройки экспорта на dpi: netflow_full_collector_type установлен неверно. +
    -  - На сервере с QoE Stor проверить, прослушиваются ли порты 1500 и 1501 командой <code>netstat -nlp | grep 1500 и netstat -nlp | grep 1501</code> Перезапустить все ресиверы на всякий случай командой <code>/var/qoestor/backend/qoestor-config.sh</code> +
    -  - Еще раз проверить [[dpi:dpi_components:qoestor#настройка_экспорта|настройки экспорта ipfix на dpi]] +
    -  - На сервере с DPIUI2 проверить [[dpi:dpi_components:dpiui:dpiui2#настройка_подключения_к_qoe_stor|настройки подключения GUI к QoE Stor]] +
    -  - На сервере с QoE Stor проверить, запущена ли СУБД ClickHouse командой <code>ps aux | grep clickhouse</code> Убедитесь, что достаточно оперативной памяти на сервере. +
    -  - На сервере с QoE Stor проверить логи clickhouse в папке /var/log/clickhouse-server/ +
    - +
    -Если есть необходимость очистить все данные в БД, то на сервере с QoE Stor надо +
    -  - Удалить БД командой <code>clickhouse-client --query="drop database qoestor"</code> +
    -  - Пересоздать БД командой <code>clickhouse-client -n < /var/qoestor/backend/etc/db/qoestor.sql</code> +
    - +
    -==== Выполнили yum –y update, не запускаются ресиверы ==== +
    - +
    -При выполнении **yum –y update** ломаются некоторые библиотеки. Ресиверы перестают запускаться. +
    -  - Удалите fastor и зависимости <code> yum remove fastor ipfixreceiver libfixbuf netsa_silk netsa-python</code> +
    -  - Установите заново, используя скрипт [[https://vasexperts.ru/install/fastor-rpm_install.sh.gz|fastor-rpm_install.sh.gz]] +
    - +
    - +
    -===== SQL и выгрузка данных в CSV, JSON, TabSeparated ===== +
    - +
    -При необходимости вы можете самостоятельно без дополнительных  инструментов сформировать собственные отчеты и выгрузить данные в любом формате CSV, JSON, TabSeparated. +
    - +
    -Данные хранятся в 4 основных логах +
    -  * qoestor.fullflow – полный netflow лог, период хранения – 24 часа +
    -  * qoestor.clicksteam – полный clickstream лог, период хранения – 24 часа +
    -  * qoestor.fullflow_agg – предагрегированный neflow лог, период хранения не ограничен  +
    -  * qoestor.clicksteam_agg – предагрегированный clickstream лог, период хранения не ограничен +
    - +
    -Формат команды следущий <code>clickhouse-client --database=qoestor  --query="Ваш sql тут"</code> +
    - +
    -По умолчанию данные выгружаются в формате TabSeparated. +
    - +
    -**Пример**. Клиент попросил лог соединений с определенным хостом в формате CSV +
    - +
    -<code>clickhouse-client --database=qoestor --query="select * from fullflow prewhere flow_start_date = '2018-10-04' where (source_ipv4 = '10.64.66.100' or destination_ipv4 = '10.64.66.100') and host = 'google.com' ORDER BY flow_start_time limit 10 format CSV"</code> +
    - +
    -Подробную информацию по SQL ClickHouse смотрите по ссылке [[https://clickhouse.yandex/docs/ru/query_language/select/]] +
    - +
    -~~DISCUSSION|Помогите нам улучшить документацию: если что-то непонятно или вы обнаружили ошибку, сообщите нам об этом.~~+