QoE Stor (Statistics collection module) [Документация VAS Experts]
Документация VAS Experts Документация VAS Experts-mobile
in

Settings

  • Show page
  • Old revisions
  • Export to PDF
  • Fold/unfold all
  • Backlinks
  • Show page
  • Old revisions
  • Export to PDF
  • Fold/unfold all
  • Backlinks

    • Differences

    This shows you the differences between two versions of the page.

    Link to this comparison view

    Both sides previous revisionPrevious revision
    Next revision    		Previous revision
    en:dpi:dpi_components:qoestor [2020/10/14 14:31] – [Version Information] arusnaken:dpi:dpi_components:qoestor [2024/09/26 15:29] (current) – created - external edit 127.0.0.1
    Line 1: Line 1:
    -====== 7 QoE Stor Module ====== +====== QoE Stor (Statistics collection module) ====== 
    -== QoE analytics data collection and storage module ==+{{indexmenu_n>6}}
      
    - +  -[[en:dpi:dpi_components:qoestor:intro]] 
    -===== Introduction ===== +  -[[en:dpi:dpi_components:qoestor:install_and_update]] 
    - +  -[[en:dpi:dpi_components:qoestor:configuration]] 
    -The module is designed to collect and store Neflow and Clickstream data. Data is used to analyze QoE in DPIUI2. +  -[[en:dpi:dpi_components:qoestor:dictionaries]] 
    - +  -[[en:dpi:dpi_components:qoestor:data_export]] 
    - +  -[[en:dpi:dpi_components:qoestor:commands]]
    -===== Architecture ===== +
    - +
    -Data from the VAS Experts DPI is received on several sockets (tcp or udp) using [[en:dpi:dpi_components:utilities:ipfixreceiver2| utility designed to collect IPFIX stream data]].   +
    - +
    -The collected data is stored in the ClickHouse database. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_arc.png?400 |}} +
    - +
    -===== Installation and Update ===== +
    - +
    -==== Recommendations on device to be used for installing QoE Stor Module  ==== +
    - +
    -<note important>Do not install the module on the same server with the DPI platform!</note> +
    - +
    -=== Minimum requirements === +
    - +
    -For the subsystem, you can use hardware or virtual machines with the characteristics listed below: +
    -  - Processor (CPU) 2.5 ГГц - 1 pce +
    -  - RAM - at least 16 GB +
    -  - Hard disk drive (SSD is highly desirable) - at least 500 GB +
    -  - Operating system - CentOS 7+ or CentOS 8+ +
    -  - Network interface (NIC) - at least 1 Gbps +
    -<note important>{{ :dpi:dpi_components:capacity_space_for_qoe_and_hardware_example.xlsx |Capacity space for QoE and Hardware example}}</note> +
    -== Processor == +
    - +
    -SSE 4.2 instruction set support is required. +
    - +
    -Choose processors with a large number of cores. Clock frequency is less important. For example, 16 cores with 2,600 MHz are better than 8 cores at 3,600 MHz. +
    - +
    -<note important>Do not disable Hyper-threading and Turbo-Boost.</note> +
    - +
    -== RAM == +
    - +
    -RAM should be no less than the amount of data requested. +
    - +
    -The more RAM server has, the better performance will be achieved when building reports. +
    - +
    -The more memory, the less disk system is stressing. +
    - +
    -Minimum prerequisites is 16 GB. +
    - +
    -Always disable the swap file. +
    - +
    -== Disk == +
    - +
    -Required disk space is at least 16 GB for every storage day, it's actual value depends on daily traffic. It is estimated that 10 Gbit/s of average daily traffic generates approximately 25 GB of data per hour in the QoE Store. +
    - +
    -If your budget makes you possible to use SSD, use it. Otherwise use HDD. SATA HDDs 7200 RPM will be suitable. +
    - +
    -When using HDD, you can combine them into RAID-10, RAID-5, RAID-6 or RAID-50. +
    - +
    -Most of the data is stored in the ''/var/lib/clickhouse/'' directory. You can mount a drive/partition on this directory. +
    - +
    -Temporary data (ipfix dumps) is stored in the ''/var/qoestor/backend/dump/'' directory. 50 GB be sufficient for this. +
    - +
    -=== Maintenance advice from Yandex ClickHouse === +
    - +
    -You could familiarize yourself with the contents of maintenance advice from Yandex ClickHouse by following the [[https://clickhouse.yandex/docs/en/operations/tips/|https://clickhouse.yandex/docs/ru/operations/tips/]] link. +
    - +
    - +
    -==== Version Information ==== +
    - +
    -=== Version v.1.7.3 (10/15/2020) === +
    - +
    -   * The ability to store data on HOT and COLD disks +
    -   * Reception and storage of NAT logs. The option to generate a NAT log from fullflow. Flexible configuration of the aggregation period and the list of fields to be aggregated +
    -   * Reception and storage of GTP logs +
    - +
    -=== Version v.1.6.0 (09/14/2020) === +
    - +
    -   * Option: substitution of the Login field with the value from the vchannel field +
    -   * Bug fix +
    - +
    -=== Version v.1.4.2 (06/02/2020) === +
    - +
    -  * Bug fix +
    - +
    -=== Version v.1.4.0 (05/04/2020) === +
    - +
    -  * Clickhouse 20.3 compatibility support +
    -  * Ability to install on CentOS 8 +
    - +
    -=== Version v.1.3.8 (04/09/2020) === +
    - +
    -  * Updated protocol dictionaries +
    -  * Added auto-update of the AS dictionary +
    - +
    -=== Version v.1.3.6 (11/25/2019) === +
    - +
    -  * Updated protocol dictionaries +
    - +
    -=== Version v.1.3.5 (11/06/2019) === +
    - +
    -  * Fixed the behavior of the dictionary of local subnets (got extra addresses) +
    -  * Adapted clickstream loading for the correct operation of the user interface in dpiui2-2.8.2 +
    - +
    -=== Version v.1.3.4 (10.25.2019) === +
    - +
    -  * Locked in clickstream +
    - +
    -=== Version v.1.3.3 (10/15/2019) === +
    - +
    -  * Update Clickhouse to the latest version (> = 19.15) +
    -  * Improved method of loading and processing logs +
    -  * Prepared base for the analysis of raw logs +
    -  * Connection logs Clear and Netflow +
    - +
    - +
    -=== Version v.1.1.1 (09/06/2019) === +
    - +
    -  * Update Clickhouse to the latest version +
    -  * Directories asnum_exclude_diс, subnets_exclude_dic and the corresponding filtering mode for these directories +
    - +
    -=== Version v.1.0.9 (21.02.2019) === +
    - +
    -  * Bug concerning incorrect recognition of trunk switches is fixed +
    -  * ''protocols_dic'' dictionary is updated +
    - +
    -=== Version v.1.0.7 hot fixes (24.12.2018) === +
    - +
    -  * ipfix re-export feature is added and is available through the ipfixreceiver configuration file: IPFIX_FULLFLOW_EXPORT and IPFIX_CLICKSTREAM_EXPORT +
    - +
    -=== Version v.1.0.6 hot fixes (04.12.2018) === +
    - +
    -  * Bugs concerning the work of the subnets_local_dic dictionary (such as "A call to function range would produce 12884901882 array elements") +
    -  * ipfixreceiver2 receiver configuration has been fixed (FileWriter queue is full. Records dropped.) +
    - +
    -=== Version v.1.0.5 (03.12.2018) === +
    - +
    -  * Host Category Dictionaries +
    -  * Transition to the ipfixreceiver2 +
    - +
    -<note important> +
    -  - Do not forget to update the installation script (in the Installation section) before upgrading. To upgrade, use the installation script. +
    -  - After the upgrade has been finished, check whether the receivers are running: <code>netstat -nlp | grep 1500</code> and <code>netstat -nlp | grep 1501</code> ipfixreceiver2 should listen corresponding sockets.  +
    -  - If receivers are not running, execute the /var/qoestor/backend/qoestor-config.sh script +
    -</note> +
    - +
    -=== Version v.1.0.4 (02.11.2018) === +
    - +
    -  * Preaggregation is implemented resulting in reducing the netflow by 6-7 times and clickstream by 3 times +
    -  * The following dictionaries are implemented: subscribers, switches, autonomous systems (AS), crc +
    -  * The following features are added: identification the traffic direction and subscribers filtering  (separation of IP hosts and IP subscribers) by AS and CIDR. This option  is meaningful only if the VAS Experts DPI is implemented using the mirror connection scheme. +
    - +
    -This QoE Stor version works with the version of DPIUI2-2.1.5 and higher +
    - +
    -<note important>If you have already installed version 1.0.0, then you should to delete the database due to complete version incompatibility before installing the new version. +
    -To do so issue the command:  <code>clickhouse-client --query="drop database qoestor"</code></note> +
    - +
    - +
    -=== Version v.1.0.0 (20.09.2018) === +
    - +
    -  * A new QoE Stor module is implemented +
    - +
    - +
    -==== Installation ==== +
    - +
    -<note warning>Before installing or upgrading, check your Internet connection. Make shure you run scripts under the root or using sudo.</note> +
    - +
    -For automatically installation or upgrading follow these steps: +
    - +
    -  - Execute [[https://vasexperts.ru/install/fastor-rpm_install.sh|fastor-rpm_install.sh]] script. <code> +
    -sudo yum install wget +
    - +
    -sudo wget https://vasexperts.ru/install/fastor-rpm_install.sh +
    - +
    -sudo sh fastor-rpm_install.sh</code> It will cause the installation of the following packages: ipfixreceiver, clickhouse, fastor. All of them will be automatically configured according to the defaults. +
    -  - Check whether the qoestor database in clickhouse is available. To do this, issue the command<code>clickhouse-client --query="show databases" | grep qoestor</code> +
    -  - If there is no database (probably the database server is not running), you should to create it by issuing the following command<code>clickhouse-client -n < /var/qoestor/backend/etc/db/qoestor.sql</code> +
    - +
    -==== Upgrading ==== +
    - +
    -Upgrading is performed using the same scripts as in the [[en:dpi:dpi_components:qoestor#инсталляция|installation]] section.  +
    - +
    -If receivers stopped after you have executed the <code>yum -y update</code> command, pleas refer to the troubleshooting section following the [[en:dpi:dpi_components:qoestor#выполнили_yum_y_update_не_запускаются_ресиверы|link]]. +
    - +
    -==== Configuration ==== +
    - +
    -=== ipfix receivers configuration === +
    - +
    - ipfix receivers configuration is implemented through the .env file: \\ <code>/var/qoestor/backend/.env</code> +
    - +
    -Default configuration looks like:\\ <code> +
    -#Ipfix form DPI 0 +
    -IPFIX_FULLFLOW_PORT_TYPE[0]=tcp +
    -IPFIX_FULLFLOW_PORT[0]=1500 +
    -#IPFIX_FULLFLOW_EXPORT[0]=10.0.0.2/9920/tcp,10.0.0.3/3440/udp +
    - +
    -IPFIX_CLICKSTREAM_PORT_TYPE[0]=tcp +
    -IPFIX_CLICKSTREAM_PORT[0]=1501 +
    -#IPFIX_CLICKSTREAM_EXPORT[0]=10.0.0.2/9921/tcp,10.0.0.3/3441/udp +
    - +
    -#Traffic direction definition +
    -# 0 - as is +
    -# 1 - by AS (for fullflow only) +
    -# 2 - by CIDR (for fullflow and clickstream) +
    -# 3 - by both: AS and CIDR +
    -# 4 - any: AS or CIDR +
    -TRAFFIC_DIR_DEF_MODE=0 +
    - +
    -#Subscriber filter +
    -# 0 - no filter +
    -# 1 - by AS (for fullflow only) +
    -# 2 - by CIDR (for fullflow and clickstream) +
    -# 3 - by both: AS and CIDR +
    -# 4 - any: AS or CIDR +
    -SUBSCRIBER_FILTER_MODE=0 +
    -</code> +
    - +
    -In the configuration above fullflow and clickstream receivers are listening 1500 and 1501 socket respectively. <<0>> in array subscript means that the receiver get the data from DPI number 0. +
    - +
    -<note tip>It is better to prefer tcp over udp because udp packets can be lost when the MTU is exceeded.</note> +
    - +
    -The values TRAFFIC_DIR_DEF_MODE = 0 and SUBSCRIBER_FILTER_MODE = 0 mean that there is no need to calculate the traffic direction and apply filters to subscribers. +
    - +
    -The IPFIX_FULLFLOW_EXPORT and IPFIX_CLICKSTREAM_EXPORT parameters allow you to configure export to third-party receivers. Format for use: ip/port/proto[,ip/port/proto. +
    - +
    -<note important>If the configuration has changed, you should run the /var/qoestor/backend/qoestor-config.sh script</note> +
    - +
    -**The following example shows how to configure data reception from several DPIs** \\ <code> +
    -#Ipfix form DPI 0 +
    -IPFIX_FULLFLOW_PORT_TYPE[0]=tcp +
    -IPFIX_FULLFLOW_PORT[0]=1500 +
    - +
    -IPFIX_CLICKSTREAM_PORT_TYPE[0]=tcp +
    -IPFIX_CLICKSTREAM_PORT[0]=1501 +
    - +
    -#Ipfix form DPI 1 +
    -IPFIX_FULLFLOW_PORT_TYPE[1]=tcp +
    -IPFIX_FULLFLOW_PORT[1]=1510 +
    - +
    -IPFIX_CLICKSTREAM_PORT_TYPE[1]=tcp +
    -IPFIX_CLICKSTREAM_PORT[1]=1511 +
    - +
    -#Ipfix form DPI 2 +
    -IPFIX_FULLFLOW_PORT_TYPE[2]=tcp +
    -IPFIX_FULLFLOW_PORT[2]=1520 +
    - +
    -IPFIX_CLICKSTREAM_PORT_TYPE[2]=tcp +
    -IPFIX_CLICKSTREAM_PORT[2]=1521 +
    -</code> +
    - +
    -**The following example corresponds to the situation when you need to identify subscribers by CIDR** +
    - +
    -This configuration makes sense only when the VAS Experts DPI is installed using port mirroring. +
    - +
    -<code> +
    -TRAFFIC_DIR_DEF_MODE=2 +
    -SUBSCRIBER_FILTER_MODE=2 +
    -</code> +
    - +
    -Be sure to configure subnets_local_dic dictionary for this configuration example! +
    - +
    -**The following example corresponds to the situation when export to third-party receivers is configured** +
    - +
    -<code> +
    -IPFIX_FULLFLOW_PORT_TYPE[0]=tcp +
    -IPFIX_FULLFLOW_PORT[0]=1500 +
    -IPFIX_FULLFLOW_EXPORT[0]=10.0.0.2/1600/tcp +
    - +
    -IPFIX_CLICKSTREAM_PORT_TYPE[0]=tcp +
    -IPFIX_CLICKSTREAM_PORT[0]=1501 +
    -IPFIX_CLICKSTREAM_EXPORT[0]=10.0.0.2/1601/tcp +
    -</code> +
    - +
    -== Restarting the receivers == +
    - +
    -All receivers can be restarted using the command: <code>/var/qoestor/backend/qoestor-config.sh</code> +
    - +
    -If you need to restart the receivers one by one, you can do this by restarting corresponding systemd service units, for example +
    - +
    -  * For CentOS 7 <code>systemctl restart qoestor_fullflow_0.service +
    -systemctl restart qoestor_clickstream_0.service</code> +
    -  * For CentOS 6 <code>service qoestor_fullflow_0 stop +
    -service qoestor_clickstream_0 stop +
    -service qoestor_fullflow_0 start +
    -service qoestor_clickstream_0 start</code> +
    - +
    -== Stoping the receivers == +
    -  * For CentOS 7 <code>systemctl stop qoestor_fullflow_0.service +
    -systemctl stop qoestor_clickstream_0.service</code> +
    -  * For CentOS 6 <code>service qoestor_clickstream_0 stop +
    -service qoestor_fullflow_0 stop</code> +
    - +
    -== Сlickhouse DB stop and start == +
    -  * Stop <code>sudo /etc/init.d/clickhouse-server stop</code> +
    -  * Start <code>sudo /etc/init.d/clickhouse-server restart</code> +
    - +
    -=== DPI configuration === +
    - +
    -== Export configuration == +
    - +
    -The DPI version must be at least 8.1. +
    - +
    -You can configure ipfix export by editing the fastdpi.conf configuration file on your DPI device. +
    -<code> +
    -netflow=8 +
    -netflow_dev=em1 +
    -netflow_timeout=10 +
    -netflow_as_direction=3 +
    -netflow_full_collector_type=2 +
    -netflow_full_port_swap=0 +
    -netflow_full_collector=YOUR_QOESTOR_IP:1500 +
    -ipfix_dev=em1 +
    -ipfix_tcp_collectors=YOUR_QOESTOR_IP:1501 +
    -</code> +
    - +
    -<note important>fastdpi restart is needed for the changes to take effect:\\ +
    -**service fastdpi restart**</note> +
    - +
    - +
    -This can be achieved also using [[en:dpi:dpi_components:dpiui:dpiui2|DPIUI2]].The dpiui2 version must be at least 2.1.0. +
    - +
    -To perform configuration using DPIUI2 you should open the section DPI CONTROL -> CONFIGURATION. Open the tab **Collection and analysis of statistics on protocols and directions**. +
    - +
    -Set neflow option to **Export of complete statistics for sessions**. See figure below. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_конфигурация_dpi.png?400 |}} +
    - +
    -Then specify socket for fullflow receiver within "netflow_full_collector" parameter: "IP address of the netflow collector with full statistics (netflow_full_collector)". "netflow_full_collector_type" should be set to "Export ipfix to udp header", whereas "netflow_full_port_swap" should be empty or equals to "Keep original port numbers". See the figure below. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_конфигурация_dpi_1.png?400 |}} +
    - +
    -Type in the clickstream receiver socket in "ipfix_udp_collectors" parameter under the "DPI CONTROL -> CONFIGURATION -> Operator's SORM -> IP or the domain name (: port) of the ipfix collector clickstream (ipfix_udp_collectors)". See the figure below. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_конфигурация_dpi_2.png?400 |}} +
    - +
    -Press Save and then restart fast_dpi. See the figure below. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_конфигурация_dpi_save_restart.png?400 |}} +
    - +
    - +
    -== ID assignment to ipfix collector == +
    - +
    -Open the section Hardware "Management -> Equipment -> Hardware settings". Type in an identifier within the "Id on ipfix collector" section (under the "Ipfix settings") for the ipfix collector. See the figure below. +
    - +
    -{{ :dpi:dpi_components:qoestor:qoestor_конфигурация_dpi_set_dpi_num.png?400 |}} +
    - +
    -=== Setting up DPIUI2 connection to the QoE Stor module === +
    - +
    -In order to browse QoE reports, you should configure the DPIUI2 connection to the QoE Stor. See the [[en:dpi:dpi_components:dpiui:dpiui2#настройка_подключения_к_qoe_stor|Setting up a connection to the QoE Stor]] section. +
    - +
    -==== Configuring the dictionaries ==== +
    - +
    -All dictionaries are in the /var/qoestor/backend/etc/db/ directory and have .txt extension +
    - +
    -Each dictionary has it's own sample.txt which can be used as a pattern. +
    - +
    -All columns within the dictionaries are separated by a tab character (\t). The number of \t should be one less than the number of columns in dictionary. Please, pay close attention to this circumstance. +
    - +
    -When files are changed, the data is loaded into the database automatically. +
    - +
    -Some useful commands when working with dictionaries: +
    - +
    -  * Speed up data updates in directories <code>clickhouse-client --database=qoestor --query="system reload dictionaries"</code> +
    -  * Check dictionaries for errors <code>clickhouse-client --database=qoestor --query="select * from system.dictionaries"</code> +
    -  * Check if there is data in the directory, for example for the ''subnets_local_dic'' <code>clickhouse-client --database=qoestor --query="select * from subnets_local_dic"</code> +
    - +
    - +
    - +
    -==== asnum_local_dic and subnets_local_dic dictionaries ==== +
    - +
    -The lists of your local AS and local subnets are specified in these dictionaries. +
    -Dictionaries are used to identify the traffic direction (true in case the DPI is installed using mirroring) and to filter subscribers (so that the hosts IP addresses do not appear in the subscriber reports) +
    - +
    -Example of **asnum_local_dic** dictionary +
    -<code> +
    -12345 LOCAL +
    -65535 UNKNOWN +
    -</code> +
    - +
    -The first column is AS number, the second one is it's name (it is displayed in reports). +
    - +
    -Example of **subnets_local_dic** dictionary +
    -<code> +
    -192.168.1.0/24 LOCAL +
    -10.64.66.0/24 LOCAL +
    -172.16.0.0 LOCAL +
    -2a02:2168:aaa:bbbb::2 LOCAL +
    -</code> +
    - +
    -The first column is IP address or CIDR, the second one is the name (it is NOT displayed in reports, but it is required by format). +
    - +
    -<note important>Do not add too large subnet. Break into small ones. Highest value (limit) is 100000000</note> +
    - +
    -==== subscribers_dic, switches_dic, crc_dic dictionaries ==== +
    - +
    -=== subscribers_dic === +
    - +
    -Dictionary of subscribers. +
    - +
    -Dictionary example +
    -<code> +
    -10.64.66.100 login 5 port1 unit_vendor cabel contract services mac +
    -10.64.66.101 login 2 port1 unit_vendor cabel contract services mac +
    -10.64.66.102 login 3 port1 unit_vendor cabel contract services mac +
    -10.64.66.103 login 4 port1 unit_vendor cabel contract services mac +
    -10.64.66.104 login 5 port1 unit_vendor cabel contract services mac +
    -10.64.66.105 login 5 port2 unit_vendor cabel contract services mac +
    -10.64.66.106 login 5 port3 unit_vendor cabel contract services mac +
    -</code> +
    - +
    -Columns: +
    -  - IP address  +
    -  - Login +
    -  - Switch ID (access switch) +
    -  - Switch port +
    -  - Subscriber device vendor +
    -  - Cable +
    -  - Contract +
    -  - Services +
    -  - Subscriber device MAC address (is reserved for future purposes) +
    - +
    -=== switches_dic === +
    - +
    -Hierarchical dictionary of devices (access switches and trunk switches) +
    - +
    -Dictionary example +
    -<code> +
    -1 Switch_1 Ethernet Region_1 Address_1 10.140.1.18 ISP_1 0 0  +
    -2 Switch_2 Ethernet Region_2 Address_2 10.140.2.18 ISP_1 0 0  +
    -3 Switch_3 Ethernet Region_3 Address_3 10.140.3.18 ISP_1 0 1 port1  +
    -4 Switch_4 Ethernet Region_4 Address_4 10.140.4.18 ISP_1 0 3 port1  +
    -5 Switch_5 Ethernet Region_5 Address_5 10.140.5.18 ISP_1 0 4 port1  +
    -</code> +
    - +
    -Columns: +
    -  - Device ID UInt64 +
    -  - Device name +
    -  - Device type +
    -  - Region +
    -  - Address +
    -  - Switch IP address +
    -  - Internet service provider +
    -  - Indicator: trunk switch indicator (1 if so). Is not currently used, you can set 0 everywhere +
    -  - Upstream Switch ID UInt64 +
    -  - Upstream Switch port +
    -  - The owner +
    - +
    -=== crc_dic === +
    - +
    -CRC Errors Dictionary (on switch ports) +
    - +
    -Dictionary example +
    -<code> +
    -2 port_1 450 +
    -5 port_1 550 +
    -5 port_2 500 +
    -4 port_1 780 +
    -</code> +
    - +
    -Columns +
    -  - Switch ID +
    -  - Switch port +
    -  - CRC value  +
    - +
    -==== urlcats_dic and urlcats_host_dic dictionaries ==== +
    - +
    -Host categories dictionaries. Designed to determine the ownership of a particular host category.  +
    - +
    -Directories are automatically downloaded from vasexperts.ru resources. +
    - +
    -To speed up the initial load, issue the following commands +
    -  - <code>sh /var/qoestor/backend/etc/cron_daily.sh</code> +
    -  - <code>clickhouse-client --database=qoestor --query="system reload dictionaries"</code> +
    - +
    -==== Transferring dumps and database to a separate drive ==== +
    - +
    -All data is stored in the section /var by default.  +
    - +
    -For example you have a separate drive connected to /home. +
    - +
    -  - Work from the root user <code>sudo su</code> +
    -  - Stop the recievers and DB <code>systemctl stop qoestor_fullflow_0.service +
    -systemctl stop qoestor_clickstream_0.service +
    -sudo /etc/init.d/clickhouse-server stop</code> +
    -  - Create dictionaries in the /home section <code>mkdir /home/qoestor +
    -mkdir /home/qoestor/clickhouse +
    -mkdir /home/qoestor/dump +
    -</code> +
    -  - Copy tha data on the new drive <code>cp -r /var/lib/clickhouse/* /home/qoestor/clickhouse +
    -cp -r /var/qoestor/backend/dump/* /home/qoestor/dump +
    -</code> +
    -  - Change the folder owner /home/qoestor/clickhouse <code>chown -R clickhouse:clickhouse /home/qoestor/clickhouse</code> +
    -  - Delete the old dictionaries <code>rm -rf /var/lib/clickhouse +
    -rm -rf /var/qoestor/backend/dump/</code> +
    -  - Create simlinks <code>ln -s /home/qoestor/clickhouse /var/lib/clickhouse +
    -ln -s /home/qoestor/dump /var/qoestor/backend/dump +
    -</code> +
    -  - Check the links <code>readlink -f /var/lib/clickhouse +
    -readlink -f /var/qoestor/backend/dump +
    -</code> +
    -  - Run the DB <code>sudo /etc/init.d/clickhouse-server restart</code> +
    -  - Start the recievers <code>sudo sh /var/qoestor/backend/qoestor-config.sh</code> +
    - +
    -===== Troubleshooting ===== +
    - +
    -==== QoE Stor module does not work, although everything was installed according to the instructions. ==== +
    - +
    -If you have installed and configured everything according to the instructions above, and the DPIUI2 "QoE Analytics" section is empty, below is a checklist of the steps to be taken before contacting our technical support: +
    - +
    -  - Check the time and timezone settings on servers with dpiui2 and QoE Stor installed. Try to specify a long period in dpiui2. If it's about the timezone, the data will appear. Set the proper time on the servers with dpiui2 and QoE Stor module installed , restart the corresponding servers (on which the dpiui2 and QoE Stor module are installed). +
    -  - Check if the database is created on the server with QoE Stor installed <code>clickhouse-client --query="show databases" | grep qoestor</code> If the database is not created, you should create it using the following command <code>clickhouse-client -n < /var/qoestor/backend/etc/db/qoestor.sql</code> +
    -  - Check if there is data in the database on the server with QoE Stor installed <code>clickhouse-client --query="select count(), min(flow_start_time), max(flow_start_time) from qoestor.fullflow"</code> и <code>clickhouse-client --query="select count(), min(time), max(time) from qoestor.clickstream"</code> +
    -  - Check the content of the receiver dumps on the server with QoE Stor installed <code>/var/qoestor/backend/dump/fullflow</code> and <code>/var/qoestor/backend/dump/clickstream</code> +
    -  - Check the receivers logs under the <code>/var/qoestor/backend/logs/</code> directory. Is there something like "oops!”? Please contact technical support, because there is most likely the components are not installed from the proper repositories". If there is "Illegal IPFIX Message Version 0x0005", then once again check the export settings on the dpi: ''netflow_full_collector_type'' is specified improperly. +
    -  - Check whether the 1500 and 1501 ports are listening on the server with QoE Stor installed <code>netstat -nlp | grep 1500 и netstat -nlp | grep 1501</code> Restart all the receivers, just to be safe, by issuing the command: <code>/var/qoestor/backend/qoestor-config.sh</code> +
    -  - Check again [[en:dpi:dpi_components:qoestor#настройка_экспорта|ipfix export settings on the dpi device]] +
    -  - Check the [[en:dpi:dpi_components:dpiui:dpiui2#настройка_подключения_к_qoe_stor|GUI connection details for QoE Stor]] on the server with DPIUI2 installed +
    -  - Check if the ClickHouse database is running on the server with QoE Stor installed by issuing following command: <code>ps aux | grep clickhouse</code> Make sure the server has enough amount of RAM. +
    -  - Check the clickhouse logs under the /var/log/clickhouse-server/ directory on the server with QoE Stor installed. +
    - +
    -If there is a need to drop all data in the database, then you should take the following steps on the server with the QoE Stor installed: +
    -  - Drop the DB by issuing: <code>clickhouse-client --query="drop database qoestor"</code> +
    -  - Recreate DB by issuing <code>clickhouse-client -n < /var/qoestor/backend/etc/db/qoestor.sql</code> +
    - +
    -==== yum –y update command is issued, but receivers are still not running ==== +
    - +
    -When running **yum –y update** some libraries can be broken resulting in receivers are not running. Follow these steps to troubleshoot: +
    -  - Remove fastor and its dependencies<code> yum remove fastor ipfixreceiver libfixbuf netsa_silk netsa-python</code> +
    -  - Reinstall it using [[https://vasexperts.ru/install/fastor-rpm_install.sh.gz|fastor-rpm_install.sh.gz]] script. +
    - +
    - +
    -===== SQL and data uploading using CSV, JSON, TabSeparated formats ===== +
    - +
    -If necessary, you can create your own reports without additional tools and upload data in any format such as CSV, JSON, TabSeparated. +
    - +
    -Data is stored in 4 main logs: +
    -  * qoestor.fullflow – full netflow log, storage period is 24 hours +
    -  * qoestor.clicksteam – full clickstream log, storage period is 24 hours +
    -  * qoestor.fullflow_agg – pre-aggregated neflow log, unlimited storage period +
    -  * qoestor.clicksteam_agg – aggregated clickstream log, unlimited storage period +
    - +
    -Use the following command format <code>clickhouse-client --database=qoestor  --query="Your_SQL_query"</code> +
    - +
    -Data is uploaded using TabSeparated format by default.  +
    - +
    -**Example**. The client asked for a log of connections to a specific host in CSV format. +
    - +
    -<code>clickhouse-client --database=qoestor --query="select * from fullflow prewhere flow_start_date = '2018-10-04' where (source_ipv4 = '10.64.66.100' or destination_ipv4 = '10.64.66.100') and host = 'google.com' ORDER BY flow_start_time limit 10 format CSV"</code> +
    - +
    -For more information on ClickHouse SQL, follow the link: [[https://clickhouse.yandex/docs/ru/query_language/select/]]. +
    - +
    -~~DISCUSSION|Help us improve the documentation: if something isn't clear or you find an error, let us know.~~+