| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| en:dpi:qoe_analytics:implementation_administration:requirements [2025/11/01 06:53] – elena.krasnobryzh | en:dpi:qoe_analytics:implementation_administration:requirements [2025/12/09 08:49] (current) – [Table] elena.krasnobryzh |
|---|
| - RAM – from 16 GB | - RAM – from 16 GB |
| - Hard drive (SSD strongly recommended) – from 500 GB | - Hard drive (SSD strongly recommended) – from 500 GB |
| - Operating system – CentOS 8.x, [[en:veos:installation|VEOS]], CentOS Stream 8.x, Oracle Linux Server 8.x, AlmaLinux 8.x | - Operating system – CentOS 8.x, [[en:veos:installation|VEOS]], CentOS Stream 8.x, Oracle Linux Server 8.x, AlmaLinux 8.x.\\ :!: When using VEOS, take into account the information in [[en:veos:changelogs]] when selecting hardware. |
| - Network interface card (NIC) – from 1Gbps | - Network interface card (NIC) – from 1Gbps |
| |
| - RAM – 64 GB | - RAM – 64 GB |
| - Hard drive (SSD strongly recommended) – from 500 GB; see details about storage capacity calculation and recommendations below | - Hard drive (SSD strongly recommended) – from 500 GB; see details about storage capacity calculation and recommendations below |
| - Operating system – CentOS 8.x, [[en:veos:installation|VEOS]], CentOS Stream 8.x, Oracle Linux Server 8.x, AlmaLinux 8.x | - Operating system – CentOS 8.x, [[en:veos:installation|VEOS]], CentOS Stream 8.x, Oracle Linux Server 8.x, AlmaLinux 8.x.\\ :!: When using VEOS, take into account the information in [[en:veos:changelogs]] when selecting hardware. |
| - Network interface card (NIC) – 2x10Gbps. It is important to consider that each DPI generates an IPFIX stream at a rate of 0.5% to 1% of real traffic speed. It is also recommended to combine ports on the QoE in a LAG for fault tolerance. | - Network interface card (NIC) – 2x10Gbps. It is important to consider that each DPI generates an IPFIX stream at a rate of 0.5% to 1% of real traffic speed. It is also recommended to combine ports on the QoE in a LAG for fault tolerance. |
| |
| |
| =====Detailed recommendations===== | =====Detailed recommendations===== |
| | CPU | **Single processor** supporting **SSE 4.2** instructions starting from [[http://en.wikipedia.org/wiki/Nehalem_(microarchitecture)|Intel Nehalem]] and [[https://ru.wikipedia.org/wiki/Zen_2_(%D0%BC%D0%B8%D0%BA%D1%80%D0%BE%D0%B0%D1%80%D1%85%D0%B8%D1%82%D0%B5%D0%BA%D1%82%D1%83%D1%80%D0%B0)|AMD EPYC Zen2]] **with 4 or more cores** and a **base clock speed of 2.5 GHz or higher**. Choose CPUs with more cores. Clock speed is less important. For example, 16 cores at 2600 MHz is better than 8 cores at 3600 MHz.\\ \\ **Do not disable Hyper-threading and Turbo-Boost**. | | | | CPU | **Single processor** supporting **SSE 4.2** instructions starting from [[http://en.wikipedia.org/wiki/Nehalem_(microarchitecture)|Intel Nehalem]] and [[https://en.wikipedia.org/wiki/Zen_2|AMD EPYC Zen2]] **with 4 or more cores** and a **base clock speed of 2.5 GHz or higher**. Choose CPUs with more cores. Clock speed is less important. For example, 16 cores at 2600 MHz is better than 8 cores at 3600 MHz.\\ \\ **Do not disable Hyper-threading and Turbo-Boost**. | |
| | RAM | From 16 GB; memory modules must be installed **in all CPU channels** on the motherboard. The memory size should not be less than the volume of queried data. The more memory, the better performance when generating reports and the lower the disk load.\\ \\ **Always disable swap file**. | | | | RAM | From 16 GB; memory modules must be installed **in all CPU channels** on the motherboard. The memory size should not be less than the volume of queried data. The more memory, the better performance when generating reports and the lower the disk load.\\ \\ **Always disable swap file**. | |
| | Disks | To optimize storage cost, multiple types of disks are used:\\ 1. **default** — fast disks for data ingestion and aggregation processes, SSD NVMe in RAID0 recommended.\\ 2. **hot** — disks for storing data likely to be queried (usually up to 3 months), SSDs in RAID-10, RAID-5, RAID-6, or RAID-50.\\ 3. **cold** — high-capacity slow disks for long-term storage, HDDs in RAID-10, RAID-5, RAID-6, or RAID-50 recommended.\\ Retention period at each level is configured via GUI. Data migration and cleanup occur automatically according to settings. A mechanism for overflow protection is also provided. The main data volume is stored in /var/lib/clickhouse. Temporary data (IPFIX dumps) are stored in /var/qoestor/backend/dump. For best performance, these directories should be located on a separate disk or array. See [[en:dpi:qoe_analytics:implementation_administration:configuration_setup:disc]].\\ For OS and QoE Stor software installation, use two drives of at least 256GB combined in RAID1 (mirror). A hardware RAID controller is required. | | | | Disks | **File system type: ext4.**\\ To optimize storage cost, multiple types of disks are used:\\ 1. **default** — fast disks for data ingestion and aggregation processes, SSD NVMe in RAID0 recommended.\\ 2. **hot** — disks for storing data likely to be queried (usually up to 3 months), SSDs in RAID-10, RAID-5, RAID-6, or RAID-50.\\ 3. **cold** — high-capacity slow disks for long-term storage, HDDs in RAID-10, RAID-5, RAID-6, or RAID-50 recommended.\\ Retention period at each level is configured via GUI. Data migration and cleanup occur automatically according to settings. A mechanism for overflow protection is also provided. The main data volume is stored in /var/lib/clickhouse. Temporary data (IPFIX dumps) are stored in /var/qoestor/backend/dump. For best performance, these directories should be located on a separate disk or array. See [[en:dpi:qoe_analytics:implementation_administration:configuration_setup:disc]].\\ For OS and QoE Stor software installation, use two drives of at least 256GB combined in RAID1 (mirror). A hardware RAID controller is required. | |
| | QoE Cluster (Sharding) | It is better to create several nodes and combine them into a cluster:\\ GUI can optimize queries so that all nodes build reports in parallel.\\ [[en:dpi:dpi_components:ipfix_balancer]] is used for even data distribution across nodes (round-robin), significantly improving performance.\\ If a node fails, the balancer automatically directs data to remaining nodes. General recommendation: more nodes and smaller data portions per node. This ensures:\\ 1. High performance\\ 2. Fault tolerance\\ 3. Scalability (by adding nodes to the cluster) | | | | QoE Cluster (Sharding) | It is better to create several nodes and combine them into a cluster:\\ GUI can optimize queries so that all nodes build reports in parallel.\\ [[en:dpi:dpi_components:ipfix_balancer]] is used for even data distribution across nodes (round-robin), significantly improving performance.\\ If a node fails, the balancer automatically directs data to remaining nodes. General recommendation: more nodes and smaller data portions per node. This ensures:\\ 1. High performance\\ 2. Fault tolerance\\ 3. Scalability (by adding nodes to the cluster) | |
| |
| =====Operation tips from Yandex ClickHouse===== | =====Operation tips from Yandex ClickHouse===== |
| You can read operation tips from Yandex ClickHouse at https://clickhouse.yandex/docs/ru/operations/tips/. | You can read operation tips from Yandex ClickHouse at https://clickhouse.com/docs/operations/tips. |
| |
