| Both sides previous revisionPrevious revision | |
| en:dpi:dpi_components:qoestor:install_and_update:hardware_recommendations [2025/01/13 13:43] – elena.krasnobryzh | en:dpi:dpi_components:qoestor:install_and_update:hardware_recommendations [2025/01/13 13:45] (current) – [Table] elena.krasnobryzh |
|---|
| It is assumed that the average daily traffic is 60% of the total peak (in+out) traffic. \\ {{ :dpi:dpi_components:qoestor:install_and_update:qoe_stor_sizing_based_on_average_speed.xlsx |In the provided calculator, you need to change the traffic value to get the storage volumes.}} | It is assumed that the average daily traffic is 60% of the total peak (in+out) traffic. \\ {{ :dpi:dpi_components:qoestor:install_and_update:qoe_stor_sizing_based_on_average_speed.xlsx |In the provided calculator, you need to change the traffic value to get the storage volumes.}} |
| ==== Detailed Recommendations ==== | ==== Detailed Recommendations ==== |
| |CPU |**One processor** supporting **SSE 4.2** instructions starting from [[http://en.wikipedia.org/wiki/Nehalem_(microarchitecture)|Intel Nehalem]] and [[https://ru.wikipedia.org/wiki/Zen_2_(%D0%BC%D0%B8%D0%BA%D1%80%D0%BE%D0%B0%D1%80%D1%85%D0%B8% D1%82%D0%B5%D0%BA%D1%82%D1%83%D1%80%D0%B0)|AMD EPYC Zen2]] **with 4 or more cores**, base clock speed of **2.5 GHz and higher**. Choose processors with more cores. Clock speed is less important. For example, 16 cores at 2600 MHz is better than 8 cores at 3600 MHz. \\ **Do not disable Hyper-threading and Turbo-Boost.**| | | CPU | **One processor** supporting **SSE 4.2** instructions starting from [[http://en.wikipedia.org/wiki/Nehalem_(microarchitecture)|Intel Nehalem]] and [[https://en.wikipedia.org/wiki/Zen_2|AMD EPYC Zen2]] **with 4 or more cores**, base clock speed of **2.5 GHz and higher**. Choose processors with more cores. Clock speed is less important. For example, 16 cores at 2600 MHz is better than 8 cores at 3600 MHz. \\ **Do not disable Hyper-threading and Turbo-Boost.** | |
| |RAM |At least 16 GB, memory modules must be installed **in all processor channels** on the motherboard. Memory should be no less than the volume of requested data. More memory improves performance when generating reports. More memory also reduces disk load.\\ **Always disable swap file.**| | | RAM | At least 16 GB, memory modules must be installed **in all processor channels** on the motherboard. Memory should be no less than the volume of requested data. More memory improves performance when generating reports. More memory also reduces disk load.\\ **Always disable swap file.** | |
| |Disks |To optimize storage costs, several types of disks are used: \\ default — fast disks for data reception and aggregation process, it is recommended to use SSD NVMe in RAID0. \\ hot — disks for storage during periods of higher likelihood of report requests on this data, typically up to 3 months, SSD disks in RAID-10, RAID-5, RAID-6, or RAID-50. \\ cold — slow, large-volume disks for long-term storage, HDD disks in RAID-10, RAID-5, RAID-6, or RAID-50 are recommended. \\ The storage duration for each level is set in the configuration via the GUI. Data migration between disks and data cleanup happens automatically based on the settings. A mechanism for overflow control is also provided to protect the database. The main volume of data is stored in the /var/lib/clickhouse directory. Temporary data (IPFIX dumps) are stored in the /var/qoestor/backend/dump directory. For better performance, it is important (recommended) that these directories are located on a separate disk or array. See [[en:dpi:dpi_components:qoestor:configuration:disc|Disk Configuration]]. \\ For OS and QoE Stor software installation, use 2 disks of at least 256GB capacity, combined in RAID 1 (mirror). A hardware RAID controller is required.| | | Disks | To optimize storage costs, several types of disks are used: \\ default — fast disks for data reception and aggregation process, it is recommended to use SSD NVMe in RAID0. \\ hot — disks for storage during periods of higher likelihood of report requests on this data, typically up to 3 months, SSD disks in RAID-10, RAID-5, RAID-6, or RAID-50. \\ cold — slow, large-volume disks for long-term storage, HDD disks in RAID-10, RAID-5, RAID-6, or RAID-50 are recommended. \\ The storage duration for each level is set in the configuration via the GUI. Data migration between disks and data cleanup happens automatically based on the settings. A mechanism for overflow control is also provided to protect the database. The main volume of data is stored in the /var/lib/clickhouse directory. Temporary data (IPFIX dumps) are stored in the /var/qoestor/backend/dump directory. For better performance, it is important (recommended) that these directories are located on a separate disk or array. See [[en:dpi:dpi_components:qoestor:configuration:disc|Disk Configuration]]. \\ For OS and QoE Stor software installation, use 2 disks of at least 256GB capacity, combined in RAID 1 (mirror). A hardware RAID controller is required. | |
| |QoE Cluster (Sharding) | It is better to create multiple nodes and combine them into a cluster:\\ GUI optimizes queries so that all nodes generate reports in parallel. \\ [[en:dpi:dpi_components:ipfix_balancer|]] is used to evenly distribute data across nodes (roundrobin), significantly improving system performance. \\ In case of node failure, the balancer will automatically distribute data to the remaining nodes. General recommendation: as many nodes as possible and as little data per node as possible. This will provide:\\ 1. High performance\\ 2. Good fault tolerance\\ 3. Scalability (by adding nodes to the cluster) | | | QoE Cluster (Sharding) | It is better to create multiple nodes and combine them into a cluster:\\ GUI optimizes queries so that all nodes generate reports in parallel. \\ [[en:dpi:dpi_components:ipfix_balancer|]] is used to evenly distribute data across nodes (roundrobin), significantly improving system performance. \\ In case of node failure, the balancer will automatically distribute data to the remaining nodes. General recommendation: as many nodes as possible and as little data per node as possible. This will provide:\\ 1. High performance\\ 2. Good fault tolerance\\ 3. Scalability (by adding nodes to the cluster) | |
| |
| ==== Tips from Yandex ClickHouse ==== | ==== Tips from Yandex ClickHouse ==== |
