| Next revision | Previous revision |
| en:dpi:dpi_components:qoestor:configuration:disc [2024/09/26 15:29] – created - external edit 127.0.0.1 | en:dpi:dpi_components:qoestor:configuration:disc [2025/09/22 07:43] (current) – elena.krasnobryzh |
|---|
| ====== Setting up disk space ====== | ====== Disk Space Configuration ====== |
| {{indexmenu_n>2}} | {{indexmenu_n>2}} |
| |
| QoE Stor uses several folders for storage: | QoE Stor uses several folders for storage: |
| |
| * /var/qoestor/backend/dump – temporary data received by receivers is stored here. After loading into the database, this data is deleting. | * /var/qoestor/backend/dump – temporary data received by receivers is stored here. After being loaded into the database, this data is deleted. |
| * /var/lib/clickhouse/ - so called DEFAULT disk. This is where the database schema and data (raw and aggregated logs) are stored. | * /var/lib/clickhouse/ - the so-called DEFAULT disk. Here, the database schema and data (raw and aggregated logs) are stored. |
| * /var/lib/clickhouse-hot/ - the so-called HOT disc. | * /var/lib/clickhouse-hot/ - the so-called HOT disk. |
| * /var/lib/clickhouse-cold/ - the so-called COLD disc. | * /var/lib/clickhouse-cold/ - the so-called COLD disk. |
| |
| HOT and COLD drives are used when you need to reduce the cost of storage. | HOT and COLD disks are used when storage system cost reduction is needed. |
| | |
| DEFAULT must be super fast (NVMe SSD, for example) in order to receive data from DPI, store it in the database in a raw log and aggregate (form an aggregated log). | DEFAULT must be very fast (e.g., NVMe SSD) to receive data from DPI, save it to the database in raw logs, and aggregate it (form aggregated logs). |
| |
| On a HOT disk, you can store an aggregated log for 1-2 weeks. A HOT disk must also be an SSD in order to build reports quickly. | The HOT disk can store aggregated logs for 1-2 weeks. The HOT disk should also be an SSD to quickly generate reports. |
| |
| Old data can be stored on the COLD disk. COLD can be a SATA HDD disk, because reports on old data are rarely built and data can be stored on a slow disk. | The COLD disk can store old data. COLD can be a SATA HDD disk, as reports on old data are rarely generated, and data can be stored on a slow disk. |
| |
| How to physically partition directories by disks, see the section [[en:dpi:dpi_components:qoestor:configuration:disc:split|Setting up storage on a separate disk ]] | For instructions on physically separating directories across disks, see below in the section [[en:dpi:dpi_components:qoestor:configuration:disc#configuring_storage_on_a_separate_disk|Configuring Storage on a Separate Disk]]. |
| |
| | ===== Configuring Storage on a Separate Disk ===== |
| |
| | By default, all data is stored in the /var partition. |
| |
| | Assume we have mounted a separate disk to /storage. |
| |
| | - Work as the root user: <code>sudo su</code> |
| | - Stop the receivers and database: <code>fastor-stop |
| | fastor-db-stop</code> |
| | - Create directories in the /storage partition: <code>mkdir /storage/qoestor |
| | mkdir /storage/qoestor/clickhouse |
| | mkdir /storage/qoestor/clickhouse-hot |
| | mkdir /storage/qoestor/clickhouse-cold |
| | mkdir /storage/qoestor/dump |
| | </code> |
| | - Copy data to the new disk: <code>cp -r /var/lib/clickhouse/* /storage/qoestor/clickhouse |
| | cp -r /var/lib/clickhouse-hot/* /storage/qoestor/clickhouse-hot |
| | cp -r /var/lib/clickhouse-cold/* /storage/qoestor/clickhouse-cold |
| | cp -r /var/qoestor/backend/dump/* /storage/qoestor/dump |
| | </code> |
| | - Change the owner of the /storage/qoestor/clickhouse folder: <code>chown -R clickhouse:clickhouse /storage/qoestor/clickhouse |
| | chown -R clickhouse:clickhouse /storage/qoestor/clickhouse-hot |
| | chown -R clickhouse:clickhouse /storage/qoestor/clickhouse-cold |
| | </code> |
| | - Remove the old directories: <code>rm -rf /var/lib/clickhouse |
| | rm -rf /var/lib/clickhouse-hot |
| | rm -rf /var/lib/clickhouse-cold |
| | rm -rf /var/qoestor/backend/dump |
| | </code> |
| | - Create symlinks: <code>ln -s /storage/qoestor/clickhouse /var/lib/clickhouse |
| | ln -s /storage/qoestor/clickhouse-hot /var/lib/clickhouse-hot |
| | ln -s /storage/qoestor/clickhouse-cold /var/lib/clickhouse-cold |
| | ln -s /storage/qoestor/dump /var/qoestor/backend/dump |
| | </code> |
| | - Check the links: <code>readlink -f /var/lib/clickhouse |
| | readlink -f /var/lib/clickhouse-hot |
| | readlink -f /var/lib/clickhouse-cold |
| | readlink -f /var/qoestor/backend/dump |
| | </code> |
| | - Start the database: <code>fastor-db-restart</code> |
| | - Start the receivers: <code>fastor-restart</code> |
