Disk space configuration

QoE Stor uses several folders for data storage:

• /var/qoestor/backend/dump – stores temporary data received by receivers. After loading into the database, this data is deleted.
• /var/lib/clickhouse/ – the so-called DEFAULT disk. Stores the database schema and data (raw and aggregated logs).
• /var/lib/clickhouse-hot/ – the so-called HOT disk.
• /var/lib/clickhouse-cold/ – the so-called COLD disk.

HOT and COLD disks are used to reduce storage costs.

The DEFAULT disk must be super fast (for example, NVMe SSD) to receive data from DPI, save it to the database as raw logs, and perform aggregation (creating aggregated logs).

The HOT disk can store aggregated logs for 1–2 weeks. It should also be an SSD to allow fast report generation.

The COLD disk can store older data. The COLD disk may be a SATA HDD, as reports on older data are rarely generated and can be stored on slower drives.

For information on how to physically separate directories by disks, see the section below on Storage setup on a separate disk.

By default, all data is stored in the /var directory.

Suppose we have connected a separate disk mounted at /storage.

1. Work as the root user

   sudo su

2. Stop receivers and the database

   fastor-stop
   fastor-db-stop

3. Create directories under /storage

   mkdir /storage/qoestor
   mkdir /storage/qoestor/clickhouse
   mkdir /storage/qoestor/clickhouse-hot
   mkdir /storage/qoestor/clickhouse-cold
   mkdir /storage/qoestor/dump

4. Copy data to the new disk

   cp -r /var/lib/clickhouse/* /storage/qoestor/clickhouse
   cp -r /var/lib/clickhouse-hot/* /storage/qoestor/clickhouse-hot
   cp -r /var/lib/clickhouse-cold/* /storage/qoestor/clickhouse-cold
   cp -r /var/qoestor/backend/dump/* /storage/qoestor/dump

5. Change ownership of the /storage/qoestor directories

   chown -R clickhouse:clickhouse /storage/qoestor/clickhouse
   chown -R clickhouse:clickhouse /storage/qoestor/clickhouse-hot
   chown -R clickhouse:clickhouse /storage/qoestor/clickhouse-cold

6. Remove old directories

   rm -rf /var/lib/clickhouse
   rm -rf /var/lib/clickhouse-hot
   rm -rf /var/lib/clickhouse-cold
   rm -rf /var/qoestor/backend/dump

7. Create symbolic links

   ln -s /storage/qoestor/clickhouse /var/lib/clickhouse
   ln -s /storage/qoestor/clickhouse-hot /var/lib/clickhouse-hot
   ln -s /storage/qoestor/clickhouse-cold /var/lib/clickhouse-cold
   ln -s /storage/qoestor/dump /var/qoestor/backend/dump

8. Verify links

   readlink -f /var/lib/clickhouse
   readlink -f /var/lib/clickhouse-hot
   readlink -f /var/lib/clickhouse-cold
   readlink -f /var/qoestor/backend/dump

9. Start the database

   fastor-db-restart

10. Start receivers

    fastor-restart