Disk Space Configuration [Документация VAS Experts]

Disk Space Configuration

QoE Stor uses several folders for storage:

  • /var/qoestor/backend/dump – temporary data received by receivers is stored here. After being loaded into the database, this data is deleted.
  • /var/lib/clickhouse/ - the so-called DEFAULT disk. Here, the database schema and data (raw and aggregated logs) are stored.
  • /var/lib/clickhouse-hot/ - the so-called HOT disk.
  • /var/lib/clickhouse-cold/ - the so-called COLD disk.

HOT and COLD disks are used when storage system cost reduction is needed.

DEFAULT must be very fast (e.g., NVMe SSD) to receive data from DPI, save it to the database in raw logs, and aggregate it (form aggregated logs).

The HOT disk can store aggregated logs for 1-2 weeks. The HOT disk should also be an SSD to quickly generate reports.

The COLD disk can store old data. COLD can be a SATA HDD disk, as reports on old data are rarely generated, and data can be stored on a slow disk.

For instructions on physically separating directories across disks, see below in the section Configuring Storage on a Separate Disk.

Configuring Storage on a Separate Disk

By default, all data is stored in the /var partition.

Assume we have mounted a separate disk to /storage.

  1. Work as the root user:
    sudo su
  2. Stop the receivers and database:
    fastor-stop
    fastor-db-stop
  3. Create directories in the /storage partition:
    mkdir /storage/qoestor
    mkdir /storage/qoestor/clickhouse
    mkdir /storage/qoestor/clickhouse-hot
    mkdir /storage/qoestor/clickhouse-cold
    mkdir /storage/qoestor/dump
  4. Copy data to the new disk:
    cp -r /var/lib/clickhouse/* /storage/qoestor/clickhouse
    cp -r /var/lib/clickhouse-hot/* /storage/qoestor/clickhouse-hot
    cp -r /var/lib/clickhouse-cold/* /storage/qoestor/clickhouse-cold
    cp -r /var/qoestor/backend/dump/* /storage/qoestor/dump
  5. Change the owner of the /storage/qoestor/clickhouse folder:
    chown -R clickhouse:clickhouse /storage/qoestor/clickhouse
    chown -R clickhouse:clickhouse /storage/qoestor/clickhouse-hot
    chown -R clickhouse:clickhouse /storage/qoestor/clickhouse-cold
  6. Remove the old directories:
    rm -rf /var/lib/clickhouse
    rm -rf /var/lib/clickhouse-hot
    rm -rf /var/lib/clickhouse-cold
    rm -rf /var/qoestor/backend/dump
  7. Create symlinks:
    ln -s /storage/qoestor/clickhouse /var/lib/clickhouse
    ln -s /storage/qoestor/clickhouse-hot /var/lib/clickhouse-hot
    ln -s /storage/qoestor/clickhouse-cold /var/lib/clickhouse-cold
    ln -s /storage/qoestor/dump /var/qoestor/backend/dump
  8. Check the links:
    readlink -f /var/lib/clickhouse
    readlink -f /var/lib/clickhouse-hot
    readlink -f /var/lib/clickhouse-cold
    readlink -f /var/qoestor/backend/dump
  9. Start the database:
    fastor-db-restart
  10. Start the receivers:
    fastor-restart

Was this information helpful?