Disk Space Configuration

QoE Stor uses several folders for data storage:

/var/qoestor/backend/dump – stores temporary data received by receivers. After loading into the database, this data is deleted.

/var/lib/clickhouse/ – the so-called DEFAULT disk. Stores the database schema and data (raw and aggregated logs).

/var/lib/clickhouse-hot/ – the so-called HOT disk.

/var/lib/clickhouse-cold/ – the so-called COLD disk.

HOT and COLD disks are used to reduce storage costs.

The DEFAULT disk must be super fast (for example, NVMe SSD) to receive data from DPI, save it to the database as raw logs, and perform aggregation (creating aggregated logs).

The HOT disk can store aggregated logs for 1–2 weeks. It should also be an SSD to allow fast report generation.

The COLD disk can store older data. The COLD disk may be a SATA HDD, as reports on older data are rarely generated and can be stored on slower drives.

For information on how to physically separate directories by disks, see the section below on Storage setup on a separate disk.

Storage setup on a separate disk

By default, all data is stored in the /var directory.

Suppose we have connected a separate disk mounted at /storage.

Work as the root user

sudo su

Stop receivers and the database

fastor-stop
fastor-db-stop

Create directories under /storage

mkdir /storage/qoestor
mkdir /storage/qoestor/clickhouse
mkdir /storage/qoestor/clickhouse-hot
mkdir /storage/qoestor/clickhouse-cold
mkdir /storage/qoestor/dump

Copy data to the new disk

cp -r /var/lib/clickhouse/* /storage/qoestor/clickhouse
cp -r /var/lib/clickhouse-hot/* /storage/qoestor/clickhouse-hot
cp -r /var/lib/clickhouse-cold/* /storage/qoestor/clickhouse-cold
cp -r /var/qoestor/backend/dump/* /storage/qoestor/dump

Change ownership of the /storage/qoestor directories

chown -R clickhouse:clickhouse /storage/qoestor/clickhouse
chown -R clickhouse:clickhouse /storage/qoestor/clickhouse-hot
chown -R clickhouse:clickhouse /storage/qoestor/clickhouse-cold

Remove old directories

rm -rf /var/lib/clickhouse
rm -rf /var/lib/clickhouse-hot
rm -rf /var/lib/clickhouse-cold
rm -rf /var/qoestor/backend/dump

Create symbolic links

ln -s /storage/qoestor/clickhouse /var/lib/clickhouse
ln -s /storage/qoestor/clickhouse-hot /var/lib/clickhouse-hot
ln -s /storage/qoestor/clickhouse-cold /var/lib/clickhouse-cold
ln -s /storage/qoestor/dump /var/qoestor/backend/dump

Verify links

readlink -f /var/lib/clickhouse
readlink -f /var/lib/clickhouse-hot
readlink -f /var/lib/clickhouse-cold
readlink -f /var/qoestor/backend/dump

Start the database

fastor-db-restart

Start receivers

fastor-restart