Sharding configuration [Документация VAS Experts]

Sharding configuration

QoE Stor can work in sharding mode: horizontal cluster scaling, in which parts of the same database are placed on different shards (nodes). Through sharding, you can: 1. Distribute the load on the IPFIX inserts from several DPIs, spreading it over the nodes. 2. At times speed up the construction of reports by increasing the machine resources-nodes: the construction of reports is performed in parallel and glued into the final report.

1 Create a cluster

To create a QoE Stor cluster, you must specify the cluster node settings on each node in the file /etc/clickhouse-server/config.d/qoestor-cluster-config.xml

Edit file on 1st node

<?xml version="1.0"?>
<yandex>
    <remote_servers>
        <qoestor_cluster>
            <shard>
                <replica>
                    <host>127.0.0.1</host>
                    <port>9000</port>
                </replica>
            </shard>
            <shard>
                <replica>
                    <host>127.0.0.2</host>
                    <port>9000</port>
                </replica>
            </shard>
            <shard>
                <replica>
                    <host>127.0.0.3</host>
                    <port>9000</port>
                </replica>
            </shard>
        </qoestor_cluster>
    </remote_servers>
</yandex>

For each shard, you must specify the host IP or its name. Save your changes.

Copy the file to the rest of the cluster nodes.

Restart the database with the

fastor-db-restart

command on each node. The cluster is ready to run.

Port 9000 must be open on all firewall cluster nodes in the internal network between the cluster nodes.

2 Adding a cluster to the GUI

1 Master node connection

Choose which node you will use as the master. Enable it in the GUI settings. Enable cluster mode. See screenshot below.

Port 8123 must be open on all cluster nodes.

2 Management via SSH

In the GUI Administrator / Hardware section, add all nodes for SSH control.

3 Setting up IPFIX receivers

In the QoE Stor configuration, configure the receivers for each node.

Reconfigure DPI to send IPFIX to the appropriate QoE cluster nodes.

Distribute the load evenly. One or more DPIs must send IPFIX to one QoE Stor node.
Do not send IPFIX from a single DPI to multiple QoE Stor nodes.