Dictionaries

Dictionary configuration

All dictionaries are located in the folder /var/qoestor/backend/etc/db/ and have the .txt extension.

Each dictionary has a sample.txt example file that can be used as a template.

All columns in the dictionaries are separated by a tab character (\t). The number of \t characters must be one less than the number of columns in the dictionary. Pay close attention to this.

When files are modified, the data is automatically reloaded into the database.

Some useful commands for working with dictionaries:

Speed up dictionary data refresh

clickhouse-client --database=qoestor --query="system reload dictionaries"

Check if there are errors in dictionaries

clickhouse-client --database=qoestor --query="select * from system.dictionaries"

Check if a dictionary contains data, for example, for subnets_local_dic

clickhouse-client --database=qoestor --query="select * from subnets_local_dic"

In dpiui2, in the QoE Analytics / Administrator / Dictionary Information section, you can view a list of dictionaries, errors, and contents. See Dictionary Information.

List of dictionaries

Dictionary	Description	Columns	Example	Notes
asnum_dic	Network AS dictionary. Cloud-based, loaded on schedule. Not editable.	1. AS number 2. AS name	0 -Reserved AS-, ZZ 1 LVLT-1, US 2 UDEL-DCN, US 3 MIT-GATEWAYS, US 4 ISI-AS, US 5 SYMBOLICS, US 6 BULL-HN, US 7 DSTL, GB 8 RICE-AS, US ...	To manually reload the dictionary, execute: sh /var/qoestor/backend/app_bash/cron_daily.sh
asnum_local_dic subnets_local_dic	Lists your local AS numbers and subnets. Used to determine traffic direction (relevant when DPI is installed on a mirror) and filter subscribers (so host IPs don’t appear in subscriber reports).	asnum_local_dic: 1. AS number 2. Display name subnets_local_dic: 1. IP or CIDR 2. Name (not displayed in reports, but required by format)	asnum_local_dic: 12345 LOCAL 65535 UNKNOWN subnets_local_dic: 192.168.1.0/24 LOCAL 10.64.66.0/24 LOCAL 172.16.0.0 LOCAL 2a02:2168:aaa:bbbb::2 LOCAL	Use parameters SUBSCRIBER_FILTER_MODE and TRAFFIC_DIR_DEF_MODE to control filtering based on these dictionaries. See Configuration. Avoid adding very large subnets; split them into smaller ones. Limit – 100000000
asnum_exclude_dic subnets_exclude_dic	Lists AS numbers and subnets (or individual IPs) to exclude from aggregated logs. Subnets listed here will be ignored when writing to the aggregated log (used for reports).	asnum_exclude_dic: 1. AS number 2. Name subnets_exclude_dic: 1. IP or CIDR 2. Name	asnum_exclude_dic: 12345 LOCAL 65535 LOCAL subnets_exclude_dic: 192.168.1.0/24 LOCAL 10.64.66.0/24 LOCAL 172.16.0.0 LOCAL 2a02:2168:aaa:bbbb::2 LOCAL	Use SUBSCRIBER_EXCLUDE_MODE to control filtering based on these dictionaries. See Configuration. Avoid adding very large subnets; split them into smaller ones. Limit – 100000000
gtp_cells_locations_dic	Contains a list of the operator’s cellular base stations with coordinates. Used for map-based reporting.	1. MCC - Mobile Country Code 2. MNC - Mobile Network Code 3. TAC/LAC - Location Area Code 4. CID/SAC/ECI - Cell ID 5. LAT - latitude 6. LON - longitude 7. CELL_ID - cell identifier (operator-defined) 8. CELL_NAME - cell name 9. CELL_DESCRIPTION - cell description	250 1 17774 4881420 55.783184763200005 37.525521418974996 cell_id_0 cell_name_0 cell_description_0 250 1 17716 3398401 55.86518562276 37.66309891922 cell_id_2 cell_name_2 cell_description_2 250 1 17716 3398411 55.86312193015 37.657007482905 cell_id_4 cell_name_4 cell_description_4 250 1 17782 2359297 55.86370294584 37.661393065359995 cell_id_5 cell_name_5 cell_description_5 250 1 1603 26057 55.80400515858 37.63568625476 cell_id_1 cell_name_1 cell_description_1 250 1 17781 2914571 55.799170861170005 37.6376949577 cell_id_6 cell_name_6 cell_description_6 250 1 1605 23895 55.869888445769995 37.64814730281 cell_id_3 cell_name_3 cell_description_3	—
net_protocols_dic	Network protocol dictionary. Not editable, updated with QoE Stor updates.	1. Port number 2. Protocol name	0 HOPOPT 1 ICMP 2 IGMP 3 GGP 4 IPv4 5 ST 6 TCP 7 CBT 8 EGP 9 IGP 10 BBN-RCC-MON 11 NVP-II ...	—
protocols_dic protocols_groups_dic	Dictionaries of application protocols and their groups. Cloud-based, loaded on schedule. Not editable.	protocols_dic: 1. Port number 2. Protocol name 3. Group ID protocols_groups_dic: 1. Group ID 2. Group name	protocols_dic: 1 tcpmux 1 2 compressnet 1 5 rje 1 7 echo 1 11 systat 7 13 daytime 7 17 qotd 7 18 msp 8 19 chargen 7 20 ftp-data 9 21 ftp 9 22 ssh 8 23 telnet 8 25 smtp 5 27 nsw-fe 7 ... protocols_groups_dic: 1 Network services 2 Web browsing 3 P2P 4 Gaming 5 E-Mail 6 Instant messengers	To manually reload dictionaries, execute: sh /var/qoestor/backend/app_bash/cron_daily.sh
settings_dic	Service dictionary of settings. Generated by scripts based on configuration files. Not editable.	1. Parameter 2. Value	TRAFFIC_DIR_DEF_MODE 0 SUBSCRIBER_FILTER_MODE 0 SUBSCRIBER_EXCLUDE_MODE 0 URLS_CATEGORIES_DIC_AUTOLOAD_ENABLED 1 ASNUM_DIC_AUTOLOAD_ENABLED 1 ULR_REPLACE_LOGIN_WITH_VCHANNEL 0 ULR_USE_DIC_WHEN_REPLACING_LOGIN 0 ULR_VCHANNEL_NAME_DIC_AUTOLOAD_ENABLED 0 ULR_VCHANNEL_NAME_DIC_URL NAT_AGG_LOG_FIELDS_TO_SAVE_BITMASK 0x654 NAT_IMPORT_FROM_FULLFLOW NAT_AGG_LOG_GROUP_TIME_INTERVAL 15 FULLFLOW_AGG_LOG_FIELDS_TO_SAVE_BITMASK 0x39fff FULLFLOW_AGG_LOG_GROUP_TIME_INTERVAL 15 CLICKSTREAM_AGG_LOG_FIELDS_TO_SAVE_BITMASK 0x30ff CLICKSTREAM_AGG_LOG_GROUP_TIME_INTERVAL 15 CLICKSTREAM_ADD_INFO_FROM_FULLFLOW 1 SUBSCRIBER_BIND_MODE 1 FILL_IP_LOGIN_BINDING_FROM_FULLFLOW 1 NAT_ADD_LOGIN_FROM_IP_LOGIN_BINDING 1 APP_ID qoestor APP_VERSION 1.9.0	—
subscribers_dic	Subscriber dictionary.	1. IP address 2. Login 3. Switch ID 4. Switch port 5. Equipment vendor 6. Cable 7. Contract 8. Services 9. MAC address (reserved for future use)	10.64.66.100 login 5 port1 unit_vendor cabel contract services mac 10.64.66.101 login 2 port1 unit_vendor cabel contract services mac 10.64.66.102 login 3 port1 unit_vendor cabel contract services mac 10.64.66.103 login 4 port1 unit_vendor cabel contract services mac 10.64.66.104 login 5 port1 unit_vendor cabel contract services mac 10.64.66.105 login 5 port2 unit_vendor cabel contract services mac 10.64.66.106 login 5 port3 unit_vendor cabel contract services mac	—
switches_dic	Hierarchical dictionary of equipment (access and backbone switches).	1. Equipment ID UInt64 2. Name 3. Type 4. Region 5. Address 6. Switch IP 7. Operator 8. Flag 9. Parent switch ID 10. Port 11. Owner	1 Switch 1 Ethernet Region1 Address 1 10.140.1.18 oper1 0 0 2 Switch 2 Ethernet Region2 Address 2 10.140.2.18 oper1 0 0 3 Switch 3 Ethernet Region3 Address 3 10.140.3.18 oper1 0 1 port1 4 Switch 4 Ethernet Region4 Address 4 10.140.4.18 oper1 0 3 port1 5 Switch 5 Ethernet Region5 Address 5 10.140.5.18 oper1 0 4 port1	—
crc_dic	Dictionary of CRC errors on switch ports.	1. Switch ID 2. Port 3. CRC value	2 port1 450 5 port1 550 5 port2 500 4 port1 780	—
urlcats_dic urlcats_host_dic	Host category dictionaries. Used to determine host category membership. Cloud-based, loaded on schedule. Not editable.	urlcats_dic: 1. Category ID 2. Category code urlcats_host_dic: 1. Host 2. Category ID	urlcats_dic: 1 unknown 2 software_tools 3 search_engine 4 0_other 5 school 6 proxys 7 tabak_alkohol 8 religia urlcats_host_dic: iris06-gold-ssl.gameloft.com 1 satfrog-tv.ddns.net:5890 1 vs824.vcdn.biz 1 cs05.trafmag.com 1	To manually reload dictionaries, execute: sh /var/qoestor/backend/app_bash/cron_daily.sh
ulr_vchannel_name_dic	Dictionary of virtual channel names. Used to generate statistics by virtual channels.	1. DPI device ID 2. Virtual channel number (vchannel) 3. Name	0 1 vchan_0_1 0 2 vchan_0_2 0 3 vchan_0_3 0 4 vchan_0_4 1 1 vchan_1_1 1 2 vchan_1_2 1 3 vchan_1_3 1 4 vchan_1_4	—

OpenCellID service

If coordinate data is missing in the gtp_cells_locations_dic dictionary for some base stations, it is possible to use a service for enriching the SSG GUI dictionary with base station data from an external source via API.

OpenCellID is a project that collects GPS coordinates of cell towers and their corresponding location area identifiers. The OpenCellID database is published under the international Creative Commons Attribution-ShareAlike 4.0 license, promoting free use and distribution of data.

The service is activated by adding the following parameters to

/var/qoestor/backend/.env

:

#Enable autoload of OpenCellID dictionary
CELL_TOWER_AUTOLOAD_ENABLED=1
# 0 - Disabled (default)
# 1 - Enabled
#Token for the LocalAPI platform
CELL_TOWER_LOADER_API_TOKEN=pk.fce9af82655c117ef42c88d36885cc8b
#Logging level
CELL_TOWER_LOG=1
# 0 - Info
# 1 - Debug
#Time interval for updating empty data for geo reports
CELL_TOWER_RESPONSE_HOUR=
#Defined in hours

A token can be obtained via https://my.unwiredlabs.com/trial. After registration, the token will be available:

When you click Show token, a free key with a limit of 100 requests per day will be displayed.