VeloDB Manager 24.x User Guide

Overview

Unlike the SSH mutual trust method of the historical version of the 23.x series, control of the 24.0.0 version of VeloDB Manager uses the Agent method. The Agent and Server communicate directly using the HTTP protocol, which can be combined with SSL encryption data to ensure security.

Main Features

VeloDB Manager provides the following main features :

• Cluster Deployment : Deploy Aache Doris or VeloDB Doris clusters on physical or virtual machines using VeloDB Manager.
• Cluster Takeover : Take over existing Apache Doris or VeloDB Doris clusters for operation and monitoring using VeloDB Manager.
• Cluster Details : View the operational status, details, and connection information of the cluster.
• Cluster Scaling : Scale FE and BE nodes in or out.
• Cluster Upgrade : Upgrade the cluster version, providing options for full-service stop upgrade and online rolling upgrade. Choose the appropriate upgrade method based on business scenarios.
• Cluster Restart : Restart the entire cluster, FE, BE, and nodes, supporting both rolling and full restarts.
• Node details : View real-time status and machine information of the node.
• Monitoring and Alerts : View monitoring metrics, set alert rules, and receive alert notifications through email, chat software, Webhook, etc.
• Parameter Configuration : Support custom editing of configuration files for nodes, with the ability to view all running parameters for individual nodes.
• Log Viewing : Support viewing and querying logs for FE and BE nodes, facilitating offline troubleshooting of cluster issues.
• Task audit : Support to view detailed information of each task, including operation time, operator, operation content and other information.
• Cluster Inspection : Support manual or regular one-click inspection of machine status, cluster operation status, timely discovery and positioning of performance bottlenecks, and provide repair suggestions.
• WebUI : Supports viewing data and information in the database, and can also perform SQL queries, data imports, permission management and other operations.
• Others : Change passwords for the root user and admin user of the cluster, stop taking over the cluster, etc.

Initialization Configuration

The first step after starting is to set up the administrator account. The administrator account has the highest authority within VeloDB Manager, enabling tasks such as user configuration, VeloDB Manager's self-configuration, upgrades, and more. Other accounts must be added with the identity of the administrator account.

1. Initialize VeloDB Manager administrator account

Note: Please remember the set administrator username and password. If there is only one administrator account in VeloDB Manager, the username and password of the administrator account cannot be reset and restored once lost.

2. Configure service component deployment information

After creating the VeloDB Manager administrator user, click Next to enter the service component configuration page and fill in the configuration information.

Setting is divided into two parts, namely Component Configuration and Package Configuration:

• Component configuration : VeloDB Manager monitors the configuration of components related to the alarm service.
  • Monitoring Alarm Service : Optional, used to configure the monitoring alarm module of VeloDB Manager. Here, Grafana, Prometheus, and Alertmanager will be installed. You need to select three available ports on the machine where VeloDB Manager is installed.
    • Email Alarm : Configure the mail server and then use the "mail alerts" channel in alerts.
    • Advanced configuration : Optional for monitoring access addresses and proxy configurations.
      • Monitoring access address: Used to solve the problem that monitoring metrics cannot be viewed when accessing VeloDB Manager through domain name.
      • Proxy configuration: If the production environment and extranet are isolated, you can set the proxy to send notifications to the public network work communication software.
• Package Configuration : Configure the local Doris Core and VeloDB Manager installation packags for creating and upgrading clusters.

3. Start VeloDB Manager

After completing the configuration, click on Start VeloDB Manager .

After the restart is complete, click to Enter VeloDB Manager , and you will enter the control page. You can initiate operations to create a new cluster or take over a cluster on the control page.

User Menu

Service Configuration

After entering VeloDB Manager, you can click on the user's avatar in the bottom left corner, select Service Configuration from the menu, and make configuration modifications. After modification, click Save and Restart , VeloDB Manager will restart and load the new configuration.

In the service module, you can see the status of each service and support the restart and start-stop operations of the service.

Access control

In User Management, system administrators can create new users and assign roles to them.

VeloDB Manager has three built-in roles, and their permission scopes are as follows :

Cluster management is the permission to perform operation and maintenance operations (parameter configuration, restart, upgrade, etc.) on the cluster. Those who have cluster management permissions must have cluster viewing permissions.

You can also create new roles and assign them permission scopes, such as the GuideAdmin role in the figure, which has administrative permissions on the Guide cluster.

Change Login Password

Click on the user avatar in the bottom left corner, select Change Sign in Password and you can modify the login password for the current user.

Upgrade VeloDB Manager

1. View the current version

Click on the user information in the lower left corner of the page to view the current version of VeloDB Manager.

2. Download the new version of VeloDB Manager installation package

The upgrade page will prompt the storage path of the target version installation package on the VeloDB Manager server.

Directly log in to the VeloDB Manager server machine and save the new version of the installation package into the directory. Then refresh the page to view and select the target version.

Click the Upgrade button to enter the upgrade page.

3. Upgrade VeloDB Manager

Note: The webserver component will also be restarted during the upgrade, so the page will briefly exit automatically. Please log in again.

New Cluster

Deploy Cluster

Deploy Apache Doris or VeloDB Doris clusters on physical or virtual machines through VeloDB Manager. VeloDB Manager will automatically handle the installation and configuration of the cluster. During the deployment process, users can customize the cluster's scale, node count, and other parameters to meet various business requirements.

After deployment, VeloDB Manager will provide the connection information for the cluster, facilitating users in quick access and utilization.

Environment Settings

First, choose the deployment method to create a new cluster, set the cluster name in Environment Settings , select the version to be deployed and set the database root user password.

Currently, VeloDB Manager supports deploying 1.2.1 version of SelectDB Doris as well as Apache Doris clusters.

Note: The version type of Clustered Deployment must be consistent with the Manager version type. For example, if the cluster version you deploy is x64 type, the Manager also needs to deploy the corresponding x64 version. Conversely, if the cluster version you deploy is arm type, Manager also needs to deploy the arm version.

Host Registration

When deploying a cluster through Manager, you need to deploy Agent on the control machine for host registration. You can refer to the Agent Installation Guide to deploy on the target host. If you want to know the detailed host registration process, see the deployment document.

After deployment, enter the Agent port and IP information (supporting IPV4 and IPV6), and VeloDB Manager will automatically detect the Agent status of the host.

Note:

1. Agent ports of all hosts in the cluster must be consistent.
2. A host can only deploy one cluster.

FE Configuration

Configure the ports, roles, deployment directories, etc. of the FE node in the FE configuration, as follows:

• Common Configuration: represents the configuration for various ports, installation directory, metadata storage directory, and log directory used by all FE nodes. The system provides default values that can be modified according to the machine environment.
• FE node: Select a node from the successfully registered hosts, specify the role of the node, FE Follower is recommended to be an odd number, expand the custom configuration, and modify various ports, priority networks, deployment directories, metadata storage directories, and log directories of the FE node.

During deployment, VeloDB Manager will check the machine environment of the host. Deployment can only proceed if the machine environment is determined to be normal. If the machine environment check is abnormal, there are several types of errors:

• The machine's configuration does not meet the deployment requirements for Doris, such as the swap partition not being closed, the maximum open file limit being less than 65535, transparent huge pages not being closed, and the virtual memory region count not meeting the requirements. Follow the prompts provided by VeloDB Manager to make the necessary modifications.
• Port occupation: Change to an unused port if the current port is occupied.
• Deployment path check: Ensure that the space in the directory where the installation path is located is greater than 20 GB, and the directory itself is not occupied.

BE Configuration

Configure the ports, roles, deployment directories, etc. of the BE node in the BE configuration. The specific instructions are as follows :

• Common Configuration: Represents the configuration for various ports, installation directory, data storage directory, log directory, and external cache used by all BE nodes. The system provides default values, and you can modify them based on the machine environment.
• BE Node: Select nodes from successfully registered hosts. The role of a regular BE node is a standard (Hybrid Node), which can perform SQL queries and manage Tablet data storage. By expanding the custom configuration, you can modify the ports, deployment directories, data storage directories, log directories, and external caches of the BE node. If necessary, you can also add a Compute Node with the role of Compute Node, which is specifically used to handle remote federated queries such as Data Lake. It can only perform SQL queries and does not save any data.

Other Configurations

Configure WebUI, automatic control mode, and table name case sensitivity in other configurations, as follows:

• WebUI configuration: WebUI requires access through a browser; ensure connectivity between the browser and the deployment machine. You can specify memory for WebUI, with a recommended range of 1 to 2 GB.

Note: WebUI is packaged in the kernel of selectdb-doris, so only the deployment version of selectdb-doris kernel will have WebUI configuration. The open source version of Clustered Deployment does not have WebUI.

• Advanced Configuration
  • Enable Automatic Management Mode, VeloDB Manager will set up automatic daemon launching for Doris processes on the machine. If you prefer no intervention from VeloDB Manager, you can choose to disable the automatic mode.
  • Table name case sensitive, set lower_case_table_names value to specify whether the table name of the cluster is case sensitive.

Click Deploy Cluster, and VeloDB Manager will perform Clustered Deployment. You can see the detailed deployment process and progress on the New Cluster page.

After successful deployment, you will see the access information of the cluster, as well as the initialized admin user password, which you can copy and save. VeloDB Manager does not save the user's password, which only appears once here.

You can see the connection information of the database in the cluster details, but you will not see the password information.

Next, click **into the cluster **to enter the VeloDB Manager to view and operate the cluster.

Takeover cluster

Take over an existing Apache Doris or VeloDB Doris cluster and manage it through VeloDB Manager for operations and monitoring. You only need to provide relevant information about the existing cluster, such as the cluster address and login credentials. VeloDB Manager can automatically take over the cluster, enabling unified management.

After the takeover, you can perform operational tasks on the cluster through VeloDB Manager, such as monitoring, scaling, restarting, etc. Additionally, you can view information about the cluster's runtime status and monitoring metrics, helping users better understand and manage the cluster.

Environment Configuration

In the Takeover Cluster Configuration page, enter the cluster name, fill in any FE node IP, HTTP port, JDBC port, database root user password, and Agent Port to take over the cluster.

Note: The hosts that take over the cluster also need to deploy the Agent in advance for host registration.

Node Settings

If you successfully connect to the cluster, you can see the information of the FE and BE nodes of the cluster. Here, VeloDB Manager will check the Agent status, machine environment, and node status of the nodes. If there is an exception, please recover first before continuing to take over the cluster. If the cluster you are taking over has deployed WebUI before, you can fill in the information of the WebUI node here, including the deployment node and Port Number. VeloDB Manager will check the node connection status and whether the process exists.

At the same time, you can also specify the automatic control mode and choose whether to set up automatic pull-up guardians for Doris processes. Finally, click on Takeover Cluster to complete the takeover.

VeloDB Manager will automatically complete the cluster takeover, which takes about 5 minutes.

You can retain the connection information for the cluster. VeloDB Manager will not modify or retrieve the user passwords in your cluster during the takeover process.

Operations

Cluster Overview

In the cluster overview, you can view the current operational status of the cluster, including the live status of FE (Frontend) and BE (Backend) nodes, as well as information about the database and data volume. You can also find the cluster's link access method in the cluster details.

In the cluster overview, you have the option to restart the entire cluster or all FE and BE nodes, as well as to scale the cluster.

Node Details

View the status, port, and version of FE (Frontend), BE (Backend), and WebUI nodes.

In addition, in the BE node, you can set labels for the BE node, and nodes with the same label form a resource group.

Host Details

The host metrics include Agent status, deployment modules, CPU, memory, disk, network, etc. The running metrics will be refreshed in real time, with a frequency of 5s/time. You can click the sort button next to the corresponding metric title to find nodes with bottlenecks in the cluster.

Cluster Scaling

In the upper right corner of the cluster page, you can enter the cluster volume expansion and contraction function to perform expansion and contraction operations on FE and BE.

When expanding the node through Manager, the expanded node must be the node that deploys Agent and registers successfully. If the expansion drop-down selection node is empty, you can drop down to select Add Host and enter the Add Host page for registration.

On the Add Host page, you can refer to the Agent Installation Guide to deploy on the target host. The Agent port of the newly added host needs to be consistent with other nodes in the cluster.

After the deployment is completed, enter the IP information (supporting IPV4 and IPV6), and VeloDB Manager will automatically detect the Agent status of the host.

After the host is successfully added, you can drop down to select the successfully registered host on the expansion page. The expanded node supports configuration port, deployment directory and other information just like the new deployment.

After confirming the expansion node information, you can enter the cluster expansion page to check the progress of the expansion.

Restart cluster

In the upper right corner of the cluster module, you can perform a restart operation. You can restart FE , BE individually, or the entire cluster.

You can choose rolling restart or full restart to restart the cluster. If you want to restart the nodes one by one, choose rolling restart. If you want to restart quickly, choose full restart.

After clicking Restart , you will enter the restart task and the task log will be displayed.

Configuration

Edit configuration

VeloDB Manager supports modifying corresponding parameters by editing configuration files. You can click on the drop-down menu in the upper right corner of the cluster module and select Parameter Configuration . On the configuration page, you can make modifications to FE or BE parameters for individual or multiple nodes.

If you modify multiple node configurations in bulk, the configurations of all nodes will be overwritten with the same content. If you need to customize some node configurations, please modify the node configurations one by one. After modification, you can choose restart mode to restart the cluster to make the configuration take effect.

Note: Port and path configurations cannot be modified here.

View Configuration

On the node page, you can view the runtime parameters for each node, displaying all the values for these runtime parameters.

Cluster Upgrade

VeloDB Manager supports upgrading the cluster with two modes: full service stop upgrade and online rolling upgrade. You can click on the drop-down menu in the upper right corner, select Cluster Upgrade and choose the target upgrade version and upgrade mode to initiate the cluster upgrade operation.

Note : backup data is optional when upgrading. If your cluster data volume is large, the backup time will be longer, and the service will not be available during the period. If you do not check backup, the upgrade does not support rollback.

Currently, VeloDB Manager supports upgrading Apache Doris and SelectDB Doris clusters above version 1.2.1. Upgrading the cluster version has restrictions related to version type and version number, as shown in the table below.

After initiating the upgrade operation, you can see the detailed upgrade process and progress on the cluster upgrade page.

Full service stop upgrade supports rollback, while online rolling upgrade is not reversible after completion. After a full service stop upgrade, you can inspect the upgraded cluster, and if any issues are detected, you have the option to initiate a rollback operation.

Monitor

Monitor

VeloDB Manager provides a comprehensive set of pre-configured monitoring metrics, helping you gain real-time insights into the operational status of the cluster.

Alarm

You can set the alarm threshold for monitoring metrics as needed. When the metric exceeds the threshold, VeloDB Manager will send an alarm notification through email, chat software, Webhook, etc., reminding you to deal with potential problems in a timely manner. When you use email as an alarm method, you need to configure your mail server in the configuration file of VeloDB Manager.

During cluster initialization, VeloDB Manager has built-in default alarm templates according to best practices. When creating an alarm policy, you can select the indicators in the template for quick configuration.

At the same time, two types of templates are built into the alarm strategy, one for the cluster dimension and the other for the host dimension. You can quickly configure according to the templates.

On the list page, you can view existing alarm policies and their alarm history.

Channel

When using the alarm robot, please ensure that you can connect to the public network.

Wecom

Configuration method: fill in the robot webhook.

1. On WeCom for PC, find the target WeCom group for receiving alarm notifications.
2. Right-click the WeCom group. In the window that appears, click Add Group Bot .
3. In the window that appears, click Create a Bot .
4. In the window that appears, enter a custom bot name and click Add .
5. Copy the webhook address.

Lark

Configuration method: fill in the robot webhook.

To make a custom bot instantly push messages from an external system to the group chat, you need to use a webhook to connect the group chat and your external system. Enter your target group and click Settings > BOTs > Add Bot . Select Custom Bot . Enter a suitable name and description for your bot and click Next .

You'll then get the webhook address.

DingTalk

Configuration method: fill in the robot webhook.

1. Run the DingTalk client on a PC, go to the DingTalk group to which you want to add a chatbot, and then click the Group Settings icon in the upper-right corner.
2. In the Group Settings panel, click Group Assistant .
3. In the Group Assistant panel, click Add Robot .
4. In the ChatBot dialog box, click the + icon in the Add Robot section. Then, click Custom .

5. In the Robot details dialog box, click Add . 6. In the Add Robot dialog box, perform the following steps: 7. Set a profile picture and a name for the chatbot. 8. Select Custom Keywords for the Security Settings parameter. Then, enter alert . 9. Read the terms of service and select I have read and accepted DingTalk Custom Robot Service Terms of Service . 10. Click Finished . 11. In the Add Robot dialog box, copy the webhook URL of the DingTalk chatbot and click Finished .

Logs

Logs serve as a crucial tool for troubleshooting issues and are an essential feature for managers. Doris logs exist in the form of files, and VeloDB Manager operates independently of the cluster. This enables you to access logs for issue diagnosis even when the cluster is unavailable.

To enter the log module, you first need to choose nodes and logs in the upper left corner. After making the selection, you can preview the logs on the right.

When viewing the logs, you can choose to view from the end or the beginning of the log file, corresponding to the options Tail/Head in the upper right corner.

Simultaneously, you can perform queries on the log content. During a query, you can choose whether the query is case-sensitive and whether it is a strict match. Using the up and down buttons on the right side of the query box, you can locate matching items on the page.

VeloDB Manager does not load logs into the frontend all at once but rather loads them on demand. However, during a query, VeloDB Manager will try to match items through full-text search. You can find all the query results on the left and quickly locate the specific location of the matching item in the logs by clicking on the entry.

Task Audit

With the development of business, the regulatory demand for task execution and responsibility tracing is increasing. The audit function of historical tasks has become an indispensable part of control tools, which can help managers trace the execution process of tasks and record the complete lifecycle of each task from initiation to completion.

The task audit module of VeloDB Manager records detailed information about each task, including task type, ID, module, status, initiator, execution progress, start time, and end time.

At the same time, by selecting a specific task in the list, you can view the execution details of the task. The webserver log of each step of the task execution is recorded in the task details.

At the same time, you can also view the execution log of the node dimension under each step.

Inspection

VeloDB Manager is equipped with built-in inspection items that may impact the stability and operational efficiency of the cluster, supporting a one-click inspection of the entire cluster.

Abnormal states in the inspection are categorized into three types:

• Execution Failure: During the execution, there is no successful return result, which could be caused by issues such as permissions, machine environment settings, or cluster availability.
• Warning: Inspection items in this category may have a significant impact on the healthy operation of the cluster. You can click View Suggestions to learn how to address and fix these issues.
• Advisory: Inspection items in this category may have some impact on the healthy operation of the cluster or pose potential risks. You can click View Suggestions to understand how to address and fix these issues.

Additionally, you can click Export to download the inspection report as a PDF file to your local system.

Inspection supports the configuration of regular inspections. You can enable regular inspections and configure the inspection cycle and notification as needed.

Settings

You can change the cluster name, the root user password for the database, the admin user password for the database, and stop hosting the cluster in the settings.

WebUI

VeloDB Manager provides a convenient WebUI access point in the left navigation bar. In WebUI, users can view data and information within the database. Additionally, they can perform operations such as SQL queries, data import, and manage permissions.

If you did not deploy WebUI during the initial deployment or if you have deployed but have not taken over, you can click on the **WebUI **in the left navigation bar and select the node from the successfully registered hosts for configuration.