VeloDB Manager 23.x User Guide

Overview

Main Features

VeloDB Manager provides the following main features:

• Cluster Deployment : Deploy Aache Doris or VeloDB Doris clusters on physical or virtual machines using VeloDB Manager.
• Cluster Takeover : Take over existing Apache Doris or VeloDB Doris clusters for operation and monitoring using VeloDB Manager.
• Cluster Details : View the operational status, details, and connection information of the cluster.
• Cluster Scaling : Scale FE and BE nodes in or out.
• Cluster Upgrade : Upgrade the cluster version, providing options for full-service stop upgrade and online rolling upgrade. Choose the appropriate upgrade method based on business scenarios.
• Cluster Restart : Restart the entire cluster, FE, BE, and nodes, supporting both rolling and full restarts.
• Node Details : View real-time status and machine information of nodes.
• Monitoring and Alerts : View monitoring metrics, set alert rules, and receive alert notifications through email, chat software, Webhook, etc.
• Parameter Configuration : Support custom editing of configuration files for nodes, with the ability to view all running parameters for individual nodes.
• Log Viewing : Support viewing and querying logs for FE and BE nodes, facilitating offline troubleshooting of cluster issues.
• Cluster Inspection : One-click check machine conditions and cluster operational status, promptly discover and locate performance bottlenecks, and provide repair suggestions.
• WebUI : Entry point to access the cluster's WebUI.
• Others : Change passwords for the root user and admin user of the cluster, stop taking over the cluster, etc.

Service Configuration and User System

After extracting the newly installed VeloDB Manager, it only requires a single line of code to start. Simultaneously, the identity credentials for managing the cluster have switched from the cluster root user to natural person users, enhancing cluster data protection and more effectively allocating permissions.

Configuration for Starting VeloDB Manager

The first step after starting is to set up the administrator account. The administrator account has the highest authority within VeloDB Manager, enabling tasks such as user configuration, VeloDB Manager's self-configuration, upgrades, and more. Other accounts must be added with the identity of the administrator account.

You can customize the username and password, which will be used for logging into the VeloDB Manager system later. Please remember the administrator username and password. In the case of having only one administrator account in VeloDB Manager, if the administrator account's username and password are lost, they can not be reset or recovered.

Next, the configuration for VeloDB Manager itself will take place.

Setting is divided into two parts, namely Component Configuration and Package Configuration:

• Component Configuration: Involves the configuration of components related to VeloDB Manager's control service and monitoring alerts. There are mainly two aspects, namely, control port and monitoring alert service.
  • Control Port: A required field. VeloDB Manager needs to occupy a port for the management and control of the cluster. Please select an open and available port on the machine where VeloDB Manager is installed.
  • Monitoring Alert Service: An optional field used to configure VeloDB Manager's monitoring alert module. This involves the installation of Grafana, Prometheus, and Alertmanager. You need to select three available ports on the machine where VeloDB Manager is installed.
    • Email Alert: Configures the email server, which can later be used as the "Email Alert" channel in the alerts.
• Package Configuration: Configures the local Doris Core and VeloDB Manager installation packages for creating and upgrading clusters.

After completing the configuration, click on Start VeloDB Manager . VeloDB Manager will restart and load the configuration. The restart of VeloDB Manager itself will not affect existing clusters (both taken over and untaken over).

After the restart is complete, click to Enter VeloDB Manager , and you will access the management page.

Service Configuration

After entering VeloDB Manager, you can click on the user's avatar in the bottom left corner, select Service Configuration from the menu, and make configuration modifications. After the modifications, VeloDB Manager will restart and load the new configuration.

Access Control

In User Management, system administrators can create new users and assign roles to them.

VeloDB Manager has three built-in roles, and their permission scopes are as follows:

Cluster Management involves operational actions on the cluster, such as parameter configuration, restart, upgrade, etc. Having cluster management permission implies having cluster viewing permission.

You can also create roles and assign them specific permission scopes.

Change Login Password

Click on the user avatar in the bottom left corner, select Change Sign in Password and you can modify the login password for the current user.

Upgrade VeloDB Manager

When there is a higher version of VeloDB Manager in the local installation package path, there will be an upgrade prompt at the VeloDB Manager version in the bottom-left menu. Click on the VeloDB Manager version to initiate the upgrade process.

New / Takeover Cluster

Deploy Cluster

Deploy Apache Doris or VeloDB Doris clusters on physical or virtual machines through VeloDB Manager. VeloDB Manager will automatically handle the installation and configuration of the cluster. During the deployment process, users can customize the cluster's scale, node count, and other parameters to meet various business requirements.

After deployment, VeloDB Manager will provide the connection information for the cluster, facilitating users in quick access and utilization.

Environment Settings

First, set the cluster name in **Environment Settings ** , choose the creation method as Deploy New Cluster , select the version to deploy, and set the root user password for the database. Currently, VeloDB Manager supports deploying VeloDB Doris version 1.2.0 and above, as well as Apache Doris clusters.

Node Settings

Afterward, configure access authorization for node machines, FE module, BE module, WebUI, and automatic management mode in Node Settings . The specific module configuration details are as follows:

• Node Access Authorization
  • Username and Password: All nodes use the same set of username and password information. Ensure that the provided username and password are available on all node machines.
  • SSH KEY : Enter the username, passphrase, and the path to the SSH KEY.
• FE Configuration
  • Common Configuration: Represents the configuration for various ports, installation directory, and metadata storage directory used by all FE nodes. The system provides default values that you can modify based on the machine environment.
  • FE Nodes: Supports adding nodes in IPV4 and IPV6 formats, specifying the role of the node (FE Follower must be an odd number), expanding custom configurations to modify various ports, priority networks, installation paths, and metadata storage directories for FE nodes.
• BE Configuration
  • Common Configuration: Represents the configuration for various ports, installation directory, and data storage directory used by all BE nodes. The system provides default values, and you can modify them based on the machine environment.
  • BE Nodes: Supports adding nodes in both IPV4 and IPV6 formats. Expand the custom configuration to modify various ports, installation paths, and data storage directory for BE nodes.
• WebUI Configuration
  • WebUI requires access through a browser; ensure connectivity between the browser and the deployment machine. You can specify memory for WebUI, with a recommended range of 1 to 2 GB.
• Advanced Configuration
  • Enable Automatic Management Mode, VeloDB Manager will set up automatic daemon launching for Doris processes on the machine. If you prefer no intervention from VeloDB Manager, you can choose to disable the automatic mode.

Note: Only VeloDB deployment includes the WebUI module; other versions do not have WebUI.

During deployment, VeloDB Manager performs a connectivity check on the machine nodes. Only when the check indicates a normal connection status can the deployment proceed.

If VeloDB Manager encounters an abnormality in the connection status of machine nodes, there are two types of errors :

• Connection Error: This error signifies that VeloDB Manager is unable to establish a connection to the node using the information provided in the Node Access Authorization . If you have chosen the username and password method, please verify that the login credentials and port number for accessing the node are correct. If you have opted for the SSH KEY method, ensure that the SSH KEY path, passphrase, and port are accurate.
• Invalid: This error type encompasses the following situations:
  • The machine's configuration does not meet the deployment requirements for Doris, such as the swap partition not being closed, the maximum open file limit being less than 65535, transparent huge pages not being closed, and the virtual memory region count not meeting the requirements. Follow the prompts provided by VeloDB Manager to make the necessary modifications.
  • Port occupation: Change to an unused port if the current port is occupied.
  • Deployment path check: Ensure that the space in the directory where the installation path is located is greater than 20 GB, and the directory itself is not occupied.

Clicking on Deploy Cluster , VeloDB Manager will initiate the cluster deployment process.

After successful deployment, you will see the access information for the cluster and the initialized admin user password. You can copy and save this information. VeloDB Manager does not store user passwords; these password details appear only once here.

You can view the database connection information in the cluster details, but you will not see password information. Next, click on Enter Cluster to access VeloDB Manager and manage the cluster.

Take Over Cluster

Take over an existing Apache Doris or VeloDB Doris cluster and manage it through VeloDB Manager for operations and monitoring. You only need to provide relevant information about the existing cluster, such as the cluster address and login credentials. VeloDB Manager can automatically take over the cluster, enabling unified management.

After the takeover, you can perform operational tasks on the cluster through VeloDB Manager, such as monitoring, scaling, restarting, etc. Additionally, you can view information about the cluster's runtime status and monitoring metrics, helping users better understand and manage the cluster.

Environment Settings

In the new cluster configuration page, enter the cluster name and choose Takeover Existing Cluster as the creation method.

Node settings

On the Node Settings page, enter the IP, HTTP port, and JDBC port of any one of the FE nodes of the cluster to be taken over. Configure access permissions for the node as well. Similarly, if the provided node access user is not the root user, follow the prompts to configure the machine environment. All nodes use the same set of username and password information, so ensure that the provided username and password are accessible on all node machines.

Note: The node username entered during access authorization needs to be consistent with the username used to deploy the Doris cluster; otherwise, the takeover of the cluster may fail.

Next, click on Connect Cluster .

If the connection to the cluster is successful, you will see the nodes of the cluster in a normal state. Here, you can configure the WebUI. If the cluster you took over had WebUI deployed previously, you can enter the information for the WebUI node here, including the deployment node and port number. VeloDB Manager will check the node's connection status and whether the process is running.

At the same time, you can specify the automatic control mode, choosing whether to set up automatic restart for Doris processes. Finally, click on **Take Over Cluster **to complete the takeover process

VeloDB Manager will automatically complete the cluster takeover, and this stage typically takes approximately 10-15 minutes.

You can retain the connection information for the cluster. VeloDB Manager will not modify or retrieve the user passwords in your cluster during the takeover process.

Operations

Cluster Overview

In the cluster overview, you can view the current operational status of the cluster, including the live status of FE (Frontend) and BE (Backend) nodes, as well as information about the database and data volume. You can also find the cluster's link access method in the cluster details.

In the cluster overview, you have the option to restart the entire cluster or all FE and BE nodes, as well as to scale the cluster.

Node Details

View the status, metrics, and machine information of FE (Frontend), BE (Backend), and WebUI nodes, you can examine detailed information for each FE, BE, and WebUI node, such as node status, operational metrics, and machine details.

Operational metrics include CPU, memory, disk, network, etc. These metrics refresh in real-time at a frequency of 5 seconds per update. You can click on the sorting button next to the corresponding metric title to identify nodes in the cluster facing bottlenecks.

Additionally, on BE nodes, you have the option to assign labels to BE nodes, where nodes with the same label form a resource group.

Cluster Scaling

In the upper right corner of the cluster page, you can access the cluster scaling feature to perform scaling operations on FE and BE nodes. The access method for newly added nodes should be consistent with the one provided during the initial creation or takeover. Nodes added during scaling support configuration options such as ports, installation paths, and other details, similar to the ones during initial deployment.

Restart Cluster

In the upper right corner of the cluster module, you can perform restart operations. You can individually restart FE or BE nodes, or restart the entire cluster.

Cluster restarts offer the option of either rolling restart or full restart. If you prefer a sequential restart of nodes, choose rolling restart. If you aim for a quick restart, opt for full restart."

After clicking Restart , you will enter the restart task and the task log will be displayed.

Configuration

Edit Configuration

VeloDB Manager supports modifying corresponding parameters by editing configuration files. You can click on the drop-down menu in the upper right corner of the cluster module and select Parameter Configuration . On the configuration page, you can make modifications to FE or BE parameters for individual or multiple nodes.

If you batch edit configurations for multiple nodes, the configurations for all nodes will be overwritten with the same content.. If you need to customize configurations for specific nodes, please modify them one by one. After confirmation, the cluster will be automatically restarted to apply the new configurations.

Note: Port and path configurations can not be modified here.

View Configuration

On the node page, you can view the runtime parameters for each node, displaying all the values for these runtime parameters.

Cluster Upgrade

VeloDB Manager supports upgrading the cluster with two modes: full service stop upgrade and online rolling upgrade. You can click on the drop-down menu in the upper right corner, select Cluster Upgrade and choose the target upgrade version and upgrade mode to initiate the cluster upgrade operation.

Currently, VeloDB Manager supports upgrading Apache Doris clusters version 1.2 and above. There are restrictions on the cluster version upgrade based on version types and version numbers, as indicated in the following table:

Full service stop upgrade supports rollback, while online rolling upgrade is not reversible after completion. After a full service stop upgrade, you can inspect the upgraded cluster, and if any issues are detected, you have the option to initiate a rollback operation.

Monitor

Monitor

VeloDB Manager provides a comprehensive set of pre-configured monitoring metrics, helping you gain real-time insights into the operational status of the cluster.

Alarm

You can set alert thresholds for monitoring metrics based on your needs. When a metric exceeds the threshold, VeloDB Manager will send alert notifications via email, chat software, Webhook, and other methods to remind you to promptly address potential issues.

When using email as the alert method, you need to configure your email server in the VeloDB Manager's configuration file.

You can view the existing alert policies and their current alert status on the list page.

Channel

When using the alerm robot, please ensure that you can connect to the public network.

Wecom

Configuration method: fill in the robot webhook.

1. On WeCom for PC, find the target WeCom group for receiving alarm notifications.
2. Right-click the WeCom group. In the window that appears, click Add Group Bot .
3. In the window that appears, click Create a Bot .
4. In the window that appears, enter a custom bot name and click Add .
5. Copy the webhook address.

Lark

Configuration method: fill in the robot webhook.

To make a custom bot instantly push messages from an external system to the group chat, you need to use a webhook to connect the group chat and your external system. Enter your target group and click Settings > BOTs > Add Bot . Select Custom Bot . Enter a suitable name and description for your bot and click Next .

You'll then get the webhook address.

DingTalk

Configuration method: fill in the robot webhook.

1. Run the DingTalk client on a PC, go to the DingTalk group to which you want to add a chatbot, and then click the Group Settings icon in the upper-right corner.
2. In the Group Settings panel, click Group Assistant .
3. In the Group Assistant panel, click Add Robot .
4. In the ChatBot dialog box, click the + icon in the Add Robot section. Then, click Custom .

5. In the Robot details dialog box, click Add . 6. In the Add Robot dialog box, perform the following steps: 7. Set a profile picture and a name for the chatbot. 8. Select Custom Keywords for the Security Settings parameter. Then, enter alert . 9. Read the terms of service and select I have read and accepted DingTalk Custom Robot Service Terms of Service . 10. Click Finished . 11. In the Add Robot dialog box, copy the webhook URL of the DingTalk chatbot and click Finished .

Logs

Logs serve as a crucial tool for troubleshooting issues and are an essential feature for managers. Doris logs exist in the form of files, and VeloDB Manager operates independently of the cluster. This enables you to access logs for issue diagnosis even when the cluster is unavailable.

To enter the log module, you first need to choose nodes and logs in the upper left corner. After making the selection, you can preview the logs on the right.

When viewing the logs, you can choose to view from the end or the beginning of the log file, corresponding to the options Tail/Head in the upper right corner.

Simultaneously, you can perform queries on the log content. During a query, you can choose whether the query is case-sensitive and whether it is a strict match. Using the up and down buttons on the right side of the query box, you can locate matching items on the page.

VeloDB Manager does not load logs into the frontend all at once but rather loads them on demand. However, during a query, VeloDB Manager will try to match items through full-text search. You can find all the query results on the left and quickly locate the specific location of the matching item in the logs by clicking on the entry.

Inspection

VeloDB Manager is equipped with built-in inspection items that may impact the stability and operational efficiency of the cluster, supporting a one-click inspection of the entire cluster.

Abnormal states in the inspection are categorized into three types:

• Execution Failure: During the execution, there is no successful return result, which could be caused by issues such as permissions, machine environment settings, or cluster availability.
• Warning: Inspection items in this category may have a significant impact on the healthy operation of the cluster. You can click View Recommendations to learn how to address and fix these issues.
• Advisory: Inspection items in this category may have some impact on the healthy operation of the cluster or pose potential risks. You can click View Recommendations to understand how to address and fix these issues.

Additionally, you can click Export to download the inspection report as a PDF file to your local system.

Settings

You can change the cluster name, the root user password for the database, the admin user password for the database, and stop hosting the cluster in the settings.

If you have deployed/taken over the cluster within an intranet but wish to allow external access to the WebUI, you can configure this in the settings under Modify External WebUI Access Address .

WebUI

VeloDB Manager provides a convenient WebUI access point in the left navigation bar. In WebUI, users can view data and information within the database. Additionally, they can perform operations such as SQL queries, data import, and manage permissions.