BYOC Warehouse Guide
What is BYOC
BYOC(Bring Your Own Cloud) warehouse provides you with VeloDB Cloud data warehouse installation and operational services within your own cloud resource pool. When you initiate a computing cluster, the corresponding virtual machine resources will start up in your VPC (Virtual Private Cloud), with the costs being charged to you by the cloud platform. Additionally, you will need to pay for the usage duration of the VeloDB service.
When should I use a BYOC warehouse?
Generally, utilizing a BYOC warehouse serves two purposes:
- Compliance: Some compliance frameworks require that users' data can only be stored within their own cloud VPC. The BYOC mode provides data warehouse service within your own cloud resource pool, complying with these requirements. In terms of security, both BYOC and SAAS (Software as a Service) warehouses have been recognized for their security by industry compliance frameworks.
- Cost: The virtual machine resources used by the BYOC warehouse are initiated within your cloud resource pool, and the costs are settled directly with the cloud platform. For customers who have significant discounts with cloud platforms, BYOC warehouses offer better cost advantages.
Although VeloDB Cloud provides a one-stop solution for data warehouse installation and operation, the BYOC warehouse is located within your own cloud resource environment, requiring you to have a basic understanding of the cloud environment, such as network segment planning, load balancing, etc.
BYOC Warehouse Architecture
The BYOC warehouse will install a control Agent, along with necessary monitoring components, within your VPC. The control Agent will fetch control commands from VeloDB Cloud through a private network connection (PrivateLink) to carry out cluster creation, scaling, and upgrading operations as commanded via the VeloDB Cloud Manager.
The control Agent code is open and auditable, ensuring that your data will be stored within your VPC and will not be transmitted externally.
(AWS) How to create BYOC Warehouse
Click on "Create Warehouse", select "BYOC" for Warehouse Mode, choose the cloud platform and the region it's located in, and select the desired version for the warehouse.
This table lists the AWS Regions and Availability Zones where VeloDB Cloud BYOC Warehouse can be deployed, along with the associated Availability Zone IDs. You will need this information to choose an available subnet when setting up AWS CloudFormation.
Cloud Platform | Region | Availability Zone ID |
---|---|---|
AWS | us-east-1 | use1-az2 |
If you have a subnet within the Availability Zones, you can select it directly; otherwise, you need to create a new subnet in the listed Availability Zones. If you wish to use VeloDB Cloud in more Regions and Availability Zones, please contact us (opens in a new tab).
Please note, the mapping of Availability Zone to Availability Zone ID is not consistent across different AWS accounts . Please select or create your subnet according to the Availability Zone ID .
Next, configure the VPC. If you have created a BYOC warehouse in a VPC previously, we can reuse the control components within that VPC to create the warehouse directly. If it's a VPC where a BYOC warehouse hasn't been created before, initialization of the VPC is required.
Choose "New VPC" and click the "Create" button. A new window will open with the CloudFormation page. VeloDB Cloud will use Cloud Formation (CLF) to create the corresponding resources, completing the environment preparation within your VPC seamlessly.
Please note that the following actions will be taken when executing CloudFormation:
IAM Role Creation
- A role will be created with permission to perform tasks like launching/terminating EC2, accessing/purchasing S3 Buckets, and creating/deleting private network connections.
- The credentials (AKSK/ARN) of this role are stored in your VPC. Notably, VeloDB Cloud does not acquire this user's access information.
EC2 Instance launch
- An EC2 instance will be launched in your VPC. This instance will be used to deploy the Agent for management and control.
Private Network Connection Establishment to VeloDB VPC
- The Agent will utilize a private network to pull management commands from the VeloDB Cloud and transmit monitoring metric data to VeloDB Cloud.
- The connection established is unidirectional.The audited code of the Agent doesn't include any functionality to fetch user data from your VPC.
The CloudFormation code provided by VeloDB is open and auditable, and will not operate on your data or other environments within your VPC.
Once CloudFormation execution is complete, VeloDB Cloud will establish a connection with the BYOC warehouse. Subsequently, you can enter the warehouse and create clusters.
(GCP) How to create BYOC Warehouse
Create VPC and Subnet in GCP
Before creating a warehouse, please create a VPC and subnet in GCP in advance. For detailed instruction, please refer to Create and manage VPC networks (opens in a new tab).
Record the VPC id and subnet id, as they will be used in subsequent operations.
Create Warehouse
Click the "+ New Warehouse" button to enter the creation page.
Fill in the following information to start creating warehouse.
Parameters | Description |
---|---|
Warehouse Name | Up to 32 characters, you can use Chinese, letters (case insensitive), numbers, _ and - |
Warehouse Mode | Select BYOC |
Cloud Platform | Select GCP (Google Cloud Platform) |
Region | us-west1 is supported |
Core Version | Select available version |
VPC | To create a new warehouse in a VPC for the first time, select New VPC. To create a new warehouse in a VPC with an existing warehouse, select the VPC name. |
Click the "Create" button to proceed to the next step.
Cloud Resource Configuration
To deploy VeloDB Cloud BYOC warehouse in Google Cloud, select your project correctly, then fill in the parameters to select where to deploy the base BYOC environment to begin the automated creation process after the VPC and subnet.
Fill in the following information to execute shell.
Parameters | Description |
---|---|
VPC ID | The VPC ID created in the previous step |
Subnet ID | The Subnet ID created in the previous step |
Zone ID | us-west1-a |
Wait for the script to complete, then return to the velodb page, and wait for the "Warehouse Initialization" step to complete.
Once Warehouse Initialization is complete, VeloDB Cloud will establish a connection with the BYOC warehouse. Subsequently, you can enter the warehouse and create clusters.
Manager Function Guide
The management of a BYOC warehouse is roughly similar to a SAAS warehouse, with minor differences based on the architecture.
Clusters
You can create clusters as usual and set automatic start or stop policy.
It's important to note that the billing for BYOC warehouse is divided into two parts:
- Cloud resource fee: The cost generated by the virtual machines, object storage, private network endpoint, etc. when starting a cluster, charged by the cloud platform.
- Compute service fee: The service fee charged for managing the cluster, billed by VeloDB Cloud, 90 days free, after that, please refer to the Pricing Page (opens in a new tab).
Connections
In the connection module, since the core components of the warehouse have entered your VPC, private connection is no longer needed. To avoid disrupting your VPC's network planning, we haven't set up public network access. You can set up load balancing and open public access on the cloud platform console yourself.
Merics
Monitoring alerts remain consistent with SAAS warehouses. You can still use our pre-set monitoring metrics and receive alerts through various channels.
Usage
The usage module will display the current usage of your warehouse, including computation (vCPU-Hour) and storage (GB-Hour), allowing you to grasp the usage situation of the warehouse in VeloDB Cloud Manager.
Settings
In the settings module, you can change the warehouse name, modify the warehouse admin user's password, upgrade the warehouse version, and delete the warehouse.
Note that after deleting the warehouse, VeloDB Cloud does not have the authority to delete the control components within your VPC. You can thoroughly delete the machine resources and private network connections generated by the control within your VPC by deleting the stack generated by CloudFormation.
Precautions
Most warehouse resources run within your cloud environment, so avoid operating cloud resources created by VeloDB Cloud directly on the cloud platform console.
The cloud resources created by VeloDB Cloud have the following three tags (Tag):
- velodb-cloud-resource : vdb-server
- Name : Specific warehouse ID
- velodb-cluster-id : Specific cluster ID
You can filter the resources created by VeloDB Cloud through the filter function on the cloud platform console.
Actions that may cause the warehouse to become abnormally unavailable include:
- Modifying the permissions of the IAM user created by VeloDB Cloud
- Modifying or deleting the virtual machines, storage buckets created by VeloDB Cloud
- Modifying or deleting the endpoint services created by VeloDB Cloud
Please note, the warehouse unavailability caused by your operations on the cloud platform console may be irrecoverable.