GCP Preparation

Add users to the organization and authorize them

Create a custom role

Notice: This custom role is the minimum set of permissions required to create a velodb cloud warehouse. If you plan to directly grant the user the Owner permission of the project, you can skip creating a custom role.

For detailed steps on creating a custom role, refer to the GCP documentation (opens in a new tab) .

Switch to the gcloud tab and follow the instructions to create a custom role using cloud shell:

Activate Cloud Shell on this page.

Execute the vim velodb_cloud_role.yaml command in Cloud Shell to create a yaml file.

Press i to enter edit mode, copy and paste the following content, press ESC, enter :wq to save and exit.

title: velodb_cloud_role
description: velodb cloud role
stage: GA
includedPermissions:
- compute.addresses.create
- compute.addresses.createInternal
- compute.addresses.delete
- compute.addresses.deleteInternal
- compute.addresses.setLabels
- compute.addresses.get
- compute.addresses.list
- compute.addresses.use
- compute.addresses.useInternal
- compute.disks.create
- compute.disks.createTagBinding
- compute.disks.delete
- compute.disks.deleteTagBinding
- compute.disks.get
- compute.disks.list
- compute.disks.resize
- compute.disks.setLabels
- compute.disks.use
- compute.firewalls.create
- compute.firewalls.createTagBinding
- compute.firewalls.delete
- compute.firewalls.deleteTagBinding
- compute.firewalls.get
- compute.firewalls.list
- compute.firewalls.update
- compute.forwardingRules.create
- compute.forwardingRules.delete
- compute.forwardingRules.get
- compute.forwardingRules.pscCreate
- compute.forwardingRules.pscDelete
- compute.globalOperations.get
- compute.instanceGroups.create
- compute.instanceGroups.delete
- compute.instanceGroups.update
- compute.instanceGroups.use
- compute.instances.attachDisk
- compute.instances.create
- compute.instances.createTagBinding
- compute.instances.delete
- compute.instances.deleteTagBinding
- compute.instances.detachDisk
- compute.instances.get
- compute.instances.getEffectiveFirewalls
- compute.instances.list
- compute.instances.listTagBindings
- compute.instances.osAdminLogin
- compute.instances.osLogin
- compute.instances.reset
- compute.instances.resume
- compute.instances.setDiskAutoDelete
- compute.instances.setLabels
- compute.instances.setMachineType
- compute.instances.setMetadata
- compute.instances.setName
- compute.instances.setServiceAccount
- compute.instances.setTags
- compute.instances.start
- compute.instances.stop
- compute.instances.suspend
- compute.instances.update
- compute.instances.use
- compute.networks.get
- compute.networks.list
- compute.networks.updatePolicy
- compute.networks.use
- compute.regionBackendServices.create
- compute.regionBackendServices.createTagBinding
- compute.regionBackendServices.delete
- compute.regionBackendServices.deleteTagBinding
- compute.regionBackendServices.get
- compute.regionBackendServices.getIamPolicy
- compute.regionBackendServices.list
- compute.regionBackendServices.update
- compute.regionBackendServices.use
- compute.regionHealthChecks.create
- compute.regionHealthChecks.delete
- compute.regionHealthChecks.useReadOnly
- compute.regionOperations.get
- compute.subnetworks.get
- compute.subnetworks.list
- compute.subnetworks.setPrivateIpGoogleAccess
- compute.subnetworks.use
- compute.zoneOperations.get
- compute.zones.get
- compute.zones.list
- iam.roles.create
- iam.roles.delete
- iam.roles.get
- iam.roles.update
- iam.serviceAccounts.actAs
- iam.serviceAccounts.create
- iam.serviceAccounts.createTagBinding
- iam.serviceAccounts.delete
- iam.serviceAccounts.deleteTagBinding
- iam.serviceAccounts.enable
- iam.serviceAccounts.get
- iam.serviceAccounts.getAccessToken
- iam.serviceAccounts.getIamPolicy
- iam.serviceAccounts.setIamPolicy
- iam.serviceAccounts.update
- resourcemanager.projects.createPolicyBinding
- resourcemanager.projects.deletePolicyBinding
- resourcemanager.projects.get
- resourcemanager.projects.getIamPolicy
- resourcemanager.projects.setIamPolicy
- servicedirectory.namespaces.create
- servicedirectory.namespaces.delete
- servicedirectory.namespaces.get
- servicedirectory.services.create
- servicedirectory.services.delete
- servicedirectory.services.get
- serviceusage.services.enable
- serviceusage.services.use
- storage.buckets.create
- storage.buckets.createTagBinding
- storage.buckets.delete
- storage.buckets.deleteTagBinding
- storage.buckets.get
- storage.buckets.getIamPolicy
- storage.buckets.setIamPolicy
- storage.hmacKeys.create
- storage.hmacKeys.delete
- storage.hmacKeys.get
- storage.hmacKeys.update
- storage.objects.delete
- storage.objects.get
- storage.objects.list

Replace PROJECT_ID with the actual project id and execute the following command
```
gcloud iam roles create velodb_cloud_role --project=PROJECT_ID \
--file=velodb_cloud_role.yaml
```
Input y to complete the creation of the custom role.

Note: For detailed permission description, please see the Permissions that Terraform templates rely on section below.

Add users to the organization and authorize them

Notice: If you already have a user, you can skip the users creation step.

In GCP, user accounts are usually managed through Google Workspace or personal Gmail accounts. If you want to add users to your organization, you usually need administrator privileges. Here are the steps to add a new user:

Add a user through Google Workspace (if using Google Workspace)
- Sign in to the Google Admin Console (opens in a new tab).
- In the left navigation bar, click Users.
- Click the Add user button at the top of the page.
- Enter the user's basic information (such as first name, last name, and email address), and then click Add.
- The new user will receive an email with a temporary password to log in to Google Workspace.
Assign a role through IAM (using the GCP console):
- Sign in to the Google Cloud Console (opens in a new tab).
- Select your project or organization.
- Navigate to IAM & Admin > IAM.
- Click the Add button at the top of the page.
- Enter the user's email address.
- Select the role above to assign and click Save.

Prepare a VPC and subnet

Notice:

Currently supported region is us-east1.

If you already have a VPC and subnet, you can skip the creation step.

Before creating a BYOC type warehouse, you need to use the above user to create a VPC and subnet in advance. The following are the specific operations.

Open the GCP VPC Network (opens in a new tab) console and switch to the region where you want to deploy the BYOC warehouse.

Click VPC networks > CREATE VPC NETWORK to enter the VPC creation page.

Input VPC Name, Select Custom mode.

Input subnet Name, Select Region, Input IPv4 range.

Note: The regions and availability zones currently supported are：

Cloud Platform	Region Name	Region ID	Availability Zone ID
GCP	US West 1	us-west1	all
GCP	US East 4	us-east4	all

Select off for Private Google Access, Flow logs, Hybrid subnet.

Uncheck the Firewall rule list.

Select Regional for Dynamic Routing mode, Select Legacy for Best path selection mode.

Click CREATE to complete the creation of the VPC and subnet.

Learn about Resource Orchestration

Note: You don't need to do anything in this chapter. If you want to learn more about how it works, you can continue reading.

When you create a BYOC type warehouse, the cloud platform's resource orchestration service will first automatically deploy the Agent to complete the private connection between the Agent and the VeloDB Cloud platform, and then the Agent will be responsible for the deployment and management of the BYOC warehouse.

Terraform template description

The Terraform resource orchestration template provided by VeloDB runs under your GCP account, and the template code is visible and auditable, and will not operate on your data and other environments in the VPC. You can get the Terraform template provided by SelectDB through the following link:

https://storage.googleapis.com/selectdb-cloud-online-us-west-1/public/gcp-byoc.tf

When you run the above Terraform template through GCP CloudShell, it will automatically create and deploy the Agent. Then the Agent will establish a private connection with VeloDB Cloud and complete the warehouse initialization process.

After the resource orchestration script is executed, you can enter the corresponding warehouse from the VeloDB Cloud platform and start creating a computing cluster for data analysis just like using a normal warehouse.

Resources Information

Compute Engine Instance
- Name: VeloDBAgent
- Purpose: Used to deploy Agent, Prometheus, FluentBit and other programs
Private Service Connect Endpoint
- Name: VeloDBEndpoint
- Purpose: Establishes private network connection (PSC) with VeloDB Manage service for one-way control instruction pulling and monitoring/logging push
VPC Firewall Rules
- Name: VeloDBSecurityGroupIngress, VeloDBSecurityGroupEgress
- Purpose: Attached to GCP instances and restricts traffic through security group rules (allows inbound traffic on port 8666 from the same subnet, allows all outbound traffic on all ports)
Cloud Storage Bucket
- Name: VeloDBBucket
- Purpose: Used for storing warehouse data
IAM & Service Account & Custom Role
- Names:
  - VeloDBControlPanelRole (control panel service account)
  - VeloDBControlPanelRolePolicy (custom role policy)
  - VeloDBBucketKey (Bucket Key)
  - VeloDBDataAccessRole (kernel service account)
- Purposes:
  - Control panel service account is bound to Agent machines with minimum required permissions. All subsequent control operations use this service account identity
  - Kernel service account is bound to MS/FE/BE machines with full owner permissions for Bucket and its data. All subsequent data-related operations use this service account identity

Permissions that Terraform templates rely on

When executing the Terraform template through CloudShell under your cloud account, cloud resources such as Compute Engine, VPC, Cloud Storage will be created or related operations will be performed, so a series of permissions are required. Before formal execution, please ensure that the user who executes this template has the corresponding permissions, otherwise the template execution may fail.

Note The execution process of the resource stack template is completely carried out under your cloud account, and the created resources also belong to your cloud account. VeloDB will not obtain your cloud account information, nor can it use the corresponding IAM permissions of the account.

The following are the permissions required based on the resources and operations defined in the template:

Permission summary:

compute.addresses.create 
compute.addresses.createInternal 
compute.addresses.delete 
compute.addresses.deleteInternal 
compute.addresses.setLabels
compute.addresses.get 
compute.addresses.list 
compute.addresses.use 
compute.addresses.useInternal 
compute.disks.create 
compute.disks.createTagBinding 
compute.disks.delete 
compute.disks.deleteTagBinding 
compute.disks.get 
compute.disks.list 
compute.disks.resize 
compute.disks.setLabels 
compute.disks.use 
compute.firewalls.create 
compute.firewalls.createTagBinding 
compute.firewalls.delete 
compute.firewalls.deleteTagBinding 
compute.firewalls.get 
compute.firewalls.list 
compute.firewalls.update 
compute.forwardingRules.create 
compute.forwardingRules.delete 
compute.forwardingRules.get 
compute.forwardingRules.pscCreate 
compute.forwardingRules.pscDelete 
compute.globalOperations.get 
compute.instanceGroups.create 
compute.instanceGroups.delete 
compute.instanceGroups.update 
compute.instanceGroups.use 
compute.instances.attachDisk 
compute.instances.create 
compute.instances.createTagBinding 
compute.instances.delete 
compute.instances.deleteTagBinding 
compute.instances.detachDisk 
compute.instances.get 
compute.instances.getEffectiveFirewalls 
compute.instances.list 
compute.instances.listTagBindings 
compute.instances.osAdminLogin 
compute.instances.osLogin 
compute.instances.reset 
compute.instances.resume 
compute.instances.setDiskAutoDelete 
compute.instances.setLabels 
compute.instances.setMachineType 
compute.instances.setMetadata 
compute.instances.setName 
compute.instances.setServiceAccount 
compute.instances.setTags 
compute.instances.start 
compute.instances.stop 
compute.instances.suspend 
compute.instances.update 
compute.instances.use 
compute.networks.get 
compute.networks.list 
compute.networks.updatePolicy 
compute.networks.use 
compute.regionBackendServices.create 
compute.regionBackendServices.createTagBinding 
compute.regionBackendServices.delete 
compute.regionBackendServices.deleteTagBinding 
compute.regionBackendServices.get 
compute.regionBackendServices.getIamPolicy 
compute.regionBackendServices.list 
compute.regionBackendServices.update 
compute.regionBackendServices.use 
compute.regionHealthChecks.create 
compute.regionHealthChecks.delete 
compute.regionHealthChecks.useReadOnly 
compute.regionOperations.get 
compute.subnetworks.get 
compute.subnetworks.list 
compute.subnetworks.setPrivateIpGoogleAccess 
compute.subnetworks.use 
compute.zoneOperations.get 
compute.zones.get 
compute.zones.list 
iam.roles.create 
iam.roles.delete 
iam.roles.get 
iam.roles.update 
iam.serviceAccounts.actAs 
iam.serviceAccounts.create 
iam.serviceAccounts.createTagBinding 
iam.serviceAccounts.delete 
iam.serviceAccounts.deleteTagBinding 
iam.serviceAccounts.enable 
iam.serviceAccounts.get 
iam.serviceAccounts.getAccessToken 
iam.serviceAccounts.getIamPolicy 
iam.serviceAccounts.setIamPolicy 
iam.serviceAccounts.update 
resourcemanager.projects.createPolicyBinding 
resourcemanager.projects.deletePolicyBinding 
resourcemanager.projects.get 
resourcemanager.projects.getIamPolicy 
resourcemanager.projects.setIamPolicy 
servicedirectory.namespaces.create 
servicedirectory.namespaces.delete 
servicedirectory.namespaces.get 
servicedirectory.services.create 
servicedirectory.services.delete 
servicedirectory.services.get 
serviceusage.services.enable 
serviceusage.services.use 
storage.buckets.create 
storage.buckets.createTagBinding 
storage.buckets.delete 
storage.buckets.deleteTagBinding 
storage.buckets.get 
storage.buckets.getIamPolicy 
storage.buckets.setIamPolicy 
storage.hmacKeys.create 
storage.hmacKeys.delete 
storage.hmacKeys.get 
storage.hmacKeys.update
storage.objects.delete 
storage.objects.get 
storage.objects.list

The specific permissions are divided as follows:

Compute Engine permissions:

Manage VM instances

compute.disks.create 
compute.disks.createTagBinding 
compute.disks.delete 
compute.disks.deleteTagBinding 
compute.disks.get 
compute.disks.list 
compute.disks.resize 
compute.disks.setLabels 
compute.disks.use 
compute.globalOperations.get 
compute.instanceGroups.create 
compute.instanceGroups.delete 
compute.instanceGroups.update 
compute.instanceGroups.use 
compute.instances.attachDisk 
compute.instances.create 
compute.instances.createTagBinding 
compute.instances.delete 
compute.instances.deleteTagBinding 
compute.instances.detachDisk 
compute.instances.get 
compute.instances.getEffectiveFirewalls 
compute.instances.list 
compute.instances.listTagBindings 
compute.instances.osAdminLogin 
compute.instances.osLogin 
compute.instances.reset 
compute.instances.resume 
compute.instances.setDiskAutoDelete 
compute.instances.setLabels 
compute.instances.setMachineType 
compute.instances.setMetadata 
compute.instances.setName 
compute.instances.setServiceAccount 
compute.instances.setTags 
compute.instances.start 
compute.instances.stop 
compute.instances.suspend 
compute.instances.update 
compute.instances.use

Manage firewall rules

compute.firewalls.create 
compute.firewalls.createTagBinding 
compute.firewalls.delete 
compute.firewalls.deleteTagBinding 
compute.firewalls.get 
compute.firewalls.list 
compute.firewalls.update

VPC Network permissions:

Get VPC-related resource information

compute.networks.get 
compute.networks.list 
compute.networks.updatePolicy 
compute.networks.use 
compute.subnetworks.get 
compute.subnetworks.list 
compute.subnetworks.setPrivateIpGoogleAccess 
compute.subnetworks.use 
compute.zoneOperations.get 
compute.zones.get 
compute.zones.list

Manage LB resources

compute.addresses.create 
compute.addresses.createInternal 
compute.addresses.delete 
compute.addresses.deleteInternal 
compute.addresses.setLabels
compute.addresses.get 
compute.addresses.list 
compute.addresses.use 
compute.addresses.useInternal 
compute.forwardingRules.create 
compute.forwardingRules.delete 
compute.forwardingRules.get 
compute.forwardingRules.pscCreate 
compute.forwardingRules.pscDelete 
compute.regionBackendServices.create 
compute.regionBackendServices.createTagBinding 
compute.regionBackendServices.delete 
compute.regionBackendServices.deleteTagBinding 
compute.regionBackendServices.get 
compute.regionBackendServices.getIamPolicy 
compute.regionBackendServices.list 
compute.regionBackendServices.update 
compute.regionBackendServices.use 
compute.regionHealthChecks.create 
compute.regionHealthChecks.delete 
compute.regionHealthChecks.useReadOnly 
compute.regionOperations.get

Cloud Storage permissions:

Manage Buckets and read and write buckets and their contents

storage.buckets.create 
storage.buckets.createTagBinding 
storage.buckets.delete 
storage.buckets.deleteTagBinding 
storage.buckets.get 
storage.buckets.getIamPolicy 
storage.buckets.setIamPolicy 
storage.hmacKeys.create 
storage.hmacKeys.delete 
storage.hmacKeys.get 
storage.hmacKeys.update
storage.objects.delete 
storage.objects.get 
storage.objects.list

IAM & ResourceManager & Service Directory permissions:

Manage roles and service accounts

iam.roles.create 
iam.roles.delete 
iam.roles.get 
iam.roles.update 
iam.serviceAccounts.actAs 
iam.serviceAccounts.create 
iam.serviceAccounts.createTagBinding 
iam.serviceAccounts.delete 
iam.serviceAccounts.deleteTagBinding 
iam.serviceAccounts.enable 
iam.serviceAccounts.get 
iam.serviceAccounts.getAccessToken 
iam.serviceAccounts.getIamPolicy 
iam.serviceAccounts.setIamPolicy 
iam.serviceAccounts.update

Get project information, manage policy and binding, manage namespace resources

resourcemanager.projects.createPolicyBinding 
resourcemanager.projects.deletePolicyBinding 
resourcemanager.projects.get 
resourcemanager.projects.getIamPolicy 
resourcemanager.projects.setIamPolicy 
servicedirectory.namespaces.create 
servicedirectory.namespaces.delete 
servicedirectory.namespaces.get 
servicedirectory.services.create 
servicedirectory.services.delete 
servicedirectory.services.get 
serviceusage.services.enable 
serviceusage.services.use

Permissions of the created service account

After the template is executed for the first time to create the resource stack, all management and control operations are performed based on the permissions of the service account. The following is an excerpt from the template.