Azure Preparation

This article mainly introduces the Azure operations involved in creating a BYOC type warehouse, including creating an IAM user and authorizing it, creating a VPC and subnet, and understanding resource orchestration and resource stacks.

Prepare an IAM user and authorize it

Before creating a BYOC warehouse, you need to prepare an Azure IAM user with relevant permissions.

During the BYOC deployment, the page will redirect to Azure. You need to use this user to log in to the Azure management console and create a resource stack in Resource Manager.

If you already have an Azure user with the Administrator role, you can skip creating a custom role and an IAM user.

Otherwise, please send this document to your Azure administrator and ask the administrator to create an IAM user for you and authorize it according to this document.

The administrator accesses the Azure console (opens in a new tab) and performs the following operations:

Create an IAM user

Search and enter the service Microsoft Entra ID.

Click + Add > User > Create New User.

Fill in relevant information. Click Review + create.

Click Create

Create a custom role

Search and enter the service Resource groups.

Click to enter the resource group where you want to create a custom role. If you don't have a resource group yet, you need to create one first.

Click Access control (IAM) on the left. Click + Add > Add custom role.

Input Custom role name. Select Start from JSON for Baseline permissions.

Copy the following text and save it as a json file. Select this file.

 {
        "roleName": "VeloDB_Cloud_Role",
        "description": "",
        "permissions": [
                    "Microsoft.Compute/register/action",
                    "Microsoft.Compute/virtualMachines/powerOff/action",
                    "Microsoft.Compute/virtualMachines/redeploy/action",
                    "Microsoft.Compute/virtualMachines/runCommand/action",
                    "Microsoft.Compute/locations/diskOperations/read",
                    "Microsoft.Compute/disks/delete",
                    "Microsoft.Compute/locations/publishers/artifacttypes/offers/skus/read",
                    "Microsoft.Compute/locations/operations/read",
                    "Microsoft.Compute/locations/vmSizes/read",
                    "Microsoft.Network/operations/read",
                    "Microsoft.Network/loadBalancers/read",
                    "Microsoft.Network/loadBalancers/health/action",
                    "Microsoft.Network/loadBalancers/backendAddressPools/queryInboundNatRulePortMapping/action",
                    "Microsoft.Network/loadBalancers/backendAddressPools/updateAdminState/action",
                    "Microsoft.Network/loadBalancers/backendAddressPools/health/action",
                    "Microsoft.Network/loadBalancers/backendAddressPools/join/action",
                    "Microsoft.Network/loadBalancers/providers/Microsoft.Insights/diagnosticSettings/read",
                    "Microsoft.Network/loadBalancers/frontendIPConfigurations/join/action",
                    "Microsoft.Network/loadBalancers/outboundRules/read",
                    "Microsoft.Network/loadBalancers/virtualMachines/read",
                    "Microsoft.Network/networkInterfaces/write",
                    "Microsoft.Network/networkInterfaces/join/action",
                    "Microsoft.Network/networkInterfaces/delete",
                    "Microsoft.Network/networkInterfaces/effectiveRouteTable/action",
                    "Microsoft.Network/networkInterfaces/effectiveNetworkSecurityGroups/action",
                    "Microsoft.Network/networkInterfaces/ipconfigurations/read",
                    "Microsoft.Network/networkInterfaces/loadBalancers/read",
                    "Microsoft.Network/networkSecurityGroups/read",
                    "Microsoft.Network/networkSecurityGroups/write",
                    "Microsoft.Network/networkSecurityGroups/delete",
                    "Microsoft.Network/networkSecurityGroups/join/action",
                    "Microsoft.Network/virtualNetworks/read",
                    "Microsoft.Network/virtualNetworks/joinLoadBalancer/action",
                    "Microsoft.Network/virtualNetworks/subnets/read",
                    "Microsoft.Network/virtualNetworks/subnets/join/action",
                    "Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action",
                    "Microsoft.Network/ipAllocations/read",
                    "Microsoft.Network/publicIPAddresses/read",
                    "Microsoft.Network/privateEndpoints/read",
                    "Microsoft.Network/privateEndpoints/write",
                    "Microsoft.Network/privateEndpoints/delete",
                    "Microsoft.Storage/storageAccounts/listkeys/action",
                    "Microsoft.Storage/storageAccounts/delete",
                    "Microsoft.Storage/storageAccounts/read",
                    "Microsoft.Storage/storageAccounts/write",
                    "Microsoft.Authorization/roleDefinitions/read",
                    "Microsoft.Authorization/roleDefinitions/write",
                    "Microsoft.Authorization/roleDefinitions/delete",
                    "Microsoft.Authorization/roleAssignments/write",
                    "Microsoft.Authorization/roleAssignments/delete",
                    "Microsoft.Authorization/roleAssignments/read",
                    "Microsoft.Resources/deploymentStacks/read",
                    "Microsoft.Resources/deploymentStacks/write",
                    "Microsoft.Resources/deploymentStacks/delete",
                    "Microsoft.Resources/deploymentStacks/manageDenySetting/action",
                    "Microsoft.Resources/deployments/read",
                    "Microsoft.Resources/deployments/write",
                    "Microsoft.Resources/deployments/delete",
                    "Microsoft.Resources/deployments/cancel/action",
                    "Microsoft.Resources/deployments/validate/action",
                    "Microsoft.Resources/deployments/whatIf/action",
                    "Microsoft.Resources/deployments/exportTemplate/action",
                    "Microsoft.Resources/deployments/operations/read",
                    "Microsoft.Resources/deployments/operationstatuses/read",
                    "Microsoft.Resources/subscriptions/resourceGroups/read"
                ],
                "notActions": [],
                "dataActions": [],
                "notDataActions": []
            }
        ]
    }
}

Note: For detailed permission description, please see below.

Click Next. Click Add assignable scopes.

Select Resource Group and Subscription. Click Select.

Click Review + create > Review + create > Create.

Add role assignment

Click to enter the resource group where you want to add role assignment.

Click Access control (IAM) on the left. Click + Add > Add role assignment.

Search and select the custom role you created in previous step. Click Next.

Select User, group, or service principal. Click + Select members and select the role. Click Next.

Click Review + assign.

Prepare a VNet and subnet

Before creating a BYOC type warehouse, you need to use the above IAM user to create a VNet and subnet in advance. The following are the specific operations.

Notice:

If you already have a VNet and subnet, you can skip the creation step.

The subnet must have a NAT gateway.

Open the Azure Portal and Select Virtual networks service.

Create VNet and subnet

Click Create.

Select Subscription and Resource Group. Input Virtual network name. Select Region. Click Next.

Note: Currently only supports (US) West US 3 region.

Click + Add a subnet. Click Edit.

Input Name. Select NAT gateway. Click Save. Click Next.

Note: If you don't have a NAT gateway available, you need to create one first.

Click Create.

Learn about Resource Orchestration and Resource Stack

Note: You don’t need to do anything in this chapter. If you want to learn more about how it works, you can continue reading.

When a user creates a BYOC type warehouse, the Agent will be automatically deployed with the help of the cloud platform's resource orchestration service to complete the private connection between the Agent and the VeloDB Cloud platform, and then the Agent will be responsible for the deployment and management of the BYOC warehouse.

Resource Orchestration Template Description

The resource orchestration template provided by VeloDB runs under your cloud account, and the template code is visible and auditable, and will not operate on your data and other environments in the VPC. You can get the resource orchestration template provided by VeloDB through the following link:

https://velodbwestus3.blob.core.windows.net/velodb-cloud-online-westus3/public/azure-westus3-byoc-cf.json

When you execute the above resource template through Azure Deployment Stacks, it will automatically create and deploy the Agent. Then the Agent will establish a private connection with VeloDB Cloud and complete the warehouse initialization process.

After the resource orchestration script is executed, you can enter the corresponding warehouse from the VeloDB Cloud platform and start creating a computing cluster for data analysis just like using a normal warehouse.

How to view resource stack information

You can view all resource information created by the VeloDB resource stack template, and view specific resources by resource name through the Azure Deployment Stacks.

Note All resources created by the resource stack template belong to your cloud account and are only used within your VPC and will not be leaked.

Virtual Machine
- VeloDBAgent: Used to deploy Agent, Prometheus, FluentBit and other programs
Private Endpoint
- VeloDBEndpoint: Establish a private network connection (PrivateLink Connection) with the VeloDB Manage service, so that you can pull control instructions and push monitoring and logs in one direction
Network SecurityGroup
- VeloDBSecurityGroupIngress, VeloDBSecurityGroupEgress: Bind to the Agent virtual machine and the end node, and limit the traffic of a specific port and a specific subnet to pass through the security group rules (allowing traffic from the 8666 port of the same subnet to enter the network and allowing traffic from all ports to leave the network)
Storage Account
- VeloDBBucket: Used to store data warehouse data
Role Definition
- VeloDBRole (Role) VeloDBBucketKey (Bucket Key): The created custom role has the minimum permissions required by the Agent, limited to the resource group, and all subsequent business operations are performed using the identity of the custom role; and the Storage Blob Data Owner role is granted to the Agent machine, and the resource is limited to the VeloDBBucket created by the resource stack;

Permissions description of resource stack template dependencies

When executing a resource stack template through the resource orchestration service (CloudFormation) under your cloud account, cloud resources such as EC2, VPC, S3, etc. will be created or related operations will be performed, so a series of IAM permissions are required. Before formal execution, please ensure that the user executing this template has the corresponding permissions, otherwise the template execution may fail.

Note The execution process of the resource stack template is completely carried out under your cloud account, and the created resources also belong to your cloud account. VeloDB will not obtain your cloud account information, nor can it use the corresponding IAM permissions of the account.

The following are the permissions required based on the resources and operations defined in the template:

Permission summary:

 {
        "roleName": "VeloDB_Cloud_Role",
        "description": "",
        "permissions": [
                    "Microsoft.Compute/register/action",
                    "Microsoft.Compute/virtualMachines/powerOff/action",
                    "Microsoft.Compute/virtualMachines/redeploy/action",
                    "Microsoft.Compute/virtualMachines/runCommand/action",
                    "Microsoft.Compute/locations/diskOperations/read",
                    "Microsoft.Compute/disks/delete",
                    "Microsoft.Compute/locations/publishers/artifacttypes/offers/skus/read",
                    "Microsoft.Compute/locations/operations/read",
                    "Microsoft.Compute/locations/vmSizes/read",
                    "Microsoft.Network/operations/read",
                    "Microsoft.Network/loadBalancers/read",
                    "Microsoft.Network/loadBalancers/health/action",
                    "Microsoft.Network/loadBalancers/backendAddressPools/queryInboundNatRulePortMapping/action",
                    "Microsoft.Network/loadBalancers/backendAddressPools/updateAdminState/action",
                    "Microsoft.Network/loadBalancers/backendAddressPools/health/action",
                    "Microsoft.Network/loadBalancers/backendAddressPools/join/action",
                    "Microsoft.Network/loadBalancers/providers/Microsoft.Insights/diagnosticSettings/read",
                    "Microsoft.Network/loadBalancers/frontendIPConfigurations/join/action",
                    "Microsoft.Network/loadBalancers/outboundRules/read",
                    "Microsoft.Network/loadBalancers/virtualMachines/read",
                    "Microsoft.Network/networkInterfaces/write",
                    "Microsoft.Network/networkInterfaces/join/action",
                    "Microsoft.Network/networkInterfaces/delete",
                    "Microsoft.Network/networkInterfaces/effectiveRouteTable/action",
                    "Microsoft.Network/networkInterfaces/effectiveNetworkSecurityGroups/action",
                    "Microsoft.Network/networkInterfaces/ipconfigurations/read",
                    "Microsoft.Network/networkInterfaces/loadBalancers/read",
                    "Microsoft.Network/networkSecurityGroups/read",
                    "Microsoft.Network/networkSecurityGroups/write",
                    "Microsoft.Network/networkSecurityGroups/delete",
                    "Microsoft.Network/networkSecurityGroups/join/action",
                    "Microsoft.Network/virtualNetworks/read",
                    "Microsoft.Network/virtualNetworks/joinLoadBalancer/action",
                    "Microsoft.Network/virtualNetworks/subnets/read",
                    "Microsoft.Network/virtualNetworks/subnets/join/action",
                    "Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action",
                    "Microsoft.Network/ipAllocations/read",
                    "Microsoft.Network/publicIPAddresses/read",
                    "Microsoft.Network/privateEndpoints/read",
                    "Microsoft.Network/privateEndpoints/write",
                    "Microsoft.Network/privateEndpoints/delete",
                    "Microsoft.Storage/storageAccounts/listkeys/action",
                    "Microsoft.Storage/storageAccounts/delete",
                    "Microsoft.Storage/storageAccounts/read",
                    "Microsoft.Storage/storageAccounts/write",
                    "Microsoft.Authorization/roleDefinitions/read",
                    "Microsoft.Authorization/roleDefinitions/write",
                    "Microsoft.Authorization/roleDefinitions/delete",
                    "Microsoft.Authorization/roleAssignments/write",
                    "Microsoft.Authorization/roleAssignments/delete",
                    "Microsoft.Authorization/roleAssignments/read",
                    "Microsoft.Resources/deploymentStacks/read",
                    "Microsoft.Resources/deploymentStacks/write",
                    "Microsoft.Resources/deploymentStacks/delete",
                    "Microsoft.Resources/deploymentStacks/manageDenySetting/action",
                    "Microsoft.Resources/deployments/read",
                    "Microsoft.Resources/deployments/write",
                    "Microsoft.Resources/deployments/delete",
                    "Microsoft.Resources/deployments/cancel/action",
                    "Microsoft.Resources/deployments/validate/action",
                    "Microsoft.Resources/deployments/whatIf/action",
                    "Microsoft.Resources/deployments/exportTemplate/action",
                    "Microsoft.Resources/deployments/operations/read",
                    "Microsoft.Resources/deployments/operationstatuses/read",
                    "Microsoft.Resources/subscriptions/resourceGroups/read"
                ],
                "notActions": [],
                "dataActions": [],
                "notDataActions": []
            }
        ]
    }
}

Virtual Machine permissions:

Manage vm instances

'Microsoft.Compute/register/action'
'Microsoft.Compute/virtualMachines/read'
'Microsoft.Compute/virtualMachines/write'
'Microsoft.Compute/virtualMachines/delete'
'Microsoft.Compute/virtualMachines/start/action'
'Microsoft.Compute/virtualMachines/powerOff/action'
'Microsoft.Compute/virtualMachines/redeploy/action'
'Microsoft.Compute/virtualMachines/restart/action'
'Microsoft.Compute/virtualMachines/deallocate/action'
'Microsoft.Compute/virtualMachines/runCommand/action'
'Microsoft.Compute/virtualMachines/attachDetachDataDisks/action'
'Microsoft.Compute/virtualMachines/vmSizes/read'
'Microsoft.Compute/virtualMachines/runCommands/read'
'Microsoft.Compute/virtualMachines/runCommands/write'
'Microsoft.Compute/virtualMachines/instanceView/read'
'Microsoft.Compute/virtualMachines/extensions/read'
'Microsoft.Compute/locations/diskOperations/read'
'Microsoft.Compute/disks/read'
'Microsoft.Compute/disks/write'
'Microsoft.Compute/disks/delete'
'Microsoft.Compute/skus/read'
'Microsoft.Compute/locations/usages/read'
'Microsoft.Compute/locations/publishers/artifacttypes/offers/skus/read'
'Microsoft.Compute/locations/publishers/artifacttypes/offers/read'
'Microsoft.Compute/locations/publishers/artifacttypes/offers/skus/versions/read'
'Microsoft.Compute/operations/read'
'Microsoft.Compute/images/read'
'Microsoft.Compute/locations/operations/read'
'Microsoft.Compute/locations/vmSizes/read'
'Microsoft.Compute/locations/runCommands/read'

Manage firewall rules

compute.firewalls.create
compute.firewalls.createTagBinding
compute.firewalls.delete
compute.firewalls.deleteTagBinding
compute.firewalls.get
compute.firewalls.list
compute.firewalls.update

Virtual networks permissions:

Get VPC-related resource information

  'Microsoft.Network/register/action'
  'Microsoft.Network/operations/read'
  'Microsoft.Network/virtualNetworks/read'
  'Microsoft.Network/virtualNetworks/joinLoadBalancer/action'
  'Microsoft.Network/virtualNetworks/subnets/read'
  'Microsoft.Network/virtualNetworks/subnets/join/action'
  'Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action'
  'Microsoft.Network/ipAllocations/read'
  'Microsoft.Network/publicIPAddresses/read'

Manage network card and security group resources

  'Microsoft.Network/register/action'
  'Microsoft.Network/operations/read'
  'Microsoft.Network/virtualNetworks/read'
  'Microsoft.Network/virtualNetworks/joinLoadBalancer/action'
  'Microsoft.Network/virtualNetworks/subnets/read'
  'Microsoft.Network/virtualNetworks/subnets/join/action'
  'Microsoft.Network/virtualNetworks/subnets/joinLoadBalancer/action'
  'Microsoft.Network/ipAllocations/read'
  'Microsoft.Network/publicIPAddresses/read'

Managing LB Resources

    'Microsoft.Network/loadBalancers/read'
    'Microsoft.Network/loadBalancers/write'
    'Microsoft.Network/loadBalancers/delete'
    'Microsoft.Network/loadBalancers/health/action'
    'Microsoft.Network/loadBalancers/backendAddressPools/queryInboundNatRulePortMapping/action'
    'Microsoft.Network/loadBalancers/backendAddressPools/updateAdminState/action'
    'Microsoft.Network/loadBalancers/backendAddressPools/health/action'
    'Microsoft.Network/loadBalancers/backendAddressPools/read'
    'Microsoft.Network/loadBalancers/backendAddressPools/write'
    'Microsoft.Network/loadBalancers/backendAddressPools/delete'
    'Microsoft.Network/loadBalancers/backendAddressPools/backendPoolAddresses/read'
    'Microsoft.Network/loadBalancers/backendAddressPools/join/action'
    'Microsoft.Network/loadBalancers/providers/Microsoft.Insights/diagnosticSettings/read'
    'Microsoft.Network/loadBalancers/providers/Microsoft.Insights/diagnosticSettings/write'
    'Microsoft.Network/loadBalancers/frontendIPConfigurations/read'
    'Microsoft.Network/loadBalancers/frontendIPConfigurations/join/action'
    'Microsoft.Network/loadBalancers/frontendIPConfigurations/loadBalancerPools/read'
    'Microsoft.Network/loadBalancers/frontendIPConfigurations/loadBalancerPools/write'
    'Microsoft.Network/loadBalancers/frontendIPConfigurations/loadBalancerPools/delete'
    'Microsoft.Network/loadBalancers/loadBalancingRules/read'
    'Microsoft.Network/loadBalancers/loadBalancingRules/health/action'
    'Microsoft.Network/loadBalancers/networkInterfaces/read'
    'Microsoft.Network/loadBalancers/outboundRules/read'
    'Microsoft.Network/loadBalancers/virtualMachines/read'
    'Microsoft.Network/loadBalancers/probes/read'
    'Microsoft.Network/loadBalancers/probes/join/action'
    'Microsoft.Network/loadBalancers/providers/Microsoft.Insights/metricDefinitions/read'

Storage Account Permissions:
- Manage storage buckets and read and write storage buckets and their contents (resources are limited to newly created storage accounts)
```
Storage Blob Data Owner
```

Azure Use Warehouse