AWS Preparation

This article mainly introduces the AWS operations involved in creating a BYOC type warehouse, including creating an IAM user and authorizing it, creating a VPC and subnet, and understanding resource orchestration and resource stacks.

Prepare an IAM user and authorize it

Before creating a BYOC warehouse, you need to prepare an AWS IAM user with relevant permissions.

During the BYOC deployment, the page will redirect to AWS. You need to use this user to log in to the AWS management console and create a resource stack in CloudFormation.

Please send this document to your AWS administrator and ask the administrator to create an IAM user for you and authorize it according to this document.

The administrator accesses the AWS Identity and Access Management(IAM) (opens in a new tab) console and performs the following operations:

Create a policy

When creating a VeloDB Cloud BYOC type warehouse, you need to execute the resource stack template through the resource orchestration service (CloudFormation), which will create cloud resources such as EC2, VPC, S3, or perform related operations, so a series of IAM permissions are required.

Click Access Management > Policies on the left to enter the permission policy management page and click Create Policy

Switch to JSON mode, clear the original text box, copy the following script, and enter the text box. For detailed permission descriptions, see the Permission Description of Resource Stack Template Dependencies section below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:RunInstances",
        "ec2:DescribeInstances",
        "ec2:TerminateInstances",
        "ec2:StopInstances",
        "ec2:StartInstances",
        "ec2:RebootInstances",
        "ec2:ModifyInstanceAttribute",
        "ec2:DescribeVolumes",
        "ec2:ModifyVolume",
        "ec2:DescribeTags",
        "ec2:CreateTags",
        "ec2:DeleteTags",
        "ec2:DescribeSecurityGroups",
        "ec2:CreateSecurityGroup",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupEgress",
        "ec2:DeleteSecurityGroup",
        "ec2:DescribeSecurityGroupRules",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets"
      ],
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": [
        "s3:Get*",
        "s3:List*",
        "s3:Put*",
        "s3:Delete*",
        "s3:CreateBucket"
      ],
      "Resource": [
        "arn:aws:s3:::selectdb*"
      ],
      "Effect": "Allow"
    },
    {
      "Action": [
        "sts:GetCallerIdentity",
        "sts:AssumeRole",
        "iam:GetUser",
        "iam:TagUser",
        "iam:CreateUser",
        "iam:DeleteUser",
        "iam:ListAccessKeys",
        "iam:CreateAccessKey",
        "iam:DeleteAccessKey",
        "iam:GetRole",
        "iam:TagRole",
        "iam:ListRoles",
        "iam:CreateRole",
        "iam:DeleteRole",
        "iam:CreatePolicy",
        "iam:GetUserPolicy",
        "iam:PutUserPolicy",
        "iam:GetRolePolicy",
        "iam:PutRolePolicy",
        "iam:DeleteUserPolicy",
        "iam:DeleteRolePolicy",
        "iam:GetInstanceProfile",
        "iam:CreateInstanceProfile",
        "iam:AddRoleToInstanceProfile",
        "iam:RemoveRoleFromInstanceProfile",
        "iam:DeleteInstanceProfile"
      ],
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "*",
      "Effect": "Allow",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": [
            "ec2.amazonaws.com",
            "lambda.amazonaws.com"
          ]
        }
      }
    },
    {
      "Action": [
        "ec2:*VpcEndpoint*",
        "elasticloadbalancing:*",
        "cloudformation:*"
      ],
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": [
        "lambda:GetFunction",
        "lambda:CreateFunction",
        "lambda:DeleteFunction",
        "lambda:InvokeFunction",
        "lambda:TagResource"
      ],
      "Resource": "*",
      "Effect": "Allow"
    }
  ]
}

Click Next, enter a name, and click OK to complete the creation of the permission policy.

Create an IAM user and authorize it

Notice: If you already have an IAM user, you can skip the creation step and authorize it directly.

Click Access Management > User on the left, enter the User Management page, click Create User, enter relevant information, and click Next.

Select the policy created in the above steps, click Next, click Create User, and complete the creation.

Create an IAM user group and authorize (optional)

If there are multiple people in the enterprise using VeloDB Cloud, you can create an IAM user group, add relevant people to the user group, and authorize them uniformly.

Click Access Management > User Groups on the left to enter the User Group Management page, click Create group, enter User group name, select the User name and Policy name to be added, and click Create user group to complete the creation.

Prepare a VPC and subnet

Before creating a BYOC type warehouse, you need to use the above IAM user to create a VPC and subnet in advance. The following are the specific operations.

Notice: If you already have a VPC and subnet, you can skip the creation step

Open the Amazon Web Services VPC (opens in a new tab) console and switch to the region where you want to deploy the BYOC warehouse.

Create VPC

Click Create VPC to enter the VPC creation page.

Select VPC only, enter the name tag, input IPv4 CDR, click Create VPC to complete the creation.

Create subnet

Click Subnets > Create subnet on the left to enter the subnet creation page.

Select the VPC created in the above steps, enter the subnet name, IPv4 subnet CIDR block, click Create Subnet, and complete the creation.

Note: The regions and availability zones currently supported by the subnet are：

Cloud Platform	Region	Availability Zone ID
AWS	us-east-1	use1-az2
AWS	us-west-2	usw2-az1
AWS	eu-west-1	euw1-az1

Learn about Resource Orchestration and Resource Stack

When a user creates a BYOC type warehouse, the Agent will be automatically deployed with the help of the cloud vendor's resource orchestration service to complete the private connection between the Agent and the VeloDB Cloud platform, and then the Agent will be responsible for the deployment and management of the BYOC warehouse.

Resource Orchestration Template Description

The resource orchestration template provided by VeloDB runs under your cloud account, and the template code is visible and auditable, and will not operate on your data and other environments in the VPC. You can get the resource orchestration template provided by VeloDB through the following link:

https://selectdb-cloud-online.s3.us-west-1.amazonaws.com/public/aws-us-east-1-byoc-cf.yaml

When you execute the above resource template through AWS CloudFormation, it will automatically create and deploy the Agent. Then the Agent will establish a private connection with VeloDB Cloud and complete the warehouse initialization process.

After the resource orchestration script is executed, you can enter the corresponding warehouse from the VeloDB Cloud platform and start creating a computing cluster for data analysis just like using a normal warehouse.

How to view resource stack information

You can switch to the region through the Amazon Cloud Technology CloudFormation (opens in a new tab) console to view all resource information created by the SelectDB resource stack template, and view specific resources by resource name.

Note All resources created by the resource stack template belong to your cloud account and are only used within your VPC and will not be leaked.

Permissions description of resource stack template dependencies

When executing a resource stack template through the resource orchestration service (CloudFormation) under your cloud account, cloud resources such as EC2, VPC, S3, etc. will be created or related operations will be performed, so a series of IAM permissions are required. Before formal execution, please ensure that the user executing this template has the corresponding permissions, otherwise the template execution may fail.

Note The execution process of the resource stack template is completely carried out under your cloud account, and the created resources also belong to your cloud account. VeloDB will not obtain your cloud account information, nor can it use the corresponding IAM permissions of the account.

The following are the permissions required based on the resources and operations defined in the template:

Permission summary:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:RunInstances",
        "ec2:DescribeInstances",
        "ec2:TerminateInstances",
        "ec2:StopInstances",
        "ec2:StartInstances",
        "ec2:RebootInstances",
        "ec2:ModifyInstanceAttribute",
        "ec2:DescribeVolumes",
        "ec2:ModifyVolume",
        "ec2:DescribeTags",
        "ec2:CreateTags",
        "ec2:DeleteTags",
        "ec2:DescribeSecurityGroups",
        "ec2:CreateSecurityGroup",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:RevokeSecurityGroupIngress",
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:RevokeSecurityGroupEgress",
        "ec2:DeleteSecurityGroup",
        "ec2:DescribeSecurityGroupRules",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets"
      ],
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": [
        "s3:Get*",
        "s3:List*",
        "s3:Put*",
        "s3:Delete*",
        "s3:CreateBucket"
      ],
      "Resource": [
        "arn:aws:s3:::selectdb*"
      ],
      "Effect": "Allow"
    },
    {
      "Action": [
        "sts:GetCallerIdentity",
        "sts:AssumeRole",
        "iam:GetUser",
        "iam:TagUser",
        "iam:CreateUser",
        "iam:DeleteUser",
        "iam:ListAccessKeys",
        "iam:CreateAccessKey",
        "iam:DeleteAccessKey",
        "iam:GetRole",
        "iam:TagRole",
        "iam:ListRoles",
        "iam:CreateRole",
        "iam:DeleteRole",
        "iam:CreatePolicy",
        "iam:GetUserPolicy",
        "iam:PutUserPolicy",
        "iam:GetRolePolicy",
        "iam:PutRolePolicy",
        "iam:DeleteUserPolicy",
        "iam:DeleteRolePolicy",
        "iam:GetInstanceProfile",
        "iam:CreateInstanceProfile",
        "iam:AddRoleToInstanceProfile",
        "iam:RemoveRoleFromInstanceProfile",
        "iam:DeleteInstanceProfile"
      ],
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "*",
      "Effect": "Allow",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": [
            "ec2.amazonaws.com",
            "lambda.amazonaws.com"
          ]
        }
      }
    },
    {
      "Action": [
        "ec2:*VpcEndpoint*",
        "elasticloadbalancing:*",
        "cloudformation:*"
      ],
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": [
        "lambda:GetFunction",
        "lambda:CreateFunction",
        "lambda:DeleteFunction",
        "lambda:InvokeFunction",
        "lambda:TagResource"
      ],
      "Resource": "*",
      "Effect": "Allow"
    }
  ]
}

EC2 and VPC permissions:

Manage EC2 instances

"ec2:RunInstances",
"ec2:DescribeInstances",
"ec2:TerminateInstances",
"ec2:StopInstances",
"ec2:StartInstances",
"ec2:RebootInstances",
"ec2:ModifyInstanceAttribute",
"ec2:DescribeVolumes",
"ec2:ModifyVolume",
"ec2:DescribeTags",
"ec2:CreateTags",
"ec2:DeleteTags",

Manage VPC

"ec2:DescribeSecurityGroups",
"ec2:CreateSecurityGroup",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupIngress",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:RevokeSecurityGroupEgress",
"ec2:DeleteSecurityGroup",
"ec2:DescribeSecurityGroupRules",
"ec2:DescribeVpcs",
"ec2:DescribeSubnets"

Manage EndPoint
```
ec2:*VpcEndpoint*
```

ELB permissions:
- Manage ELB resources
```
elasticloadbalancing:*
```

S3 permissions:

Manage S3 buckets and read and write buckets and their contents (for specific buckets)

 "Action": [
        "s3:Get*",
        "s3:List*",
        "s3:Put*",
        "s3:Delete*",
        "s3:CreateBucket"
],
"Resource": [
        "arn:aws:s3:::selectdb*"
],
"Effect": "Allow"

IAM & STS permissions:

"iam:GetUser",
"iam:TagUser",
"iam:CreateUser",
"iam:DeleteUser",
"iam:ListAccessKeys",
"iam:CreateAccessKey",
"iam:DeleteAccessKey",
"iam:GetRole",
"iam:TagRole",
"iam:ListRoles",
"iam:CreateRole",
"iam:DeleteRole",
"iam:CreatePolicy",
"iam:GetUserPolicy",
"iam:PutUserPolicy",
"iam:GetRolePolicy",
"iam:PutRolePolicy",
"iam:DeleteUserPolicy",
"iam:DeleteRolePolicy",
"iam:GetInstanceProfile",
"iam:CreateInstanceProfile",
"iam:AddRoleToInstanceProfile",
"iam:RemoveRoleFromInstanceProfile",
"iam:DeleteInstanceProfile"

"sts:GetCallerIdentity",
"sts:AssumeRole",

Permissions of sub-users created by resource stack templates

After the resource stack template is executed for the first time, a sub-user will be created for subsequent management of data warehouse related components in your VPC. The following is a description of the permissions of the sub-user.

Note The created sub-user belongs to your cloud account and is only used in your VPC and will not be leaked.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "ec2:RunInstances",
        "ec2:DescribeInstances",
        "ec2:TerminateInstances",
        "ec2:StopInstances",
        "ec2:StartInstances",
        "ec2:RebootInstances",
        "ec2:ModifyInstanceAttribute",
        "ec2:DescribeVolumes",
        "ec2:ModifyVolume",
        "ec2:DescribeTags",
        "ec2:CreateTags",
        "ec2:DeleteTags",
        "ec2:CreateSecurityGroup",
        "ec2:DescribeSecurityGroups",
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:AuthorizeSecurityGroupEgress",
        "ec2:DeleteSecurityGroup",
        "ec2:DescribeSecurityGroupRules",
        "ec2:DescribeVpcs",
        "ec2:DescribeSubnets",
        "ec2:*VpcEndpoint*"
      ],
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": [
        "s3:Get*",
        "s3:List*",
        "s3:Put*",
        "s3:Delete*"
      ],
      "Resource": [
        "arn:aws:s3:::selectdb-cloud-online-09314684accbb0d91/*",
        "arn:aws:s3:::selectdb-cloud-online-09314684accbb0d91",
        "arn:aws:s3:::selectdb-import-data-us-east-1/*",
        "arn:aws:s3:::selectdb-import-data-us-east-1"
      ],
      "Effect": "Allow"
    },
    {
      "Action": [
        "sts:GetCallerIdentity",
        "sts:AssumeRole",
        "iam:CreateInstanceProfile",
        "iam:AddRoleToInstanceProfile"
      ],
      "Resource": "*",
      "Effect": "Allow"
    },
    {
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "*",
      "Effect": "Allow",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": [
            "ec2.amazonaws.com",
            "lambda.amazonaws.com"
          ]
        }
      }
    },
    {
      "Action": [
        "elasticloadbalancing:*"
      ],
      "Resource": "*",
      "Effect": "Allow"
    }
  ]
}

AWS GCP