Amazon MSK

Connect VeloDB Cloud to an Amazon MSK cluster and ingest data from Kafka topics over a private multi-VPC connection.

note

This guide describes the SaaS setup, where VeloDB Cloud connects to your MSK cluster over a managed multi-VPC private connection that you set up through the import UI in Step 5.

For BYOC, VeloDB Cloud runs in your own VPC, so you set up network access to the MSK cluster yourself, using either multi-VPC private connectivity or VPC peering (see Access from within AWS but outside the cluster's VPC). Then create the ingestion job with a Routine Load SQL statement (see CREATE ROUTINE LOAD). The import UI for BYOC is coming soon.

Prerequisites

Before you begin, ensure you have:

• An AWS account with permissions to create and configure MSK clusters, Secrets Manager secrets, and IAM policies
• A VeloDB Cloud warehouse (SaaS or BYOC) in the same AWS region as the MSK cluster
• MSK version 3.x or earlier (version 4.x and above is not supported)

---

Step 1: Prepare an MSK cluster

note

Most customers already have an MSK cluster. If you do, skip creating a new one. Confirm the cluster has the settings below (ACL policy disabled, SASL/SCRAM authentication enabled, and a Secrets Manager secret associated), apply any that are missing, then enable multi-VPC connectivity in Step 2. For enabling multi-VPC connectivity on an existing cluster, follow Get started using multi-VPC private connectivity.

If you are creating a new cluster, creation takes approximately 40 minutes. Configure the following settings during creation to avoid time-consuming updates later.

Disable ACL policy

To enable SASL/SCRAM cross-VPC connectivity, you must disable the default ACL behavior before creating the cluster. In the MSK console, go to Cluster configurations and create a new configuration with the following property:

allow.everyone.if.no.acl.found=false

Apply this configuration when setting up the cluster properties.

Enable authentication

During cluster creation, under Security settings, enable both:

• SASL/SCRAM authentication: required for cross-VPC access with username/password credentials
• IAM role-based authentication: required for granting ACL permissions via kafka-cli

Associate an AWS Secrets Manager secret

MSK uses Secrets Manager to store SASL/SCRAM credentials. Associate a secret with the cluster during or after creation. For the full AWS requirements, see Sign-in credentials authentication with AWS Secrets Manager.

Requirements:

• The secret name must start with AmazonMSK_
• The Encryption key must not be aws/secretsmanager. Use a customer-managed KMS key or the default AWS managed key for another service. Using aws/secretsmanager causes the MSK association to fail.

If you don't have an existing secret, create one in AWS Secrets Manager with the name prefixed AmazonMSK_, then associate it with the cluster.

When creating a new secret, choose Other type of secret and enter the username and password as key/value pairs:

Update the security group (inbound)

In the MSK cluster's security group, add inbound rules to allow traffic on ports 9096-9098 (the SASL/SCRAM and IAM broker ports). Port 9096 is the SASL/SCRAM broker port, and port 9098 is the IAM broker port used for the ACL grant in the next step. Set the source to the MSK cluster's own security group, and run the ACL grant from a machine that uses this security group in the same VPC as the cluster. For the full MSK port mapping, see Port information.

note

A second security group update is required in Step 4 to open ports 14098-14100 for VeloDB Cloud's multi-VPC private connection.

Grant ACL authorization for the associated user

Run the following commands from a machine inside the same VPC as the MSK cluster. You need the Kafka CLI, which requires the Java JDK.

1. Configure AWS credentials. Run the following to verify credentials are set up:

   aws sts get-caller-identity

   If credentials are not configured, add an IAM role to the EC2 instance or run aws configure.

2. Add client-msk-iam.properties to the Kafka CLI config directory:

   security.protocol=SASL_SSL
   sasl.mechanism=AWS_MSK_IAM
   sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
   sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

3. Download the IAM auth library and place it in the Kafka CLI libs directory:

   aws-msk-iam-auth-1.1.9-all.jar

4. Get the IAM Private endpoint. In the MSK console, click View client information and copy the IAM bootstrap server endpoint.

   

5. Grant ACL permissions to the SASL/SCRAM username. Replace the placeholders with your MSK cluster's IAM Private endpoint and the associated username:

   ./bin/kafka-acls.sh --bootstrap-server <MSK_IAM_PRIVATE_ENDPOINT> \
     --command-config ./config/client-msk-iam.properties \
     --add --allow-principal User:<USERNAME> \
     --operation All --topic '*' --group '*' --cluster

   

---

Step 2: Enable multi-VPC connectivity

Enabling multi-VPC connectivity takes approximately 40 minutes.

In the MSK cluster's Networking settings, enable Multi-VPC connectivity and select SASL/SCRAM authentication as the authentication type. For background, see Amazon MSK multi-VPC private connectivity in a single Region.

note

If the SASL/SCRAM option is not available, the cluster ACL policy has not been disabled. Update the cluster configuration (allow.everyone.if.no.acl.found=false) and try again.

---

Step 3: Grant VeloDB Cloud account access

In the MSK console, go to Security and click Edit cluster policy. Attach the following resource-based policy to allow VeloDB Cloud to create a VPC connection and retrieve broker information. Replace the Resource ARN with your cluster's ARN. For the cluster policy requirements, see Permissions for multi-VPC private connectivity.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::757278738533:root"
      },
      "Action": [
        "kafka:CreateVpcConnection",
        "kafka:GetBootstrapBrokers",
        "kafka:DescribeCluster",
        "kafka:DescribeClusterV2"
      ],
      "Resource": "arn:aws:kafka:<REGION>:<ACCOUNT_ID>:cluster/<CLUSTER_NAME>/<CLUSTER_ID>"
    }
  ]
}

---

Step 4: Update the security group for VeloDB access

Add inbound rules to the MSK cluster's security group to allow traffic on ports 14098-14100, using the MSK cluster's own security group as the source. VeloDB Cloud connects over the multi-VPC private endpoint from a different AWS account. Because this access is cross-account, the multi-VPC network interfaces run in your VPC under the cluster's security group, so a self-referencing rule allows the traffic. You can't reference a security group or CIDR from VeloDB's account. These ports are distinct from the same-VPC broker ports opened in Step 1.

---

Step 5: Connect VeloDB Cloud to MSK

In the AWS Console:

1. Navigate to your MSK cluster and copy the cluster ARN.

   

2. Go to View client information and copy the Private endpoint (multi-VPC) bootstrap servers for SASL/SCRAM. You'll need these in the next step.

   

In VeloDB Cloud:

3. Go to Import and click Add Import Job. Enable Reverse Private Endpoint and click Set up Connection.

   

4. In the Set up Connection panel, select SASL/SCRAM as the authentication type and enter the cluster ARN.

   

5. Fill in the import job details:

   • Brokers: paste the multi-VPC bootstrap servers copied in step 2
   • Authentication: select SASL, enter the SASL/SCRAM username and password, and set the SASL Mechanism to SCRAM-SHA-512

   

6. Click Next to complete the connection setup. VeloDB Cloud establishes a private endpoint (multi-VPC) to the MSK cluster.

---

Step 6: Access the MSK cluster

Once the connection is established, the MSK cluster appears in the Reverse Private Endpoint table with Available connectivity status.

Configure import jobs to consume data from Kafka topics in your MSK cluster via the Private endpoint (multi-VPC).