Import from S3

Use an S3 import job to load files from Amazon S3 into VeloDB Cloud.

Prerequisites

• A VeloDB Cloud warehouse.
• The bucket name, object path, file pattern, and file format.
• Credentials with read permission on the bucket or path. You can use Access Key authentication with AK/SK or a cross-account IAM role.
• A target database and target table, or permission to create them during the wizard.

Create the import job

1. Open Import from the ingestion group, then click + Add Import Job.
2. Select Amazon S3.
3. Enter the job name, bucket or path, region settings, and credentials. For credentials, choose Access Key and enter AK/SK, or choose IAM role and enter the cross-account role ARN. For setup details, see Use an IAM role.
4. Configure the file pattern and file format.
5. Select or create the target database and table.
6. Configure job settings such as sync interval and strict mode.
7. Run the pre-checks. When all checks succeed, start the job.

View job details

In the import job list, click an S3 import job to open its details page, where you can view information such as the data source, target files, and target table.

Use an IAM role

For production workloads, use a cross-account IAM role instead of long-lived access keys.

1. In the S3 import wizard, select IAM role as the authentication method and copy the VeloDB ARN shown in the authorization guidelines.

2. In your AWS account, create an IAM policy that grants read access to the S3 bucket or prefix used by the import job:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:GetObjectVersion"
               ],
               "Resource": "arn:aws:s3:::<bucket>/<prefix>/*"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetBucketLocation"
               ],
               "Resource": "arn:aws:s3:::<bucket>"
           }
       ]
   }

3. Create an IAM role with a custom trust policy that allows VeloDB Cloud to assume the role. Replace <velodb-arn> with the ARN from the S3 import wizard:

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": "<velodb-arn>"
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }

4. Attach the S3 read policy to the IAM role.

5. Copy the role ARN, then paste it into Role ARN in the S3 import wizard.

Troubleshooting

• If object listing fails, verify the bucket path, endpoint, region, and read permissions.
• If files are skipped, review the file pattern and path configuration.
• If rows fail to load, check file format, delimiters, encoding, strict-mode behavior, and target column types.