Skip to main content

Import from Object Storage

Load files from S3-compatible object storage services into VeloDB Cloud with an object storage import job. The service must expose an S3-compatible API that the warehouse can reach.

Note:

Import from object storage is currently in Public Preview.

Prerequisites​

  • A VeloDB Cloud warehouse.
  • The storage service's endpoint and region settings, bucket name, object path, file pattern, and file format.
  • Credentials with read permission on the bucket or path. Use Access Key authentication with an access key and secret key (AK/SK). For storage in AWS, you can also use a cross-account IAM role.
  • A target database and target table, or permission to create them during the wizard.

Create the import job​

  1. Log in to the VeloDB Cloud console.
  2. In the upper-left corner, select the warehouse that you want to use.
  3. In the left navigation pane, click Data Integration. Click Add Import Job on the initial page. If integration records already exist, in the upper-right corner, choose Add Data > Import data into VeloDB instead.
  4. Select Object Storage S3, then click Continue.
  5. Enter the job name, bucket or path, and the endpoint and region settings for your storage service. For credentials, choose Access Key and enter AK/SK. For storage in AWS, you can instead choose IAM role and enter the cross-account role ARN, as described in Use an IAM role.
  6. Configure the file pattern and file format.
  7. Select or create the target database and table.
  8. Configure job settings such as sync interval and strict mode.
  9. Run the pre-checks. When all checks succeed, start the job.

View job details​

On Data Integration, select Import Jobs. In the job list, click an object storage import job to open its details page, where you can view information such as the data source, target files, and target table.

Use an IAM role​

This authentication method applies to storage in AWS. For production workloads in AWS, use a cross-account IAM role instead of long-lived access keys. For other S3-compatible object storage services, use the access credentials provided by the storage service.

  1. In the S3 import wizard, select IAM role as the authentication method and copy the VeloDB ARN shown in the authorization guidelines.

  2. In your AWS account, create an IAM policy that grants read access to the S3 bucket or prefix used by the import job:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:GetObject",
                    "s3:GetObjectVersion"
                ],
                "Resource": "arn:aws:s3:::<bucket>/<prefix>/*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket",
                    "s3:GetBucketLocation"
                ],
                "Resource": "arn:aws:s3:::<bucket>"
            }
        ]
    }

  3. Create an IAM role with a custom trust policy that allows VeloDB Cloud to assume the role. Replace <velodb-arn> with the ARN from the S3 import wizard:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "<velodb-arn>"
                },
                "Action": "sts:AssumeRole"
            }
        ]
    }

  4. Attach the S3 read policy to the IAM role.

  5. Copy the role ARN, then paste it into Role ARN in the S3 import wizard.

Troubleshooting​

  • If object listing fails, verify the bucket path, endpoint, region, and read permissions.
  • If files are skipped, review the file pattern and path configuration.
  • If rows fail to load, check file format, delimiters, encoding, strict-mode behavior, and target column types.