S3

This document describes the parameters required for accessing AWS S3. These parameters apply to:

• Catalog properties.
• Table Valued Function properties.
• Broker Load properties.
• Export properties.
• Outfile properties.

Parameter Overview

Property Name	Legacy Name	Description	Default	Required
s3.endpoint		S3 service access endpoint, e.g., s3.us-east-1.amazonaws.com	None	No
s3.access_key		AWS Access Key for authentication	None	No
s3.secret_key		AWS Secret Key for authentication	None	No
s3.region		S3 region, e.g., us-east-1. Strongly recommended	None	Yes
s3.use_path_style		Whether to use path-style access	FALSE	No
s3.connection.maximum		Maximum number of connections for high concurrency scenarios	50	No
s3.connection.request.timeout		Request timeout (milliseconds), controls connection acquisition timeout	3000	No
s3.connection.timeout		Connection establishment timeout (milliseconds)	1000	No
s3.role_arn		Role ARN specified when using Assume Role mode	None	No
s3.external_id		External ID used with s3.role_arn	None	No

Authentication Configuration

Doris supports the following two methods to access S3:

1. Direct Access Key and Secret Key

"s3.access_key"="your-access-key",
"s3.secret_key"="your-secret-key",
"s3.endpoint"="s3.us-east-1.amazonaws.com",
"s3.region"="us-east-1"

2. Assume Role Mode

Suitable for cross-account and temporary authorization access. Automatically obtains temporary credentials through role authorization.

"s3.role_arn"="arn:aws:iam::123456789012:role/demo-role",
"s3.external_id"="external-identifier",
"s3.endpoint"="s3.us-east-1.amazonaws.com",
"s3.region"="us-east-1"

If both Access Key and Role ARN are configured, Access Key mode takes precedence.

For instructions on AWS authentication and authorization configuration, please refer to the document aws-authentication-and-authorization

Accessing S3 Directory Bucket

This feature is supported since version 3.1.0.

Amazon S3 Express One Zone (also known as Directory Bucket) provides higher performance, but has a different endpoint format.

• Regular bucket: s3.us-east-1.amazonaws.com
• Directory Bucket: s3express-usw2-az1.us-west-2.amazonaws.com

For more available regions, refer to: AWS Official Documentation

Example:

"s3.access_key"="ak",
"s3.secret_key"="sk",
"s3.endpoint"="s3express-usw2-az1.us-west-2.amazonaws.com",
"s3.region"="us-west-2"

Permission Policies

Depending on the use case, permissions can be categorized into read-only and read-write policies.

1. Read-only Permissions

Only allows reading objects from S3. Suitable for LOAD, TVF, querying EXTERNAL CATALOG, and other scenarios.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
              "s3:GetObject",
              "s3:GetObjectVersion",
            ],
            "Resource": "arn:aws:s3:::<your-bucket>/your-prefix/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::<your-bucket>"
        }    
    ]
}

2. Read-write Permissions

Based on read-only permissions, additionally allows deleting, creating, and modifying objects. Suitable for EXPORT, OUTFILE, and EXTERNAL CATALOG write-back scenarios.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
              "s3:PutObject",
              "s3:GetObject",
              "s3:GetObjectVersion",
              "s3:DeleteObject",
              "s3:DeleteObjectVersion",
              "s3:AbortMultipartUpload",      
              "s3:ListMultipartUploadParts"
            ],
            "Resource": "arn:aws:s3:::<your-bucket>/<your-prefix>/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetLifecycleConfiguration"
            ],
            "Resource": "arn:aws:s3:::<your-bucket>"
        }    
    ]
}

Notes

1. Placeholder Replacement

   • <bucket> → Your S3 Bucket name.
   • <account-id> → Your AWS account ID (12-digit number).

2. Principle of Least Privilege

   • If only querying, do not grant write permissions.