Import from Kafka

Use a Kafka import job to continuously ingest records from Kafka-compatible topics into VeloDB Cloud.

Kafka import covers:

For Amazon MSK, complete the AWS-side setup in Amazon MSK before creating the import job.

Prerequisites

Kafka import is supported only for VeloDB Cloud warehouses running the 26.x series.
A VeloDB Cloud warehouse that can reach the Kafka bootstrap servers.
Topic names and the expected start position.
Authentication settings, such as SASL/SCRAM credentials, TLS settings, or the mechanism required by your Kafka service.
A target database and target tables, or permission to create them during the wizard.

Open Import from the ingestion group, then click + Add Import Job.
Select Apache Kafka, Confluent Cloud, or Amazon MSK.
Enter the job name, bootstrap servers, authentication settings, and TLS settings if required.
If the Kafka cluster is in a private VPC, enable Private Link and configure the PrivateLink connection in the wizard. For setup details, see Grant VeloDB access to your VPC.
Select the topic or topics to ingest.
Configure the start position.
Select or create the target database and tables.
Configure job settings such as concurrency and max batch interval.
Run the pre-checks. When all checks succeed, start the job.

In the import job list, click a Kafka import job to open its details page.

For a Kafka import job, the details page provides monitoring views such as:

You can also open the corresponding Task Details to inspect import task information.

If bootstrap checks fail, verify bootstrap server addresses, advertised listeners, and network access to the brokers.
If authentication fails, verify the SASL mechanism, username, password, TLS certificates, and cluster policy.
If ingestion fails after the job starts, check topic permissions, message format, and target table schema.