Data Integration
Use Data Integration in the VeloDB Cloud Console to bring data into a warehouse or connect to data that stays in an external system. It provides one place to create and monitor integrations.
Choose how to connect your data
| Method | Use when | Where to view it |
|---|---|---|
| Import | Copy data or continuously ingest changes from MySQL, PostgreSQL, Kafka-compatible streams, or Amazon S3 into warehouse tables. | Import Jobs |
| External catalog | Query an external database or data lake without copying its underlying data into warehouse tables. | External Catalogs |
| Migration | Move selected databases and tables from an existing Apache Doris or VeloDB Enterprise deployment into VeloDB Cloud. | Migration Jobs |
| Upload a local file | Load a local CSV, JSON, or Parquet file into a warehouse table. | Data Uploads |
| Add sample data | Load a built-in dataset to try queries without preparing your own source. | Data Uploads |
Available methods depend on your warehouse's deployment type, cloud provider, version, and enabled capabilities. Upload file is available for SaaS warehouses, but not BYOC warehouses. The external catalog entry is available for supported AWS warehouses.
Supported sources and versions
The following table lists the minimum VeloDB Cloud version and availability for import jobs, external catalogs, and migration jobs. No version restriction means that the integration has no minimum Cloud version requirement. Source-specific prerequisites still apply.
| Method | Data source | Minimum Cloud version | Availability |
|---|---|---|---|
| Import job | MySQL, PostgreSQL | 26.1.0 | VeloDB Cloud |
| Import job | Amazon RDS for PostgreSQL, Amazon Aurora for PostgreSQL, Amazon RDS for MySQL, Amazon Aurora for MySQL | 26.1.0 | VeloDB Cloud on AWS only |
| Import job | Amazon S3 | 26.0.2 | VeloDB Cloud |
| Import job | Apache Kafka | No version restriction | VeloDB Cloud |
| Import job | Confluent Cloud | No version restriction | VeloDB Cloud |
| Import job | Amazon MSK | No version restriction | VeloDB Cloud on AWS only |
| External catalog | MySQL, PostgreSQL, TPC-DS, TPC-H | 4.1.1 | VeloDB Cloud on AWS only |
| External catalog | AWS Glue, Hive Metastore, Amazon S3 Tables, Iceberg REST Catalog, Apache Polaris, Databricks Unity Catalog | 4.1.1 | VeloDB Cloud on AWS only |
| Migration job | Apache Doris | No version restriction | VeloDB Cloud |
| Migration job | VeloDB Enterprise | No version restriction | VeloDB Cloud |
Before you begin
- Select the destination warehouse and make sure it has a running cluster for operations that load or query data.
- Use an account with the permissions needed for the operation. Read-only access disables creation actions, and creating import jobs requires load privileges.
- For external sources, prepare connection details and credentials with the required source-side permissions. Check the source-specific guide for version and configuration requirements.
- Configure network access from the warehouse to the source. See Connect for Import for SaaS and BYOC network paths.
- For ingestion, identify the target database and tables, or make sure you can create them in the wizard.
Loading data uses warehouse compute and storage. Size sample datasets and migration workloads for your available resources. See Billing for usage and charges.
Open Data Integration
- Log in to the VeloDB Cloud Console.
- Select the warehouse that you want to use.
- In the left navigation pane, click Data Integration.
If the warehouse has no integration records, the page shows cards for creating your first integration. Once records exist, the page shows category counts and lists, with an Add Data menu for creating more integrations.
| To create | On the initial page | From Add Data |
|---|---|---|
| An import job | Click Add Import Job. | Select Import data into VeloDB. |
| An external catalog | Click Add External Catalog. | Select Connect external data. |
| A migration job | Click Add Migration Job. | Select Migrate to VeloDB Cloud. |
| A local file upload | Click Upload file. | Select Upload file. |
| A sample dataset load | Click Add sample data. | Select Add sample data. |
For import jobs, select a source and follow its guide:
For external catalogs and migration jobs, follow these guides:
Monitor and manage integrations
Select a category on Data Integration to view its records:
- Import Jobs: Find an ingestion job and open its details to inspect progress, metrics, and errors. Available actions depend on the source and job state, and can include pausing, resuming, editing, retrying, and deleting the job.
- External Catalogs: Find the catalog connected to your external source. Use Catalogs to browse its databases and tables, and SQL Editor to query them.
- Migration Jobs: Open a migration job to inspect progress and per-table results. Pause a running job, or edit and resume a paused or failed job. After a successful migration, view its migration report.
- Data Uploads: Check local file uploads and sample dataset loads. Deleting a record removes only the job record, not its imported table or data. Jobs in progress cannot be deleted.
After an import or migration completes, query the target tables and compare representative rows and row counts with the source. For continuous ingestion, also check that new source changes arrive in the target tables.
Common errors
| Symptom | What to check |
|---|---|
| A creation action is disabled | Check whether your account is read-only and whether it has the required warehouse SQL privileges. Import creation requires load privileges. |
| An integration method is missing | Check the deployment type, cloud provider, and supported capabilities. For example, BYOC warehouses do not show Upload file. |
| A source connection or pre-check fails | Verify the source host, port, credentials, network rules, and source-side permissions. For private sources, verify the configured private network path. |
| A file preview fails or columns are misaligned | Check the file format, CSV separator and quoting options, and whether the file contains valid records. |
| A job fails after starting | Open the job details and inspect its error. Check source availability, target column types, field mappings, and loading settings before retrying or resuming. |