Lance Catalog

注記

This is an experimental feature.

Lance Catalog is supported starting from Apache Doris 4.2.

Lance is a columnar data format designed for analytics and AI workloads. Doris can use a Lance Catalog to discover databases and tables in a Lance Namespace and directly query Lance datasets stored on a local file system, S3-compatible object storage, or Alibaba Cloud OSS.

Doris currently provides read-only access to Lance. Creating, writing, updating, or deleting Lance tables is not supported.

Use Cases

Use case	Description
Query Lance data directly	Analyze existing Lance datasets with Doris SQL without migrating or importing the data first.
AI vector search	Reuse vector indexes already present in Lance, run searches through vector_search(), and combine them with scalar filtering and Doris SQL analytics.
Data integration	Read Lance data into Doris internal tables for further processing, joins, or long-term storage.
Namespace management	Use a Filesystem Catalog for a simple directory-based layout. Use a REST Catalog when Namespaces, table locations, or temporary storage credentials need centralized management.

Feature Overview

Feature	Support
Filesystem Catalog	Supports warehouses on a local file system, file://, s3://, or oss://
REST Catalog	Supports Lance REST Namespace with no authentication, Bearer Token, API Key, or custom HTTP headers
Metadata access	Supports SHOW DATABASES, SHOW TABLES, DESC, and SHOW INDEX (Filesystem Catalogs only)
System tables (table$...)	Not supported; use SHOW INDEX for index metadata
Data queries	Supports column pruning, parallel Lance Fragment scans, and snapshot-consistent reads of the current version
Predicate pushdown	Supports pushing compatible scalar predicates down to Lance
File TVFs	Supports querying Lance datasets directly through s3() and local()
Vector search	Uses physical Lance index segments as parallel splits, keeps uncovered Fragments as Flat Search splits, and performs a Doris global Top-K merge
Writing to Lance	Not supported
Time Travel	Not supported
Full-Text Search / Hybrid Search	Not supported

Lance Version and Compatibility

The Doris BE data reader is built with lance-c v0.1.6, which embeds the Lance Rust crates at version 7.0.0-beta.7 (Lance commit e0e977a6). The Doris FE reads Namespace and dataset metadata with the lance-java client at version 9.1.0-beta.3 (Lance commit e934cc2c). These versions identify the reader implementations integrated with Doris. They are different from the Lance data_storage_version recorded in a dataset.

The following table describes the file-format compatibility of this reader:

data_storage_version	Read support	Notes
0.1 / legacy	Supported	Original Lance file format.
2.0 (writer-option alias 0.3)	Supported	An earlier version of the Lance v2 file format.
2.1 / stable	Supported; default stable format	In the embedded Lance version, the stable writer option and the default format for new datasets both resolve to 2.1.
2.2	Supported	The embedded Lance version treats this as a stable format, but it is not the default writer format.
2.3 / next	Experimental; not guaranteed	The embedded Lance version marks 2.3 as unstable, and the next writer option resolves to 2.3.
A later or unknown version	Not supported	Opening or scanning the dataset may fail with an unsupported storage-version error.

Lance SDK release numbers and file-format versions are independent. A dataset written by an older or newer Lance SDK is readable only when its storage format, required table feature flags, index format, and Arrow/Lance data types are all understood by the versions embedded in Doris. Consequently:

• Doris is expected to read datasets written with the 0.1, 2.0, 2.1, and 2.2 storage formats, subject to the type limitations documented below.
• Forward compatibility is not guaranteed. A dataset written or modified by a later Lance release may be unreadable if it uses a newer storage format, an unknown required manifest feature, a newer index format, or an unsupported extension type.
• For datasets that must remain readable by this Doris release, use the current default stable format, 2.1, and do not use next. If a newer writer or optional Lance feature is introduced, validate the resulting dataset with the target Doris release before using it in production.

Configure a Catalog

Syntax

CREATE CATALOG [IF NOT EXISTS] catalog_name PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "<filesystem|rest>",
    {CatalogProperties},
    {StorageProperties},
    {CommonProperties}
);

Common Properties

Property	Required	Default	Description
type	Yes	-	Must be lance.
lance.catalog.type	No	filesystem	Catalog type. Valid values are filesystem and rest.
lance.namespace.parent	No	Empty	Limits access to the specified Lance Namespace and its child Namespaces. With the default delimiter, for example, production$analytics represents a two-level Namespace.
lance.namespace.delimiter	No	$	Delimiter used to parse lance.namespace.parent. It is also passed to the REST Namespace client. This property does not change how multilevel Namespaces are displayed in Doris.
lance.namespace.root_database	No	default	Doris database name to which the root Lance Namespace is mapped.

Filesystem Catalog

A Filesystem Catalog discovers Lance Namespaces and tables directly from a warehouse directory.

Property	Required	Description
warehouse	Yes	Root path of the Lance warehouse. Local absolute paths, file:// URIs, s3:// URIs, and oss:// URIs are supported.

Select the example that matches the storage system hosting the warehouse:

S3 / S3-Compatible

CREATE CATALOG lance_fs_s3 PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "filesystem",
    "warehouse" = "s3://my-bucket/lance",
    "s3.endpoint" = "https://s3.us-east-1.amazonaws.com",
    "s3.region" = "us-east-1",
    "s3.access_key" = "<ak>",
    "s3.secret_key" = "<sk>"
);

You can omit s3.endpoint for AWS S3. For S3-compatible storage such as MinIO, set the service endpoint and add "use_path_style" = "true" when the service requires path-style access.

Alibaba Cloud OSS

CREATE CATALOG lance_fs_oss PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "filesystem",
    "warehouse" = "oss://my-bucket/lance",
    "oss.endpoint" = "https://oss-cn-beijing.aliyuncs.com",
    "oss.region" = "cn-beijing",
    "oss.access_key" = "<ak>",
    "oss.secret_key" = "<sk>"
);

For temporary STS credentials, add "oss.session_token" = "<token>". The qualified form oss://my-bucket.oss-cn-beijing.aliyuncs.com/lance is also accepted and normalized to the bucket path. OSS-HDFS is not currently supported.

Local File System

CREATE CATALOG lance_fs_local PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "filesystem",
    "warehouse" = "/data/lance"
);

warehouse must be an absolute path. The FE and every BE executing the query must access the same path. In a multi-node deployment, mount the same shared directory on all relevant nodes.

For S3 and OSS, warehouse must include a bucket, such as s3://bucket/path or oss://bucket/path.

REST Catalog

A REST Catalog obtains Namespaces, table locations, and storage access parameters through Lance REST Namespace. A REST Catalog neither requires nor permits the warehouse property.

Property	Required	Default	Description
lance.rest.uri	Yes	-	REST service URI. It must use http:// or https://.
lance.rest.security.type	No	none	Authentication type. Valid values are none, bearer, and api_key.
lance.rest.bearer-token	Yes for Bearer authentication	-	Bearer Token.
lance.rest.api-key	Yes for API Key authentication	-	API Key sent in the x-api-key header.
lance.rest.header.<header-name>	No	-	Custom HTTP header sent to the REST service. Use the dedicated authentication properties above for authentication headers.

The REST Namespace returns the storage location of each table. Configure default access properties for the storage system hosting those tables:

S3 / S3-Compatible

CREATE CATALOG lance_rest_s3 PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "rest",
    "lance.rest.uri" = "https://lance.example.com",
    "lance.rest.security.type" = "bearer",
    "lance.rest.bearer-token" = "<token>",
    "s3.endpoint" = "https://s3.us-east-1.amazonaws.com",
    "s3.region" = "us-east-1",
    "s3.access_key" = "<ak>",
    "s3.secret_key" = "<sk>"
);

This configuration applies to s3:// Lance tables returned by the REST Namespace. For S3-compatible storage such as MinIO, replace s3.endpoint with the service endpoint and add "use_path_style" = "true" when required.

Alibaba Cloud OSS

CREATE CATALOG lance_rest_oss PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "rest",
    "lance.rest.uri" = "https://lance.example.com",
    "lance.rest.security.type" = "bearer",
    "lance.rest.bearer-token" = "<token>",
    "oss.endpoint" = "https://oss-cn-beijing.aliyuncs.com",
    "oss.region" = "cn-beijing",
    "oss.access_key" = "<ak>",
    "oss.secret_key" = "<sk>"
);

This configuration applies to oss:// Lance tables returned by the REST Namespace. For temporary STS credentials, add "oss.session_token" = "<token>".

For unauthenticated REST access, omit lance.rest.security.type and the authentication property. For API Key authentication, replace them with "lance.rest.security.type" = "api_key" and "lance.rest.api-key" = "<api-key>".

If the REST Namespace vends temporary storage credentials for a table, Doris gives them precedence over the Catalog credentials, allowing the S3 or OSS access keys to be omitted from the Catalog. For OSS, the Namespace may vend oss_endpoint, oss_access_key_id, oss_secret_access_key, oss_region, and oss_security_token, or the corresponding native names endpoint, access_key_id, access_key_secret, region, and security_token.

注意

The current BE Reader does not support Lance tables whose versions are managed by REST Namespace (Managed Versioning).

Namespace Mapping

Lance supports multilevel Namespaces, while a Doris Catalog represents each Namespace as a database name:

Lance Namespace	Doris Database Name
Root Namespace	default; configurable through lance.namespace.root_database
doris	doris
doris.analytics	doris.analytics

Doris joins the levels of a multilevel Namespace with . to form a database name. Use backticks when referencing a database name that contains .:

SHOW TABLES FROM lance_catalog.`doris.analytics`;

SELECT *
FROM lance_catalog.`doris.analytics`.user_features;

Use lance.namespace.parent to limit a Catalog to a Namespace subtree. For example:

CREATE CATALOG lance_analytics PROPERTIES (
    "type" = "lance",
    "lance.catalog.type" = "filesystem",
    "warehouse" = "s3://my-bucket/lance",
    "lance.namespace.parent" = "production$analytics",
    "s3.region" = "us-east-1"
);

Doris then displays only the tables and child Namespaces below production.analytics.

Query Lance Tables

After creating a Catalog, you can browse and query Lance tables in the same way as other external tables:

SHOW DATABASES FROM lance_catalog;

SHOW TABLES FROM lance_catalog.default;

DESC lance_catalog.default.user_profiles;

SELECT user_id, name, age
FROM lance_catalog.default.user_profiles
WHERE age >= 18
ORDER BY user_id
LIMIT 100;

You can also load data from Lance into a Doris internal table:

INSERT INTO internal.demo.user_profiles
SELECT user_id, name, age
FROM lance_catalog.default.user_profiles;

For a regular Catalog query, Doris pins a Lance dataset version during planning and generates scan tasks by Fragment. A query therefore reads a consistent snapshot, while multiple Scanners can read different Fragments in parallel without every Scanner repeatedly scanning the entire dataset.

Inspect Lance Indexes

For a table in a Filesystem Catalog, SHOW INDEX displays the logical scalar and vector indexes recorded in the Lance dataset. The variant statements SHOW INDEXES, SHOW KEY, and SHOW KEYS produce the same result:

SHOW INDEX FROM lance_catalog.default.items;

Doris reads the index metadata from the latest dataset snapshot at execution time and sorts the result by index name and column position. System indexes that Lance maintains internally, such as __lance_frag_reuse and __lance_mem_wal, are not displayed. A table without indexes returns an empty result.

SHOW INDEX returns the standard 13-column result set. For a Lance table, only the following columns carry values; the other columns are always empty:

Column	Value for a Lance table
Table	Table name.
Key_name	Name of the Lance logical index.
Seq_in_index	Position of the column in the index, starting from 1. An index on multiple fields produces one row per field.
Column_name	Indexed field. An index on a nested field displays the field path joined with ., and a path segment containing characters other than letters, digits, and _ is quoted with backticks, for example attributes.`child.with.dot`.
Index_type	Index type reported by the Lance SDK, such as BTree, IVF_FLAT, IVF_SQ, IVF_PQ, IVF_HNSW_FLAT, IVF_HNSW_SQ, or IVF_HNSW_PQ.
Properties	JSON object with a fixed set of index details: metric_type and target_partition_size at the top level, type, num_bits, num_sub_vectors, and rotation_type under compression, and construction_ef, max_connections, and max_level under hnsw. Keys are sorted, and the value is {} when the index details contain none of these fields.

The following rows illustrate an IVF_PQ vector index and a BTree index on a nested field:

Table          Key_name              Seq_in_index  Column_name                  Index_type  Properties
vs_ivf_pq_f32  embedding_ivf_pq_f32  1             embedding                    IVF_PQ      {"compression":{"num_bits":4,"num_sub_vectors":4,"type":"pq"},"metric_type":"L2"}
nested_index   nested_label_btree    1             attributes.`child.with.dot`  BTree       {}

SHOW INDEX only inspects existing indexes; it does not create one. Create Lance indexes with the Lance SDK or another Lance writer outside Doris. If the recorded index metadata is inconsistent, for example when an index references an unknown field or two indexes share an identical name, the statement fails instead of returning partial metadata.

SHOW INDEX requires the SHOW privilege on the table. For a REST Catalog, the statement is rejected with the error SHOW INDEX is not supported for Lance REST catalogs.

Type Mapping

Lance / Arrow Type	Doris Type	Description
bool	BOOLEAN
int8	TINYINT
uint8	SMALLINT	Losslessly widened unsigned integer
int16	SMALLINT
uint16	INT	Losslessly widened unsigned integer
int32	INT
uint32	BIGINT	Losslessly widened unsigned integer
int64	BIGINT
uint64	LARGEINT	Losslessly widened unsigned integer
float16	FLOAT	Widened to a 32-bit floating-point value
float32	FLOAT
float64	DOUBLE
decimal128(P,S)	DECIMAL(P,S)	Maximum precision is 38
decimal256(P,S)	DECIMAL(P,S)	Maximum precision is 76
utf8, large_utf8	TEXT
binary, large_binary	VARBINARY(2147483647)
fixed_size_binary(N)	VARBINARY(N)	Preserves the fixed byte width
date32(day), date64(ms)	DATE	A date64 value must represent a complete calendar day
time32(s)	TIME(0)
time32(ms)	TIME(3)
time64(us), time64(ns)	TIME(6)	Nanosecond precision is truncated to microseconds
Timezone-naive timestamp(s)	DATETIME	Not converted according to the Session Time Zone
Timezone-naive timestamp(ms)	DATETIME(3)	Not converted according to the Session Time Zone
Timezone-naive timestamp(us), timestamp(ns)	DATETIME(6)	Nanosecond precision is truncated to microseconds
Timezone-aware timestamp	TIMESTAMPTZ(0-6)	Preserves the instant and displays it in the Doris Session Time Zone
struct	STRUCT	Child fields are mapped recursively
list, large_list, fixed_size_list	ARRAY	Element types are mapped recursively
map	MAP	Key and value types are mapped recursively

The following types are not currently supported:

• Arrow null and duration.
• Arrow/Lance Extension types with ARROW:extension:name metadata, including Lance Blob v2, Arrow JSON Extension, and Lance BFloat16 Extension.
• Complex types whose child types cannot be mapped recursively.
• Arrow Dictionary types that preserve the Dictionary marker.

For an unsupported top-level column, DESC on a Catalog table and DESC FUNCTION on a Lance file TVF both preserve the column and display unknown type: UNSUPPORTED_TYPE. If any child of a complex type cannot be mapped, the whole top-level complex column is marked unsupported. Queries can still project only supported columns. Doris reports an error during analysis when SQL projects an unsupported column. For example:

SELECT * EXCEPT(blob_col, json_col)
FROM lance_catalog.default.all_types;

注記

Some Lance Java SDK versions may lose the Dictionary marker while reading a Schema and expose a Dictionary column as its physical index type. This behavior does not mean that Doris supports the logical Dictionary values and must not be relied upon.

Predicate Pushdown

Doris converts semantically compatible predicates into Substrait expressions and passes them to Lance for evaluation during reads. The Doris BE does not evaluate a condition again after the entire condition has been pushed down. Conditions that cannot be pushed down safely remain in Doris.

Data Types Supported for Pushdown

Lance / Arrow Type	Pushdown Support
bool	Equality, null checks, and logical operations; ordering comparisons are not supported
int8/16/32/64	Supported
uint8/16/32/64	Supported
float32/64	Supported
decimal128	Precision 1 through 38, with Scale from 0 through Precision
utf8, large_utf8	Supported
date32(day)	Supported
Timezone-naive timestamp(s/ms/us)	Supported

Predicates on other readable types, including float16, decimal256, Binary, date64, Time, nanosecond Timestamp, timezone-aware Timestamp, and complex types, currently remain in Doris.

Operators Supported for Pushdown

SQL Predicate	Pushdown Condition
=, !=, <>, <, <=, >, >=	Direct comparison between a column and a constant. The constant may be on the left side.
<=>	Direct null-safe equality comparison between a column and a constant; preserves a non-NULL, two-valued result inside NOT, AND, or OR
IN, NOT IN	Non-empty constant list that does not contain NULL
IS NULL, IS NOT NULL	Direct column reference
Boolean column, NOT Boolean column	Direct Boolean column reference
LIKE, NOT LIKE	Direct utf8 or large_utf8 column and a string literal pattern without backslash escapes, an explicit ESCAPE clause, or an embedded NUL character
starts_with, ends_with	Built-in function with a direct utf8 or large_utf8 column and a string literal
AND	Top-level conjuncts can be pushed down independently, with unsupported conjuncts retained in Doris
OR	Both branches must be fully convertible
NOT	The operand must be fully convertible

The following forms are generally not pushed down:

• Functions other than the supported built-in string functions, or arithmetic expressions applied to a column. A user-defined function named like, starts_with, or ends_with is not treated as the corresponding built-in function.
• String predicates whose pattern contains an embedded NUL character. LIKE and NOT LIKE patterns that use a backslash escape or an explicit ESCAPE clause also remain in Doris.
• An empty IN list or an IN list containing NULL.
• An OR or NOT expression in which only part of the expression can be converted.
• A data type or constant value that cannot be converted to Lance without loss.

Use lancePushdownPredicate in EXPLAIN to inspect the conditions that are actually pushed down:

EXPLAIN
SELECT user_id
FROM lance_catalog.default.user_profiles
WHERE active
  AND starts_with(name, 'A')
  AND country LIKE 'C%';

Query Lance with File TVFs

If you only need to read a Lance dataset at a known path, you can use the s3() or local() TVF without creating a Catalog. uri or file_path must point to the root directory of a Lance dataset, rather than an internal data file.

S3 TVF

SELECT user_id, name
FROM s3(
    "uri" = "s3://my-bucket/lance/user_profiles.lance",
    "s3.endpoint" = "http://127.0.0.1:9000",
    "s3.access_key" = "admin",
    "s3.secret_key" = "password",
    "s3.region" = "us-east-1",
    "use_path_style" = "true",
    "format" = "lance"
)
WHERE user_id > 100;

For an S3 TVF, the FE obtains the Schema, current version, and Fragment list. Doris pins that version and scans its Fragments in parallel.

Local TVF

SELECT user_id, name
FROM local(
    "file_path" = "/data/lance/user_profiles.lance",
    "backend_id" = "10001",
    "format" = "lance"
);

file_path is passed as written to the Lance Reader on the target BE. Doris does not prepend user_files_secure_path or expand this path as a Glob, so it must point directly to the root directory of one Lance dataset that the target BE can access. An absolute path is recommended.

Local TVF Schema discovery and execution each open the latest dataset version independently. The version resolved during Schema discovery is not currently pinned for the subsequent scan. If the dataset changes between query analysis and execution, the discovered Schema and scanned snapshot can differ. Avoid modifying the dataset while a Local TVF query is being analyzed and executed. The current Local Lance TVF uses one Scanner.

Lance file TVFs have the following additional limitations:

• Only s3() and local() are supported. Other file TVFs, such as HDFS and HTTP, are not currently supported.
• path_partition_keys is not supported.
• One TVF path can represent only one Lance dataset.
• DESC FUNCTION can display a Schema containing unsupported types, but SQL cannot project unsupported columns.

Vector Search

vector_search() is a relational TVF that performs Top-K search on a vector column in a Lance table. It can use an existing Lance vector index or perform Flat Search.

Syntax and Example

SELECT user_id, label, _distance
FROM vector_search(
    "table" = "lance_catalog.default.items",
    "column" = "embedding",
    "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
    "top_k" = "10",
    "offset" = "3",
    "metric" = "l2",
    "nprobes" = "20",
    "refine_factor" = "10",
    "filter" = "category = 'book'",
    "use_index" = "true"
)
ORDER BY _distance ASC, user_id;

The relation schema of vector_search() contains all columns from the Lance source table plus the _distance column generated by the Lance Scanner for the nearest-neighbor query. The final SQL result contains only columns projected by SELECT. Doris exposes _distance as FLOAT. It is a distance, not a generic similarity score: a lower value means that two vectors are closer. The source table must not already contain a column named _distance. A SQL relation does not guarantee final display order, so explicitly specify ORDER BY _distance ASC when deterministic nearest-neighbor ordering is required. Adding a unique column as a tie-breaker is recommended for rows with the same distance.

table must parse as exactly three catalog.database.table name parts. A multilevel Lance Namespace maps to one Doris database name containing ., so quote the database part with backticks. For example, use the following value for table items in Namespace doris.analytics:

"table" = "lance_catalog.`doris.analytics`.items"

Do not use the unquoted form lance_catalog.doris.analytics.items; it parses as four name parts and is rejected. A table-only name or database.table name is also rejected.

Parameters

Parameter	Required	Default	Description
table	Yes	-	Fully qualified, three-part catalog.database.table name. If a multilevel Namespace maps to a database name containing ., quote the database part with backticks. It must identify a table in a Lance Catalog, and the user must have the SELECT privilege on the table.
column	Yes	-	Vector column name. fixed_size_list<float16|float32|float64|uint8|int8> is currently supported.
query_vector	Yes	-	JSON number array. Its dimension must match the vector column, and each value must be representable by the vector element type.
top_k	No	10	Number of results returned after skipping offset. It must be a positive integer.
offset	No	0	Number of nearest neighbors skipped inside the vector search. It must be a non-negative integer. top_k + offset must not exceed the maximum unsigned 32-bit integer.
metric	No	hamming for uint8, l2 for the other supported types	Distance metric: l2, cosine, dot, or hamming. dot_product is an alias for dot. uint8 vectors support only hamming; the other currently supported vector element types support l2, cosine, and dot. An unset metric is treated as l2 when Doris selects a vector index, so querying an index built with cosine or dot requires setting metric explicitly.
filter	No	-	Lance SQL condition evaluated before vector candidates are generated; that is, a Prefilter.
nprobes	No	Minimum 1, with no maximum	Number of IVF index partitions to probe. It must be a positive integer. When unset, Lance starts with one partition and can probe additional partitions when a Prefilter leaves too few candidates. Setting it explicitly to N fixes both the minimum and maximum number of probes to N.
refine_factor	No	Refinement disabled	Candidate refinement multiplier. It must be a positive integer. When unset, Lance does not recompute distances from the original vectors, so _distance from a quantized index may be approximate. When set to N, Lance first retrieves (top_k + offset) × N candidates, recomputes their exact distances from the original vectors, and reorders them. Refinement reads the original vector data for these candidates. As N increases, more candidates are read and evaluated, which can significantly increase I/O and reduce query performance. Setting it to 1 still enables refinement and therefore differs from leaving it unset.
ef	No	floor(1.5 × (top_k + offset))	Candidate width retained during HNSW graph search. It must be a positive integer. If refine_factor is also set, the default is floor(1.5 × (top_k + offset) × refine_factor). It has no effect on non-HNSW indexes.
use_index	No	true	When true, Doris prefers a Lance vector index compatible with the vector column and distance metric, and automatically uses Flat Search if no usable index is available. When false, Doris disables vector indexes and performs Flat Search over the data.

These defaults correspond to the Lance Scanner behavior currently integrated with Doris. When metric is omitted, Doris treats the query as l2 while selecting a vector index, so an index built with cosine or dot is not selected. When no index is selected, or if "use_index" = "false", uint8 vectors use hamming, while the other currently supported vector element types use l2.

Supported Vector Index Types

The embedded lance-c v0.1.6 explicitly supports the following Lance vector index combinations:

Index type	Description	Main query parameters
IVF_FLAT	IVF partitions with original-vector distance computation inside each partition	nprobes
IVF_SQ	IVF with Scalar Quantization	nprobes, refine_factor
IVF_PQ	IVF with Product Quantization	nprobes, refine_factor
IVF_HNSW_FLAT	IVF with HNSW whose graph nodes retain original vectors	nprobes, ef
IVF_HNSW_SQ	IVF and HNSW with Scalar Quantization	nprobes, ef, refine_factor
IVF_HNSW_PQ	IVF and HNSW with Product Quantization	nprobes, ef, refine_factor

vector_search() only uses vector indexes that already exist in Lance. It does not create indexes or let users specify an index type or name. With use_index=true, Doris automatically selects an index compatible with the vector column and distance metric. Data without a usable index, including data not covered by the selected index, automatically uses Flat Search and is not omitted. With use_index=false, all data uses Flat Search.

Supported Vector Element Types and Distance Metrics

The distance metrics supported by a vector index depend on the vector element type. Choose a supported combination from the following table; unsupported combinations cannot use a vector index.

Vector element type	l2	cosine	dot	hamming
float16	Supported [1]	Supported	Supported	Not supported
float32	Supported	Supported	Supported	Not supported
float64	Supported	Supported	Supported	Not supported
uint8	Not supported	Not supported	Not supported	IVF_FLAT and IVF_HNSW_FLAT only [2]
int8	Flat Search only [3]	Flat Search only [3]	Flat Search only [3]	Not supported

Unless a footnote states otherwise, "Supported" means that the combination works with all six index types listed above. "Flat Search only" means that the vectors can be queried, but no vector index is used.

1. Building an l2 index over float16 data with large values may take too long or fail. Keep vector values within a controlled range. If index creation fails, use Flat Search or, when appropriate for the application, use cosine instead.
2. Lance treats uint8 vectors as binary vectors and supports only the hamming metric. Product Quantization and Scalar Quantization do not support uint8, so only IVF_FLAT and IVF_HNSW_FLAT are available.
3. int8 vectors cannot currently use a vector index and are limited to Flat Search. The embedded Lance version also has a known issue where searching a nullable int8 vector column may terminate the BE process. Avoid these queries until Doris upgrades to a lance-c release containing the lance#7498 fix.

The Query Metric Must Match the Index Metric

The query metric must match the metric used to build the index. Otherwise, Doris uses Flat Search. The results remain correct, but performance is usually lower because the vectors must be scanned directly.

When metric is omitted, Doris treats it as l2 while selecting an index. Therefore, explicitly set metric when using an index built with cosine, dot, or hamming. For example, a uint8 vector index requires "metric" = "hamming"; otherwise, the index is not used.

Use EXPLAIN to confirm index usage. A lanceSearchIndexSegments value greater than 0 indicates indexed search, while 0 indicates Flat Search.

Doris considers only one vector index per vector column. Keep at most one vector index on each vector column to avoid ambiguous selection among indexes built with different metrics.

Prefilter and Post-Filter

Prefilter and Post-filter can be used in the same query:

SELECT user_id, category, _distance
FROM vector_search(
    "table" = "lance_catalog.default.items",
    "column" = "embedding",
    "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
    "top_k" = "10",
    "filter" = "category = 'book'"
)
WHERE user_id > 100
ORDER BY _distance ASC, user_id;

Filter type	Example condition	When it runs	Effect on results
Prefilter	TVF parameter "filter" = "category = 'book'"	Before Lance generates vector candidates	Searches for nearest neighbors only among rows where category = 'book'.
Post-filter	Outer WHERE user_id > 100	After Lance generates candidates and before Doris applies the final TopN	Removes non-matching rows from the generated candidates. It does not replenish candidates, so the final result may contain fewer than top_k rows.

If user_id > 100 must also participate in candidate generation, include it in the TVF parameter, for example "filter" = "category = 'book' AND user_id > 100", instead of using an outer WHERE. Even if the optimizer moves the outer WHERE into the Doris Lance Scan, it remains a Post-filter and is not converted into a Lance Prefilter.

Lance reads and evaluates columns referenced only by filter internally. If such a column is not referenced by SELECT or another Doris expression, it does not have to be returned to Doris.

Two-Phase TopN Read and Lazy Materialization

Vector search commonly produces more candidate rows than the final top_k. Reading wide output columns such as title or payload during the search wastes work because most candidate rows are discarded by TopN. Two-phase reading defers these columns: Phase 1 reads only the columns required for filtering, ordering, and TopN; after global TopN, Phase 2 reads the remaining output columns only for the retained rows.

This reduces storage I/O, network transfer, and memory usage. The benefit is usually greatest when top_k is small, the search produces many candidates, or the query returns wide string, JSON, or similar columns.

Phase	Data read
Phase 1	Columns used internally by vector search, _distance, columns required by Post-filter or ordering, and an internal Row Location.
Phase 2	Top-level columns used only by the final projection, read only for rows retained by global TopN.

Trigger Conditions

Doris uses two-phase reading only when all of the following conditions are met:

1. topn_lazy_materialization_threshold is greater than 0. Its default value is 1024.
2. top_k does not exceed the threshold.
3. At least one top-level column can be deferred. A column used only by the final SELECT projection can be deferred. A column used by a Post-filter, ORDER BY, or another expression before TopN must be read in Phase 1. Nested subcolumns cannot currently be deferred.

In the following query, category must be read in Phase 1 for the Post-filter, while user_id, title, and payload can be fetched after TopN:

SET topn_lazy_materialization_threshold = 1024;

SELECT user_id, title, payload, _distance
FROM vector_search(
    "table" = "lance_catalog.default.items",
    "column" = "embedding",
    "query_vector" = "[0.1, 0.2, 0.3, 0.4]",
    "top_k" = "10",
    "offset" = "3"
)
WHERE category = 'book';

A Prefilter column referenced in the TVF filter is used internally by Lance. Its presence in the filter string alone does not make it a Phase-1 result column returned to Doris.

Disabling Two-Phase Reading

Use the following session setting to disable two-phase reading:

SET topn_lazy_materialization_threshold = -1;

Doris automatically uses a single-phase read when top_k exceeds the threshold or no column can be deferred, so manual disabling is usually unnecessary. Consider disabling it and comparing performance in the following cases:

• top_k is large, so Phase 2 still reads most candidate rows.
• The query returns only a few narrow columns, leaving little I/O for two-phase reading to save.
• Random Row-ID reads have high latency on the underlying storage, and the extra Phase-2 Fetch costs more than deferred reading saves.

Disabling this optimization changes only when output columns are read. It does not disable vector indexes or change Prefilter, Post-filter, or TopN semantics. Compare both settings with representative queries and storage conditions.

Current Execution Model

The execution order of vector_search() is:

Pinned dataset snapshot
  -> FE Split planning
       -> Indexed coverage: one Split per physical Index Segment -> ANN Search
       -> Uncovered or unindexed data: one Split per Fragment -> Flat Search
  -> Per Split: Lance Prefilter -> ANN/Flat Search -> at most K+n candidates
  -> Doris Scan Post-filter
  -> Doris local TopN
  -> Exchange
  -> Doris global TopN (applies offset=n and limit=K)
  -> Optional lazy-materialization Fetch

Current Limitations and Recommendations

• Lance Catalogs and Lance TVFs are read-only. CREATE TABLE, INSERT, UPDATE, DELETE, TRUNCATE TABLE, and writing data back to Lance are not supported.
• Queries always read the current version selected during planning. SQL cannot select a Version or perform Time Travel.
• For tables containing unsupported column types, explicitly list the columns to read instead of projecting unsupported columns through SELECT *.
• For regular scans, inspect lancePushdownPredicate in EXPLAIN to verify which conditions have been pushed down.
• Create a vector index in Lance that matches the intended query before running indexed vector search. For small datasets or validation, set "use_index" = "false" to perform Flat Search.
• Use SHOW INDEX on a Filesystem Catalog table to inspect its logical Lance indexes, for example to verify the index name, type, and indexed fields before indexed vector search. SHOW INDEX is not supported for REST Catalogs.
• For deterministic vector result ordering, explicitly use ORDER BY _distance ASC and add a unique tie-breaker.
• Use the vector_search() filter parameter when filtering must occur before vector candidates are generated. An outer WHERE filters only the candidates already generated by each search Split and runs before Doris global TopN, so allow for a final result with fewer than top_k rows.
• Use EXPLAIN to inspect lanceSearchFragments and lanceSearchIndexSegments. The former is the number of visible Fragments in the pinned snapshot; the latter is the number of physical Index Segment splits selected by the FE. Additional fallback Fragment splits may also be present.