VeloDB Enterprise
Release Notes
Enterprise Core

Enterprise Core

Enterprise Core 2.1.x

Enterprise Core 2.1.4

June 27, 2024

  • Query optimizer supports the FE Flame Graph tool, simultaneous use of SELECT DISTINCT with aggregate functions, rewriting single-table queries without GROUP BY, and high-concurrency point queries.
  • Lakehouse integration supports Paimon's native reader to handle Deletion Vectors, using Resources in Table-Valued Functions (TVF), and achieving data masking through the Ranger plugin.
  • Asynchronous materialized view construction now supports partition roll-up, trigger-based updates, specifying store_row_column and Storage Medium, and transparent rewriting supports single-table asynchronous materialized views and AGG_STATE type aggregate roll-up.
  • Other feature enhancements include the addition of the replace_empty function, support for the show storage policy using statement, and JVM metrics on the BE side.
  • Several optimizations have been made, including improving the accuracy of memory estimation consumed by the Segment Cache and supporting the creation of inverted indexes for Chinese column names.
  • Various issues have been fixed in areas such as the query optimizer, query execution, materialized views, and semi-structured data analysis.

Enterprise Core 2.1.3

May 17, 2024

  • Support INSERT INTO hive table in Hive Catalog.
  • Add show views statement to query views.
  • Workload group support bind to specific BE hosts.
  • Broker Load spport compressed JSON format.
  • Truncate function can use column as scale argument.
  • Add new function uuid_to_int and int_to_uuid.
  • Support create mtmv based on other mtmv.
  • Support rewrite by mv nested materialized view.
  • Add BypassWorkloadGroup to pass query queue.
  • Add function strcmp.
  • Support hll functions hll_from_base64, hll_to_base64.

Enterprise Core 2.1.2

April 18, 2024

  • Add processlist table in information_schema database, users could use this table to query active connections.
  • Add a new table valued function LOCAL to allow access file system like shared storage.
  • Set the default value of the data_consistence property of EXPORT to partition to make export more stable during load.
  • Some of MySQL Connector (eg, dotnet MySQL.Data) rely on variable's column type to make connection.
  • Add rollup table name in profile to help find the mv selection problem.
  • Add test connection function to DB2 database to allow user check the connection when create DB2 Catalog.
  • Add DNS Cache for FQDN to accelerate the connect process among BEs in K8s env.
  • Refresh external table's rowcount async to make the query plan more stable.

Enterprise Core 2.1.1

April 8, 2024

  • Change float type output format to improve float type serialization performance.
  • Fix issues during rolling upgrade from 2.0.x to 2.1.x, including backend node core problems and JDBC Catalog query errors.
  • Enable proxy protocol to support IP transparency. Using this protocol, IP transparency for load balancing can be achieved, so that after load balancing, Doris can still obtain the client's real IP and implement permission control such as whitelisting.
  • Add new system table backend_active_tasks to monitor the realtime query statics on every BE.
  • Add inverted index support for CCR.
  • Support arrow serialization for varint type.
  • Fixed 20 bugs, including occasional core issues in the BE during the restore process.

Enterprise Core 2.1.0

March 18, 2024

  • Further improvement in the performance of complex SQL queries has been achieved, with over 100% performance enhancement on the TPC-DS 1TB test dataset, positioning query performance at the forefront of the industry.
  • Performance improvements in data lake analytics scenarios, with 4-6 times better performance compared to Trino and Spark, have been made. Additionally, compatibility with multiple SQL dialects has been introduced, enabling seamless migration from existing systems to Apache Doris.
  • For data science and other forms of large-scale data reading scenarios, a high-speed reading interface based on Arrow Flight has been introduced, resulting in a 100-fold improvement in data transfer efficiency.
  • In semi-structured data analysis scenarios, new Variant and IP data types have been introduced, along with enhancements to a series of analytical functions, making storage and analysis of complex semi-structured data more convenient.
  • The introduction of asynchronous materialized views based on multiple tables has improved query performance. This includes support for transparent rewriting acceleration, automatic refreshing, external-to-internal table materialized views, and direct querying of materialized views. Leveraging these capabilities, materialized views can also be used for data warehouse tiered modeling, job scheduling, and data processing.
  • In terms of data storage, capabilities such as auto-increment columns, automatic partitioning, MemTable forwarding, and server-side batching have been introduced to improve the efficiency of real-time data writing at scale.
  • Further improvements have been made in workload management, enhancing the isolation capability of Workload Group resource groups and adding the ability to view SQL resource usage at runtime, thereby enhancing stability in multi-load scenarios.

Enterprise Core 2.0.x

Enterprise Core 2.0.12

July 1, 2024

  • No longer set the default table comment to the table type. Instead, set it to be empty by default, for example, change COMMENT 'OLAP' to COMMENT ''. This new behavior is more friendly for BI software that relies on table comments.
  • Change the type of the @@autocommit variable from BOOLEAN to BIGINT to prevent errors from certain MySQL clients (such as .NET MySQL.Data).
  • Remove the disable_nested_complex_type parameter and allow the creation of nested ARRAY, MAP, and STRUCT types by default.
  • The HMS catalog supports the SHOW CREATE DATABASE command.
  • Add more inverted index metrics to the query profile.
  • Cross-Cluster Replication (CCR) supports inverted indices.

Enterprise Core 2.0.11

June 20, 2024

  • Add trino jdbc catalog type mapping for JSON and TIME.
  • FE exit when failed to transfer to (non)master to prevent unknown state and too many logs.
  • Write audit log while doing drop stats table.
  • Ignore min/max column stats if table is partially analyzed to avoid inefficient query plan.
  • Support minus operation for set like set1 - set2.
  • Improve perfmance of LIKE and REGEXP clause with concat(col, pattern_str), eg. col1 LIKE concat('%', col2, '%').
  • Add query options for short circuit queries for upgrade compatibility.
  • Since the inverted index is now mature and stable, it can replace the old BITMAP INDEX. Therefore, any newly created BITMAP INDEX will automatically switch to an INVERTED INDEX, while existing BITMAP INDEX will remain unchanged. Users can disable this automatic switch by setting the FE configuration enable_create_bitmap_index_as_inverted_index to false.

Enterprise Core 2.0.10

May 20, 2024

  • This enhancement introduces the read_only and super_read_only variables to the database system, ensuring compatibility with MySQL's read-only modes.
  • When the check status is not IO_ERROR, the disk path should not be added to the broken list. This ensures that only disks with actual I/O errors are marked as broken.
  • When performing a Create Table As Select (CTAS) operation from an external table, convert the varchar column to string type.
  • Support mapping Paimon column type "Row" to Doris type "Struct"
  • Choose disk tolerate with little skew when creating tablet
  • Write editlog for set replica drop to avoid confusing status on follower FE

Enterprise Core 2.0.9

April 24, 2024

  • Allow predicate appearing on both key and value mv columns
  • Enable mv with bitmap_union(bitmap_from_array())
  • Introduce a configuration to forcibly replicate allocation for all olap tables within the cluster
  • Add timezone support for date literals in the new optimizer Nereids
  • Enable the use of "slop" in full-text search's match_phrase to specify word distances
  • Display index ID in SHOW PROC INDEXES
  • Incorporate a secondary argument in first_value/last_value to overlook NULL values
  • Allow the use of 0 as the offset parameter in the lead/lag function

Enterprise Core 2.0.8

April 12, 2024

  • Make Inverted Index work with TopN opt in Nereids
  • Limit the max string length to 1024 while collecting column stats to control BE memory usage
  • JDBC Catalog close when JDBC client is not empty
  • Accept all Iceberg database and do not check the name format of database
  • Refresh external table's rowcount async to avoid cache miss and unstable query plan
  • Simplify the isSplitable method of hive external table to avoid too many hadoop metrics

Enterprise Core 2.0.7

March 26, 2024

  • Support make miss slot as null alias when converting outer join to anti join to speed up query.
  • Add DEFAULT_ENCRYPTION column in information_schema table and add processlist table for better compatibility for BI tools.
  • Automatically test connectivity by default when creating a JDBC Catalog.
  • Enhance auto resume to keep routine load stable.
  • Use lowercase by default for Chinese tokenizer in inverted index.
  • Add error msg if exceeded maximum default value in repeat function.
  • Skip hidden file and dir in hive table.
  • Reduce file meta cache size and disable cache for some cases to avoid OOM.
  • Reduce jvm heap memory consumed by profiles of BrokerLoadJob.
  • Remove sort which is under table sink to speed up query like INSERT INTO t1 SELECT * FROM t2 ORDER BY k.

Enterprise Core 2.0.6

March 14, 2024

  • Support match a function with alias in materialized-view.
  • Add a command to drop a tablet replica safely on backend.
  • Add row count cache for external table.
  • Support analyze rollup to gather statistics for optimizer.
  • Improve tablet schema cache memory by using deterministic way to serialize protobuf.
  • Improve show column stats performance.
  • Support estimate row count for iceberg and paimon.
  • Support sqlserver timestamp type read for JDBC catalog.

Enterprise Core 2.0.3

December 8, 2023

  • Supports automatic statistics collection, which aids the optimizer in understanding data distribution characteristics. This enables the selection of more optimal plans, significantly improving query efficiency. Starting from version 2.0.3, this feature is officially supported and is enabled by default throughout the day.
  • Data Lake supports JAVA UDF, JDBC, Hudi MOR, and more system support for complex data types.
  • Cross-Cluster Replication (CCR) now supports features like throttling and table truncation enhancements.
  • Additional built-in functions such as SHA, JSON have been added.
  • Over 20 performance improvements including inverted index, case when, predicate pushdown, etc.
  • Enhanced distributed replica management, including skipping deleted partitions, colocate group, continuous write balancing failure, and inability to balance cold-hot tiered tables.
  • Stability improvements in complex data types, inverted index, materialized views, import and compaction, Data Lake compatibility, SQL planning, and more.

Enterprise Core 2.0.2

October 8, 2023

  • Improved usability, including optimized priority network matching logic and support for role-based authorization at the row level.
  • Enhanced statistics data collection in the new optimizer, Nereids, including the elimination of file caching during the execution of analysis tasks and support for basic JDBC external table statistics collection.
  • Performance optimization and enhancement in inverted index queries, including the addition of BKD indices for improved efficiency and optimization of multi-word conjunction queries.
  • Improved support for multiple types of data sources in the multi-catalog feature for data lakes, including JDBC, HDFS, Hive, MySQL, MaxCompute, and more.
  • Optimization of array functions, with the array_union now supporting multiple parameters.

Enterprise Core 2.0.1

September 10, 2023

  • Improved the functionality and stability of complex data types such as arrays and maps, including nested complex types in inner tables and nesting outer tables with ORC/Parquet formats.
  • Enhanced performance in inverted index queries, covering tokenization, numerical processing, predicate pushdown, and more.
  • Improved query performance, including enhancements in bitmap operations, LIKE queries, scans, and aggregate functions.
  • Refined and stabilized Cross-Cluster Replication (CCR) functionality.
  • Accelerated and improved accuracy in the collection of statistics by the query optimizer, resulting in enhanced automatic query optimization.
  • Enhanced functionality and performance in the multi-catalog feature for data lakes, including performance optimizations for Iceberg and support for complex data types.

Enterprise Core 1.2.x

Enterprise Core 1.2.8

September 05, 2023

  • Fixed several decimal-related issues.
  • Resolved the problem where "show tables" couldn't display tables for which the user had select permissions.
  • Addressed issues related to replica scheduling.
  • Fixed several query planning problems.
  • Addressed an issue of file handle leakage in BE processes under certain circumstances.
  • Fixed a problem with table creation timing out in certain scenarios.
  • Resolved errors when reading ORC format files.
  • Fixed an issue where closing the FileSystem in Broker caused read errors.
  • Optimized the logic for calculating replica sizes in Auto Bucket.
  • Fixed a NullPointerException issue in Spark Load under certain circumstances.