Enterprise Core
Enterprise Core 2.1.x
Enterprise Core 2.1.5
July 25, 2024
Behavior changes
- The default connection pool size for the JDBC Catalog has been increased from 10 to 30 to prevent connection exhaustion in high-concurrency scenarios.
- The system's reserved memory (low water mark) has been adjusted to min(6.4GB, MemTotal * 5%) to mitigate BE OOM issues.
- When processing multiple statements in a single request, only the last statement's result is returned if the CLIENT_MULTI_STATEMENTS flag is not set.
- Direct modifications to data in asynchronous materialized views are no longer permitted.
- A session variable use_max_length_of_varchar_in_ctas has been added to control the behavior of varchar and char type length generation during CTAS (Create Table As Select). The default value is true. When set to false, the derived varchar length is used instead of the maximum length.
- Statistics collection now defaults to enabling the functionality of estimating the number of rows in Hive tables based on file size.
- Transparent rewrite for asynchronous materialized views is now enabled by default.
- Transparent rewrite utilizes partitioned materialized views. If partitions fail, the base tables are unioned with the materialized view to ensure data correctness.
New features
- Lakehouse: The session variable read_csv_empty_line_as_null can be used to control whether empty lines are ignored when reading CSV format files. By default, empty lines are ignored. When set to true, empty lines will be read as rows where all columns are null.Compatibility with Presto's complex type output format can be enabled by setting serde_dialect="presto".
- Multi-Table Materialized View: Supports non-deterministic functions in materialized view building. Atomically replaces definitions of asynchronous materialized views. Views creation statements can be viewed via SHOW CREATE MATERIALIZED VIEW. Transparent rewrites for multi-dimensional aggregation and non-aggregate queries. Supports DISTINCT aggregations with key columns and partitioning for roll-ups. Support for partitioning materialized views to roll up partitions using date_trunc. Partitioned table-valued functions (TVFs) are supported.
- Semi-Structured Data Management: Tables using the VARIANT type now support partial column updates. PreparedStatement support is now enabled by default. The VARIANT type can be exported to CSV format. explode_json_object function transposes JSON Object rows into columns. The ES Catalog now maps ES NESTED or OBJECT types to the Doris JSON type. By default, support_phrase is enabled for inverted indexes with specified analyzers to improve the performance of match_phrase series queries.
- Query Optimizer: Support for explaining DELETE FROM statements. Support for hint form of constant expression parameters. Memory Management: Added an HTTP API to clear the cache. Permissions: Support for authorization of resources within Table-Valued Functions (TVFs).
Improvements
- Lakehouse: Upgraded Paimon to version 0.8.1.Fixes ClassNotFoundException for org.apache.commons.lang.StringUtils when querying Paimon tables.Added support for Tencent Cloud LakeFS. Optimized the timeout duration when fetching file lists for external table queries. Configurable via the session variable fetch_splits_max_wait_time_ms. Improved default connection logic for SQLServer JDBC Catalog.Added serde properties to the show create table statements for Hive tables.Changed the default cache time for Hive table lists on the FE from 1 day to 4 hours. Data export (Export/Outfile) now supports specifying compression formats for Parquet and ORC.When creating a table using CTAS+TVF, partition columns in the TVF are automatically mapped to Varchar(65533) instead of String, allowing them to be used as partition columns for internal tables. Optimized the number of metadata accesses for Hive write operations. ES Catalog now supports mapping nested/object types to Doris's Json type. Improved error messages when connecting to Oracle using older versions of the ojdbc driver. When Hudi tables return an empty set during Incremental Read, Doris now also returns an empty set instead of error. Fixed an issue where inner-outer table join queries could lead to FE timeouts in some cases. Fixed an issue with FE metadata replay errors during upgrades from older versions to newer versions when the Hive metastore event listener is enabled.
- Multi-Table Materialized View: Automate key column selection for asynchronous materialized views. Support date_trunc in materialized view partition definitions. Enable transparent rewrites across nested materialized view aggregations. Asynchronous materialized views remain available when schema changes do not affect the correctness of their data. Improve planning speed for transparent rewrites. When calculating the availability of asynchronous materialized views, the current refresh status is no longer taken into account.
- Semi-Structured Data Management: Optimize DESC performance for viewing VARIANT sub-columns through sampling. Support for special JSON data with empty keys in the JSON type.
- Inverted Index: Reduce latency by minimizing the invocation of inverted index exists to avoid delays in accessing object storage.Optimize the overhead of the inverted index query process.Prevent inverted indices in materialized views.
- Query Optimizer: When both sides of a comparison expression are literals, the string literal will attempt to convert to the type of the other side. Refactored the sub-path pushdown functionality for the variant type, now better supporting complex pushdown scenarios. Optimized the logic for calculating the cost of materialized views, enabling more accurate selection of lower-cost materialized views. Improved the SQL cache planning speed when using user variables in SQL. Optimized the row estimation logic for NOT NULL expressions, resulting in better performance when NOT NULL is present in queries. Optimized the null rejection derivation logic for LIKE expressions. Improved error messages when querying a specific partition fails, making it clearer which table is causing the issue.
- Query Execution: Improved the performance of the bitmap_union operator up to 3 times in certain scenarios.Enhanced the reading performance of Arrow Flight in ARM environments.Optimized the execution performance of the explode, explode_map, and explode_json functions.
- Data Loading: Support setting max_filter_ratio for INSERT INTO ... FROM TABLE VALUE FUNCTION
Bug fixes
- Various issues have been fixed in areas such as lakehouse, multi-table materialized view, semi-structured data analysis, inverted index, query optimizer, query execution and storage management.
Enterprise Core 2.1.4
June 27, 2024
- Query optimizer supports the FE Flame Graph tool, simultaneous use of SELECT DISTINCT with aggregate functions, rewriting single-table queries without GROUP BY, and high-concurrency point queries.
- Lakehouse integration supports Paimon's native reader to handle Deletion Vectors, using Resources in Table-Valued Functions (TVF), and achieving data masking through the Ranger plugin.
- Asynchronous materialized view construction now supports partition roll-up, trigger-based updates, specifying store_row_column and Storage Medium, and transparent rewriting supports single-table asynchronous materialized views and AGG_STATE type aggregate roll-up.
- Other feature enhancements include the addition of the replace_empty function, support for the show storage policy using statement, and JVM metrics on the BE side.
- Several optimizations have been made, including improving the accuracy of memory estimation consumed by the Segment Cache and supporting the creation of inverted indexes for Chinese column names.
- Various issues have been fixed in areas such as the query optimizer, query execution, materialized views, and semi-structured data analysis.
Enterprise Core 2.1.3
May 17, 2024
- Support INSERT INTO hive table in Hive Catalog.
- Add show views statement to query views.
- Workload group support bind to specific BE hosts.
- Broker Load spport compressed JSON format.
- Truncate function can use column as scale argument.
- Add new function uuid_to_int and int_to_uuid.
- Support create mtmv based on other mtmv.
- Support rewrite by mv nested materialized view.
- Add BypassWorkloadGroup to pass query queue.
- Add function strcmp.
- Support hll functions hll_from_base64, hll_to_base64.
Enterprise Core 2.1.2
April 18, 2024
- Add processlist table in information_schema database, users could use this table to query active connections.
- Add a new table valued function LOCAL to allow access file system like shared storage.
- Set the default value of the data_consistence property of EXPORT to partition to make export more stable during load.
- Some of MySQL Connector (eg, dotnet MySQL.Data) rely on variable's column type to make connection.
- Add rollup table name in profile to help find the mv selection problem.
- Add test connection function to DB2 database to allow user check the connection when create DB2 Catalog.
- Add DNS Cache for FQDN to accelerate the connect process among BEs in K8s env.
- Refresh external table's rowcount async to make the query plan more stable.
Enterprise Core 2.1.1
April 8, 2024
- Change float type output format to improve float type serialization performance.
- Fix issues during rolling upgrade from 2.0.x to 2.1.x, including backend node core problems and JDBC Catalog query errors.
- Enable proxy protocol to support IP transparency. Using this protocol, IP transparency for load balancing can be achieved, so that after load balancing, Doris can still obtain the client's real IP and implement permission control such as whitelisting.
- Add new system table backend_active_tasks to monitor the realtime query statics on every BE.
- Add inverted index support for CCR.
- Support arrow serialization for varint type.
- Fixed 20 bugs, including occasional core issues in the BE during the restore process.
Enterprise Core 2.1.0
March 18, 2024
- Further improvement in the performance of complex SQL queries has been achieved, with over 100% performance enhancement on the TPC-DS 1TB test dataset, positioning query performance at the forefront of the industry.
- Performance improvements in data lake analytics scenarios, with 4-6 times better performance compared to Trino and Spark, have been made. Additionally, compatibility with multiple SQL dialects has been introduced, enabling seamless migration from existing systems to Apache Doris.
- For data science and other forms of large-scale data reading scenarios, a high-speed reading interface based on Arrow Flight has been introduced, resulting in a 100-fold improvement in data transfer efficiency.
- In semi-structured data analysis scenarios, new Variant and IP data types have been introduced, along with enhancements to a series of analytical functions, making storage and analysis of complex semi-structured data more convenient.
- The introduction of asynchronous materialized views based on multiple tables has improved query performance. This includes support for transparent rewriting acceleration, automatic refreshing, external-to-internal table materialized views, and direct querying of materialized views. Leveraging these capabilities, materialized views can also be used for data warehouse tiered modeling, job scheduling, and data processing.
- In terms of data storage, capabilities such as auto-increment columns, automatic partitioning, MemTable forwarding, and server-side batching have been introduced to improve the efficiency of real-time data writing at scale.
- Further improvements have been made in workload management, enhancing the isolation capability of Workload Group resource groups and adding the ability to view SQL resource usage at runtime, thereby enhancing stability in multi-load scenarios.
Enterprise Core 2.0.x
Enterprise Core 2.0.14
August 8, 2024
- Adds a REST interface to retrieve the most recent query profile:
curl http://user:password@127.0.0.1:8030/api/profile/text
. - Optimizes the primary key point query performance for MOW tables with sequence columns.
- Enhances the performance of inverted index queries with many conditions.
- Automatically enables the support_phrase option when creating a tokenized inverted index to accelerate match_phrase phrase queries.
- Supports simplified SQL hints, for example: SELECT /*+ query_timeout(3000) */ * FROM t.
- Automatically retries reading from object storage when encountering a 429 error to improve stability.
- LEFT SEMI / ANTI JOIN terminates subsequent matching execution upon matching a qualifying data row to enhance performance.
- Prevents coredump when returning illegal data to MySQL results.
- Unifies the output of type names in lowercase to maintain compatibility with MySQL and be more friendly to BI tools.
Enterprise Core 2.0.13
July 23, 2024
- SQL input is treated as multiple statements only when the
CLIENT_MULTI_STATEMENTS
setting is enabled on the client side, enhancing compatibility with MySQL. - A new BE configuration allow_zero_date has been added, allowing dates with all zeros. When set to false, 0000-00-00 is parsed as NULL, and when set to true, it is parsed as 0000-01-01. The default value is false to maintain consistency with previous behavior.
- LogicalWindow and LogicalPartitionTopN support multi-field predicate pushdown to improve performance.
- The ES Catalog now maps ES nested or object types to Doris JSON types.
- Queries with LIMIT end reading data earlier to reduce resource consumption and improve performance.
- Special JSON data with empty keys is now supported.
- Stability and usability of routine load have been improved, including load balancing, automatic recovery, exception handling, and more user-friendly error messages.
- BE load balancing selection of hard disk strategy and speed optimization.
- Stability and usability of the JDBC catalog have been improved, including encryption, thread pool connection count configuration, and more user-friendly error messages.
Enterprise Core 2.0.12
July 1, 2024
- No longer set the default table comment to the table type. Instead, set it to be empty by default, for example, change
COMMENT 'OLAP'
toCOMMENT ''
. This new behavior is more friendly for BI software that relies on table comments. - Change the type of the
@@autocommit
variable fromBOOLEAN
toBIGINT
to prevent errors from certain MySQL clients (such as .NET MySQL.Data). - Remove the
disable_nested_complex_type
parameter and allow the creation of nestedARRAY
,MAP
, andSTRUCT
types by default. - The HMS catalog supports the SHOW CREATE DATABASE command.
- Add more inverted index metrics to the query profile.
- Cross-Cluster Replication (CCR) supports inverted indices.
Enterprise Core 2.0.11
June 20, 2024
- Add trino jdbc catalog type mapping for JSON and TIME.
- FE exit when failed to transfer to (non)master to prevent unknown state and too many logs.
- Write audit log while doing drop stats table.
- Ignore min/max column stats if table is partially analyzed to avoid inefficient query plan.
- Support minus operation for set like
set1 - set2
. - Improve perfmance of LIKE and REGEXP clause with concat(col, pattern_str), eg.
col1 LIKE concat('%', col2, '%')
. - Add query options for short circuit queries for upgrade compatibility.
- Since the inverted index is now mature and stable, it can replace the old BITMAP INDEX. Therefore, any newly created
BITMAP INDEX
will automatically switch to anINVERTED INDEX
, while existingBITMAP INDEX
will remain unchanged. Users can disable this automatic switch by setting the FE configurationenable_create_bitmap_index_as_inverted_index
to false.
Enterprise Core 2.0.10
May 20, 2024
- This enhancement introduces the
read_only
andsuper_read_only
variables to the database system, ensuring compatibility with MySQL's read-only modes. - When the check status is not IO_ERROR, the disk path should not be added to the broken list. This ensures that only disks with actual I/O errors are marked as broken.
- When performing a Create Table As Select (CTAS) operation from an external table, convert the
varchar
column tostring
type. - Support mapping Paimon column type "Row" to Doris type "Struct"
- Choose disk tolerate with little skew when creating tablet
- Write editlog for
set replica drop
to avoid confusing status on follower FE
Enterprise Core 2.0.9
April 24, 2024
- Allow predicate appearing on both key and value mv columns
- Enable mv with bitmap_union(bitmap_from_array())
- Introduce a configuration to forcibly replicate allocation for all olap tables within the cluster
- Add timezone support for date literals in the new optimizer Nereids
- Enable the use of "slop" in full-text search's match_phrase to specify word distances
- Display index ID in SHOW PROC INDEXES
- Incorporate a secondary argument in first_value/last_value to overlook NULL values
- Allow the use of 0 as the offset parameter in the lead/lag function
Enterprise Core 2.0.8
April 12, 2024
- Make Inverted Index work with TopN opt in Nereids
- Limit the max string length to 1024 while collecting column stats to control BE memory usage
- JDBC Catalog close when JDBC client is not empty
- Accept all Iceberg database and do not check the name format of database
- Refresh external table's rowcount async to avoid cache miss and unstable query plan
- Simplify the isSplitable method of hive external table to avoid too many hadoop metrics
Enterprise Core 2.0.7
March 26, 2024
- Support make miss slot as null alias when converting outer join to anti join to speed up query.
- Add DEFAULT_ENCRYPTION column in information_schema table and add processlist table for better compatibility for BI tools.
- Automatically test connectivity by default when creating a JDBC Catalog.
- Enhance auto resume to keep routine load stable.
- Use lowercase by default for Chinese tokenizer in inverted index.
- Add error msg if exceeded maximum default value in repeat function.
- Skip hidden file and dir in hive table.
- Reduce file meta cache size and disable cache for some cases to avoid OOM.
- Reduce jvm heap memory consumed by profiles of BrokerLoadJob.
- Remove sort which is under table sink to speed up query like INSERT INTO t1 SELECT * FROM t2 ORDER BY k.
Enterprise Core 2.0.6
March 14, 2024
- Support match a function with alias in materialized-view.
- Add a command to drop a tablet replica safely on backend.
- Add row count cache for external table.
- Support analyze rollup to gather statistics for optimizer.
- Improve tablet schema cache memory by using deterministic way to serialize protobuf.
- Improve show column stats performance.
- Support estimate row count for iceberg and paimon.
- Support sqlserver timestamp type read for JDBC catalog.
Enterprise Core 2.0.3
December 8, 2023
- Supports automatic statistics collection, which aids the optimizer in understanding data distribution characteristics. This enables the selection of more optimal plans, significantly improving query efficiency. Starting from version 2.0.3, this feature is officially supported and is enabled by default throughout the day.
- Data Lake supports JAVA UDF, JDBC, Hudi MOR, and more system support for complex data types.
- Cross-Cluster Replication (CCR) now supports features like throttling and table truncation enhancements.
- Additional built-in functions such as SHA, JSON have been added.
- Over 20 performance improvements including inverted index, case when, predicate pushdown, etc.
- Enhanced distributed replica management, including skipping deleted partitions, colocate group, continuous write balancing failure, and inability to balance cold-hot tiered tables.
- Stability improvements in complex data types, inverted index, materialized views, import and compaction, Data Lake compatibility, SQL planning, and more.
Enterprise Core 2.0.2
October 8, 2023
- Improved usability, including optimized priority network matching logic and support for role-based authorization at the row level.
- Enhanced statistics data collection in the new optimizer, Nereids, including the elimination of file caching during the execution of analysis tasks and support for basic JDBC external table statistics collection.
- Performance optimization and enhancement in inverted index queries, including the addition of BKD indices for improved efficiency and optimization of multi-word conjunction queries.
- Improved support for multiple types of data sources in the multi-catalog feature for data lakes, including JDBC, HDFS, Hive, MySQL, MaxCompute, and more.
- Optimization of array functions, with the array_union now supporting multiple parameters.
Enterprise Core 2.0.1
September 10, 2023
- Improved the functionality and stability of complex data types such as arrays and maps, including nested complex types in inner tables and nesting outer tables with ORC/Parquet formats.
- Enhanced performance in inverted index queries, covering tokenization, numerical processing, predicate pushdown, and more.
- Improved query performance, including enhancements in bitmap operations, LIKE queries, scans, and aggregate functions.
- Refined and stabilized Cross-Cluster Replication (CCR) functionality.
- Accelerated and improved accuracy in the collection of statistics by the query optimizer, resulting in enhanced automatic query optimization.
- Enhanced functionality and performance in the multi-catalog feature for data lakes, including performance optimizations for Iceberg and support for complex data types.
Enterprise Core 1.2.x
Enterprise Core 1.2.8
September 05, 2023
- Fixed several decimal-related issues.
- Resolved the problem where "show tables" couldn't display tables for which the user had select permissions.
- Addressed issues related to replica scheduling.
- Fixed several query planning problems.
- Addressed an issue of file handle leakage in BE processes under certain circumstances.
- Fixed a problem with table creation timing out in certain scenarios.
- Resolved errors when reading ORC format files.
- Fixed an issue where closing the FileSystem in Broker caused read errors.
- Optimized the logic for calculating replica sizes in Auto Bucket.
- Fixed a NullPointerException issue in Spark Load under certain circumstances.