Release Note 2.1.6 #40443
Description
Search before asking
- I had searched in the issues and found no similar issues.
Description
Behavior changes
- Removed the delete_if_exists option from create repository. [chore](backup) Remove delete_if_exists properties for creating repository #38192
- Added the enable_prepared_stmt_audit_log session variable to control whether JDBC prepared statements record audit logs, with the default being no recording. [Cherry-pick](branch-2.1) Pick "[Enhancement](audit log) Add print audit log sesssion variable #38419" #38624 [cherry-pick](branch-2.1) Pick "[Enhancement](audit log) Set print audit log sesssion variable default value to false #38865" #39009
- Implemented fd limit and memory constraints for segment cache. [improvement](segmentcache) limit segment cache by fd limit or memory… #39689
- When the FE configuration item sys_log_mode is set to BRIEF, file location information is added to the logs. ([fix](fe-log) add position info in async mode #39419 #39571)
- Changed the default value of the session variable max_allowed_packet to 16MB. ([opt](variables) enlarge the default value of max_allowed_packet #38697)
- When a single request contains multiple statements, semicolons must be used to separate them. [fix](syntax) multi statements must delim with semicolon #38670
- Added support for statements to begin with a semicolon. [opt](Nereids) support parse sql with semicolon at beginning #39399
- Aligned type formatting with MySQL in statements such as show create table. [fix](compatibility) type toSql should return lowercase string #38012
- When the new optimizer planning times out, it no longer falls back to prevent the old optimizer from using longer planning times. [opt](Nereids) do not fallback if nereids failed because timeout #39499
New features
Lakehouse
- Supported writeback for Iceberg tables: https://doris.apache.org/zh-CN/docs/2.1/lakehouse/datalake-building/iceberg-build
- SQL interception rules now support external tables: https://doris.apache.org/zh-CN/docs/2.1/admin-manual/query-admin/sql-interception
- Added the system table file_cache_statistics to view BE data cache metrics: https://doris.apache.org/docs/2.1/admin-manual/system-tables/file_cache_statistics
Async Materialized View - Supported transparent rewriting during inserts: [feature](mtmv) Support querying rewrite by materialized view when insert and insert overwrite dml #38115
- Supported transparent rewriting when variant types exist in queries: [feature](mtmv) Support variants rewrite by materialized view #37929
Semi-Structured Data Management - Supported casting ARRAY MAP to JSON type: [feature](cast) Support cast complex type to json #36548
- Supported the json_keys function: [Feature](json_functions) support json-keys #36411
- Supported specifying the JSON path $. when importing JSON: [Feature](JsonReader) support sepecify
$.as root column in json path #38213 - ARRAY MAP STRUCT types now support replace_if_not_null: [improve](nested_types)support nested type with agg replace_if_not_null #38304
- ARRAY MAP STRUCT types now support adjusting column order: [improve](nestedtype) support schema change for nested type #39210
- Added the multi_match function to match keywords across multiple fields, with support for inverted index acceleration: [feature](inverted index) multi_match function add #37722
Query Optimizer
- Filled in the original database name, table name, column name, and alias for returned columns in the MySQL protocol. [opt](compatibility) fill schema and table name in protocol column def #38126
- Supported the aggregation function group_concat with both order by and distinct simultaneously. [fix](nereids)support group_concat with distinct and order by #38080
- SQL cache now supports reusing cached results for queries with different comments. [enhancement](Nereids) support reuse sql cache between different comment #40049
- In partition pruning, supported including date_trunc and date functions in filter conditions. [Feat](nereids)Support date_trunc function in partition prune #38025 [Feat](nereids) support date function in partition prune #38743
- Allowed using the database name where the table resides as a qualifier prefix for table aliases. [chore](nereids) Added compatibility with mysql alias filter #38640
- Supported hint-style comments. [feat](Nereids) support nereids hint position detaction #39113
Others - Added the system table table_properties for viewing table properties: https://doris.apache.org/docs/2.1/admin-manual/system-tables/table_properties
- Introduced deadlock and slow lock detection in FE: https://doris.apache.org/zh-CN/docs/2.1/admin-manual/maint-monitor/frontend-lock-manager
Improvements
Lakehouse
- Reimplemented the external table metadata caching mechanism. For details, refer to the documentation: https://doris.apache.org/zh-CN/docs/2.1/lakehouse/metacache/
- Added the session variable keep_carriage_return with a default value of false. By default, reading Hive Text format tables treats both \r\n and \n as newline characters. ([feature](csv)Supports reading CSV data using LF and CRLF as line separators. (#37687) #38099)
- Optimized memory statistics for Parquet/ORC file read/write operations. ([Feature](multi-catalog) Add memory tracker for orc reader/writer and arrow parquet writer。 #37257)
- Supported pushing down IN/NOT IN predicates for Paimon tables. ([bugfix](paimon)add support for 'in' and 'not in' #38390)
- Enhanced the optimizer to support Time Travel syntax for Hudi tables. ([bugfix](external)Modify the default value of
pushdowncountfor 2.1 (#37754) #38591) - Optimized Kerberos authentication-related processes. ([fix](kerberos)fix and refactor ugi login for kerberos and simple authentication #37301)
- Enabled reading Hive tables after Rename column operations. ([feature](hive)Support reading renamed Parquet Hive and Orc Hive tables. (#38432) #38809)
- Optimized the reading performance of partition columns for external tables. ([opt](serde)Optimize the filling of fixed values into block columns without repeated deserialization. (#37377) (#38245) #38810)
- Improved the data shard merging strategy during external table query planning to avoid performance degradation caused by a large number of small shards. ([opt](catalog) merge scan range to avoid too many splits (#38311) #38964)
- Added attributes such as location to SHOW CREATE DATABASE/TABLE. ([improvement](iceberg)add some description for show create for 2.1 #39179 #39644)
- Supported complex types in MaxCompute Catalog. ([fix](multi-catalog)fix max compute array and map type read offset #39822)
- Optimized the file cache loading strategy by using asynchronous loading to avoid long BE startup times. ([improve](cache) File cache async init #39036)
- Improved the file cache eviction strategy, such as evicting locks held for extended periods. ([opt](file-cache) add evict file number per round #39721)
Async Materialized View
- Supported hourly, weekly, and quarterly partition roll-up construction. [enhance](mtmv)mtmv date trunc support hour #37678
- For materialized views based on Hive external tables, the metadata cache is now updated before refresh to ensure the latest data is obtained during each refresh. [enhance](mtmv)refresh hms table before run mtmv task #38212
- Improved the performance of transparent rewrite planning in storage-compute separation mode by batch fetching metadata. [enhance](mtmv)Improve the performance of obtaining partition/table versions #39301
- Enhanced the performance of transparent rewrite planning by prohibiting duplicate enumerations. [improvement](mtmv) Only Generate rewritten plan when generate mv plan for performance #39541
- Improved the performance of transparent rewrite for refreshing materialized views based on Hive external table partitions. [enhance](mtmv)hive cache add partitionId to partitionName Map #38525
Semi-Structured Data Management
- Optimized memory allocation for TOPN queries to improve performance. [opt](inverted index) Optimization of the initialization process in topn #37429
- Enhanced the performance of string processing in inverted indexes. [opt](inverted index) opt value extraction from column to string #37395
- Optimized the performance of inverted indexes in MOW tables. [opt](inverted index) mow supports index optimization #37428
- Supported specifying the row-store page_size during table creation to control compression effectiveness. [Optimize](Row store) allow to set row_store_page_size for tables, change default value to 16KB #37145
Query Optimizer
- Adjusted the row count estimation algorithm for mark joins, resulting in more accurate cardinality estimates for mark joins. [fix](nereids) refine row count estimation for mark join #38270
- Optimized the cost estimation algorithm for semi/anti joins, enabling more accurate selection of semi/anti join orders. [opt](nereids) refine left semi/anti cost under short-cut opt #37951
- Adjusted the filter estimation algorithm for cases where some columns have no statistical information, leading to more accurate cardinality estimates. [fix](nereids)filter estimation for slot=unknown #39592
- Modified the instance calculation logic for set operation operators to prevent insufficient parallelism in extreme cases. [fix](nereids) fix union all instance number #39999
- Adjusted the usage strategy of bucket shuffle, achieving better performance when data is not sufficiently shuffled. [opt](nereids) recover adoptive bucket shuffle #36784
- Enabled early filtering of window function data, supporting multiple window functions in a single projection. [opt](nereids) support partitionTopn for multi window exprs #38393
- When a NullLiteral exists in a filter condition, it can now be folded into False, further converted to an EmptySet to reduce unnecessary data scanning and computation. [improvement](Nereids) add where Null rule to create empty relation as where false #38135
- Expanded the scope of predicate derivation, reducing data scanning in queries with specific patterns. [Feat](nereids) add pull up literal when infer predicates #37314
- Supported partial short-circuit evaluation logic in partition pruning to improve partition pruning performance, achieving over 100% improvement in specific scenarios. [enhancement](nereids) Speedup partition pruner #38191
- Enabled the computation of arbitrary scalar functions within user variables. [fix](variable) support all type functions #39144
- Maintained error messages consistent with MySQL when alias conflicts exist in queries. [chore](nereids) Added compatibility with mysql alias conflict #38104
Query Execution
- Adapted AggState for compatibility from 2.1 to 3.x and fixed coredump issues. [Compatibility](agg-state) do not throw eception when DataTypeAggState meet old be exec version #37104
- Refactored the strategy selection for local shuffle when no joins are involved. [Improvement](scan) Refine ignoring data distribution without join #37282
- Modified the scanner for internal table queries to an asynchronous approach to prevent blocking during internal table queries. [Improvement](schema scan) Use async scanner for schema scanners #38403
- Optimized the block merge process when building hash tables in Join operators. [opt](hash_join) Merging all build blocks at once will cause performance issue #37471
- Reduced the lock holding time for MultiCast operations. [opt](MultiCast) Avoid copying while holding a lock #37462
- Optimized gRPC's keepAliveTime and added a connection monitoring mechanism, reducing the probability of query failures due to RPC errors during query execution. [Improvement](rpc) set grpc channel's keepAliveTime and remove proxy on InterruptedExcep… #37304
- Cleaned up all dirty pages in jemalloc when memory limits are exceeded. [opt](memory) Purge all unused Jemalloc dirty pages for all arenas when exceed limit #37164
- Improved the performance of aes_encrypt/decrypt functions when handling constant types. [improve](function) opt aes_encrypt/decrypt function to handle const column #37194
- Optimized the performance of json_extract functions when processing constant data. [opt](fucntion) improve json_extract function handle const column #36927
- Optimized the performance of ParseUrl functions when processing constant data. [opt](function) opt ParseUrl function by process the input const column #36882
Backup Recovery / Cross-Cluster Synchronization
- Restore now supports deleting redundant tablets and partition options. [feature](restore) Support clean_tables/clean_partitions properties for restore job #39028 #39363
- Checks storage connectivity when creating a repository. [enhance](Backup) Do connectivity check when creating repository (#38350) #39538
- Enables binlog to support DROP TABLE, allowing CCR to incrementally synchronize DROP TABLE operations. [chore](fe) Returns dropped tables in GetMeta request #38541
Compaction
- Improves the issue where high-priority compaction tasks were not subject to task concurrency control limits. [enhancement](compaction) Control the parallelism for urgent compacton tasks (#37782) #38189
- Automatically reduces compaction memory consumption based on data characteristics. [enhancement](compaction) optimizing memory usage for compaction (#37099) #37486
- Fixes an issue where the sequential data optimization strategy could lead to incorrect data in aggregate tables or MOR UNIQUE tables. [fix](compaction) fix duplicate key in agg/mor table caused by ordered data compaction (#38224) #38299
- Optimizes the rowset selection strategy during compaction during replica replenishment to avoid triggering -235 errors. [fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets (#38728) #39262
MOW (Merge-On-Write)
- Optimizes slow column updates caused by concurrent column updates and compactions. [branch-2.1] Picks "[opt](merge-on-write) Skip the alignment process of some rowsets in partial update #38487" #38682
- Fixes an issue where segcompaction during bulk data imports could lead to incorrect MOW data. [fix](MoW) fix MoW & segcompaction conflict on cache of temp segment … #38992 [fix](merge-on-write) segcompaction should process delete bitmap if necessary (#38369) #39707
- Fixes data loss in column updates that may occur after BE restarts. [branch-2.1] Picks "[Fix](partial update) Persist partial_update_info in RocksDB in case of BE restart after a partial update has commited #38331" #39035
Storage Management
- Adds FE configuration to control whether queries under hot-cold tiering prefer local data replicas. Pick "[enhance](Cooldown) Use config to control whether use cooldown replica for scanning first" #38322
- Optimizes expired BE report messages to include newly created tablets. [fix](clone) fix stale tablet report miss the new cloning replica #38695 #38839 [improvement](report) report handler discard old report tasks #39469 #39605
- Optimizes replica scheduling priority strategy to prioritize replicas with missing data. [improvement](tablet scheduler) Adjust tablet sched priority to help load data succ #38528 #38884
- Prevents tablets with unfinished ALTER jobs from being balanced. [improvement](balance) don't balance tablet which has unfinish alter job #39121 #39202
- Enables modifying the number of buckets for tables with list partitioning. [fix](fe) LIST partition table support modify default bucket num #39688
- Prefers querying from online disk services. [improvement](query) prefer to chose tablet on alive disk #39467 #39654
- Improves error messages for materialized view base tables that do not support deletion during synchronization. [enhancement](delete) Add a hint msg for forbidden delete when MV or rollup exists (#39505) #39857
- Improves error messages for single columns exceeding 4GB. [improvemen](overflow) Provide the user with a suggestion to avoid th… (#39631) #39897
- Fixes an issue where aborted transactions were omitted when plan errors occurred during INSERT statements. [fix](insert) Pick Insert init plan error may forget to abort txn (#34220) #38260
- Fixes exceptions during SSL connection closure. [cherry-pick](SSL) Fix ssl connection close 2.1 (#38587) #38677
- Fixes an issue where table locks were not held when aborting transactions using labels. [fix](txn) fix abortTxn by label does not acquire table write lock (#38777) #38842
- Fixes gson pretty causing large image issues. [fix](gson) Fix GSON pretty printting setting #37211 #39135
- Fixes an issue where the new optimizer did not check for bucket values of 0 in CREATE TABLE statements. [fix](create table) fix nereids create table no check buckets=0 #38971 #38999
- Fixes errors when Chinese column names are included in DELETE condition predicates. [fix](delete) Only apply regex check on delete condition str for non-lsc tables (#39357) #39500
- Fixes frequent tablet balancing issues in partition balancing mode. [fix](partition rebalancer) fix migrate tablets between backends back and forth #39333 #39606
- Fixes an issue where partition storage policy attributes were lost. [BUG] fix partition storage policy info lost (#38700) #39677
- Fixes incorrect statistics when importing multiple tables within a transaction. [fix](statistics) correct update rows when doing multi-table load #39548
- Fixes errors when deleting random bucket tables. [cherry-pick](branch-2.1) fix delete random distributed tbl #39830
- Fixes issues where FE fails to start due to non-existent UDFs. [fix](udf) Fix exception thrown when replayDropFunction #39868
- Fixes inconsistencies in the last failed version between FE master and slave. [fix](replay) fix replay ReplicaPersistInfo no update lastFailedVersion and lastSuccVersion #39918 #39947
- Fixes an issue where related tablets may still be in schema change state when schema change jobs are canceled. [fix](schema-change) Fix potential data race when a schema change jobs is set to cancelled but the table state is still SCHEMA_CHANGE (#39164) #39327
- Fixes errors when modifying type and column order in a single statement schema change (SC). [fix](schema-change) Fix wrong intput column for cast validity check (#38894) #39107
Data Loading
- Improves error messages for -238 errors during imports. [chore](rowset writer) print rowset rows number when meet too many segments (#39091) #39182
- Allows importing to other partitions while restoring a partition. [enhancement](Load)allow load data to the other partitions when some partitions are restoring #39915
- Optimizes the strategy for FE to select BEs during group commit. [cherry-pick](branch-2.1) Pick "[Enhancement](group commit)Optimize be select for group commit #35558" #37830 [cherry-pick](branch-2.1) Pick "[Fix](group commit) Fix multiple cluster group commit BE select strategy (#38644)" #39010
- Avoids printing stack traces for some common streamload error messages. [Opt](load) don't print stack when some errors occur for stream load #38332 #38418
- Improves handling of issues where offline BEs may affect import errors. [fix](load data) decommission replica don't load data when it misses versions #38198 #38256
Permissions
- Optimizes access performance after enabling the Ranger authentication plugin. ([enhance](auth)support cache ranger datamask and row filter (#37723) #38575)
- Optimizes permission strategies for Refresh Catalog/Database/Table operations, allowing users to perform these operations with only SHOW permissions. ([enhance](auth)modify priv of refresh catalog/db/table #39008)
Bug fixes
Lakehouse
- Fixes the issue where switching catalogs may result in an error of not finding the database. ([fix](multicatalog) make lastdbofcatalog a session variable #38114)
- Addresses exceptions caused by attempting to read non-existent data on S3. ([fix](external) record not found file number #38253)
- Resolves the issue where specifying an abnormal path during export operations may lead to incorrect export locations. ([Fix](outfile) FE check the hdfs URI of outfile #38602)
- Fixes the timezone issue for time columns in Paimon tables. ([bugfix](paimon)Fixed the reading of timestamp with time zone type data #37716)
- Temporarily disables the Parquet PageIndex feature to avoid certain erroneous behaviors.
- Corrects the selection of Backend nodes in the blacklist during external table queries. ([fix](catalog) remove backend in black list from candidate backends for external table #38984)
- Resolves errors caused by missing subcolumns in Parquet Struct column types. ([bug](parquet)Fix the problem that the parquet reader reads the missing sub-columns of the struct and fails. (#38718) #39192)
- Addresses several issues with predicate pushdown in Jdbc Catalog. ([branch-2.1][fix](expr) Enhance SQL Expression Handling by Introducing printSqlInParens to CompoundPredicate #39082)
- Fixes issues where some historical Parquet formats led to incorrect query results. ([Fix](parquet-reader) Fix and optimize parquet min-max filtering. #39375)
- Improves compatibility with ojdbc6 drivers for Oracle JDBC Catalog. ([branch-2.1][improvement](jdbc catalog) Compatible with ojdbc6 by adding version check #39408)
- Resolves potential FE memory leaks caused by Refresh Catalog/Database/Table operations. ([improvement](external catalog)Optimize the process of refreshing catalog for 2.1 (#39205) #39186) ([fix](ctas) fix NPE when ctas with old planner and varchar issue (#39744) #39871)
- Fixes thread leaks in JDBC Catalog under certain conditions. ([branch-2.1][improvement](jdbc catalog) Force all resources to be closed in the close method #39666) ([fix](jdbc catalog) Fix Memory Leak by Enabling Weak References in HikariCP #39582)
- Addresses potential event processing failures after enabling Hive Metastore event subscription. ([fix](hive)Modify the Hive notification event processing method when using meta cache and add parameters to the Hive catalog. #39239)
- Disables reading Hive Text format tables with custom escape characters and null formats to prevent data errors. ([fix](hive) report error with escape char and null format (#39700) #39869)
- Resolves issues accessing Iceberg tables created via the Iceberg API under certain conditions. ([fix](catalog) refactor location path and support default fs #39116 #39203)
- Fixes the inability to read Paimon tables stored on HDFS clusters with high availability enabled. ([fix](paimon) fix not able to read paimon data from hdfs with HA (#39806) #39876)
- Addresses errors that may occur when reading Paimon table Deletion Vectors after enabling file caching. ([fix](paimon) fix crash when enable cache with paimon deletion vector(#39877) #39875)
- Resolves potential deadlocks when reading Parquet files under certain conditions. ([Fix](parquet-reader) Fix definition level rle decode dead loop in parquet-reader. (#39523) #39945)
Async Materialized View
- Fixes the inability to use SHOW CREATE MATERIALIZED VIEW on follower FEs. ([fix](mtmv)fix can not show create mtmv use follower fe #38794)
- Unifies the object type of asynchronous materialized views in metadata as tables to enable proper display in data tools. ([enhance](mtmv)change mysql table type of mtmv to table #38797)
- Resolves the issue where nested asynchronous materialized views always perform full refreshes. ([fix](mtmv)fix second level MTMV will refresh all partition #38698)
- Fixes the issue where canceled tasks may show as running after restarting FEs. ([fix](mtmv)Fix cancelled tasks with running status #39424)
- Addresses incorrect use of contexts, which may lead to unexpected failures of materialized view refresh tasks. ([fix](mtmv) mtmv getPlanBySql should not reuse ctx's StatementContext #39690)
- Resolves issues that may cause varchar type write failures due to unreasonable lengths when creating asynchronous materialized views based on external tables. ([fix](mtmv)transfer col in mysql varchar to text when create MTMV #37668)
- Fixes the potential invalidation of asynchronous materialized views based on external tables after FE restarts or catalog rebuilds. ([fix](mtmv)use name instead of id in meta of MTMV #39355)
- Prohibits the use of partition rollup for materialized views with list partitions to prevent the generation of incorrect data. ([enhance](mtmv)Disable mtmv list rollup #38124)
- Fixes incorrect results when literals exist in the select list during transparent rewriting for aggregation rollup. ([fix](mtmv) Fix select literal result wrongly in group by when use materialized view #38958)
- Addresses potential errors during transparent rewriting when queries contain filters like a = a. ([fix](mtmv) Fix result wrong when query rewrite by mv if query contains null_unsafe equals expression #39629)
- Fixes issues where transparent rewriting for direct external table queries fails. ([fix](mtmv) Fix rewrite fail when query direct external table without group by #39041)
Semi-Structured Data Management
- Removes support for prepared statements in the old optimizer. ([Refactor](Prepared Statement) remove implementation in legacy planner #39465)
- Fixes issues with JSON escape character handling. ([fix](serde)fix string deserialize with unescaped char #37251)
- Resolves issues with duplicate processing of JSON fields. ([fix](simdjson) fix simdjson reader will parse given root twice when data is object array #38490)
- Fixes issues with some ARRAY and MAP functions. ([fix](arrayfuncs) fix array_min/array_max func #39307, [fix](funcs) map struct construct funcs #39699, [fix](table-func) fix explode_map with alias #39757)
- Resolves complex combinations of inverted index queries and LIKE queries. ([Fix](inverted index) fix wrong no need read data when same column in inverted index and like function #36687)
Query Optimizer
- Fixed the potential partition pruning error issue when 'OR' condition exists in partition filter conditions. [Fix](nereids) fix partition_prune or expression evaluate wrongly #38897
- Fixed the potential partition pruning error issue when complex expressions are involved. [Fix](nereids) fix condition function partition prune #39298
- Fixed the issue where nullable in agg_state subtypes might be planned incorrectly, leading to execution errors. [fix](agg_state) adjust nullable should apply on agg_state inner type too #37489
- Fixed the issue where nullable in set operation operators might be planned incorrectly, leading to execution errors. [fix](Nereids) set operation output nullable maybe wrong #39109
- Fixed the incorrect execution priority issue of intersect operator. [opt](set operation) INTERSECT should evaluated before others #39095
- Fixed the NPE issue that may occur when the maximum valid date literal exists in the query. [fix](nereids)prevent null pointer exception if datetime value overflows #39482
- Fixed the occasional planning error that results in an illegal slot error during execution. [fix](Nereids) should push project through limit after eliminate union node #39640
- Fixed the issue where repeatedly referencing columns in cte may lead to missing data in some columns in the result. [fix](Nereids) producer to consumer should be multimap in cte #39850
- Fixed the occasional planning error issue when 'case when' exists in the query. [fix](nereids) fix The children format needs to be [WhenClause+, DefaultValue?] #38491
- Fixed the issue where IP types cannot be implicitly converted to string types. [fix](Nereids) support implicit cast ip types to string #39318
- Fixed the potential planning error issue when using multi-dimensional aggregation and the same column and its alias exist in the select list. [Fix](nereids) fix normalize repeat alias rewrite #38166
- Fixed the issue where boolean types might be handled incorrectly when using be constant folding. [fix](Nereids) cast to boolean wrong when constant folding by be #39019
- Fixed the planning error issue caused by 'default_cluster:' as a prefix for the database name in expressions. [Fix](nereids) fix bind expression compare dbname ignore cluster #39114
- Fixed the potential deadlock issue caused by 'insert into'. [fix](Nereids) remove db readlock before get table from db #38660
- Fixed the potential planning error issue caused by not holding table locks throughout the planning process. [fix](Nereids) lock table when generate distribute plan #38950
- Fixed the issue where char(0), varchar(0) are not handled correctly when creating tables. [Fix](nereids) change char(0) to char(1), varchar(0) to varchar(65533) when create table #38427
- Fixed the issue where 'show create table' may incorrectly display hidden columns. [chore](sql) Forbid show hidden columns and create table with hidden column #38796
- Fixed the issue where columns with the same name as hidden columns are not prohibited when creating tables. [chore](sql) Forbid show hidden columns and create table with hidden column #38796
- Fixed the occasional planning error issue when executing 'insert into as select' with CTEs. [fix](nereids) fix insert stmt throw MultiCastDataSink cannot be cast to DataStreamSink #38526
- Fixed the issue where 'insert into values' cannot automatically fill null default values. [fix](Nereids) fix insert into table with null literal default value #39122
- Fixed the NPE issue caused by using cte in delete without using using. [fix](Nereids) npe when delete with cte and without using #39379
- Fixed the issue where deleting from a randomly distributed aggregation model table fails. [bug](delete) fix delete random distributed tbl #37985
Query Execution
- Fixed the issue where the pipeline execution engine gets stuck in multiple scenarios, causing queries not to end. [fix](local exchange) Fix local exchange blocked by a huge data block #38657 [fix](pipeline) Fix blocked tasks if query is canceled before opening #38206 [fix](pipeline) Fix mem control in local exchanger #38885
- Fixed the coredump issue caused by null and non-null columns in set difference calculations. [Bug](fix) fix coredump case in (not null, null) execpt (not null, t null) case #38737
- Fixed the incorrect result issue of the width_bucket function. [fix](function) the bucket number arg of width_bucket should be a positive integer value #37892
- Fixed the query error issue when a single row of data is large and the result set is also large (exceeding 2GB). [fix](sink) The issue with 2GB limit of protocol buffer #37990
- Fixed the incorrect result issue of stddev with DecimalV2 type. [fix](function) stddev with DecimalV2 type will result in an error #38731
- Fixed the coredump issue caused by the MULTI_MATCH_ANY function. [Fix](function) fix coredump for MULTI_MATCH_ANY #37959
- Fixed the issue where insert overwrite auto partition causes transaction rollback. [Fix](insert-overwrite) Fix insert overwrite auto detect transaction safe #38103
- Fixed the incorrect result issue of the convert_tz function. [fix](function) fix wrong result when convert_tz is out of bound #37358 [fix](function) Fix wrong nullable processing for convert_tz #38764
- Fixed the coredump issue when using the collect_set function with window functions. [Bug](agg) fix collect_set function core dump without arena pool #38234
- Fixed the coredump issue caused by the mod function with abnormal input. [Bug](function) fix mod function cause core dump #37999
- Fixed the issue where executing the same expression in multiple threads may lead to incorrect Java UDF results. [bug](expr)fix operator do_projections should use local_state intermediate_projections #38612
- Fixed the overflow issue caused by the incorrect return type of the conv function. [bug](function) fix conv function get wrong result as parse overflow #38001
- Fixed the unstable result issue of the histogram function. [fix](hist) Fix unstable result of aggregrate function hist #38608
Backup & Recovery / CCR
- Fixed the issue where the data version after backup and recovery may be incorrect, leading to unreadability. [fix](restore) Reset next version for restored partitions #38343
- Fixed the issue of using restore version across versions. [fix](backup) Read BackupMeta with the specified meta version #38396
- Fixed the issue where the job is not canceled when backup fails. [bugfix](backup)(cooldown) cancel backup properly when be backup failed (#38724) #38993
- Fixed the NPE issue in ccr during the upgrade from 2.1.4 to 2.1.5, causing the FE to fail to start. [fix](binlog) Fix NPE when recover binlogs (#39909) #39910
- Fixed the issue where views and materialized views cannot be used after restore. [chore](backup) Fix the db name of the restored view #38072 [fix](restore) Reset the db name of the materialized index stmt #39710 #39848
Storage Management
- Fixed possible memory leaks in routineload when loading multiple tables from a single stream. [fix](multi table) fix single stream multi table memory leak (#38255) #38824
- Fixed the issue where delimiters and escape characters in routineload were not effective. [fix](routine load) fix enclose and escape can not set in routine load job (#38402) #38825
- Fixed incorrect show routineload results when the routineload task name contained uppercase letters. [fix](routine load) fix show routine load task result incorrect (#38523) #38826
- Fixed the issue where the offset cache was not reset when changing the routineload topic. [fix](routine load) reset Kafka progress cache when routine load job topic change #38474
- Fixed the potential exception triggered by show routineload under concurrent scenarios. [fix](routine load) add read lock to fix some concurrent bugs (#39242) #39525
- Fixed the issue where routineload might import data repeatedly. [fix](routine load) should update progress before handle transaction state transform (#39311) #39526
- Fixed the data error caused by setNull when enabling group commit via JDBC. [fix](group commit) Fix prepare stmt setNull return too many filtered rows error (#38262) #38276
- Fixed the potential NPE issue when enabling group commit insert to a non-master FE. [cherry-pick](branch-2.1) Fix some group commit forward to master problems #38345
- Fixed incorrect error handling during internal data writing in group commit. [fix](group commit) fix repaly wal check label status (#38883) #38997
- Fixed the coredump that might be triggered when the group commit execution plan failed. [fix](group commit) fix group commit core if be inject FragmentMgr.exec_plan_fragment.failed (#39339) #39396
- Fixed the issue where concurrent imports into auto partition tables might report non-existent tablets. [branch-2.1](load) fix miss writer in concurrency incremental open (#38605) #38793
- Fixed potential load stream leakage issues. [fix](move-memtable) close stream when cancel load stream stub (#38912) #39039
- Fixed the issue where transactions were opened for insert into select with no data. [fix](planner) Empty table source should not begin a transaction to avoid infinite transaction (#38991) #39108
- Ignored the single-replica import configuration when using memtable prefetching. [fix](move-memtable) ignore single replica load when move memtable (#32845) #39154
- Fixed the issue where background imports of stream load records might be abnormally aborted upon encountering db deletion. [fix](stream load) do not throw exception but skip record when can not find database (#39360) #39527
- Fixed inaccurate error messages when data errors occurred in strict mode. [Fix](load) Fix the incorrect src value printed in the error log when strict mode is true #39447 #39587
- Fixed the issue where streamload did not return an error URL upon encountering erroneous data. [fix](load) fix no error url for stream load #38325 #38417
- Fixed the issue with the combined use of insert overwrite and auto partition. [branch-2.1](insert-overwrite) Fix insert overwrite auto detect transaction safe (#38103) #38442
- Fixed parsing errors when CSV encountered data where the line delimiter was enclosed by the enclosing character. [fix](csv reader) fix csv parser incorrect if enclosing line_delimiter (#38347) #38445
Data Exporting
- Fixed the issue where enabling the delete_existing_files property during export operations might result in duplicate deletion of exported data. ([Fix](Export) Export delete multiple times when specify the
delete_existing_filesproperty () #39304)
Permissions
- Fixed the incorrect requirement of ALTER TABLE permission when creating a materialized view. [fix](auth)fix create mv not need alter priv of table #38011
- Fixed the issue where the db was explicitly displayed as empty when showing routine load. [fix](auth)fix show routine load db is null #38365
- Fixed the incorrect requirement of CREATE permission on the original table when using CREATE TABLE LIKE. [fix](auth)fix create table like need create_priv of existed table #37879
- Fixed the issue where grant operations did not check if the object existed. [enhance](privilege)Grant check name #39597
Use case
No response
Related issues
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct