Release Note 3.0.7

[gavinchou]

Behavior Changes

• Adjust the permission requirements for show frontends and show backends to align with the corresponding RESTful API, i.e., requiring the SELECT_PRIV permission on the information_schema database ([fix](restapi) Unify Permission Requirements for Executing SHOW FRONTENDS/BACKENDS And NODE RestAPI #50140)
• Admin and root users with specified domains are no longer considered system users ([fix](auth)Only treat admin@% and root@% as system users. #50904)
• Storage: The default number of concurrent transactions per database is adjusted to 10000 ([chore](conf) change max_running_txn_num_per_db to 10000 #51367, branch-3.0: [chore](conf) change max_running_txn_num_per_db to 10000 #51367 #52380)

New Features

Lakehouse

(No specific content)

Asynchronous Materialized Views

(No specific content)

Query Optimizer

• Support MySQL's aggregate roll-up syntax GROUP BY ... WITH ROLLUP ([improvement](nereids)Support GROUP BY ... WITH ROLLUP syntax #51948)

Query Execution

• Like statement supports escape syntax ([Feature](function) support like with escape clause (#52146) #52540)

Semi-structured Data Management

• Support building non-tokenized inverted indexes and ngram bloomfilter indexes only for new data by setting the session variable enable_add_index_for_new_data=true ([feature](index change)Support light index change for inverted index without parser #52251, [feature](index change)Support light index change for ngram bf index #48461)

Storage

(No new features; relevant changes see Behavior Changes)

New Functions

• Added data functions: cot/sec/cosec ([Feature](function) support function cot/sec/cosec #52872)

Improvements

Data Ingestion

• Optimize error message prompts for SHOW CREATE LOAD ([chore](load) optimize show create load error message #53694, branch-3.0: [chore](load) optimize show create load error message #53694 #53730)

Primary Key Model

• Add segment key bounds truncation capability to avoid single large import failures ([opt](rowset meta) truncate segments key bounds if too large to avoid RowsetMetaCloudPB exceeds fdb's 100KB limits #45287, branch-3.0: [opt](rowset meta) truncate segments key bounds if too large to avoid RowsetMetaCloudPB exceeds fdb's 100KB limits (#45287) #51595)

Storage

• Enhance the reliability of compaction and imported data ([fix](cloud) compaction and schema change potential data race when retrying prepare rowset #51048, [fix](cloud) compaction and schema change potential data race when retrying prepare rowset (#51048) #51852, [fix](cloud) potential data race when retrying prepare/commit rowset for load #51129, branch-3.0: [fix](cloud) potential data race when retrying prepare/commit rowset for load #51129 #51483)
• Optimize balance speed ([opt](cloud) Optimize balance speed by reducing the complexity of the rebalance algorithm #51733, branch-3.0: [opt](cloud) Optimize balance speed by reducing the complexity of the rebalance algorithm #51733 #52813, [fix](cloud) Fix ConcurrentModificationException in cloud rebalance #52013, branch-3.0: [fix](cloud) Fix ConcurrentModificationException in cloud rebalance #52013 #52309)
• Optimize table creation speed ([opt](create table) Fixed table creation becomes slower as the number of tablets increases #52688, branch-3.0: [opt](create table) Fixed table creation becomes slower as the number of tablets increases #52688 #52918)
• Optimize compaction default parameters and observability ([enhance](compaction) limit time series table max version using maximum of current backend #53244, [enhance](compaction) limit time series table max version using maximum of current backend (#53244) #53562, [enhance](compaction) optimize mow base compaction parameters #52321, branch-3.0: [enhance](compaction) optimize mow base compaction parameters #52321 #52605, [Enhancement](Compaction) Support auto set cumu compaction threads num base on cpu num #53133, branch-3.0: [Enhancement](Compaction) Support auto set cumu compaction threads num base on cpu num #53133 #53215, [Enhancement](Compaction) Make base compaction use the same tablet selection strategy as cumulative compaction #51649, branch-3.0: [Enhancement](Compaction) Make base compaction use the same tablet selection strategy as cumulative compaction #51649 #52389, [bvar](cloud-mow) Add bvar for mow compaction get delete bitmap lock backoff sleep time #52044, branch-3.0: [bvar](cloud-mow) Add bvar for mow compaction get delete bitmap lock backoff sleep time (#52044) #52297)
• Optimize the issue of query error -230 ([opt](rowset) Remote fetch rowsets to avoid -230 error when capturing rowsets (#52995) #52440, [fix](replica) Get tablet replica infos should return all primary backends except for warmup jobs #54131)
• Add system table backend_tablets (branch-3.0: [Enhancement] add information_schema backend_tablets table #52195)
• Optimize the performance of querying information_schema.tables from follower nodes in cloud mode ([opt](query) accelerate query information_schema.tables from follower node in cloud mode #51240, [opt](query) accelerate query information_schema.tables from follower node in cloud mode (#51240) #51405)

Storage-Compute Decoupled

• Enhance observability of Meta-service recycler ([enhance](meta-service)add bvar for fdb process status #52882, branch-3.0: [enhance](meta-service)add bvar for fdb process status #52882 #53100, branch-3.0: [Enhancement](systable)add information_schema backend_configuration table #51542, [enhance](meta-service)collect fdb kv meta range info as metrics #52430, branch-3.0: [enhance](meta-service)collect fdb kv meta range info as metrics #52430 #53116, [enhance](meta-service)add bytes for kv stats #52729, branch-3.0 [enhance](meta-service)add bytes for kv stats (#52729) #53351, [enhance](meta-service) add real request ip for be rpc #53114, branch-3.0 [enhance](meta-service) add real request ip for be rpc #53114. #53320, branch-3.0: [enhance](meta-service)add bvar for ms kv get del put count #52714, [enhance](metrics)add some compaction metrics #50910, branch-3.0: [enhance](metrics)add some compaction metrics #50910 #51487, [Feature](recycler) Add recycler metrics for recycler layer #51409, branch-3.0: [Feature](recycler) Add recycler metrics for recycler layer #51409 #51884, [feat](recycler) Add http api for statistics recycler metrics #52523, branch-3.0: [feat](recycler) Add http api for statistics recycler metrics #52523 #53117)
• Support cross-compute group incremental preheating during import compaction ([feature](cloud) support event driven or periodic warm up #52370, branch-3.0: [feature](cloud) support event driven or periodic warm up #52370 #52514, [fix](warmup) fix show warm up tables #53406, [fix](warmup) fix warm up jobs missing last batch #53860, [fix](warmup) fix warm up jobs missing last batch (#53860) #53861, [fix](load) fix bad load id in injection #52339, branch-3.0: [fix](load) fix bad load id in injection #52339 #52426, [test](p2) allow protocol prefix in S3 endpoint format in test_broker_load #53525, [test](p2) allow protocol prefix in S3 endpoint format in test_broker_load (#53525) #53530, [fix](warmup) avoid calling recycle_cache after rebalance #53339, [fix](warmup) avoid calling recycle_cache after rebalance (#53339) #53523, [metrics](warmup) add some metrics for warmup jobs #52991, [fix](warmup) prevent NPE when upgrading from older versions #53555, branch-3.0: [fix](warmup) prevent NPE when upgrading from older versions #53555 #53666, [feat](warmup) display tables in SHOW WARM UP JOB results #51594, [feat](warmup) display tables in SHOW WARM UP JOB results (#51594) #52291, [fix](vcg) use "vcg cancel" as cancel message for warm up jobs #53752)
• Optimize Storage vault connectivity check ([feat](storage vault) Check storage vault connectivity for BE when starting #51175, branch-3.0: [feat](storage vault) Check storage vault connectivity for BE when starting #51175 #52319, [fix](be) Fix check_storage_vault deadlock #52541, branch-3.0: [fix](be) Fix check_storage_vault deadlock #52541 #52602, [fix](cloud) Sync storage resource once in read path when rowset._storage_resource.fs is null #53075, branch-3.0: [fix](cloud) Sync storage resource once in read path when rowset._storage_resource.fs is null #53075 #53227)
• Support updating storage backend information via MS API ([feat](cloud) Support alter operation for obj_info and s3 vault obj_info #51162, branch-3.0: [feat](cloud) Support alter operation for obj_info and s3 vault obj_info #51162 #51685)

Lakehouse

• Optimize ORC zlib decompression performance in x86 environment and fix potential issues ([fix & opt](orc) ORC-1525: Fix bad read in RleDecoderV2::readByte and Decompress zlib by libdeflate. #51775)
• Optimize the default number of concurrent threads for external table reading ([opt](multi-catalog) Optimize remote scan concurrency. #51415)
• Optimize error messages for Catalogs that do not support DDL operations ([fix](iceberg)Table operations are not supported for catalogs of the dlf type.  #50696)

Asynchronous Materialized Views

• Optimize the performance of transparent rewriting planning ([opt](mtmv) optimize mtmv rewrite performance #49514)

Query Optimizer

• The group_concat function now allows parameters of non-string types ([opt](group_concat) allow args be types other than string #52805)
• The sum and avg functions allow parameters of non-numeric types ([opt](Nereids) aggregate function sum support string type as parameter #49954)
• Expand the scope of support for delayed materialization in TOP-N queries, enabling delayed materialization when querying partial columns (branch-3.0: [opt](Nereids) support defer materialization with project #52522)
• When creating partitions, list partitions allow inclusion of MAX_VALUE ([fix](nereids)allow in partition contains MAX_VALUE #46076)
• Optimize the performance of sampling and collecting statistical information for aggregate model tables ([improvement](statistics)Agg table set preagg on when doing sample analyzing. #49918)
• Optimize the accuracy of NDV values when sampling and collecting statistical information ([improvement](statistics)Eliminate null values while sample analyzing ndv. #50574)

Inverted Index

• Unify the order of properties displayed for inverted indexes in show create table ([enhancement](inverted index) ensure consistent ordering of properties for inverted index show #51467)
• Add per-condition profile metrics (such as hit rows and execution time) for inverted index filter conditions to facilitate performance analysis ([feature](inverted index) Add profile statistics for each condition in inverted index filters #47504)
• Enhance the display of inverted index-related information in profiles ([opt](inverted index) Enhance I/O statistics collection for the inverted index in file cache scenarios #48950, [opt](inverted index) uniform profile naming convention #48826, [fix](inverted index) enhance inverted index profile #51495)

Permissions

• Ranger supports setting permissions for storage vault and compute group ([enhance](auth)ranger support storage vault and compute group #47925)

Bug Fixes

Data Ingestion

• Fix the correctness issue that may occur when importing CSV files with multi-character separators ([fix](csv reader) fix data loss when concurrency read using multi char line delimiter #53374, branch-3.0: [fix](csv reader) fix data loss when concurrency read using multi char line delimiter (#53374) #53634)
• Fix the issue where the result of ROUTINE LOAD task display is incorrect after modifying task properties ([fix](job) fix show routine load job result incorrect after alter job property #53038, branch-3.0: [fix](job) fix show routine load job result incorrect after alter job property #53038 #53098)
• Fix the issue where the one-stream multi-table import plan becomes invalid after primary node restart or Leader switch ([fix](load) fix multi table load plan fail after restart master Fe or leader change #53799, branch-3.0: [fix](load) fix multi table load plan fail after restart master Fe or leader change (#53799) #53829)
• Fix the issue where all scheduling tasks are blocked because ROUTINE LOAD tasks cannot find available BE nodes ([fix](job) fix routine load task scheduler block for one job can not find any BE #52654, branch-3.0: [fix](job) fix routine load task scheduler block for one job can not find any BE (#52654) #52791)
• Fix the concurrent read-write conflict issue of runningTxnIds ([fix](load) fix concurrent read and write to runningTxnIds #51615, branch-3.0: [fix](load) fix concurrent read and write to runningTxnIds #51615 #51639)

Primary Key Model

• Optimize the import performance of mow tables under high-frequency concurrent imports ([Opt](cloud-mow) Skip MS RPC retry's backoff when encounter fdb txn conflict when mow load get ms delete bitmap lock  #52360, branch-3.0: [Opt](cloud-mow) Skip MS RPC retry's backoff when encounter fdb txn conflict when mow load get ms delete bitmap lock #52360 #52439, [improve](cloud-mow) batch get tablet stats when get_delete_bitmap_update_lock #47281, branch-3.0: [improve](cloud-mow) batch get tablet stats when get_delete_bitmap_update_lock (#47281) #52225)
• mow table full compaction releases space of deleted data ([Opt](compaction) Prune rows with delete sign=1 in full compaction #51874, branch-3.0: [Opt](compaction) Prune rows with delete sign=1 in full compaction #51874 #52256)
• Fix the potential import failure issue of mow tables in extreme scenarios ([Opt](cloud-mow) Retry to commit txn when encounter stale calc delete bitmap response regardless of status code #52547, branch-3.0: [Opt](cloud-mow) Retry to commit txn when encounter stale calc delete bitmap response regardless of status code (#52547) #52848)
• Optimize the compaction performance of mow tables ([Opt](cloud-mow) Do fast retry when commit compaction job for mow tablet #52476, branch-3.0; [Opt](cloud-mow) Do fast retry when commit compaction job for mow tablet (#52476) #52952)
• Fix the potential correctness issue of mow tables during concurrent imports and schema changes ([Fix](mow) Fix DeleteBitmap's assignment operator and constructor #52582, branch-3.0: [Fix](mow) Fix DeleteBitmap's assignment operator and constructor #52582 #52974)
• Fix the issue where schema change on empty mow tables may cause import stuck or schema change failure ([fix](mow) fix update delete bitmap lock not removed if schema change for empty tablet #51780, branch-3.0: [fix](mow) fix update delete bitmap lock not removed if schema change for empty tablet (#51780) #52166)
• Fix the memory leak issue of mow delete bitmap cache ([fix](mow) fix potential mem leak for DeleteBitmap::get_agg #52718, [fix](chore) fix cache release core #52756, branch-3.0:[fix](mow) fix potential mem leak for DeleteBitmap::get_agg (#52718, #52756) #52931)
• Fix the potential correctness issue of mow tables after schema change ([Fix](cloud-mow) Remove potential existing split delete bitmap KVs before update them in schema change #51353, branch-3.0: [Fix](cloud-mow) Remove potential existing split delete bitmap KVs before update them in schema change (#51353) #51531)

Storage

• Fix the missing rowset issue in clone process caused by compaction ([Fix](Compaction) Fix full clone failure when rowset missing #53984, branch-3.0: [Fix](Compaction) Fix full clone failure when rowset missing #53984 #54162, [Fix](Clone) Fix compaction and mow failure when missing rowset  #52812, [Cherry-Pick](branch-3.0) Pick "[Fix](Clone) Fix compaction and mow failure when missing rowset (#52812)" #53497, [Enhancement](Log) Missing rowset clone should not print stack log #53193, [Cherry-Pick](branch-3.0) Pick "[Enhancement](Log) Missing rowset clone should not print stack log (#53193)" #53527)
• Fix the issue of inaccurate size calculation and default value for autobucket ([fix](auto bucket)Set the estimated partition size to 5G in non cloud #51258, branch-3.0: [fix](auto bucket)Set the estimated partition size to 5G in non cloud #51258 #51682, [fix](auto bucket)Fix auto bucket calc bucketnum err when partition size is invalid #52801, branch-3.0: [fix](auto bucket)Fix auto bucket calc bucketnum err when partition size is invalid #52801 #53250)
• Fix the potential correctness issue caused by bucket columns ([fix](schema-change) Forbid dropping distribution columns (branch-3.0) #54037, [fix](schema-change) Rebuild distribution info according to original order #54024, [fix](schema-change) Rebuild distribution info should update default distribution info #54072, [fix](schema-change) Rebuild distribution info according to original order (#54024) (#54072) #54109)
• Fix the issue where single-column tables cannot be renamed ([fix](schema-change) Fix single column table could not rename columns #47275, branch-3.0: [fix](schema-change) Fix single column table could not rename columns #47275 #52340)
• Fix the potential memory leak issue of memtable ([fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902, branch-3.0: [fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable (#52902) #52965)
• Fix the inconsistent error reporting issue for unsupported operations in empty table transaction writes ([fix](txn load) fix delete in txn load #52133, branch-3.0: [fix](txn load) fix delete in txn load (#52133) #52635)

Storage-Compute Decoupled

• Several fixes for File cache ([enhancement](cloud) monitor evict size of file cache active gc #51197, branch-3.0: [enhancement](cloud) monitor evict size of file cache active gc #51197 #51793, [regression](filecache) fix regression failures part2 #53783, [regression](filecache) fix regression failures part2 (#53783) #53915, [fix](filecache) fix load_cache_info_into_memory crash #51684, branch-3.0: [fix](filecache) fix load_cache_info_into_memory crash #51684 #51904, [optimization](filecache) speed up filecache warm up #51776, [optimization](filecache) speed up filecache warm up #51776 #52556, [fix](cloud) fix file cache types priority order #51463, branch-3.0: [fix](cloud) fix file cache types priority order #51463 #51603, [enhancement](filecache) fix default capacity and add reset_capacity validation #51711, branch-3.0: [enhancement](filecache) fix default capacity and add reset_capacity validation #51711 #52152)
• Fix the issue where cumulative point may roll back during schema process ([fix](cloud) Fix roll-backed cumulative point of new tablet when doing schema change #53402, branch-3.0: [fix](cloud) Fix roll-backed cumulative point of new tablet when doing schema change #53402 #53446)
• Fix the issue where background tasks affect automatic restart ([fix](cloud)Fix auto start affected by daemon jobs #51729, branch-3.0: [fix](cloud)Fix auto start affected by daemon jobs #51729 #52519)
• Fix the unhandled exception issue in data recycling process in azure environment ([fix](recycler) Fix two errors for recycler #53042, branch-3.0: [fix](recycler) Fix two errors for recycler #53042 #53224)
• Fix the issue where file cache is not cleaned up in time when compacting a single rowset ([feat](cloud) Add unused rowset state for CloudTablet #51674)

Lakehouse

• Fix the transaction commit failure issue for Iceberg table writes in Kerberos environment ([fix](iceberg)Fix the thread pool issue used for commit. #51508)
• Fix the query issue for hudi in kerberos environment ([fix](hudi catalog) Fix the Kerberos authentication error when querying hudi table #51713)
• Fix the potential deadlock issue in multi-Catalog scenarios ([fix](catalog) fix deadlock of catalog and database #53626)
• Fix the metadata inconsistency issue caused by concurrent Catalog refresh in some cases ([fix](catalog) synchronize reset methods in catalog classes and remove Lombok annotations #51787)
• Fix the issue where ORC footer is read multiple times in some cases ([Optimize](orc-reader) Optimize stripe footer multiple reads in orc reader. #51277)
• Fix the issue where Table Valued Function cannot read compressed json files ([fix](tvf) support compressed json file for tvf and refactor code #51983)
• SQL Server Catalog supports identifying IDENTITY column information ([improvement](jdbc catalog) Optimize the acquisition of indentity type in SQLServer #51285)
• SQL Convertor supports specifying multiple URLs for high availability ([Enhancement](sql-dialect) Support multiple sql-converter service urls #52636)

Asynchronous Materialized Views

• Fix the issue where partition compensation may be performed incorrectly when the query is optimized to an empty result set ([fix](mtmv) Fix compensate union wrongly when direct query is empty relation #51700)

Query Optimizer

• Fix the issue where factors other than sql_select_limit affect DML execution results ([fix](sql_select_limit) sql_select_limit should not affect DML #53379)
• Fix the issue where materialized CTEs may report errors in extreme cases when starting local shuffle ([fix](coordinator) fix cte with local shuffle throw exception #52870)
• Fix the issue where prepared insert statements cannot be executed on non-master nodes ([fix](Prepared Statment) Fix exec prepared insert stmt in non master error #48689)
• Fix the result error issue when casting ipv4 to string ([fix](nereids) fix cast ipv4 to string #51546)

Permissions

• When a user has multiple roles, the permissions of the multiple roles will be merged before authorization (branch-3.0:[fix](auth)fix when authentication, the permissions of multiple roles… #52948)

Query Execution

• Fix issues with some json functions ([fix](json) Add . after in JSON path to support correct token parsing (#52543) #52744, [fix](test) Avoid the configuration item disable_datev1 affecting other test cases #52915, [fix](json) incorrect results of json_contains (#53291) #53364)
• Fix the potential BE Core issue when the asynchronous thread pool is full ([fix](pipeline) premature exit causing core dump during concurrent pr… #52365)
• Fix the incorrect result issue of hll_to_base64 ([Bug](function) fix wrong result of hll_to_base64  #51831)
• Fix the result error issue when casting decimal256 to float ([fix](decimal256) fix casting decimal256 to float #54140)
• Fix two memory leak issues ([bugfix](memleak) fix memory leak for tabletschema and result cache (#51931) #51952, [bugfix](memleak) fix memleak in arrow input stream #51929, branch-3.0: [Fix](field) Fix potential memory leak and wrong binary reading about JsonbField (#50174) #52542)
• Fix the be core issue caused by bitmap_from_base64 ([Bug](function) fix bitmap_from_base64 function cause heap-buffer-overflow error #53018)
• Fix the potential be core issue caused by array_map function ([bug](function) fix array_map cause coredump as NULL #51618)
• Fix the potential error issue of split_by_regexp function ([bug](function) fix split_by_regexp meet empty string return error #51293)
• Fix the potential result error issue of bitmap_union function under extremely large data volumes ([Bug](aggregate) fix bitmap_union return error result in query sql #52033)
• Fix the potential core issue of format round function under some boundary values ([branch-3.0](core) Fix format round would core under boundary conditions #53855)

Inverted Index

• Fix the memory leak issue of inverted indexes in abnormal situations ([fix](inverted index) fix index memeory leak for inverted index #53235)
• Fix the error reporting issue when writing and querying empty index files ([fix](inverted index) fix error when writing empty index file #51984, [fix](inverted index) fix error when open empty index file #51393)
• Capture IO exceptions in inverted index string reading to avoid process crash due to exceptions ([fix](inverted index) catch IO exception to avoid coredump in inverted index string reader #51844)

Complex Data Types

• Fix the potential type inference error when Variant Nested data types conflict ([fix](variant)fix variant type conflicts in nested types #52696)
• Fix the parameter type inference error of map function ([fix](variant)fix variant type conflicts in nested types #52696)
• Fix the issue where data is incorrectly converted to NULL when specifying '$.' as the path in jsonpath ([Fix](JsonReader) Fix the issue where the null bitmap of the JSON reader was not initialized when the JSON path is specified as '$.’ #52211)
• Fix the issue where the serialization format cannot be restored when a subfield of Variant contains . (brach-3.0 cherry-pick [Fix](Variant) fix serialize with json key contains . as name #51930)

Others

• Fix the insufficient length issue of the IP field in the auditlog table ([opt](auditlog) Use varchar(1024) for column frontend_ip of audit log table #52762, [opt](auditlog) Use varchar(1024) for column frontend_ip of audit log table #52762 #52984)
• Fix the issue where the query id recorded in the audit log is that of the previous query when SQL parsing fails ([fix](audit)Fixed an issue that the audit log would record the previous queryId when parseSQL fails. #53107)

[gavinchou]

行为变更

• 调整 show frontends 和 show backends 的权限需求，使其与对应的 RESTful API 保持一致，即需要 information_schema 库的 SELECT_PRIV 权限 ([fix](restapi) Unify Permission Requirements for Executing SHOW FRONTENDS/BACKENDS And NODE RestAPI #50140)
• 指定 domain 的 admin 和 root 用户不再视为系统用户 ([fix](auth)Only treat admin@% and root@% as system users. #50904)
• 存储：单库默认并发事务数调整为 10000 ([chore](conf) change max_running_txn_num_per_db to 10000 #51367, branch-3.0: [chore](conf) change max_running_txn_num_per_db to 10000 #51367 #52380)

新特性

Lakehouse

（无具体内容）

异步物化视图

（无具体内容）

查询优化器

• 支持 MySQL 的聚合上卷语法 GROUP BY ... WITH ROLLUP ([improvement](nereids)Support GROUP BY ... WITH ROLLUP syntax #51948)

查询执行

• 新增数据函数：cot/sec/cosec ([Feature](function) support function cot/sec/cosec #52872)
• Like 语句支持 escape 语法 ([Feature](function) support like with escape clause (#52146) #52540)

半结构化数据管理

• 通过设置会话变量 enable_add_index_for_new_data=true，支持仅对新增数据构建不分词倒排索引和 ngram bloomfilter 索引 ([feature](index change)Support light index change for inverted index without parser #52251, [feature](index change)Support light index change for ngram bf index #48461)

存储

（无新增特性，相关变更见行为变更）

改进

导入

• 优化 SHOW CREATE LOAD 错误信息提示 ([chore](load) optimize show create load error message #53694, branch-3.0: [chore](load) optimize show create load error message #53694 #53730)

主键

• 新增 segment key bounds 截断能力，避免单次大导入失败的问题 ([opt](rowset meta) truncate segments key bounds if too large to avoid RowsetMetaCloudPB exceeds fdb's 100KB limits #45287, branch-3.0: [opt](rowset meta) truncate segments key bounds if too large to avoid RowsetMetaCloudPB exceeds fdb's 100KB limits (#45287) #51595)

存储

• 增强 compaction 和导入数据的可靠性 ([fix](cloud) compaction and schema change potential data race when retrying prepare rowset #51048, [fix](cloud) compaction and schema change potential data race when retrying prepare rowset (#51048) #51852, [fix](cloud) potential data race when retrying prepare/commit rowset for load #51129, branch-3.0: [fix](cloud) potential data race when retrying prepare/commit rowset for load #51129 #51483)
• 优化 balance 速度 ([opt](cloud) Optimize balance speed by reducing the complexity of the rebalance algorithm #51733, branch-3.0: [opt](cloud) Optimize balance speed by reducing the complexity of the rebalance algorithm #51733 #52813, [fix](cloud) Fix ConcurrentModificationException in cloud rebalance #52013, branch-3.0: [fix](cloud) Fix ConcurrentModificationException in cloud rebalance #52013 #52309)
• 优化建表速度 ([opt](create table) Fixed table creation becomes slower as the number of tablets increases #52688, branch-3.0: [opt](create table) Fixed table creation becomes slower as the number of tablets increases #52688 #52918)
• 优化 compaction 默认参数及可观测性 ([enhance](compaction) limit time series table max version using maximum of current backend #53244, [enhance](compaction) limit time series table max version using maximum of current backend (#53244) #53562, [enhance](compaction) optimize mow base compaction parameters #52321, branch-3.0: [enhance](compaction) optimize mow base compaction parameters #52321 #52605, [Enhancement](Compaction) Support auto set cumu compaction threads num base on cpu num #53133, branch-3.0: [Enhancement](Compaction) Support auto set cumu compaction threads num base on cpu num #53133 #53215, [Enhancement](Compaction) Make base compaction use the same tablet selection strategy as cumulative compaction #51649, branch-3.0: [Enhancement](Compaction) Make base compaction use the same tablet selection strategy as cumulative compaction #51649 #52389, [bvar](cloud-mow) Add bvar for mow compaction get delete bitmap lock backoff sleep time #52044, branch-3.0: [bvar](cloud-mow) Add bvar for mow compaction get delete bitmap lock backoff sleep time (#52044) #52297)
• 优化查询报错 -230 的问题 ([opt](rowset) Remote fetch rowsets to avoid -230 error when capturing rowsets (#52995) #52440, [fix](replica) Get tablet replica infos should return all primary backends except for warmup jobs #54131)
• 增加系统表 backend_tablets (branch-3.0: [Enhancement] add information_schema backend_tablets table #52195)
• 优化 cloud 模式下从 follower 节点查询 information_schema.tables 的性能 ([opt](query) accelerate query information_schema.tables from follower node in cloud mode #51240, [opt](query) accelerate query information_schema.tables from follower node in cloud mode (#51240) #51405)

存算分离

• 增强 Meta-service recycler 可观测性 ([enhance](meta-service)add bvar for fdb process status #52882, branch-3.0: [enhance](meta-service)add bvar for fdb process status #52882 #53100, branch-3.0: [Enhancement](systable)add information_schema backend_configuration table #51542, [enhance](meta-service)collect fdb kv meta range info as metrics #52430, branch-3.0: [enhance](meta-service)collect fdb kv meta range info as metrics #52430 #53116, [enhance](meta-service)add bytes for kv stats #52729, branch-3.0 [enhance](meta-service)add bytes for kv stats (#52729) #53351, [enhance](meta-service) add real request ip for be rpc #53114, branch-3.0 [enhance](meta-service) add real request ip for be rpc #53114. #53320, branch-3.0: [enhance](meta-service)add bvar for ms kv get del put count #52714, [enhance](metrics)add some compaction metrics #50910, branch-3.0: [enhance](metrics)add some compaction metrics #50910 #51487, [Feature](recycler) Add recycler metrics for recycler layer #51409, branch-3.0: [Feature](recycler) Add recycler metrics for recycler layer #51409 #51884, [feat](recycler) Add http api for statistics recycler metrics #52523, branch-3.0: [feat](recycler) Add http api for statistics recycler metrics #52523 #53117)
• 支持导入 compaction 过程进行跨 compute group 增量预热 ([feature](cloud) support event driven or periodic warm up #52370, branch-3.0: [feature](cloud) support event driven or periodic warm up #52370 #52514, [fix](warmup) fix show warm up tables #53406, [fix](warmup) fix warm up jobs missing last batch #53860, [fix](warmup) fix warm up jobs missing last batch (#53860) #53861, [fix](load) fix bad load id in injection #52339, branch-3.0: [fix](load) fix bad load id in injection #52339 #52426, [test](p2) allow protocol prefix in S3 endpoint format in test_broker_load #53525, [test](p2) allow protocol prefix in S3 endpoint format in test_broker_load (#53525) #53530, [fix](warmup) avoid calling recycle_cache after rebalance #53339, [fix](warmup) avoid calling recycle_cache after rebalance (#53339) #53523, [metrics](warmup) add some metrics for warmup jobs #52991, [fix](warmup) prevent NPE when upgrading from older versions #53555, branch-3.0: [fix](warmup) prevent NPE when upgrading from older versions #53555 #53666, [feat](warmup) display tables in SHOW WARM UP JOB results #51594, [feat](warmup) display tables in SHOW WARM UP JOB results (#51594) #52291, [fix](vcg) use "vcg cancel" as cancel message for warm up jobs #53752)
• 优化 Storage vault 连通性检查 ([feat](storage vault) Check storage vault connectivity for BE when starting #51175, branch-3.0: [feat](storage vault) Check storage vault connectivity for BE when starting #51175 #52319, [fix](be) Fix check_storage_vault deadlock #52541, branch-3.0: [fix](be) Fix check_storage_vault deadlock #52541 #52602, [fix](cloud) Sync storage resource once in read path when rowset._storage_resource.fs is null #53075, branch-3.0: [fix](cloud) Sync storage resource once in read path when rowset._storage_resource.fs is null #53075 #53227)
• 支持通过 MS API 更新存储后端信息 ([feat](cloud) Support alter operation for obj_info and s3 vault obj_info #51162, branch-3.0: [feat](cloud) Support alter operation for obj_info and s3 vault obj_info #51162 #51685)

Lakehouse

• 优化 x86 环境下 ORC zlib 的解压性能并修复潜在问题 ([fix & opt](orc) ORC-1525: Fix bad read in RleDecoderV2::readByte and Decompress zlib by libdeflate. #51775)
• 优化外表读取的默认并发线程数 ([opt](multi-catalog) Optimize remote scan concurrency. #51415)
• 优化不支持 DDL 操作的 Catalog 的报错信息 ([fix](iceberg)Table operations are not supported for catalogs of the dlf type.  #50696)

异步物化视图

• 优化透明改写规划的性能 ([opt](mtmv) optimize mtmv rewrite performance #49514)

查询优化器

• group_concat 函数现在允许参数为非字符串类型 ([opt](group_concat) allow args be types other than string #52805)
• sum 和 avg 函数允许参数为非数值类型 ([opt](Nereids) aggregate function sum support string type as parameter #49954)
• 扩展 TOP-N 查询延迟物化的支持范围，当查询部分列时也能延迟物化 (branch-3.0: [opt](Nereids) support defer materialization with project #52522)
• 创建分区时，list 分区允许包含 MAX_VALUE ([fix](nereids)allow in partition contains MAX_VALUE #46076)
• 优化采样收集聚合模型表统计信息的性能 ([improvement](statistics)Agg table set preagg on when doing sample analyzing. #49918)
• 优化采样收集统计信息时 NDV 值的准确性 ([improvement](statistics)Eliminate null values while sample analyzing ndv. #50574)

倒排索引

• 统一 show create table 中倒排索引展示的 properties 顺序 ([enhancement](inverted index) ensure consistent ordering of properties for inverted index show #51467)
• 为倒排索引过滤条件新增逐条件的 profile 指标（如命中行数与执行时间），便于性能分析 ([feature](inverted index) Add profile statistics for each condition in inverted index filters #47504)
• 增强 profile 中倒排索引相关信息展示 ([opt](inverted index) Enhance I/O statistics collection for the inverted index in file cache scenarios #48950, [opt](inverted index) uniform profile naming convention #48826, [fix](inverted index) enhance inverted index profile #51495)

权限

• Ranger 支持设置 storage vault 和 compute group 的权限 ([enhance](auth)ranger support storage vault and compute group #47925)

缺陷修复

导入

• 修复导入 CSV 文件使用多字符分隔符可能导致的正确性问题 ([fix](csv reader) fix data loss when concurrency read using multi char line delimiter #53374, branch-3.0: [fix](csv reader) fix data loss when concurrency read using multi char line delimiter (#53374) #53634)
• 修复修改任务属性后显示 ROUTINE LOAD 任务结果不正确的问题 ([fix](job) fix show routine load job result incorrect after alter job property #53038, branch-3.0: [fix](job) fix show routine load job result incorrect after alter job property #53038 #53098)
• 修复主节点重启或 Leader 切换后一流多表导入计划失效的问题 ([fix](load) fix multi table load plan fail after restart master Fe or leader change #53799, branch-3.0: [fix](load) fix multi table load plan fail after restart master Fe or leader change (#53799) #53829)
• 修复 ROUTINE LOAD 任务因找不到可用 BE 节点导致所有调度任务阻塞的问题 ([fix](job) fix routine load task scheduler block for one job can not find any BE #52654, branch-3.0: [fix](job) fix routine load task scheduler block for one job can not find any BE (#52654) #52791)
• 修复 runningTxnIds 并发读写冲突问题 ([fix](load) fix concurrent read and write to runningTxnIds #51615, branch-3.0: [fix](load) fix concurrent read and write to runningTxnIds #51615 #51639)

主键

• 优化 mow 表在高频并发导入下的导入性能 ([Opt](cloud-mow) Skip MS RPC retry's backoff when encounter fdb txn conflict when mow load get ms delete bitmap lock  #52360, branch-3.0: [Opt](cloud-mow) Skip MS RPC retry's backoff when encounter fdb txn conflict when mow load get ms delete bitmap lock #52360 #52439, [improve](cloud-mow) batch get tablet stats when get_delete_bitmap_update_lock #47281, branch-3.0: [improve](cloud-mow) batch get tablet stats when get_delete_bitmap_update_lock (#47281) #52225)
• mow 表 full compaction 释放被删除数据的空间 ([Opt](compaction) Prune rows with delete sign=1 in full compaction #51874, branch-3.0: [Opt](compaction) Prune rows with delete sign=1 in full compaction #51874 #52256)
• 修复 mow 表在极端场景下可能出现的导入失败问题 ([Opt](cloud-mow) Retry to commit txn when encounter stale calc delete bitmap response regardless of status code #52547, branch-3.0: [Opt](cloud-mow) Retry to commit txn when encounter stale calc delete bitmap response regardless of status code (#52547) #52848)
• 优化 mow 表 compaction 性能 ([Opt](cloud-mow) Do fast retry when commit compaction job for mow tablet #52476, branch-3.0; [Opt](cloud-mow) Do fast retry when commit compaction job for mow tablet (#52476) #52952)
• 修复 mow 表在有并发导入和 sc 时可能的正确性问题 ([Fix](mow) Fix DeleteBitmap's assignment operator and constructor #52582, branch-3.0: [Fix](mow) Fix DeleteBitmap's assignment operator and constructor #52582 #52974)
• 修复 mow 空表执行 schema change 可能导致导入卡住或 schema change 失败的问题 ([fix](mow) fix update delete bitmap lock not removed if schema change for empty tablet #51780, branch-3.0: [fix](mow) fix update delete bitmap lock not removed if schema change for empty tablet (#51780) #52166)
• 修复 mow delete bitmap cache 内存泄漏问题 ([fix](mow) fix potential mem leak for DeleteBitmap::get_agg #52718, [fix](chore) fix cache release core #52756, branch-3.0:[fix](mow) fix potential mem leak for DeleteBitmap::get_agg (#52718, #52756) #52931)
• 修复 mow 表在 sc 后可能的正确性问题 ([Fix](cloud-mow) Remove potential existing split delete bitmap KVs before update them in schema change #51353, branch-3.0: [Fix](cloud-mow) Remove potential existing split delete bitmap KVs before update them in schema change (#51353) #51531)

存储

• 修复 compaction 导致的 clone 过程 missing rowset 问题 ([Fix](Compaction) Fix full clone failure when rowset missing #53984, branch-3.0: [Fix](Compaction) Fix full clone failure when rowset missing #53984 #54162, [Fix](Clone) Fix compaction and mow failure when missing rowset  #52812, [Cherry-Pick](branch-3.0) Pick "[Fix](Clone) Fix compaction and mow failure when missing rowset (#52812)" #53497, [Enhancement](Log) Missing rowset clone should not print stack log #53193, [Cherry-Pick](branch-3.0) Pick "[Enhancement](Log) Missing rowset clone should not print stack log (#53193)" #53527)
• 修复 autobucket 计算 size 不准确及默认值问题 ([fix](auto bucket)Set the estimated partition size to 5G in non cloud #51258, branch-3.0: [fix](auto bucket)Set the estimated partition size to 5G in non cloud #51258 #51682, [fix](auto bucket)Fix auto bucket calc bucketnum err when partition size is invalid #52801, branch-3.0: [fix](auto bucket)Fix auto bucket calc bucketnum err when partition size is invalid #52801 #53250)
• 修复分桶列可能导致的正确性问题 ([fix](schema-change) Forbid dropping distribution columns (branch-3.0) #54037, [fix](schema-change) Rebuild distribution info according to original order #54024, [fix](schema-change) Rebuild distribution info should update default distribution info #54072, [fix](schema-change) Rebuild distribution info according to original order (#54024) (#54072) #54109)
• 修复单列表不能 rename 的问题 ([fix](schema-change) Fix single column table could not rename columns #47275, branch-3.0: [fix](schema-change) Fix single column table could not rename columns #47275 #52340)
• 修复 memtable 可能的内存泄漏问题 ([fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable #52902, branch-3.0: [fix](load) Convert RowInBlock* to shared_ptr to fix potential memory leaks in MemTable (#52902) #52965)
• 修复空表事务写对不支持行为的报错不统一问题 ([fix](txn load) fix delete in txn load #52133, branch-3.0: [fix](txn load) fix delete in txn load (#52133) #52635)

存算分离

• File cache 相关修复 ([enhancement](cloud) monitor evict size of file cache active gc #51197, branch-3.0: [enhancement](cloud) monitor evict size of file cache active gc #51197 #51793, [regression](filecache) fix regression failures part2 #53783, [regression](filecache) fix regression failures part2 (#53783) #53915, [fix](filecache) fix load_cache_info_into_memory crash #51684, branch-3.0: [fix](filecache) fix load_cache_info_into_memory crash #51684 #51904, [optimization](filecache) speed up filecache warm up #51776, [optimization](filecache) speed up filecache warm up #51776 #52556, [fix](cloud) fix file cache types priority order #51463, branch-3.0: [fix](cloud) fix file cache types priority order #51463 #51603, [enhancement](filecache) fix default capacity and add reset_capacity validation #51711, branch-3.0: [enhancement](filecache) fix default capacity and add reset_capacity validation #51711 #52152)
• 修复 schema 过程中 cumulative point 可能回滚的问题 ([fix](cloud) Fix roll-backed cumulative point of new tablet when doing schema change #53402, branch-3.0: [fix](cloud) Fix roll-backed cumulative point of new tablet when doing schema change #53402 #53446)
• 修复后台任务影响自动重启的问题 ([fix](cloud)Fix auto start affected by daemon jobs #51729, branch-3.0: [fix](cloud)Fix auto start affected by daemon jobs #51729 #52519)
• 修复 azure 环境中数据回收过程未处理的异常问题 ([fix](recycler) Fix two errors for recycler #53042, branch-3.0: [fix](recycler) Fix two errors for recycler #53042 #53224)
• 修复单 rowset 做 compaction 未及时清理 file cache 的问题 ([feat](cloud) Add unused rowset state for CloudTablet #51674)

Lakehouse

• 修复 Kerberos 环境下 Iceberg 表写入事务提交失败的问题 ([fix](iceberg)Fix the thread pool issue used for commit. #51508)
• 修复 kerberos 环境下查询 hudi 的问题 ([fix](hudi catalog) Fix the Kerberos authentication error when querying hudi table #51713)
• 修复多 Catalog 情况下潜在的死锁问题 ([fix](catalog) fix deadlock of catalog and database #53626)
• 修复某些情况下并发刷新 Catalog 导致元数据不一致的问题 ([fix](catalog) synchronize reset methods in catalog classes and remove Lombok annotations #51787)
• 修复 ORC footer 某些情况下会被多次读取的问题 ([Optimize](orc-reader) Optimize stripe footer multiple reads in orc reader. #51277)
• 修复 Table Valued Function 无法读取压缩格式 json 文件的问题 ([fix](tvf) support compressed json file for tvf and refactor code #51983)
• SQL Server Catalog 支持识别 IDENTITY 列信息 ([improvement](jdbc catalog) Optimize the acquisition of indentity type in SQLServer #51285)
• SQL Convertor 支持指定多个 url 以实现高可用 ([Enhancement](sql-dialect) Support multiple sql-converter service urls #52636)

异步物化视图

• 修复当查询被优化为空集结果时，可能错误进行分区补偿的问题 ([fix](mtmv) Fix compensate union wrongly when direct query is empty relation #51700)

查询优化器

• 修复 sql_select_limit 以外的影响 DML 执行结果的问题 ([fix](sql_select_limit) sql_select_limit should not affect DML #53379)
• 修复开始 local shuffle 时，物化的 CTE 在极端情况下可能执行报错的问题 ([fix](coordinator) fix cte with local shuffle throw exception #52870)
• 修复 prepare 的 insert 语句无法在非 master 节点执行的问题 ([fix](Prepared Statment) Fix exec prepared insert stmt in non master error #48689)
• 修复 cast ipv4 到 string 的结果错误问题 ([fix](nereids) fix cast ipv4 to string #51546)

权限

• 当一个用户拥有多个角色时，会合并多个角色的权限后再执行鉴权 (branch-3.0:[fix](auth)fix when authentication, the permissions of multiple roles… #52948)

查询执行

• 修复部分 json 函数问题 ([fix](json) Add . after in JSON path to support correct token parsing (#52543) #52744, [fix](test) Avoid the configuration item disable_datev1 affecting other test cases #52915, [fix](json) incorrect results of json_contains (#53291) #53364)
• 修复异步线程池满时可能导致 BE Core 的问题 ([fix](pipeline) premature exit causing core dump during concurrent pr… #52365)
• 修复 hll_to_base64 结果不正确的问题 ([Bug](function) fix wrong result of hll_to_base64  #51831)
• 修复 decimal256 转换为 float 时结果错误的问题 ([fix](decimal256) fix casting decimal256 to float #54140)
• 修复两处内存泄漏问题 ([bugfix](memleak) fix memory leak for tabletschema and result cache (#51931) #51952, [bugfix](memleak) fix memleak in arrow input stream #51929, branch-3.0: [Fix](field) Fix potential memory leak and wrong binary reading about JsonbField (#50174) #52542)
• 修复 bitmap_from_base64 导致的 be core 问题 ([Bug](function) fix bitmap_from_base64 function cause heap-buffer-overflow error #53018)
• 修复 array_map 函数可能导致的 be core 问题 ([bug](function) fix array_map cause coredump as NULL #51618)
• 修复 split_by_regexp 函数可能的错误问题 ([bug](function) fix split_by_regexp meet empty string return error #51293)
• 修复超大数据量下 bitmap_union 函数可能的结果错误问题 ([Bug](aggregate) fix bitmap_union return error result in query sql #52033)
• 修复 format round 函数在部分边界值下可能 core 的问题 ([branch-3.0](core) Fix format round would core under boundary conditions #53855)

倒排索引

• 修复倒排索引在异常情况下产生的内存泄漏问题 ([fix](inverted index) fix index memeory leak for inverted index #53235)
• 修复写入和查询空索引文件时报错的问题 ([fix](inverted index) fix error when writing empty index file #51984, [fix](inverted index) fix error when open empty index file #51393)
• 捕获倒排索引字符串读取中的 IO 异常，避免因异常导致进程崩溃 ([fix](inverted index) catch IO exception to avoid coredump in inverted index string reader #51844)

复杂数据类型

• 修复 Variant Nested 嵌套数据类型冲突时可能导致的类型推断错误 ([fix](variant)fix variant type conflicts in nested types #52696)
• 修复 map 函数参数类型推导错误 ([fix](variant)fix variant type conflicts in nested types #52696)
• 修复 jsonpath 中指定 '$.' 作为 path 导致数据错误变为 NULL 的问题 ([Fix](JsonReader) Fix the issue where the null bitmap of the JSON reader was not initialized when the JSON path is specified as '$.’ #52211)
• 修复 Variant 的子字段包含 . 时，序列化格式无法还原的问题 (brach-3.0 cherry-pick [Fix](Variant) fix serialize with json key contains . as name #51930)

其他

• 修复 auditlog 表 IP 字段长度不足的问题 ([opt](auditlog) Use varchar(1024) for column frontend_ip of audit log table #52762, [opt](auditlog) Use varchar(1024) for column frontend_ip of audit log table #52762 #52984)
• 修复 SQL 解析错误时，审计日志中记录的 query id 为上一次执行查询的 query id 的问题 ([fix](audit)Fixed an issue that the audit log would record the previous queryId when parseSQL fails. #53107)