[enhancement](Nereids) fast compute hash code of deep expression tree to reduce conflict (#38981) #39132
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e#38123) ## Proposed changes pick apache#38096
## Proposed changes backport: apache#37936 Issue Number: close #xxx <!--Describe your changes.-->
…n MySQL Handshake (apache#38140) pick(apache#38086) Issue Number: close apache#38065 This PR introduces a fix to the SSL and CLIENT_LONG_FLAG handling in the MySQL handshake process. It ensures that the flags are correctly set according to the protocol requirements and resolves the issues related to decimal data fetching when SSL is enabled.
…he#38152) ## Proposed changes backport: apache#37495 Issue Number: close #xxx <!--Describe your changes.-->
## Proposed changes pick apache#38151 <!--Describe your changes.-->
## Proposed changes ### log ``` at org.apache.doris.DorisFE.main(DorisFE.java:95) ~[doris-fe.jar:1.2-SNAPSHOT] 2024-07-09 11:56:53,826 WARN (mysql-nio-pool-0|316) [ConnectProcessor.handleQueryException():524] Process one query failed because unknown reason: java.lang.NoSuchMethodError: 'void javax.ws.rs.core.NewCookie.<init>(java.lang.String, java.lang.String, java.lang.String, java.lang.String, int, java.lang.String, int, java.util.Date, boolean, boolean)' at org.glassfish.jersey.message.internal.CookiesParser$MutableNewCookie.getImmutableNewCookie(CookiesParser.java:130) ~[jersey-common-2.35.jar:?] at org.glassfish.jersey.message.internal.CookiesParser.parseNewCookie(CookiesParser.java:176) ~[jersey-common-2.35.jar:?] at org.glassfish.jersey.message.internal.HttpHeaderReader.readNewCookie(HttpHeaderReader.java:335) ~[jersey-common-2.35.jar:?] at org.glassfish.jersey.message.internal.NewCookieProvider.fromString(NewCookieProvider.java:87) ~[jersey-common-2.35.jar:?] at org.glassfish.jersey.message.internal.NewCookieProvider.fromString(NewCookieProvider.java:34) ~[jersey-common-2.35.jar:?] at javax.ws.rs.core.NewCookie.valueOf(NewCookie.java:126) ~[jsr311-api-1.1.1.jar:?] at com.sun.jersey.api.client.ClientResponse.getCookies(ClientResponse.java:783) ~[jersey-bundle-1.19.3.jar:1.19.3] at org.apache.ranger.admin.client.RangerAdminRESTClient.setCookieReceivedFromRoleDownloadSession(RangerAdminRESTClient.java:1364) ~[ranger-plugins-common-2.4.0.jar:2.4.0] at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1220) ~[ranger-plugins-common-2.4.0.jar:2.4.0] at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167) ~[ranger-plugins-common-2.4.0.jar:2.4.0] at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183) ~[ranger-plugins-common-2.4.0.jar:2.4.0] at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123) ~[ranger-plugins-common-2.4.0.jar:2.4.0] at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495) ~[ranger-plugins-common-2.4.0.jar:2.4.0] at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144) ~[ranger-plugins-common-2.4.0.jar:2.4.0] at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245) ~[ranger-plugins-common-2.4.0.jar:2.4.0] at org.apache.doris.catalog.authorizer.ranger.hive.RangerHivePlugin.<init>(RangerHivePlugin.java:25) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessController.<init>(RangerHiveAccessController.java:59) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessControllerFactory.createAccessController(RangerHiveAccessControllerFactory.java:28) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.privilege.AccessControllerManager.createAccessController(AccessControllerManager.java:102) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.ExternalCatalog.initAccessController(ExternalCatalog.java:326) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.CatalogFactory.createCatalog(CatalogFactory.java:163) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.CatalogFactory.createFromStmt(CatalogFactory.java:91) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.CatalogMgr.createCatalog(CatalogMgr.java:250) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:381) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:2992) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1019) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:619) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:537) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:385) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:241) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:260) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:288) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:342) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:833) ~[?:?] ``` ### changes JAX-RS (Java API for RESTful Web Services) is the implementation of the JSR 311 specification, providing the Java API for creating RESTful web services. JAX-RS 1.0 was the first version based on the JSR 311 specification. Subsequent versions of JAX-RS, such as JAX-RS 2.0, have extended and improved upon the initial JSR 311 specification while maintaining compatibility. JSR 311 is the specification, and JAX-RS is the actual implementation of that specification. They share the same package name, but when both coexist in a project, it can lead to loading exceptions. Therefore, I excluded the jsr311-api jar and retained only jax-rs. (cherry picked from commit 400c1fd) ## Proposed changes Issue Number: close apache#37575 <!--Describe your changes.-->
pick from master apache#36562 ## Proposed changes [test](mtmv)Add group by aggregate negative case
…adata to avoid holding the lock for an extended period.apache#38162 (apache#38163) ## Proposed changes apache#38162 <!--Describe your changes.-->
## Proposed changes Issue Number: close #xxx <!--Describe your changes.--> Co-authored-by: yiguolei <yiguolei@gmail.com>
## Proposed changes Issue Number: close #xxx <!--Describe your changes.--> Co-authored-by: yiguolei <yiguolei@gmail.com>
## Proposed changes Issue Number: close #xxx <!--Describe your changes.--> Co-authored-by: stephen <hello-stephen@qq.com>
… StreamLoadContext when operate_txn_2pc apache#38033 (apache#38187) cherry pick from apache#38033
…n tasks (apache#37782) (apache#38189) ## Proposed changes For some urgent compaction tasks, their submittion should take parallelism into account. Currently, we apply the control policy for data loading in specific. Other source of urgent tasks are considered as eager.
…apache#38200) ## Proposed changes pick apache#38206 <!--Describe your changes.-->
…itory (apache#38192) delete_if_exists is a temporary solution introduced in apache#25847, to avoid concurrent testing conflicts. Cherry-pick apache#38190
… (apache#38233) For large cluster, too many parallel tasks will cause performance issue. So this PR limit the max parallel tasks in Doris. pick apache#38196 <!--Describe your changes.-->
…ce_olap_table_replication_num apache#38211 (apache#38232) cherry pick from apache#38211
…fo level for debug convenience (apache#38133) (apache#38230) ## Proposed changes As title.
…tionSpecGather (apache#36877) (apache#38164) pick from master apache#36877
…38131) Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
…versions apache#38198 (apache#38256) cherry pick from apache#38198
…che#38056) pick from master apache#36902 ## Proposed changes [fix](mtmv)Fix mtmv name to resolve conflicts Co-authored-by: Dongyang Li <hello_stephen@qq.com>
…f periodic tasks. (apache#38264) … Since there might be delays in execution, taskCount should be set to >=1. ## Proposed changes Issue Number: close #xxx apache#38263
…pache#39004) (apache#39059) fix conflict create table by different regression tests, introduced by apache#38796
…nt error message (apache#38995) (apache#39068) pick apache#38995
… in RocksDB in case of BE restart after a partial update has commited apache#38331" (apache#39035) picks apache#38331 and apache#39066
…ache#39070) pick from master apache#38749 Co-authored-by: stephen <hello-stephen@qq.com>
apache#39072) …38731) apache#38731 The stddev function has a separate implementation for the DecimalV2 type, but there are issues with the implementation. Given that there is almost no existing data for DecimalV2, it will be removed here. For be, upgrading to this situation will result in an error directly. ``` SELECT STDDEV(data) FROM DECIMALV2_10_0_DATA; ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Agg Function stddev(decimal(10,0)) is not implemented ``` After removing DecimalV2, parameters of type DecimalV2 will be converted to double for calculations. ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
…void infinite transaction (apache#38991) (apache#39108) ## Proposed changes Issue Number: close apache#38956 As title.
…pache#38894) (apache#39107) ## Proposed changes 1. Use column idx of ref block instead of new block to indicate the ref column. 2. Rename some variables to clarify their meanings. 3. Clarify some log msg. 4. Add a minimal case to verify the change.
…tions apache#39015 (apache#39099) ## Proposed changes apache#39015 ### Description: This issue proposes the addition of new features to the project, including a deadlock detection tool and monitored lock implementations. These features will help in identifying and debugging potential deadlocks and monitoring lock usage. Features: #### AbstractMonitoredLock: A monitored version of Lock that tracks and logs lock acquisition and release times. Functionality: Overrides lock(), unlock(), tryLock(), and tryLock(long timeout, TimeUnit unit) methods. Logs information about lock acquisition time, release time, and any failure to acquire the lock within the specified timeout. ##### eg ```log 2024-08-07 12:02:59 [ Thread-2:2006 ] - [ WARN ] Thread ID: 12, Thread Name: Thread-2 - Lock held for 1912 ms, exceeding hold timeout of 1000 ms Thread stack trace: at java.lang.Thread.getStackTrace(Thread.java:1564) at org.example.lock.AbstractMonitoredLock.afterUnlock(AbstractMonitoredLock.java:49) at org.example.lock.MonitoredReentrantLock.unlock(MonitoredReentrantLock.java:32) at org.example.ExampleService.timeout(ExampleService.java:17) at org.example.Main.lambda$test2$1(Main.java:39) at java.lang.Thread.run(Thread.java:750) ``` #### DeadlockCheckerTool: Uses ScheduledExecutorService for periodic deadlock checks. Logs deadlock information including thread names, states, lock info, and stack traces. **ThreadMXBean accesses thread information in the local JVM, which is already in memory, so accessing it is less expensive than fetching data from external resources such as disk or network. Thread state cache: The JVM typically maintains a cache of thread states, reducing the need for real-time calculations or additional data processing.** ##### eg ```log Thread Name: Thread-0 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Lock Owner Name: Thread-1 Lock Owner Id: 12 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@1d653213 Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$3(Main.java:79) at org.example.Main$$Lambda$1/1221555852.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) 2024-08-07 14:11:28 [ pool-1-thread-1:2001 ] - [ WARN ] Deadlocks detected: Thread Name: Thread-1 Thread State: WAITING Lock Name: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Lock Owner Name: Thread-0 Lock Owner Id: 11 Waited Time: -1 Blocked Time: -1 Lock Info: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Blocked by: java.util.concurrent.locks.ReentrantLock$NonfairSync@13a2dfcf Stack Trace: at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at org.example.lock.MonitoredReentrantLock.lock(MonitoredReentrantLock.java:22) at org.example.Main.lambda$testDeadLock$4(Main.java:93) at org.example.Main$$Lambda$2/1556956098.run(Unknown Source) at java.lang.Thread.run(Thread.java:750) ``` ##### benchmark ``` @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(1) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 15889.407 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 678.061 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.000 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 668.249 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.080 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.075 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.006 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 20.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 6.000 ms LockBenchmark.testNativeLock thrpt 2 103130.635 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 ≈ 10⁻⁴ MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁶ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts @WarmUp(iterations = 5, time = 1, timeUnit = TimeUnit.SECONDS) @measurement(iterations = 2, time = 2, timeUnit = TimeUnit.SECONDS) @threads(100) Benchmark Mode Cnt Score Error Units LockBenchmark.testMonitoredLock thrpt 2 10994.606 ops/ms LockBenchmark.testMonitoredLock:·gc.alloc.rate thrpt 2 488.508 MB/sec LockBenchmark.testMonitoredLock:·gc.alloc.rate.norm thrpt 2 56.002 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space thrpt 2 481.390 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Eden_Space.norm thrpt 2 55.163 B/op LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space thrpt 2 0.020 MB/sec LockBenchmark.testMonitoredLock:·gc.churn.PS_Survivor_Space.norm thrpt 2 0.002 B/op LockBenchmark.testMonitoredLock:·gc.count thrpt 2 18.000 counts LockBenchmark.testMonitoredLock:·gc.time thrpt 2 9.000 ms LockBenchmark.testNativeLock thrpt 2 558652.036 ops/ms LockBenchmark.testNativeLock:·gc.alloc.rate thrpt 2 0.016 MB/sec LockBenchmark.testNativeLock:·gc.alloc.rate.norm thrpt 2 ≈ 10⁻⁴ B/op LockBenchmark.testNativeLock:·gc.count thrpt 2 ≈ 0 counts ``` ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
) ## Proposed changes bp apache#39024
…pache#35094) (apache#39120) ## Proposed changes bp apache#35094 Co-authored-by: Luennng <luennng@gmail.com>
…che#39019) (apache#39128) pick from master apache#39019 not add case because be return wrong answer for this select cast(2.0 as boolean); -- should return 1 not 2
pick from master apache#38758 ensure header do not be changed unexpectedly
apache#39112) ``` Fail to serialize doris.PFetchDataResult ``` If the size of `PFetchDataResult` is greater than 2G, protocol buffer cannot serialize the message. pick apache#37990
…pache#39111) ## Proposed changes pick apache#36392
… (apache#39129) pick from master apache#39109 when first regulator child output nullable is not right, we may get wrong nullable output, and lead be crash
## Proposed changes backport : apache#34848 Issue Number: close #xxx <!--Describe your changes.-->
… to reduce conflict (apache#38981) The Expression.hashCode default is getClass().hashCode(), just contains one level information, so the lots of expressions which is same type will return the same hash code and conflict, then compare deeply in the HashMap cause inefficient and hold table lock for long time. This pr support fast compute hash code by the bottom literal and slot, reduce the compare expression time because of the conflict of hash code In my test case, the sql planner time can reduce from 20 minutes(not finished) to 35 seconds (cherry picked from commit d5049ec)
924060929
requested review from
CalvinKirs,
dataroaring,
gavinchou,
morningman and
platoneko
as code owners
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cherry pick from #38981