[fix](group commit) Fix the timeout to wait wal finished when schema change

[mymeiyi]

Proposed changes

1. modify regression case to add some schema change;

2. when schema change, there is a dead lock:

   • the schema change job hold table write lock, and wait for BE wal num is 0
   • the group_commit commit failed because can not get table write lock
   • the replay wal failed because can not get table read lock to generate a plan fragment

   so move the wait wal out of the table write lock.

3. If BE encounter memory hard limit, the load will stuck until the hard limit is ended. This may lead to the commit time interval of group commit is much larger than users' config, so improve the check timely.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[mymeiyi]

run buildall

[github-actions]

clang-tidy review says "All clean, LGTM! 👍"

[doris-robot]

TeamCity be ut coverage result:
Function Coverage: 35.23% (8914/25304)
Line Coverage: 26.97% (73295/271720)
Region Coverage: 26.15% (37861/144809)
Branch Coverage: 22.96% (19275/83968)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9c652bd0c2af789a85b1ebcf5b4f1c4b769f017b_9c652bd0c2af789a85b1ebcf5b4f1c4b769f017b/report/index.html

[doris-robot]

ClickBench: Total hot run time: 32.49 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9c652bd0c2af789a85b1ebcf5b4f1c4b769f017b, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.07	0.08
query5	0.49	0.51	0.52
query6	1.41	0.86	0.82
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.49	0.45	0.45
query10	0.51	0.51	0.50
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.64	0.65	0.63
query14	0.92	1.15	0.94
query15	0.85	0.86	0.85
query16	0.38	0.38	0.38
query17	1.04	1.02	1.05
query18	0.23	0.24	0.24
query19	1.83	1.80	1.79
query20	0.02	0.01	0.01
query21	15.42	0.66	0.65
query22	4.29	6.03	3.06
query23	18.35	1.37	1.27
query24	1.86	0.25	0.27
query25	0.16	0.10	0.10
query26	0.27	0.18	0.17
query27	0.09	0.09	0.09
query28	13.36	1.02	1.01
query29	12.54	3.39	3.39
query30	0.25	0.07	0.06
query31	2.83	0.40	0.39
query32	3.26	0.49	0.49
query33	2.74	2.81	2.78
query34	17.19	4.47	4.55
query35	4.47	4.63	4.53
query36	0.66	0.48	0.47
query37	0.22	0.19	0.18
query38	0.19	0.19	0.20
query39	0.05	0.05	0.04
query40	0.20	0.15	0.15
query41	0.11	0.06	0.06
query42	0.06	0.07	0.06
query43	0.05	0.04	0.06
Total cold run time: 109.83 s
Total hot run time: 32.49 s

[mymeiyi]

run feut

[mymeiyi]

run p0

[dataroaring]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.

[Yukang-Lian]

LGTM