[fix](meta) add sync new image timeout #25768

CanGuan · 2023-10-23T07:58:30Z

Add timeout config for sync

Proposed changes

Issue Number: close #25764

Add sync_new_image_timeout config, for adjusting synchronize image timeout from master

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Add timeout config for sync

morningman · 2023-10-23T10:46:32Z

Have you try it? can it really solve the timeout issue?

morningman · 2023-10-23T10:46:45Z

fe/fe-common/src/main/java/org/apache/doris/common/Config.java

+            "adjust by the size of image file in the ${meta_dir}/image and the network environment between nodes. " +
+            "The default values is 3000 ms."
+    })
+    public static int sync_new_image_timeout = 30000;


Suggested change

public static int sync_new_image_timeout = 30000;

public static int sync_new_image_timeout_s = 300;

morningman · 2023-10-23T10:46:56Z

fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java

                String filename = Storage.IMAGE + "." + version;
                File dir = new File(this.imageDir);
-                MetaHelper.getRemoteFile(url, HTTP_TIMEOUT_SECOND * 1000, MetaHelper.getFile(filename, dir));
+                MetaHelper.getRemoteFile(url, Config.sync_new_image_timeout, MetaHelper.getFile(filename, dir));


Suggested change

MetaHelper.getRemoteFile(url, Config.sync_new_image_timeout, MetaHelper.getFile(filename, dir));

MetaHelper.getRemoteFile(url, Config.sync_new_image_timeout_s * 1000, MetaHelper.getFile(filename, dir));

Add timeout config for sync

CanGuan · 2023-10-23T11:15:55Z

Have you try it? can it really solve the timeout issue?

We found this bug by adding new follower to our doris cluster. By tracing this bug, we found the sync image timeout is only 5s and no way to modify it. After fixing and deploying to our cluster, it fix the problem. Here are the success log

In our case (image file is 1G), Follower takes 22s to synchronize the file from FE master.

refer to issue #25764

Add timeout config for sync

dataroaring · 2023-10-24T07:06:51Z

fe/fe-common/src/main/java/org/apache/doris/common/Config.java

+            + "adjust by the size of image file in the ${meta_dir}/image and the network environment between nodes. "
+            + "The default values is 300s."
+    })
+    public static int sync_new_image_timeout_s = 300;


sync_image_timeout_second is a better name.

morningman

LGTM

morningman · 2023-10-24T10:42:50Z

run buildall

fix checkstyle

CanGuan · 2023-10-24T13:05:43Z

LGTM

Please review again, chore code for checkstyle

morningman

LGTM

morningman · 2023-10-25T07:17:03Z

run buildall

dataroaring

LGTM

github-actions · 2023-10-25T07:19:11Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-10-25T07:19:14Z

PR approved by anyone and no changes requested.

doris-robot · 2023-10-25T08:01:45Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.56 seconds
stream load tsv: 572 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162517629 Bytes

The image file of our cluster reaches 2.3G. After the checkpoint, Followers synchronize the image timeout, resulting in the continuous increase of the bdb directory. related pr: #25768

Add timeout config for sync

…#26003) The image file of our cluster reaches 2.3G. After the checkpoint, Followers synchronize the image timeout, resulting in the continuous increase of the bdb directory. related pr: apache#25768

Add timeout config for sync

…#26003) The image file of our cluster reaches 2.3G. After the checkpoint, Followers synchronize the image timeout, resulting in the continuous increase of the bdb directory. related pr: apache#25768

[fix](meta) add sync new image timeout (apache#25764)

c860d2d

Add timeout config for sync

CanGuan changed the title ~~[fix](meta) add sync new image timeout (#25764)~~ [fix](meta) add sync new image timeout Oct 23, 2023

morningman reviewed Oct 23, 2023

View reviewed changes

[fix](meta) change config name (apache#25764)

ee3ba71

Add timeout config for sync

guanzhaoyun added 2 commits October 23, 2023 19:34

[fix](meta) change config name (apache#25764)

58eab3b

Add timeout config for sync

[fix](meta) fix checkstyle problems (apache#25764)

da024b3

Add timeout config for sync

dataroaring requested changes Oct 24, 2023

View reviewed changes

guanzhaoyun and others added 2 commits October 24, 2023 15:50

[fix](meta) change config name (apache#25764)

e38d5b4

Merge branch 'master' into master

e8dcefa

morningman added the dev/2.0.3 label Oct 24, 2023

morningman previously approved these changes Oct 24, 2023

View reviewed changes

guanzhaoyun and others added 3 commits October 24, 2023 18:59

[fix](meta) add sync new image timeout (apache#25764)

1a078be

fix checkstyle

Merge branch 'master' into master

ba19683

Merge remote-tracking branch 'origin/master'

570226c

CanGuan dismissed morningman’s stale review via 570226c October 24, 2023 11:02

CanGuan and others added 3 commits October 24, 2023 20:22

Merge branch 'master' into master

3c99d36

[fix](meta) add sync new image timeout (apache#25764)

5f1ec32

fix checkstyle

Merge remote-tracking branch 'origin/master'

e68c081

morningman approved these changes Oct 25, 2023

View reviewed changes

morningman self-assigned this Oct 25, 2023

dataroaring approved these changes Oct 25, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 25, 2023

github-actions bot added the reviewed label Oct 25, 2023

morningman added the usercase Important user case type label label Oct 25, 2023

morningman merged commit 0fb57a6 into apache:master Oct 25, 2023

luozenglin mentioned this pull request Oct 27, 2023

[fix](meta) fix follower sync image timeout after checkpoint. #26003

Merged

dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023

[fix](meta) add sync new image timeout (apache#25768)

abc3a3e

Add timeout config for sync

xiaokang pushed a commit to xiaokang/doris that referenced this pull request Oct 31, 2023

[fix](meta) add sync new image timeout (apache#25768)

b8582a4

Add timeout config for sync

xiaokang pushed a commit to xiaokang/doris that referenced this pull request Nov 1, 2023

[fix](meta) add sync new image timeout (apache#25768)

dbe002f

Add timeout config for sync

xiaokang added dev/2.0.3-merged and removed dev/2.0.3 labels Nov 1, 2023

luozenglin mentioned this pull request Nov 2, 2023

[cherry-pick](meta) fix follower sync image timeout after checkpoint. (#26003) #26276

Merged

gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023

[fix](meta) add sync new image timeout (apache#25768)

09dbe4b

Add timeout config for sync

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[fix](meta) add sync new image timeout (apache#25768)

8f372ed

Add timeout config for sync

nsnhuang mentioned this pull request Nov 14, 2024

[fix](config) fe config sync_image_timeout_second should not be masterOnly #43954

Merged

	public static int sync_new_image_timeout = 30000;
	public static int sync_new_image_timeout_s = 300;

	MetaHelper.getRemoteFile(url, Config.sync_new_image_timeout, MetaHelper.getFile(filename, dir));
	MetaHelper.getRemoteFile(url, Config.sync_new_image_timeout_s * 1000, MetaHelper.getFile(filename, dir));

[fix](meta) add sync new image timeout #25768

[fix](meta) add sync new image timeout #25768

Uh oh!

Conversation

CanGuan commented Oct 23, 2023

Proposed changes

Further comments

Uh oh!

morningman commented Oct 23, 2023

Uh oh!

morningman Oct 23, 2023

Choose a reason for hiding this comment

Uh oh!

CanGuan Oct 23, 2023

Choose a reason for hiding this comment

Uh oh!

morningman Oct 23, 2023

Choose a reason for hiding this comment

Uh oh!

CanGuan Oct 23, 2023

Choose a reason for hiding this comment

Uh oh!

CanGuan commented Oct 23, 2023

Uh oh!

dataroaring Oct 24, 2023

Choose a reason for hiding this comment

Uh oh!

CanGuan Oct 24, 2023

Choose a reason for hiding this comment

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented Oct 24, 2023

Uh oh!

CanGuan commented Oct 24, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

morningman commented Oct 25, 2023

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 25, 2023

Uh oh!

github-actions bot commented Oct 25, 2023

Uh oh!

doris-robot commented Oct 25, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants