[fix](replica) handle replica version missing info to avoid -214 error #8209

morningman · 2022-02-23T10:38:50Z

Proposed changes

Issue Number: close #8208

Problem Summary:

In the original tablet reporting information, the version missing information is done by combining
two pieces of information as follows:

the maximum consecutive version number
the version_miss field

The logic of this approach is confusing and inconsistent with the logic of checking for missing versions when querying.

After the change, we directly use the version checking logic used in the query, and set version_miss to true
if a missing version is found

and on the FE processing side. Originally, only the bad replica information was syncronized among FEs,
but not the version missing information. As a result, the non-master FE is not aware of the missing version information.

In the new design, we deprecate the original log persistence class BackendTabletsInfo and use the new
BackendReplicasInfo to record replica reporting information and write both bad and version missing
information to metadata so that other FEs can synchronize these information.

Checklist(Required)

Does it affect the original behavior: (No)
Has unit tests been added: (Yes)
Has document been added or modified: (No Need)
Does it need to update dependencies: (No)
Are there any changes that cannot be rolled back: (No)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

yangzhg · 2022-03-03T05:37:16Z

be/src/olap/tablet.cpp

 }

-OLAPStatus Tablet::check_version_integrity(const Version& version) {
+OLAPStatus Tablet::check_version_integrity(const Version& version, bool quite) {


what does quite mean?

if !quite, it will print log for missing versions.

You mean quiet? slient may be better

yangzhg

LGTM

yangzhg

LGTM

github-actions · 2022-03-09T03:23:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2022-03-09T03:23:49Z

PR approved by anyone and no changes requested.

maybe typo

yangzhg

LGTM

github-actions · 2022-03-09T03:36:21Z

PR approved by at least one committer and no changes requested.

apache#8209) In the original tablet reporting information, the version missing information is done by combining two pieces of information as follows: 1. the maximum consecutive version number 2. the `version_miss` field The logic of this approach is confusing and inconsistent with the logic of checking for missing versions when querying. After the change, we directly use the version checking logic used in the query, and set `version_miss` to true if a missing version is found and on the FE processing side. Originally, only the **bad replica** information was syncronized among FEs, but not the **version missing** information. As a result, the non-master FE is not aware of the missing version information. In the new design, we deprecate the original log persistence class `BackendTabletsInfo` and use the new `BackendReplicasInfo` to record replica reporting information and write both **bad** and **version missing** information to metadata so that other FEs can synchronize these information.

#8444) 1. This bug was introduced by #8209. Error in fe.warn.log: ``` java.lang.IllegalStateException: 560278 at com.google.common.base.Preconditions.checkState(Preconditions.java:508) ~[spark-dpp-0.15-SNAPSHOT.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.TabletInvertedIndex.getReplica(TabletInvertedIndex.java:462) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.Catalog.replayBackendReplicasInfo(Catalog.java:6941) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:626) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.Catalog.replayJournal(Catalog.java:2446) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:116) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:74) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.common.util.Daemon.run(Daemon.java:116) [palo-fe.jar:0.15-SNAPSHOT] ``` Since the reporting of a tablet and the deletion of a tablet are two independent events and are not mutually exclusive, it may happen that the tablet is deleted first and the reporting is done later. 2. Change the tablet report info. Now, the version of a tablet report from BE is the largest continuous version. Eg, versions: [1,2,3,5,7], the report version of this tablet will be 3.

apache#8209) In the original tablet reporting information, the version missing information is done by combining two pieces of information as follows: 1. the maximum consecutive version number 2. the `version_miss` field The logic of this approach is confusing and inconsistent with the logic of checking for missing versions when querying. After the change, we directly use the version checking logic used in the query, and set `version_miss` to true if a missing version is found and on the FE processing side. Originally, only the **bad replica** information was syncronized among FEs, but not the **version missing** information. As a result, the non-master FE is not aware of the missing version information. In the new design, we deprecate the original log persistence class `BackendTabletsInfo` and use the new `BackendReplicasInfo` to record replica reporting information and write both **bad** and **version missing** information to metadata so that other FEs can synchronize these information.

this bug was introduced from apache#8209

this bug was introduced from #8209

#8209) In the original tablet reporting information, the version missing information is done by combining two pieces of information as follows: 1. the maximum consecutive version number 2. the `version_miss` field The logic of this approach is confusing and inconsistent with the logic of checking for missing versions when querying. After the change, we directly use the version checking logic used in the query, and set `version_miss` to true if a missing version is found and on the FE processing side. Originally, only the **bad replica** information was syncronized among FEs, but not the **version missing** information. As a result, the non-master FE is not aware of the missing version information. In the new design, we deprecate the original log persistence class `BackendTabletsInfo` and use the new `BackendReplicasInfo` to record replica reporting information and write both **bad** and **version missing** information to metadata so that other FEs can synchronize these information.

apache#8209) In the original tablet reporting information, the version missing information is done by combining two pieces of information as follows: 1. the maximum consecutive version number 2. the `version_miss` field The logic of this approach is confusing and inconsistent with the logic of checking for missing versions when querying. After the change, we directly use the version checking logic used in the query, and set `version_miss` to true if a missing version is found and on the FE processing side. Originally, only the **bad replica** information was syncronized among FEs, but not the **version missing** information. As a result, the non-master FE is not aware of the missing version information. In the new design, we deprecate the original log persistence class `BackendTabletsInfo` and use the new `BackendReplicasInfo` to record replica reporting information and write both **bad** and **version missing** information to metadata so that other FEs can synchronize these information.

apache#8444) 1. This bug was introduced by apache#8209. Error in fe.warn.log: ``` java.lang.IllegalStateException: 560278 at com.google.common.base.Preconditions.checkState(Preconditions.java:508) ~[spark-dpp-0.15-SNAPSHOT.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.TabletInvertedIndex.getReplica(TabletInvertedIndex.java:462) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.Catalog.replayBackendReplicasInfo(Catalog.java:6941) ~[palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:626) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.catalog.Catalog.replayJournal(Catalog.java:2446) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.master.Checkpoint.doCheckpoint(Checkpoint.java:116) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.master.Checkpoint.runAfterCatalogReady(Checkpoint.java:74) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) [palo-fe.jar:0.15-SNAPSHOT] at org.apache.doris.common.util.Daemon.run(Daemon.java:116) [palo-fe.jar:0.15-SNAPSHOT] ``` Since the reporting of a tablet and the deletion of a tablet are two independent events and are not mutually exclusive, it may happen that the tablet is deleted first and the reporting is done later. 2. Change the tablet report info. Now, the version of a tablet report from BE is the largest continuous version. Eg, versions: [1,2,3,5,7], the report version of this tablet will be 3.

this bug was introduced from apache#8209

morningman added the dev/backlog waiting to be merged in future dev branch label Feb 23, 2022

yangzhg reviewed Mar 3, 2022

View reviewed changes

yangzhg previously approved these changes Mar 8, 2022

View reviewed changes

morningman and others added 4 commits March 9, 2022 10:54

1

202e6e3

2

013d3b5

2

f2564f2

3

934cc61

morningman dismissed yangzhg’s stale review via 934cc61 March 9, 2022 03:00

morningman force-pushed the bug214 branch from 540d97c to 934cc61 Compare March 9, 2022 03:00

rebase

92818a7

yangzhg previously approved these changes Mar 9, 2022

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 9, 2022

github-actions bot added the reviewed label Mar 9, 2022

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 9, 2022

fix

244ba7e

yangzhg approved these changes Mar 9, 2022

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 9, 2022

morningman merged commit 826467e into apache:master Mar 9, 2022

morningman mentioned this pull request Mar 11, 2022

[fix](report) fix bug that tablet may already be delete when reporting #8444

Merged

morningman pushed a commit to morningman/doris that referenced this pull request Mar 12, 2022

[fix] BE crash when reporting tablet

88c5343

this bug was introduced from apache#8209

morningman mentioned this pull request Mar 12, 2022

[fix] BE crash when reporting tablet #8453

Merged

morningman added a commit that referenced this pull request Mar 12, 2022

[fix] BE crash when reporting tablet (#8453)

5f8e948

this bug was introduced from #8209

morningman added a commit that referenced this pull request Mar 12, 2022

[fix] BE crash when reporting tablet (#8453)

32da525

this bug was introduced from #8209

morningman added dev/merged-1.0.1-deprecated PR has been merged into dev-1.0.1 and removed dev/backlog waiting to be merged in future dev branch labels Mar 14, 2022

zhengte pushed a commit to zhengte/incubator-doris that referenced this pull request Mar 15, 2022

[fix] BE crash when reporting tablet (apache#8453)

e0f494b

this bug was introduced from apache#8209

morningman mentioned this pull request Apr 27, 2022

[Bug] failed to initialize storage reader. tablet=924991.xxxx, res=-214, backend=xxxx #9267

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](replica) handle replica version missing info to avoid -214 error #8209

[fix](replica) handle replica version missing info to avoid -214 error #8209

Uh oh!

morningman commented Feb 23, 2022

Uh oh!

yangzhg Mar 3, 2022

Uh oh!

morningman Mar 5, 2022

Uh oh!

yangzhg Mar 9, 2022

Uh oh!

yangzhg left a comment

Uh oh!

yangzhg left a comment

Uh oh!

github-actions bot commented Mar 9, 2022

Uh oh!

github-actions bot commented Mar 9, 2022

Uh oh!

yangzhg left a comment

Uh oh!

github-actions bot commented Mar 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[fix](replica) handle replica version missing info to avoid -214 error #8209

[fix](replica) handle replica version missing info to avoid -214 error #8209

Uh oh!

Conversation

morningman commented Feb 23, 2022

Proposed changes

Problem Summary:

Checklist(Required)

Further comments

Uh oh!

yangzhg Mar 3, 2022

Choose a reason for hiding this comment

Uh oh!

morningman Mar 5, 2022

Choose a reason for hiding this comment

Uh oh!

yangzhg Mar 9, 2022

Choose a reason for hiding this comment

Uh oh!

yangzhg left a comment

Choose a reason for hiding this comment

Uh oh!

yangzhg left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 9, 2022

Uh oh!

github-actions bot commented Mar 9, 2022

Uh oh!

yangzhg left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 9, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants