Skip to content

[Bug] Fix some bugs of load job scheduler #4869

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yangzhg merged 3 commits into from
Nov 10, 2020
Merged

[Bug] Fix some bugs of load job scheduler #4869

yangzhg merged 3 commits into from
Nov 10, 2020

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Nov 9, 2020 •
edited
Loading

Proposed changes

  1. The fix load meta bug logic should be removed since 0.12.
  2. The load task thread pool's waiting queue should be as long as desired pending jobs num.
  3. Submit the load task outside database lock to prevent holding lock for long time.
  4. Use 2 thread pools for loading and pending load tasks, separately. To avoid dead lock submission.

Types of changes

  • Bugfix (non-breaking change which fixes an issue)

Checklist

@morningman
[Bug] Fix some bugs of load job scheduler
c1d48e7 
1. The fix load meta bug logic should be removed since 0.12.
2. The load task thread pool's waiting queue should be as long as desired pending jobs num.
3. Submit the load task outside database lock to prevent holding lock for long time.
@morningman
2
593a239
@morningman
3
3a136fd
@morningman morningman added kind/fix Categorizes issue or PR as related to a bug. area/load Issues or PRs related to all kinds of load labels Nov 9, 2020
@EmmyMiao87
Copy link
Contributor

EmmyMiao87 commented Nov 9, 2020

Waiting for checking Unit test

EmmyMiao87
EmmyMiao87 approved these changes Nov 9, 2020
View reviewed changes
Copy link
Contributor

@EmmyMiao87 EmmyMiao87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman added the approved Indicates a PR has been approved by one committer. label Nov 9, 2020
@yangzhg yangzhg merged commit 4204a87 into apache:master Nov 10, 2020
@yangzhg yangzhg mentioned this pull request Feb 9, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@EmmyMiao87 EmmyMiao87 EmmyMiao87 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. area/load Issues or PRs related to all kinds of load kind/fix Categorizes issue or PR as related to a bug.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Bug][Load] Load task submit logic has some errors.

3 participants

@morningman @EmmyMiao87 @yangzhg