Skip to content

ddl: support concurrent ddl#32169

Merged
hawkingrei merged 27 commits into
pingcap:masterfrom
hawkingrei:concurrent_ddl_workspace
Jul 20, 2022
Merged

ddl: support concurrent ddl#32169
hawkingrei merged 27 commits into
pingcap:masterfrom
hawkingrei:concurrent_ddl_workspace

Conversation

@hawkingrei
Copy link
Copy Markdown
Member

@hawkingrei hawkingrei commented Feb 8, 2022 •
edited by xiongjiwei
Loading

Signed-off-by: Weizhen Wang wangweizhen@pingcap.com
Signed-off-by: xiongjiwei xiongjiwei1996@outlook.com
Signed-off-by: wjhuang2016 huangwenjun1997@gmail.com

What problem does this PR solve?

Issue Number: ref #32031

this is a big PR and we split it into many commits, every commit almost has a single purpose. I will introduce them briefly, you may need to reference the doc #33629

  • init ddl tables
    create tidb_ddl_job, tidb_ddl_reorg, tidb_ddl_history tables with raw meta write, these 3 tables is use to replace the ddl job queue and reorg and history hash table. you can see this part in doc

  • setup concurrent ddl env and add ddl worker pool
    this commit adds the ddl worker pool definition, the ddl job manager will find a job and ship it to a worker in the worker pool.
    Also, this commit provides a sessionctx wrapper, only use in ddl relate. it just wraps begin, commit and execute.

  • add ddl manager to handle ddl job
    this commit implements the ddl manager, which is used for

    • find a runnable ddl job
    • ship the job to the worker

    you can ref the doc. Also, it adds a function HandleDDLJob which will do the ddl job, and it is a replacer of HandleDDLJobQueue. The last thing of this commit is adding schemaVersionManager to update the schema version in a new txn, it prevents the txn conflict with the schema version key.

  • reorg handler for concurrent ddl
    just implements the partner of the reorg information.

  • manage ddl jobs for concurrent ddl
    same as above, the partner of add job, delete job and many other related to history job

  • change ddl interface caller
    because many of the functions need a session now, we just change the caller

  • add metrics for concurrent ddl
    add metrics

  • migrate ddl between table and queue
    support switch between the old and new ddl framework, migrate the existing ddl job between queue and table

What is changed and how it works?

Check List

Tests

  • Unit test
  • Integration test
  • Manual test (add detailed scripts or steps below)
  • No code

Release note

None

@hawkingrei hawkingrei added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Feb 8, 2022
@ti-chi-bot
Copy link
Copy Markdown
Member

ti-chi-bot commented Feb 8, 2022 •
edited
Loading

[REVIEW NOTIFICATION]

This pull request has been approved by:

  • wjhuang2016
  • zimulala

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

@ti-chi-bot ti-chi-bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Feb 8, 2022
@sre-bot
Copy link
Copy Markdown
Contributor

sre-bot commented Feb 8, 2022

Please follow PR Title Format:

  • pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

  • *: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

@hawkingrei hawkingrei force-pushed the concurrent_ddl_workspace branch 2 times, most recently from f064240 to c8495c6 Compare February 9, 2022 06:54
@sre-bot
Copy link
Copy Markdown
Contributor

sre-bot commented Feb 9, 2022

Please follow PR Title Format:

  • pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

  • *: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

1 similar comment
@sre-bot
Copy link
Copy Markdown
Contributor

sre-bot commented Feb 9, 2022

Please follow PR Title Format:

  • pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

  • *: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

@hawkingrei hawkingrei changed the title [DNM] implemant concurrent ddl [DNM] ddl: implemant concurrent ddl Feb 9, 2022
@hawkingrei hawkingrei force-pushed the concurrent_ddl_workspace branch 18 times, most recently from c3c2de0 to fc3f2f1 Compare February 10, 2022 06:46
@xiongjiwei
Copy link
Copy Markdown
Contributor

xiongjiwei commented Jul 20, 2022

/run-tics-test

@xiongjiwei
Copy link
Copy Markdown
Contributor

xiongjiwei commented Jul 20, 2022

/run-integration-cdc-test

@xiongjiwei
Copy link
Copy Markdown
Contributor

xiongjiwei commented Jul 20, 2022

/run-integration-ddl-test

@sre-bot
Copy link
Copy Markdown
Contributor

sre-bot commented Jul 20, 2022

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/d8bfb403d6eeffbd1c50392b3ecc4957062db1e5

@xiongjiwei
Copy link
Copy Markdown
Contributor

xiongjiwei commented Jul 20, 2022

/run-integration-cdc-test

@xiongjiwei
Copy link
Copy Markdown
Contributor

xiongjiwei commented Jul 20, 2022

/merge

@ti-chi-bot
Copy link
Copy Markdown
Member

ti-chi-bot commented Jul 20, 2022

This pull request has been accepted and is ready to merge.

DetailsCommit hash: d8bfb40

@hawkingrei
Copy link
Copy Markdown
Member Author

hawkingrei commented Jul 20, 2022

/hold

@sre-bot
Copy link
Copy Markdown
Contributor

sre-bot commented Jul 20, 2022

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/e542bccbcdb00c88a77b18ef4ee3225b3e50298f

@sre-bot
Copy link
Copy Markdown
Contributor

sre-bot commented Jul 20, 2022

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/2ff012c648b0118b6604516afd4f0f5ecd1c985f

@wjhuang2016
Copy link
Copy Markdown
Member

wjhuang2016 commented Jul 20, 2022

/unhold

@wjhuang2016
Copy link
Copy Markdown
Member

wjhuang2016 commented Jul 20, 2022

/run-build

@wjhuang2016
Copy link
Copy Markdown
Member

wjhuang2016 commented Jul 20, 2022

/run-mysql-test

@wjhuang2016
Copy link
Copy Markdown
Member

wjhuang2016 commented Jul 20, 2022

/run-unit-test

1 similar comment
@wjhuang2016
Copy link
Copy Markdown
Member

wjhuang2016 commented Jul 20, 2022

/run-unit-test

@wjhuang2016
Copy link
Copy Markdown
Member

wjhuang2016 commented Jul 20, 2022

/run-unit-test

@sre-bot
Copy link
Copy Markdown
Contributor

sre-bot commented Jul 20, 2022

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/2e4b1d1df3498a4ffa1363d9aa6ae848fdd3c964

@sre-bot
Copy link
Copy Markdown
Contributor

sre-bot commented Jul 20, 2022

TiDB MergeCI notify

🔴 Bad News! New failing [2] after this pr merged.
These new failed integration tests seem to be caused by the current PR, please try to fix these new failed integration tests, thanks!

CI Name Result Duration Compare with Parent commit
idc-jenkins-ci/integration-cdc-test 🟥 failed 34, success 2, total 36 1 hr 11 min New failing
idc-jenkins-ci-tidb/integration-common-test 🟥 failed 2, success 9, total 11 48 min New failing
idc-jenkins-ci-tidb/common-test 🟢 all 12 tests passed 19 min Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2 🟢 all 28 tests passed 7 min 11 sec Existing passed
idc-jenkins-ci-tidb/integration-ddl-test 🟢 all 6 tests passed 6 min 42 sec Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1 🟢 all 26 tests passed 6 min 12 sec Existing passed
idc-jenkins-ci-tidb/tics-test 🟢 all 1 tests passed 6 min 2 sec Existing passed
idc-jenkins-ci-tidb/mybatis-test 🟢 all 1 tests passed 3 min 44 sec Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test 🟢 all 1 tests passed 3 min 14 sec Existing passed
idc-jenkins-ci-tidb/plugin-test 🟢 build success, plugin test success 4min Existing passed

@3pointer 3pointer mentioned this pull request Jul 21, 2022
Merged
12 tasks
This was referenced Aug 24, 2022
Merged
Closed
@YuJuncen YuJuncen mentioned this pull request Oct 18, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@overvenus overvenus overvenus left review comments

@3pointer 3pointer 3pointer left review comments

@zimulala zimulala zimulala approved these changes

@XuHuaiyu XuHuaiyu XuHuaiyu approved these changes

@wjhuang2016 wjhuang2016 wjhuang2016 approved these changes

@tangenta tangenta tangenta approved these changes

+1 more reviewer

@xiongjiwei xiongjiwei xiongjiwei left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

release-note-none Denotes a PR that doesn't merit a release note. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. status/can-merge Indicates a PR has been approved by a committer. status/LGT2 Indicates that a PR has LGTM 2.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.