dumpling: update dumpling document on how to reduce memory usage by lichunzhu · Pull Request #5860 · pingcap/docs-cn

lichunzhu · 2021-03-26T08:43:30Z

First-time contributors' checklist

I've signed Contributor License Agreement that's required for repo owners to accept my contribution.

What is changed, added or deleted? (Required)

Update dumpling document on how to reduce memory usage. Make sure users can get this important infomation.

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

This PR is translated from:
Other reference link(s):

Do your changes match any of the following descriptions?

Delete files
Change aliases
Need modification after applied to another branch
Might cause conflicts after applied to another branch

TomShawn · 2021-03-26T08:47:08Z

/translation doing

TomShawn · 2021-03-26T12:22:08Z

/label needs-cherry-pick-4.0
/label needs-cherry-pick-5.0
/status PTAL
/size small
/assign TomShawn
/label require-LGT1

TomShawn · 2021-03-26T12:22:29Z

@lichunzhu Please involve a technical review~

TomShawn · 2021-03-29T03:07:51Z

/cc 3pointer overvenus

overvenus · 2021-03-29T05:27:17Z

  --filetype sql \
  --threads 32 \
  -o /tmp/test \
+  -r 200000 \


I think it's too large and may cause TiDB OOM.

If we suggest TiDB uses 1GB memory, and the default threads is 4, then avg_row_length is
1GB / 4 / 200000 = 1342. It's more than 1KB. In most conditions avg_row_length is less than this.

overvenus · 2021-03-29T05:28:37Z

  --filetype sql \
  --threads 32 \
  -o /tmp/test \
+  -r 200000 \


Please also add -r to other examples, e.g., ### 导出到 csv 文件

added in 0c0706b

overvenus · 2021-03-29T12:09:41Z

/lgtm

ti-chi-bot · 2021-03-29T12:09:43Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

overvenus

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Details

Reviewer can indicate their review by writing /lgtm in a comment.
Reviewer can cancel approval by writing /lgtm cancel in a comment.

3pointer · 2021-03-29T12:19:22Z

+
+> **注意：**
+>
+> 导出较大表（单表大小超过 10 GB）时 `-r`, `-F` 参数均**强烈建议**开启。


-r -F 不能同时生效，也描述下吧

现在是可以同时生效的

3pointer · 2021-03-30T02:42:56Z

/lgtm

overvenus · 2021-04-06T08:43:18Z

@TomShawn PTAL, thanks!

TomShawn · 2021-04-06T09:13:22Z

+
+> **注意：**
+>
+> 导出较大表（单表大小超过 10 GB）时 `-r`, `-F` 参数均**强烈建议**开启。


Suggested change

> 导出较大表（单表大小超过 10 GB）时 `-r`, `-F` 参数均**强烈建议**开启。

> 如果导出的单表大小超过 10 GB，**强烈建议**指定 `-r` 和 `-F` 参数。

Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com>

TomShawn · 2021-04-06T09:52:37Z

/lgtm

TomShawn · 2021-04-06T09:52:51Z

/merge

ti-chi-bot · 2021-04-06T09:52:53Z

This pull request has been accepted and is ready to merge.

Details

Commit hash: 6df2aa6

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-04-06T09:55:27Z

cherry pick to release-4.0 in PR #5962

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot · 2021-04-06T09:56:14Z

cherry pick to release-5.0 in PR #5963

…) (#5963)

…) (#5962)

TomShawn · 2021-04-13T08:42:50Z

/translation done
/remove-translation doing

update dumpling documents about concurrency

0500f00

ti-chi-bot added missing-translation-status This PR does not have translation status info. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Mar 26, 2021

ti-chi-bot added translation/doing This PR’s assignee is translating this PR. and removed missing-translation-status This PR does not have translation status info. labels Mar 26, 2021

ti-chi-bot assigned TomShawn Mar 26, 2021

ti-chi-bot added status/PTAL This PR is ready for reviewing. size/small Changes of a small size. needs-cherry-pick-4.0 require-LGT1 Indicates that the PR requires an LGTM. labels Mar 26, 2021

ti-chi-bot requested review from 3pointer and overvenus March 29, 2021 03:07

overvenus reviewed Mar 29, 2021

View reviewed changes

address comments

0c0706b

ti-chi-bot removed the size/small Changes of a small size. label Mar 29, 2021

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Mar 29, 2021

3pointer reviewed Mar 29, 2021

View reviewed changes

TomShawn reviewed Apr 6, 2021

View reviewed changes

Apply suggestions from code review

6df2aa6

Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com>

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Apr 6, 2021

ti-chi-bot merged commit b8655e3 into pingcap:master Apr 6, 2021

ti-srebot pushed a commit to ti-srebot/docs-cn that referenced this pull request Apr 6, 2021

cherry pick pingcap#5860 to release-4.0

70901a9

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Apr 6, 2021

dumpling: update dumpling document on how to reduce memory usage (#5860) #5962

Merged

11 tasks

ti-srebot pushed a commit to ti-srebot/docs-cn that referenced this pull request Apr 6, 2021

cherry pick pingcap#5860 to release-5.0

e87095d

Signed-off-by: ti-srebot <ti-srebot@pingcap.com>

ti-srebot mentioned this pull request Apr 6, 2021

dumpling: update dumpling document on how to reduce memory usage (#5860) #5963

Merged

11 tasks

lichunzhu deleted the updateDumplingDoc branch April 6, 2021 09:59

ti-chi-bot pushed a commit that referenced this pull request Apr 6, 2021

dumpling: update dumpling document on how to reduce memory usage (#5860…

52ef4b9

…) (#5963)

ti-chi-bot pushed a commit that referenced this pull request Apr 6, 2021

dumpling: update dumpling document on how to reduce memory usage (#5860…

3c21cc7

…) (#5962)

TomShawn mentioned this pull request Apr 13, 2021

dumpling: update dumpling document on how to reduce memory usage pingcap/docs#5309

Merged

10 tasks

ti-chi-bot added translation/done This PR has been translated from English into Chinese and updated to pingcap/docs-cn in a PR. and removed translation/doing This PR’s assignee is translating this PR. labels Apr 13, 2021

This was referenced Apr 19, 2021

dumpling: update dumpling document on how to reduce memory usage (#5309) pingcap/docs#5380

Merged

dumpling: update dumpling document on how to reduce memory usage (#5309) pingcap/docs#5381

Merged

	> 导出较大表（单表大小超过 10 GB）时 `-r`, `-F` 参数均强烈建议开启。
	> 如果导出的单表大小超过 10 GB，强烈建议指定 `-r` 和 `-F` 参数。

Conversation

lichunzhu commented Mar 26, 2021

First-time contributors' checklist

What is changed, added or deleted? (Required)

Which TiDB version(s) do your changes apply to? (Required)

What is the related PR or file link(s)?

Do your changes match any of the following descriptions?

Uh oh!

TomShawn commented Mar 26, 2021

Uh oh!

TomShawn commented Mar 26, 2021

Uh oh!

TomShawn commented Mar 26, 2021

Uh oh!

TomShawn commented Mar 29, 2021

Uh oh!

overvenus Mar 29, 2021

Choose a reason for hiding this comment

Uh oh!

lichunzhu Mar 29, 2021

Choose a reason for hiding this comment

Uh oh!

overvenus Mar 29, 2021

Choose a reason for hiding this comment

Uh oh!

lichunzhu Mar 29, 2021

Choose a reason for hiding this comment

Uh oh!

overvenus commented Mar 29, 2021

Uh oh!

ti-chi-bot commented Mar 29, 2021

Uh oh!

3pointer Mar 29, 2021

Choose a reason for hiding this comment

Uh oh!

lichunzhu Mar 29, 2021

Choose a reason for hiding this comment

Uh oh!

3pointer commented Mar 30, 2021

Uh oh!

overvenus commented Apr 6, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TomShawn Apr 6, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

TomShawn commented Apr 6, 2021

Uh oh!

TomShawn commented Apr 6, 2021

Uh oh!

ti-chi-bot commented Apr 6, 2021

Uh oh!

ti-srebot commented Apr 6, 2021

Uh oh!

ti-srebot commented Apr 6, 2021

Uh oh!

TomShawn commented Apr 13, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants