Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions TOC.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,7 @@
+ [BR Use Cases](/br/backup-and-restore-use-cases.md)
+ TiDB Binlog
+ [Overview](/tidb-binlog/tidb-binlog-overview.md)
+ [Quick Start](/tidb-binlog/get-started-with-tidb-binlog.md)
+ [Deploy](/tidb-binlog/deploy-tidb-binlog.md)
Comment thread
ran-huang marked this conversation as resolved.
+ [Maintain](/tidb-binlog/maintain-tidb-binlog-cluster.md)
+ [Configure](/tidb-binlog/tidb-binlog-configuration-file.md)
Expand Down
38 changes: 0 additions & 38 deletions load-misuse-handling.md

This file was deleted.

55 changes: 46 additions & 9 deletions loader-overview.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,15 @@
---
title: Loader Instructions
summary: Use Loader to load data to TiDB.
aliases: ['/docs/dev/loader-overview/','/docs/dev/reference/tools/loader/']
aliases: ['/docs/dev/loader-overview/','/docs/dev/reference/tools/loader/','/docs/dev/load-misuse-handling/','/docs/dev/reference/tools/error-case-handling/load-misuse-handling/','/tidb/dev/load-misuse-handling']
---

# Loader Instructions

> **Warning:**
>
> Loader is no longer maintained. Its features are completely superseded by [TiDB Lightning TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md). It is strongly recommend to use TiDB Lightning instead.

## What is Loader?

Loader is a data import tool to load data to TiDB.
Expand All @@ -18,11 +22,11 @@ Since tools like mysqldump will take us days to migrate massive amounts of data,

## What can Loader do?

+ Multi-thread import data
+ Multi-threaded data import

+ Support table level concurrent import and scattered hot spot write
+ Support table-level concurrent import and scattered hotspot write

+ Support concurrent import of a single large table and scattered hot spot write
+ Support concurrent import of a single large table and scattered hotspot write

+ Support Mydumper data format

Expand Down Expand Up @@ -56,23 +60,23 @@ Since tools like mysqldump will take us days to migrate massive amounts of data,

-d string: the storage directory of data that need to import (default: "./")

-h string: the host of TiDB (default: "127.0.0.1")
-h string: the host IP of TiDB (default: "127.0.0.1")

-p string: the account and password of TiDB

-status-addr string: It can be used by Prometheus to pull Loader metrics, and is also the pprof address of Loader (default: ":8272")

-t int: the number of thread,increase this as TiKV nodes increase (default: 16)
-t int: the number of threads. Each worker restores one file at a time. (default: 16)

-u string: the user name of TiDB (default: "root")
```

### Configuration file

Apart from command line parameters, you can also use configuration files. The format is shown as below:
Apart from command-line parameters, you can also use configuration files. The format is shown as below:

```toml
# Loader log level, which can be set as "debug", "info", "warn", "error" and "fatal" (default: "info")
# Loader log level, which can be set as "debug", "info", "warn", "error", and "fatal" (default: "info")
log-level = "info"

# Loader log file
Expand All @@ -87,7 +91,7 @@ status-addr = ":8272"
# The checkpoint data is saved to TiDB, and the schema name is defined here.
checkpoint-schema = "tidb_loader"

# Number of threads restoring concurrently for worker pool (default: 16). Each worker restore one file at a time.
# Number of threads restoring concurrently for worker pool (default: 16). Each worker restores one file at a time.
pool-size = 16

# The target database information
Expand Down Expand Up @@ -150,3 +154,36 @@ pattern-table = "table_*"
target-schema = "example_db"
target-table = "table"
```

## Error: ```Try adjusting the `max_allowed_packet` variable```

The following error is reported during full data import:

```
packet for query is too large. Try adjusting the 'max_allowed_packet' variable
```

### Reasons

* Both MySQL client and MySQL/TiDB Server have `max_allowed_packet` quotas. If any of the `max_allowed_packet` quotas is violated, the client receives a corresponding error message. Currently, the latest version of Loader and TiDB Server all have a default `max_allowed_packet` quota of `64M`.
* Please use the latest version, or the latest stable version of the tool. See the [download page](/download-ecosystem-tools.md).
* The full data import processing module in Loader or DM does not support splitting `dump sqls` files. This is because Mydumper has the simple code implementation, as shown in the code comment `/* Poor man's data dump code */`. To support correctly splitting `dump sqls` files, you need to implement a sound parser based on TiDB parser, but you will encounter the following issues:
* A large amount of workload.
* The task is difficult. It is not easy to ensure the correctness.
* Significant performance reduction.

### Solutions

* For the above reasons, it is recommended to use the `-s, --statement-size` option which Mydumper offers to control the size of `Insert Statement`: `Attempted size of INSERT statement in bytes, default 1000000`.

According to the default configuration of `--statement-size`, the size of `Insert Statement` that Mydumper generates is as close as `1M`. This default configuration ensures that this error will not occur in most cases.

Sometimes the following `WARN` log appears during the dump process. The `WARN` log itself does not affect the dump process but indicates that the dumped table might be a wide table.

```
Row bigger than statement_size for xxx
```

* If a single row of a wide table exceeds 64M, you need to modify and enable the following two configurations:
* Execute `set @@global.max_allowed_packet=134217728` (`134217728 = 128M`) in TiDB Server.
* Add `max-allowed-packet=128M` to db configuration in the Loader configuration file according to your situation, and then restart the progress or task.