From 1e9a620c25a2b83ca1a447f9950236124e05990d Mon Sep 17 00:00:00 2001 From: kennytm Date: Tue, 30 Jun 2020 03:17:01 +0800 Subject: [PATCH 1/4] *: replaced black-white-list by table-filter --- TOC.md | 3 +- br/backup-and-restore-tool.md | 37 +++ table-filter.md | 252 ++++++++++++++++++ .../tidb-lightning-configuration.md | 8 +- tidb-lightning/tidb-lightning-glossary.md | 16 +- tidb-lightning/tidb-lightning-table-filter.md | 130 --------- 6 files changed, 305 insertions(+), 141 deletions(-) create mode 100644 table-filter.md delete mode 100644 tidb-lightning/tidb-lightning-table-filter.md diff --git a/TOC.md b/TOC.md index d06064b0dbab1..6379bb538d641 100644 --- a/TOC.md +++ b/TOC.md @@ -151,7 +151,7 @@ + [Configure](/tidb-lightning/tidb-lightning-configuration.md) + Key Features + [Checkpoints](/tidb-lightning/tidb-lightning-checkpoints.md) - + [Table Filter](/tidb-lightning/tidb-lightning-table-filter.md) + + [Table Filter](/table-filter.md) + [CSV Support](/tidb-lightning/migrate-from-csv-using-tidb-lightning.md) + [TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md) + [Web Interface](/tidb-lightning/tidb-lightning-web-interface.md) @@ -418,6 +418,7 @@ + [Errors Codes](/error-codes.md) + [TiCDC Overview](/ticdc/ticdc-overview.md) + [TiCDC Open Protocol](/ticdc/ticdc-open-protocol.md) + + [Table Filter](/table-filter.md) + FAQs + [TiDB FAQs](/faq/tidb-faq.md) + [TiDB Lightning FAQs](/tidb-lightning/tidb-lightning-faq.md) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 327e4f45c24d2..5d8350782f0ac 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -269,6 +269,25 @@ For descriptions of other options, see [Back up all cluster data](#back-up-all-t A progress bar is displayed in the terminal during the backup operation. When the progress bar advances to 100%, the backup is complete. Then the BR also checks the backup data to ensure data safety. +### Back up with table filter + +To back up multiple tables with more complex criteria, execute the `br backup full` command and specify the [table filters](/table-filter.md) with `--filter` or `-f`. + +**Usage example:** + +Back up the data of all tables in the form `db*.tbl*` to the `/tmp/backup` path on each TiKV node and write the `backupmeta` file to this path. + +{{< copyable "shell-regular" >}} + +```shell +br backup full \ + --pd "${PDIP}:2379" \ + --filter 'db*.tbl*' \ + --storage "local:///tmp/backup" \ + --ratelimit 120 \ + --log-file backuptable.log +``` + ### Back up data to Amazon S3 backend If you back up the data to the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access Amazon S3. @@ -443,6 +462,24 @@ br restore table \ In the above command, `--table` specifies the name of the table to be restored. For descriptions of other options, see [Restore all backup data](#restore-all-the-backup-data) and [Restore a database](#restore-a-database). +### Restore with table filter + +To restore multiple tables with more complex criteria, execute the `br restore full` command and specify the [table filters](/table-filter.md) with `--filter` or `-f`. + +**Usage example:** + +Restore a subset of tables backed up in the `/tmp/backup` path to the cluster. + +{{< copyable "shell-regular" >}} + +```shell +br restore full \ + --pd "${PDIP}:2379" \ + --filter 'db*.tbl*' \ + --storage "local:///tmp/backup" \ + --log-file restorefull.log +``` + ### Restore data from Amazon S3 backend If you restore data from the Amazon S3 backend, instead of `local` storage, you need to specify the S3 storage path in the `storage` sub-command, and allow the BR node and the TiKV node to access Amazon S3. diff --git a/table-filter.md b/table-filter.md new file mode 100644 index 0000000000000..242b1ada6a6b4 --- /dev/null +++ b/table-filter.md @@ -0,0 +1,252 @@ +--- +title: Table Filter +summary: Usage of table filter feature in TiDB tools. +category: reference +aliases: ['/docs/dev/tidb-lightning/tidb-lightning-table-filter/', '/docs/dev/reference/tools/tidb-lightning/table-filter/', '/tidb-lightning-table-filter/'] +--- + +# Table Filter + +The TiDB ecosystem tools operate on the all databases by default, but oftentimes only a subset is needed. Say, we only want to work with the schemas in the form `foo*` and `bar*` and nothing else. + +All TiDB tools since 4.0 share a common filter syntax to define subsets, which we specify here. + +## Usage + +### CLI + +Table filters can be supplied to the tools using multiple `-f` or `--filter` command line parameters. Each filter is in the form `db.table`, where each part can be a wildcard (further explained in the next section). The following lists example usage in each tool. + +* [BR](/br/backup-and-restore-tool.md): + + {{< copyable "shell-regular" >}} + + ```bash + ./br backup full -f 'foo*.*' -f 'bar*.*' -s 'local:///tmp/backup' + # ^~~~~~~~~~~~~~~~~~~~~~~ + ./br restore full -f 'foo*.*' -f 'bar*.*' -s 'local:///tmp/backup' + # ^~~~~~~~~~~~~~~~~~~~~~~ + ``` + +* [Dumpling](/export-or-backup-using-dumpling.md): + + {{< copyable "shell-regular" >}} + + ```bash + ./dumpling -f 'foo*.*' -f 'bar*.*' -P 3306 -o /tmp/data/ + # ^~~~~~~~~~~~~~~~~~~~~~~ + ``` + +* [Lightning](/tidb-lightning/tidb-lightning-overview.md): + + {{< copyable "shell-regular" >}} + + ```bash + ./tidb-lightning -f 'foo*.*' -f 'bar*.*' -d /tmp/data/ --backend tidb + # ^~~~~~~~~~~~~~~~~~~~~~~ + ``` + +### TOML configuration files + +Table filters in TOML files are specified as [array of strings](https://toml.io/en/v1.0.0-rc.1#section-15). The following lists example usage in each tool. + +* Lightning: + + ```toml + [mydumper] + filter = ['foo*.*', 'bar*.*'] + ``` + +* CDC: + + ```toml + [filter] + rules = ['foo*.*', 'bar*.*'] + + [[sink.dispatchers]] + matcher = ['db1.*', 'db2.*', 'db3.*'] + dispatcher = 'ts' + ``` + +## Syntax + +### Plain table names + +Each table filter rule consists of a "schema pattern" and a "table pattern", separated by a dot (`.`). Tables which fully-qualified name matching the rules are accepted. + +``` +db1.tbl1 +db2.tbl2 +db3.tbl3 +``` + +A plain name must only consist of valid [identifier characters](/schema-object-names.md), that is + +* digits (`0` to `9`) +* letters (`a` to `z`, `A` to `Z`) +* `$` +* `_` +* non ASCII characters (U+0080 to U+10FFFF) + +All other ASCII characters are reserved. Some punctuations have special meanings, described below. + +### Wildcards + +Each part of the name can be a wildcard symbol described in [fnmatch(3)](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_13): + +* `*` — matches zero or more characters +* `?` — matches one character +* `[a-z]` — matches one character between "a" and "z" inclusively +* `[!a-z]` — matches one character except "a" to "z". + +``` +db[0-9].tbl[0-9a-f][0-9a-f] +data.* +*.backup_* +``` + +"Character" here means a Unicode code point, for instance + +* U+00E9 (é) is 1 character. +* U+0065 U+0301 (é) are 2 characters. +* U+1F926 U+1F3FF U+200D U+2640 U+FE0F (🤦🏿‍♀️) are 5 characters. + +### File import + +Include an `@` at the beginning of the rule to specify a file name. The table filter parser treats each line of the imported file as additional filter rules. + +For example, if a file `config/filter.txt` has content: + +``` +employees.* +*.WorkOrder +``` + +the following two invocations are equivalent: + +```bash +./dumpling -f '@config/filter.txt' +./dumpling -f 'employees.*' -f '*.WorkOrder' +``` + +A filter file cannot further import another file. + +### Comments and blank lines + +Inside a filter file, leading and trailing white-spaces of every line are trimmed. Furthermore, blank lines (empty strings) are ignored. + +A leading `#` marks a comment and is ignored. `#` not at start of line may be considered syntax error. + +``` +# this line is a comment +db.table # but this part is not comment and may cause error +``` + +### Exclusion + +An `!` at the beginning of the rule means the pattern after it is used to exclude tables from being processed. This effectively turns the filter into a block list. + +``` +*.* +#^ note: must add the *.* to include all tables first +!*.Password +!employees.salaries +``` + +### Escape character + +Precede any special character by a `\` to turn it into an identifier character. + +``` +db\.with\.dots.* +``` + +For simplicity and future compatibility, the following sequences are prohibited: + +* `\` at the end of the line after trimming whitespaces (use `[ ]` to match a literal whitespace at the end). +* `\` followed by any ASCII alphanumeric character (`[0-9a-zA-Z]`). In particular, C-like escape sequences like `\0`, `\r`, `\n` and `\t` currently are meaningless. + +### Quoted identifier + +Besides `\`, special characters can also be suppressed by quoting using `"` or `` ` ``. + +``` +"db.with.dots"."tbl\1" +`db.with.dots`.`tbl\2` +``` + +The quote characters can be included within an identifier by doubling the character. + +``` +"foo""bar".`foo``bar` +# equivalent to: +foo\"bar.foo\`bar +``` + +Quoted identifier cannot span multiple lines. + +It is invalid to partially quote an identifier: + +``` +"this is "invalid*.* +``` + +### Regular expression + +In case very complex rules are needed, each pattern can be written as a regular expression delimited with `/`: + +``` +/^db\d{2,}$/./^tbl\d{2,}$/ +``` + +These regular expressions use the [Go dialect](https://pkg.go.dev/regexp/syntax?tab=doc). The pattern is matched if the identifier contains a substring matching the regular expression. For instance, `/b/` matches `db01`. + +> **Note:** +> +> Every `/` in the regex must be escaped as `\/`, including inside `[…]`. You cannot place an unescaped `/` between `\Q…\E`. + +## Multiple rules + +When a table name matches none of the rules in the filter list, the default behavior is to ignore such unmatched tables. + +To build a block list, an explicit `*.*` must be used as the first rule, otherwise all tables will be excluded. + +```bash +# every table will be filtered out +./dumpling -f '!*.Password' + +# only the "Password" table is filtered out, the rest are included. +./dumpling -f '*.*' -f '!*.Password' +``` + +In a filter list, if a table name matches multiple patterns, the last match decides the outcome. For instance, given + +``` +# rule 1 +employees.* +# rule 2 +!*.dep* +# rule 3 +*.departments +``` + +We get: + +| Table name | Rule 1 | Rule 2 | Rule 3 | Outcome | +|-----------------------|--------|--------|--------|------------------| +| irrelevant.table | | | | Default (reject) | +| employees.employees | ✓ | | | Rule 1 (accept) | +| employees.dept_emp | ✓ | ✓ | | Rule 2 (reject) | +| employees.departments | ✓ | ✓ | ✓ | Rule 3 (accept) | +| else.departments | | ✓ | ✓ | Rule 3 (accept) | + +> **Note:** +> +> In TiDB tools, the system schemas are always excluded regardless of the table filter settings. The system schemas are: +> +> * `INFORMATION_SCHEMA` +> * `PERFORMANCE_SCHEMA` +> * `METRICS_SCHEMA` +> * `INSPECTION_SCHEMA` +> * `mysql` +> * `sys` diff --git a/tidb-lightning/tidb-lightning-configuration.md b/tidb-lightning/tidb-lightning-configuration.md index 6b06422ed8b66..2965294d1f805 100644 --- a/tidb-lightning/tidb-lightning-configuration.md +++ b/tidb-lightning/tidb-lightning-configuration.md @@ -164,6 +164,9 @@ strict-format = false # parallel. max-region-size is the maximum size of each chunk after splitting. # max-region-size = 268_435_456 # Byte (default = 256 MB) +# Only import tables if these wildcard rules are matched. See the corresponding section for details. +filter = ['*.*'] + # Configures how CSV files are parsed. [mydumper.csv] # Separator between fields, should be an ASCII character. @@ -256,10 +259,6 @@ analyze = true switch-mode = "5m" # Duration between which an import progress is printed to the log. log-progress = "5m" - -# Table filter options. See the corresponding section for details. -#[black-white-list] -# ... ``` ### TiKV Importer @@ -351,6 +350,7 @@ min-available-ratio = 0.05 | -V | Prints program version | | | -d *directory* | Directory of the data dump to read from | `mydumper.data-source-dir` | | -L *level* | Log level: debug, info, warn, error, fatal (default = info) | `lightning.log-level` | +| -f *rule* | [Table filter rules](/table-filter.md) (can be specified multiple times) | `mydumper.filter` | | --backend *backend* | [Delivery backend](/tidb-lightning/tidb-lightning-tidb-backend.md) (`importer` or `tidb`) | `tikv-importer.backend` | | --log-file *file* | Log file path (default = a temporary file in `/tmp`) | `lightning.log-file` | | --status-addr *ip:port* | Listening address of the TiDB Lightning server | `lightning.status-port` | diff --git a/tidb-lightning/tidb-lightning-glossary.md b/tidb-lightning/tidb-lightning-glossary.md index 8ac7191e9600a..1fd772c5c953d 100644 --- a/tidb-lightning/tidb-lightning-glossary.md +++ b/tidb-lightning/tidb-lightning-glossary.md @@ -35,12 +35,6 @@ Back end is the destination where TiDB Lightning sends the parsed result. Also s See [TiDB Lightning TiDB-backend](/tidb-lightning/tidb-lightning-tidb-backend.md) for details. -### Black-white list - -A configuration list that specifies which tables to be imported and which should be excluded. - -See [TiDB Lightning Table Filter](/tidb-lightning/tidb-lightning-table-filter.md) for details. - ## C @@ -103,6 +97,16 @@ Engines use TiKV Importer's `import-dir` as temporary storage, which are sometim See also [data engine](/tidb-lightning/tidb-lightning-glossary.md#data-engine) and [index engine](/tidb-lightning/tidb-lightning-glossary.md#index-engine). + + +## F + +### Filter + +A configuration list that specifies which tables to be imported or excluded. + +See [Table Filter](/table-filter.md) for details. + ## I diff --git a/tidb-lightning/tidb-lightning-table-filter.md b/tidb-lightning/tidb-lightning-table-filter.md deleted file mode 100644 index 2b0f33326da86..0000000000000 --- a/tidb-lightning/tidb-lightning-table-filter.md +++ /dev/null @@ -1,130 +0,0 @@ ---- -title: TiDB Lightning Table Filter -summary: Use black and white lists to filter out tables, ignoring them during import. -category: reference -aliases: ['/docs/dev/tidb-lightning/tidb-lightning-table-filter/','/docs/dev/reference/tools/tidb-lightning/table-filter/'] ---- - -# TiDB Lightning Table Filter - -TiDB Lightning supports setting up black and white lists to ignore certain databases and tables. This can be used to skip cache tables, or manually partition the data source on a shared storage to allow multiple Lightning instances work together without interfering each other. - -The filtering rule is similar to MySQL `replication-rules-db`/`replication-rules-table`. - -## Filtering databases - -```toml -[black-white-list] -do-dbs = ["pattern1", "pattern2", "pattern3"] -ignore-dbs = ["pattern4", "pattern5"] -``` - -* If the `do-dbs` array in the `[black-white-list]` section is not empty, - * If the name of a database matches *any* pattern in the `do-dbs` array, the database is included. - * Otherwise, the database is skipped. -* Otherwise, if the name matches *any* pattern in the `ignore-dbs` array, the database is skipped. -* If a database’s name matches *both* the `do-dbs` and `ignore-dbs` arrays, the database is included. - -The pattern can either be a simple name, or a regular expression in [Go dialect](https://golang.org/pkg/regexp/syntax/#hdr-syntax) if it starts with a `~` character. - -> **Note:** -> -> The system databases `INFORMATION_SCHEMA`, `PERFORMANCE_SCHEMA`, `mysql` and `sys` are always black-listed regardless of the table filter settings. - -## Filtering tables - -```toml -[[black-white-list.do-tables]] -db-name = "db-pattern-1" -tbl-name = "table-pattern-1" - -[[black-white-list.do-tables]] -db-name = "db-pattern-2" -tbl-name = "table-pattern-2" - -[[black-white-list.do-tables]] -db-name = "db-pattern-3" -tbl-name = "table-pattern-3" - -[[black-white-list.ignore-tables]] -db-name = "db-pattern-4" -tbl-name = "table-pattern-4" - -[[black-white-list.ignore-tables]] -db-name = "db-pattern-5" -tbl-name = "table-pattern-5" -``` - -* If the `do-tables` array is not empty, - * If the qualified name of a table matched *any* pair of patterns in the `do-tables` array, the table is included. - * Otherwise, the table is skipped -* Otherwise, if the qualified name matched *any* pair of patterns in the `ignore-tables` array, the table is skipped. -* If a table’s qualified name matched *both* the `do-tables` and `ignore-tables` arrays, the table is included. - -Note that the database filtering rules are applied before Lightning considers the table filtering rules. This means if a database is ignored by `ignore-dbs`, all tables inside this database are not considered even if they matches any `do-tables` array. - -## Example - -To illustrate how these rules work, suppose the data source contains the following tables: - -``` -`logs`.`messages_2016` -`logs`.`messages_2017` -`logs`.`messages_2018` -`forum`.`users` -`forum`.`messages` -`forum_backup_2016`.`messages` -`forum_backup_2017`.`messages` -`forum_backup_2018`.`messages` -`admin`.`secrets` -``` - -Using this configuration: - -```toml -[black-white-list] -do-dbs = [ - "forum_backup_2018", # rule A - "~^(logs|forum)$", # rule B -] -ignore-dbs = [ - "~^forum_backup_", # rule C -] - -[[black-white-list.do-tables]] # rule D -db-name = "logs" -tbl-name = "~_2018$" - -[[black-white-list.ignore-tables]] # rule E -db-name = "~.*" -tbl-name = "~^messages.*" - -[[black-white-list.do-tables]] # rule F -db-name = "~^forum.*" -tbl-name = "messages" -``` - -First apply the database rules: - -| Database | Outcome | -|---------------------------|--------------------------------------------| -| `` `logs` `` | Included by rule B | -| `` `forum` `` | Included by rule B | -| `` `forum_backup_2016` `` | Skipped by rule C | -| `` `forum_backup_2017` `` | Skipped by rule C | -| `` `forum_backup_2018` `` | Included by rule A (rule C will not apply) | -| `` `admin` `` | Skipped since `do-dbs` is not empty and this does not match any pattern | - -Then apply the table rules: - -| Table | Outcome | -|--------------------------------------|--------------------------------------------| -| `` `logs`.`messages_2016` `` | Skipped by rule E | -| `` `logs`.`messages_2017` `` | Skipped by rule E | -| `` `logs`.`messages_2018` `` | Included by rule D (rule E will not apply) | -| `` `forum`.`users` `` | Skipped, since `do-tables` is not empty and this does not match any pattern | -| `` `forum`.`messages` `` | Included by rule F (rule E will not apply) | -| `` `forum_backup_2016`.`messages` `` | Skipped, since database is already skipped | -| `` `forum_backup_2017`.`messages` `` | Skipped, since database is already skipped | -| `` `forum_backup_2018`.`messages` `` | Included by rule F (rule E will not apply) | -| `` `admin`.`secrets` `` | Skipped, since database is already skipped | From e0b5990090a3d29a7fced9bc6a31aa9948e6dee1 Mon Sep 17 00:00:00 2001 From: kennytm Date: Tue, 30 Jun 2020 21:16:44 +0800 Subject: [PATCH 2/4] Apply suggestions from code review Co-authored-by: Ran --- br/backup-and-restore-tool.md | 4 ++-- table-filter.md | 40 +++++++++++++++++------------------ 2 files changed, 22 insertions(+), 22 deletions(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 5d8350782f0ac..8998f6fe13b11 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -275,7 +275,7 @@ To back up multiple tables with more complex criteria, execute the `br backup fu **Usage example:** -Back up the data of all tables in the form `db*.tbl*` to the `/tmp/backup` path on each TiKV node and write the `backupmeta` file to this path. +The following command backs up the data of all tables in the form `db*.tbl*` to the `/tmp/backup` path on each TiKV node and writes the `backupmeta` file to this path. {{< copyable "shell-regular" >}} @@ -468,7 +468,7 @@ To restore multiple tables with more complex criteria, execute the `br restore f **Usage example:** -Restore a subset of tables backed up in the `/tmp/backup` path to the cluster. +The following command restores a subset of tables backed up in the `/tmp/backup` path to the cluster. {{< copyable "shell-regular" >}} diff --git a/table-filter.md b/table-filter.md index 242b1ada6a6b4..09d47f9de7d3c 100644 --- a/table-filter.md +++ b/table-filter.md @@ -2,26 +2,26 @@ title: Table Filter summary: Usage of table filter feature in TiDB tools. category: reference -aliases: ['/docs/dev/tidb-lightning/tidb-lightning-table-filter/', '/docs/dev/reference/tools/tidb-lightning/table-filter/', '/tidb-lightning-table-filter/'] +aliases: ['/docs/dev/tidb-lightning/tidb-lightning-table-filter/','/docs/dev/reference/tools/tidb-lightning/table-filter/','/tidb/dev/tidb-lightning-table-filter/'] --- # Table Filter -The TiDB ecosystem tools operate on the all databases by default, but oftentimes only a subset is needed. Say, we only want to work with the schemas in the form `foo*` and `bar*` and nothing else. +The TiDB ecosystem tools operate on all the databases by default, but oftentimes only a subset is needed. For example, you only want to work with the schemas in the form of `foo*` and `bar*` and nothing else. -All TiDB tools since 4.0 share a common filter syntax to define subsets, which we specify here. +Since TiDB 4.0, all TiDB ecosystem tools share a common filter syntax to define subsets. This document describes how to use the table filter feature. ## Usage ### CLI -Table filters can be supplied to the tools using multiple `-f` or `--filter` command line parameters. Each filter is in the form `db.table`, where each part can be a wildcard (further explained in the next section). The following lists example usage in each tool. +Table filters can be applied to the tools using multiple `-f` or `--filter` command line parameters. Each filter is in the form of `db.table`, where each part can be a wildcard (further explained in the [next section](#wildcards)). The following lists the example usage in each tool. * [BR](/br/backup-and-restore-tool.md): {{< copyable "shell-regular" >}} - ```bash + ```shell ./br backup full -f 'foo*.*' -f 'bar*.*' -s 'local:///tmp/backup' # ^~~~~~~~~~~~~~~~~~~~~~~ ./br restore full -f 'foo*.*' -f 'bar*.*' -s 'local:///tmp/backup' @@ -48,7 +48,7 @@ Table filters can be supplied to the tools using multiple `-f` or `--filter` com ### TOML configuration files -Table filters in TOML files are specified as [array of strings](https://toml.io/en/v1.0.0-rc.1#section-15). The following lists example usage in each tool. +Table filters in TOML files are specified as [array of strings](https://toml.io/en/v1.0.0-rc.1#section-15). The following lists the example usage in each tool. * Lightning: @@ -57,7 +57,7 @@ Table filters in TOML files are specified as [array of strings](https://toml.io/ filter = ['foo*.*', 'bar*.*'] ``` -* CDC: +* [TiCDC](/ticdc/ticdc-overview.md): ```toml [filter] @@ -72,7 +72,7 @@ Table filters in TOML files are specified as [array of strings](https://toml.io/ ### Plain table names -Each table filter rule consists of a "schema pattern" and a "table pattern", separated by a dot (`.`). Tables which fully-qualified name matching the rules are accepted. +Each table filter rule consists of a "schema pattern" and a "table pattern", separated by a dot (`.`). Tables whose fully-qualified name matches the rules are accepted. ``` db1.tbl1 @@ -80,7 +80,7 @@ db2.tbl2 db3.tbl3 ``` -A plain name must only consist of valid [identifier characters](/schema-object-names.md), that is +A plain name must only consist of valid [identifier characters](/schema-object-names.md), such as: * digits (`0` to `9`) * letters (`a` to `z`, `A` to `Z`) @@ -88,7 +88,7 @@ A plain name must only consist of valid [identifier characters](/schema-object-n * `_` * non ASCII characters (U+0080 to U+10FFFF) -All other ASCII characters are reserved. Some punctuations have special meanings, described below. +All other ASCII characters are reserved. Some punctuations have special meanings, as described in the next section. ### Wildcards @@ -105,7 +105,7 @@ data.* *.backup_* ``` -"Character" here means a Unicode code point, for instance +"Character" here means a Unicode code point, such as: * U+00E9 (é) is 1 character. * U+0065 U+0301 (é) are 2 characters. @@ -113,9 +113,9 @@ data.* ### File import -Include an `@` at the beginning of the rule to specify a file name. The table filter parser treats each line of the imported file as additional filter rules. +To import a file as the filter rule, include an `@` at the beginning of the rule to specify the file name. The table filter parser treats each line of the imported file as additional filter rules. -For example, if a file `config/filter.txt` has content: +For example, if a file `config/filter.txt` has the following content: ``` employees.* @@ -135,7 +135,7 @@ A filter file cannot further import another file. Inside a filter file, leading and trailing white-spaces of every line are trimmed. Furthermore, blank lines (empty strings) are ignored. -A leading `#` marks a comment and is ignored. `#` not at start of line may be considered syntax error. +A leading `#` marks a comment and is ignored. `#` not at start of line is considered syntax error. ``` # this line is a comment @@ -155,7 +155,7 @@ An `!` at the beginning of the rule means the pattern after it is used to exclud ### Escape character -Precede any special character by a `\` to turn it into an identifier character. +To turn a special character into an identifier character, precede it with a backslash `\`. ``` db\.with\.dots.* @@ -175,7 +175,7 @@ Besides `\`, special characters can also be suppressed by quoting using `"` or ` `db.with.dots`.`tbl\2` ``` -The quote characters can be included within an identifier by doubling the character. +The quotation mark can be included within an identifier by doubling itself. ``` "foo""bar".`foo``bar` @@ -183,7 +183,7 @@ The quote characters can be included within an identifier by doubling the charac foo\"bar.foo\`bar ``` -Quoted identifier cannot span multiple lines. +Quoted identifiers cannot span multiple lines. It is invalid to partially quote an identifier: @@ -203,7 +203,7 @@ These regular expressions use the [Go dialect](https://pkg.go.dev/regexp/syntax? > **Note:** > -> Every `/` in the regex must be escaped as `\/`, including inside `[…]`. You cannot place an unescaped `/` between `\Q…\E`. +> Every `/` in the regular expression must be escaped as `\/`, including inside `[…]`. You cannot place an unescaped `/` between `\Q…\E`. ## Multiple rules @@ -219,7 +219,7 @@ To build a block list, an explicit `*.*` must be used as the first rule, otherwi ./dumpling -f '*.*' -f '!*.Password' ``` -In a filter list, if a table name matches multiple patterns, the last match decides the outcome. For instance, given +In a filter list, if a table name matches multiple patterns, the last match decides the outcome. For instance: ``` # rule 1 @@ -230,7 +230,7 @@ employees.* *.departments ``` -We get: +The filtered outcome is as follows: | Table name | Rule 1 | Rule 2 | Rule 3 | Outcome | |-----------------------|--------|--------|--------|------------------| From 4a3d6bfc3c86a398dfc3de9d8f9acf589217cca8 Mon Sep 17 00:00:00 2001 From: kennytm Date: Tue, 30 Jun 2020 21:18:58 +0800 Subject: [PATCH 3/4] table-filter: replace ```bash by ```shell --- table-filter.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/table-filter.md b/table-filter.md index 09d47f9de7d3c..80f44ac522ec8 100644 --- a/table-filter.md +++ b/table-filter.md @@ -32,7 +32,7 @@ Table filters can be applied to the tools using multiple `-f` or `--filter` comm {{< copyable "shell-regular" >}} - ```bash + ```shell ./dumpling -f 'foo*.*' -f 'bar*.*' -P 3306 -o /tmp/data/ # ^~~~~~~~~~~~~~~~~~~~~~~ ``` @@ -41,7 +41,7 @@ Table filters can be applied to the tools using multiple `-f` or `--filter` comm {{< copyable "shell-regular" >}} - ```bash + ```shell ./tidb-lightning -f 'foo*.*' -f 'bar*.*' -d /tmp/data/ --backend tidb # ^~~~~~~~~~~~~~~~~~~~~~~ ``` From 6dc3551f7c0000f559fca21c5a1213b035e2c687 Mon Sep 17 00:00:00 2001 From: kennytm Date: Wed, 1 Jul 2020 16:45:41 +0800 Subject: [PATCH 4/4] Update br/backup-and-restore-tool.md Co-authored-by: 3pointer --- br/backup-and-restore-tool.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/br/backup-and-restore-tool.md b/br/backup-and-restore-tool.md index 8998f6fe13b11..70d0bd9b015ba 100644 --- a/br/backup-and-restore-tool.md +++ b/br/backup-and-restore-tool.md @@ -285,7 +285,7 @@ br backup full \ --filter 'db*.tbl*' \ --storage "local:///tmp/backup" \ --ratelimit 120 \ - --log-file backuptable.log + --log-file backupfull.log ``` ### Back up data to Amazon S3 backend