Skip to content
This repository was archived by the owner on Jul 24, 2024. It is now read-only.
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
365 commits
Select commit Hold shift + click to select a range
9b71f1c
restore: write index kv pairs and data kv pairs to separate engine (#…
lonng Mar 8, 2019
9c6a5f6
Support CSV (#111)
kennytm Mar 11, 2019
acc626d
update version of tidb to latest release-2.1 (#138)
WangXiangUSTC Mar 11, 2019
e15468e
tidb-lightning-ctl: added --import-engine and --cleanup-engine comman…
kennytm Mar 13, 2019
6a64363
*: parameter type fix (#141)
amyangfei Mar 14, 2019
5c69d10
remove changelog and get change logs from release (#146)
IANTHEREAL Mar 15, 2019
59b0f24
test: fix cleanup script to enable re-run integration test locally (#…
amyangfei Mar 15, 2019
76bd3e1
restore: fix #140 lightning breaks on tidb-master (#147)
lonng Mar 15, 2019
852c5e9
fix deadlock and goroutine leak in chunk restore (#149)
IANTHEREAL Mar 18, 2019
d2e15ca
restore: eager release index engine worker after imported (#150)
lonng Mar 20, 2019
9991839
checkpoint: fix checkpoint not updated in some scenario (#151)
amyangfei Mar 22, 2019
c3886a8
Revert "fix deadlock and goroutine leak in chunk restore (#149)" (#152)
IANTHEREAL Mar 25, 2019
80f7c41
restore: add local checksum log (#153)
lonng Mar 27, 2019
82ed51b
tests: add kill lightning test in checkpoint_chunks case (#158)
amyangfei Apr 3, 2019
7bae12e
*: parse the data source directly into data and skip the KV encoder (…
kennytm Apr 3, 2019
43fdfc3
tests: fix a test failure due to conflict between #145 and #158 (#159)
kennytm Apr 4, 2019
c14d62a
restore: give priority to small tables for importing (#156)
lonng Apr 9, 2019
3f468b3
config: Allow overriding some config from command line (#157)
kennytm Apr 9, 2019
c103cd1
go.mod: upgrade dependencies, esp TiDB -> v3.0.0-beta.1 (#160)
kennytm Apr 15, 2019
fcbafa0
Fix interpretation of integers in a BIT column (#161)
kennytm Apr 16, 2019
ea3b27e
*: replace gofail with new failpoint implementation (#165)
lonng Apr 17, 2019
a515465
*: use pingcap/log (zap) for logging (#162)
kennytm Apr 19, 2019
2f304e1
main: sync log before exit + update failpoint dep (#168)
kennytm Apr 22, 2019
9adb660
kv: fix handling of column default values (#170)
kennytm Apr 25, 2019
9b895b6
tests,restore: re-enable the exotic_filenames test (#172)
kennytm Apr 26, 2019
889ce1b
config: reduce default table-concurrency from 8 to 6 (#175)
kennytm Apr 30, 2019
748f7e2
Support table routing rules (merging sharded tables) (#95)
kennytm May 4, 2019
c8ead05
tests: removes the emoji from a test database name (#179)
kennytm May 6, 2019
d477eaf
*: fix failpoint-ctl path, unify failpoint runtime and ctl to same ve…
amyangfei May 6, 2019
a3cda85
restore: fix the potential null pointer exception when logging progre…
kennytm May 6, 2019
877e839
kv,restore: log which value caused conversion failure (#154)
kennytm May 6, 2019
dd21072
restore: retry if deliver KVs to importer (#176)
lonng May 8, 2019
f8385c7
config: automatically discover tidb.pd-addr and tidb.port if not prov…
kennytm May 8, 2019
7bb11c2
*: added linters (#183)
kennytm May 8, 2019
00d4c68
common: improve unit test coverage of 'common' package (#186)
lonng May 13, 2019
973244e
common: improve unit test coverage of 'kv' package (#187)
lonng May 14, 2019
f4512ae
Make the parsers stricter, and improve unit test coverage of `mydump`…
kennytm May 14, 2019
08657c6
lightning: improve unit test coverage of 'lightning' package (#188)
lonng May 14, 2019
4d01a9b
README: update coverage status badge (#189)
lonng May 14, 2019
e98dbf8
config: improve unit test coverage of 'config' package (#192)
lonng May 16, 2019
78bb37c
Add unit tests for kv/importer and restore/checkpoints, plus some bug…
kennytm May 21, 2019
bcddb57
Add unit tests for 'restore.go' (TableRestore and chunkRestore) (#193)
kennytm May 27, 2019
3256747
go.mod: update dependencies (#197)
kennytm Jun 5, 2019
9fec239
config,lightning: Implements server mode (#198)
kennytm Jun 18, 2019
9d55221
go.mod: update dependencies (#200)
kennytm Jul 2, 2019
8601657
Post restore config fix (#202)
ericsyh Jul 3, 2019
adda917
Introduce a basic web interface (#199)
kennytm Jul 5, 2019
7097f66
Update README.md (#207)
lance6716 Jul 10, 2019
54796d7
Improve errors and logs on syntax error / conversion failure (#201)
kennytm Jul 11, 2019
ba6a2d2
tests,web: exclude these directories from the Go module (#209)
kennytm Jul 15, 2019
69e76c2
lightning/restore: fix ColumnPermutation calculation (#210)
lonng Jul 15, 2019
f3af17a
build(deps): bump lodash from 4.17.11 to 4.17.14 in /web (#213)
dependabot[bot] Jul 16, 2019
5218dd6
*: adjust solution for TOOL-1420 and add a test case (#214)
kennytm Jul 18, 2019
cf08f2f
tests: add test case for simple partitioned tables (#206)
kennytm Jul 19, 2019
666c34e
checkpoints: remove node_id field and rename the schema on keep-after…
kennytm Jul 24, 2019
034ba60
config: attempt to solve TOOL-1405 and modify old test cases (#217)
lance6716 Jul 25, 2019
ae59251
restore: fix gc life time not recovered after table restore (#218)
amyangfei Jul 25, 2019
af34025
*: abstract the Importer communication into an interface (#215)
kennytm Jul 29, 2019
e8d463a
restore: update and restore GCLifeTime once when parallel (#220)
lance6716 Aug 7, 2019
4eb74ec
*: support MySQL backend (#221)
kennytm Aug 14, 2019
1577f31
config: always skip the system databases (#225)
kennytm Aug 15, 2019
ce5fa5f
backend: update mysql backend to tidb backend (#228)
WangXiangUSTC Aug 21, 2019
56c3c04
lightning/common: add unit test (#229)
WangXiangUSTC Aug 22, 2019
befa68a
mock: update rpc endpoint (#226)
lance6716 Aug 23, 2019
ab23466
cmd: do not exit(1) if failed to sync log (#230)
kennytm Aug 26, 2019
7ecd9ad
mock: update rpc endpoint (#234)
lance6716 Aug 26, 2019
d49fec6
backend: dynamically calculate the maximum auto-inc ID (#227)
kennytm Aug 30, 2019
aaac2b4
checkpoint: fix empty map become nil after unmarshall (#237)
lance6716 Sep 6, 2019
b7d4675
config: increase default concurrency (#244)
lance6716 Sep 29, 2019
b38da1e
backend/tidb: use REPLACE INTO or INSERT IGNORE INTO to provide idemp…
lance6716 Oct 8, 2019
3052939
*: use fixed timestamp to ensure import stability wrt CURRENT_TIMESTA…
kennytm Oct 9, 2019
7dbd0c6
*: update dependencies (tidb -> 3.0.4) (#246)
kennytm Oct 18, 2019
0edbc12
improve the log when encountering invalid checkpoint (#247)
kennytm Nov 3, 2019
605760d
config: adding `[tidb] max-allowed-packet` config (#248)
kennytm Nov 20, 2019
8d57e70
Fix +incompatible suffix not allowed error reported by go mod (#249)
suzaku Dec 12, 2019
99e1ed0
allow use a separate pauser instead of global pauser (#251)
Dec 16, 2019
61a64ad
Add password as command line argument (#253)
Dec 17, 2019
42b7585
metrics: copy the grafana board into this repo (#256)
kennytm Jan 10, 2020
f9f6d8b
*: Update dependencies and fix unit test on Windows (#254)
kennytm Jan 14, 2020
b865826
config: synchronize actual default value with the toml file (#255)
kennytm Feb 4, 2020
bab3bc1
lightning: ensure the web interface still works outside server mode (…
kennytm Feb 5, 2020
4348bb1
Upgrade TiDB to 4.0.0-beta, and recognize @@tidb_row_format_version =…
kennytm Feb 10, 2020
9068e31
Support TLS; Reduce the need of config.toml in integration tests (#270)
kennytm Feb 27, 2020
e57567b
backend: define a reusable BufStore when creating a new session (#274)
kennytm Feb 29, 2020
619c2dc
Update Chinese doc url (#276)
kennytm Mar 5, 2020
093ce9f
Set session var for every new conn (#280)
july2993 Mar 10, 2020
3c8f4d7
lightning: split large csv file if possible (#272)
XuHuaiyu Mar 12, 2020
bf3c830
send a batch of kv in encodeLoop (#279)
XuHuaiyu Mar 16, 2020
f53609a
Replace CSV Ragel parser by a hand-written parser copied from encodin…
kennytm Mar 16, 2020
0f54c30
backend: fix issue 282 (#283)
kennytm Mar 16, 2020
98bc849
optimize the performance of lightning (#281)
july2993 Mar 16, 2020
56d0f7d
Some SwitchMode improvements (#287)
kennytm Apr 13, 2020
c0cbeb5
go.mod,web: update dependencies (#289)
kennytm Apr 13, 2020
b21f9ed
Support store version format generated by `git describe --tags` (#295)
solotzg Apr 20, 2020
8153e31
Warn for single large file, change switch mode log level to info (#315)
lichunzhu May 11, 2020
f3ce13e
restore: fix typo (#304)
kennytm May 11, 2020
3c3fa29
print lightning log to local file (#313)
lichunzhu May 11, 2020
c10636e
*: avoid accessing internal ports when backend=tidb (#312)
kennytm May 11, 2020
18cd93c
config: remove strict mode from default SQL mode (#316)
kennytm May 11, 2020
edfa861
check table id when loading checkpoint (#317)
lichunzhu May 11, 2020
391db4a
update tidb-tools to latest (#319)
3pointer May 14, 2020
18b82fb
support alter random && update tidb dependency to latest (#324)
3pointer Jun 5, 2020
cc190c1
backend: add local kv storage backend to get rid of importer (#326)
glorv Jun 10, 2020
99f6933
make lightning compatible with allow_auto_random_explicit_insert (#328)
3pointer Jun 11, 2020
368d52e
config: update example config file (#331)
glorv Jun 11, 2020
5b3b811
Fix test cases on release-3.0 (#330)
kennytm Jun 17, 2020
369eaed
optimize parse csv and local backend write tikv (#334)
glorv Jun 19, 2020
1dbbf30
fix checkpoint cleanup (#336)
glorv Jun 24, 2020
f8a9b65
lightning: fix web page not showing when not using server mode (#337)
kennytm Jun 24, 2020
cd6210d
config,mydumper: replace black-white-list by table-filter (#332)
kennytm Jun 24, 2020
4af19d5
optimize encoder and adjust some config (#338)
glorv Jun 28, 2020
25256f7
log: fix log file path (#345)
glorv Jul 8, 2020
3d6a4db
fix local backend index split range (#347)
glorv Jul 10, 2020
31c4b90
add log for environment http proxy setting (#340)
glorv Jul 10, 2020
f38aa6e
do not always change auto increment id (#348)
glorv Jul 14, 2020
73e48bb
server: check open file ulimit for local backend (#343)
glorv Jul 14, 2020
26a0f71
restore: do not rebase auto-id or generate auto row id for table with…
glorv Jul 15, 2020
32c2868
Fix verbose log message for shell (#352)
Jul 22, 2020
d00b370
local: fix batch split retry alway failed error (#356)
glorv Jul 24, 2020
aaee763
backend: fix handling of empty binary literals (#357)
kennytm Jul 24, 2020
94f7216
add log when execute statement failed (#359)
glorv Jul 28, 2020
1dc2d6a
check checkpoint schema (#354)
glorv Jul 28, 2020
d1024a2
parser: fix csv parse header with empty line (#364)
glorv Aug 6, 2020
f376161
restore: fix missing colum infos when restore from checkpoint (#362)
glorv Aug 7, 2020
d3dc3ea
local: fix import with common handle (#367)
glorv Aug 13, 2020
f18863a
restore: don't switch mode in tidb backend (#368)
glorv Aug 13, 2020
52a84b0
restore: support split csv source file with header (#363)
glorv Aug 19, 2020
b1b93fd
restore: support file level routing (#366)
glorv Aug 20, 2020
daa94d0
tidb_tools: update dependency (#371)
3pointer Aug 20, 2020
417d55f
restore: check header columns (#372)
glorv Aug 24, 2020
9fde4aa
web: update dependencies (#374)
kennytm Aug 24, 2020
0630432
backend: update committs from unix timestamp to pd tso (#379)
3pointer Aug 27, 2020
81e6118
update pd dependencies (#380)
glorv Aug 31, 2020
ee94dc5
local: return error if write to tikv returns no leader info (#381)
glorv Aug 31, 2020
63c72b3
encoder: check string value for tidb encoder (#378)
glorv Aug 31, 2020
c017df5
Fix running unit tests on Windows (#375)
kennytm Sep 1, 2020
e4da649
checkpoint: verify checkpoint when resume from checkpoint (#376)
glorv Sep 1, 2020
75fcfbe
test: change integration test script to allow run tests in parallel (…
glorv Sep 2, 2020
3cee9d5
backend: split and ingest region size more precise (#369)
glorv Sep 3, 2020
134c944
wait checkpoint finished if exit before success (#386)
glorv Sep 3, 2020
d23cf02
restore: support restore from s3 (#361)
glorv Sep 4, 2020
3c11d2e
backend: fix sample when split region size is small (#387)
glorv Sep 4, 2020
12fe8b1
backend: use peer address as grpc addr for tiflash store (#392)
glorv Sep 8, 2020
6e39bb8
loader: fix store.WalkDir return inaccurate file size for soft link s…
glorv Sep 11, 2020
81965be
fix chunk checkpoint may reset offset and row id (#395)
glorv Sep 11, 2020
8ec7eff
make tiflash test more stable (#397)
glorv Sep 14, 2020
b4dc94b
test: fix integration test for 3.x version (#390)
glorv Sep 16, 2020
4ea4e76
test: make start tiflash optional in integration test (#398)
glorv Sep 16, 2020
6e4c898
restore: support restore apache parquet format source files (#373)
glorv Sep 16, 2020
e8c9b88
backend: fix load partition table with local backend (#402)
glorv Sep 17, 2020
cc3f809
lightning: support dynamically modifying the log level (#393)
kennytm Sep 21, 2020
e29c752
Check Lightning version when reusing checkpoint (#383)
kennytm Sep 21, 2020
e4de23b
support new collation for kv encoder (#407)
glorv Sep 25, 2020
bce9977
mydump: support multi bytes csv delimiter and separator (#406)
glorv Sep 27, 2020
43b0914
backend: always retry ingest and get region if it's retryble (#405)
glorv Sep 28, 2020
a83a72e
Add license scan report and status (#399)
fossabot Sep 28, 2020
5972027
mydump: fix infinite loop in ExportStatement when Read() returns non-…
kennytm Oct 9, 2020
4fa18d1
lightning: start the HTTP server when receiving SIGUSR1 (#415)
kennytm Oct 9, 2020
e35cdcd
backend/tidb: fix issue 410 (#412)
kennytm Oct 9, 2020
e9b79ee
config: fix error on `-d 'C:\Windows\Path'` (#411)
kennytm Oct 10, 2020
aa029d8
local: fix infinity loop in retry get region (#418)
glorv Oct 12, 2020
da84e5d
backend: speed up uploading by open multi TCP connections (#400)
YuJuncen Oct 13, 2020
60c1c27
post-restore: add optional level for post-restore operations (#421)
glorv Oct 19, 2020
2660ffe
backend/local: do not retry epochNotMatch error when ingest sst (#419)
glorv Oct 19, 2020
44d81ba
restore: disable some pd scheduler during restore (#408)
glorv Oct 23, 2020
144bacc
log: simplify some warn log and do retry write for epoch not match er…
glorv Oct 23, 2020
c11e6bd
fix test (#426)
glorv Oct 23, 2020
5b212bf
backend: fix a bug about wrong column info (#420)
Oct 23, 2020
78e064d
restore: better estimate task remain time progress log (#377)
glorv Oct 23, 2020
aa83de1
checksum: use gc ttl api for checksum gc safepoint in v4.0 cluster (#…
glorv Oct 23, 2020
3ecec63
backend/tidb: add rebase auto id for tidb backend (#428)
glorv Oct 27, 2020
6887f4e
fix autoid for v4.0.0 (#430)
glorv Oct 27, 2020
4d97351
make: hide go.mod to resolve cyclic dependency with tidb (#439)
lance6716 Nov 2, 2020
8013216
config: support encode PostOpLevel and Duration as input (#441)
lance6716 Nov 4, 2020
eb609ec
restore: fix several bugs related to column permutations (#437)
glorv Nov 4, 2020
a1ac978
dep: update uuid dependency to latest google/uuid (#452)
lance6716 Nov 6, 2020
176d053
tidb-lightning-ctl: change default of -d to 'noop://' (#453)
kennytm Nov 6, 2020
25a3711
restore: fix the bug that gc life time ttl does not take effect (#448)
glorv Nov 9, 2020
66281df
config: filter out all system schemas by default (#459)
kennytm Nov 9, 2020
66ac09a
backend: fix auto random default value for primary key (#457)
glorv Nov 9, 2020
f4354c9
mydumper: fix parquet data parser (#435)
glorv Nov 9, 2020
51d0e7b
backend/local: use range properties to optimize region range estimate…
glorv Nov 10, 2020
780143a
fix pd service id is empty (#460)
glorv Nov 10, 2020
0e8cef2
fix s3 parquet reader (#461)
glorv Nov 10, 2020
963829f
fix service gc ttl again (#465)
glorv Nov 11, 2020
1d396c9
mydumper: verify file routing config (#470)
glorv Nov 12, 2020
ccf7f3c
config: allow four byte-size config to be specified using human-reada…
kennytm Nov 13, 2020
7e17f97
test: change double type syntax (#474)
lance6716 Nov 16, 2020
a78137f
restore: add `glue.Glue` interface and other function (#456)
lance6716 Nov 16, 2020
da8b24e
glue: add GlueCheckpointDB and remove external TiDB usage (#478)
lance6716 Nov 23, 2020
2644a19
*: replace context.Backend with app context (#468)
glorv Nov 23, 2020
f5b98ef
restore: wait sub task finish before exit (#485)
glorv Nov 24, 2020
16c7217
mydumper: convert parquet columns to lower case (#479)
glorv Nov 25, 2020
91ca438
test: fix an unstable integration test (#492)
glorv Nov 25, 2020
05bad62
mydumper: optimize parquet reader performance (#482)
glorv Nov 25, 2020
a15013c
restore: let the tikv checksum manager respect the DistSQLScanConcurr…
kennytm Nov 25, 2020
fbb62fa
support restore view (#417)
glorv Nov 26, 2020
cbad36a
backend/local: more robust range retry strategy (#476)
glorv Nov 26, 2020
8a276c0
backend/local: batch split region with batch limit (#487)
glorv Nov 27, 2020
b460fcf
kill tiflash by name (#499)
glorv Dec 1, 2020
95e4aa2
.github: let challenge-bot recognize the default SIG (#498)
kennytm Dec 1, 2020
a7fb063
backend: support stored generated columns in local/importer backends …
kennytm Dec 5, 2020
e363572
backend/local: check and return iter.Error when pebble is not valid (…
glorv Dec 8, 2020
a0b12ac
backend,restore: duplicate more important system variables from downs…
kennytm Dec 8, 2020
e0ffa0f
backend/local: set pebble db max file limit (#501)
glorv Dec 11, 2020
7cb623e
mydump: fix issue 519 (#521)
kennytm Dec 11, 2020
2fdd481
backend/local: remove useless and buggy truncate key (#516)
glorv Dec 11, 2020
4fd9cd6
restore: apply adjust max-pending-peer-count when stop pd schedulers …
glorv Dec 11, 2020
56bc32d
backend/local: fix next key (#523)
glorv Dec 14, 2020
ff241c3
backend: import planner/core package to initialize expression.Rewrite…
Dec 20, 2020
e9a1bfb
*: add some error description (#527)
glorv Dec 21, 2020
c2b6dde
test: fix unstable integration test auto_random_default (#529)
glorv Dec 21, 2020
7622f91
post-process: support run table analyze after all tables are finished…
glorv Dec 22, 2020
c3fc041
encode retry split key (#531)
glorv Dec 23, 2020
8510040
restore: fix error lost in create schema (#530)
3pointer Dec 23, 2020
463e728
mydumper: update br to apply auto retry s3 read error (#533)
glorv Dec 24, 2020
f842ab8
Sort index rather than insert it into skiplist (#520)
Little-Wallace Dec 28, 2020
1c04266
*: redact log and error messages, add log-redact parameter (#538)
lichunzhu Dec 28, 2020
1f02abc
mydumper: do not remove more than 1 sep if trim last sep is true (#535)
glorv Dec 29, 2020
b23840d
restore: add error retry for checksum by tikv (#537)
glorv Dec 31, 2020
d1becd4
post-process: allow run checksum at last and restrict the number of c…
glorv Jan 4, 2021
d89fe86
restore: don't change TiDB config to support lightning via SQL (#545)
lance6716 Jan 4, 2021
1709cc7
restore: check row value count to avoid unexpected encode result (#528)
glorv Jan 6, 2021
a401ad0
restore: Try to create tables in parallel (#502)
hidehalo Jan 6, 2021
0529577
Compatible for disk quota (#543)
Little-Wallace Jan 7, 2021
d9254b1
*: add method to check whether need local SST of table (#491)
lance6716 Jan 8, 2021
161f005
backend/local: skip split regions if engine total size is smaller tha…
glorv Jan 11, 2021
0c9788e
config: change redact log parameter name (#547)
lichunzhu Jan 13, 2021
deb5243
backend/tidb: temporarily disable the strict-mode value check in tidb…
glorv Jan 18, 2021
90dbfe3
test: fix invalid failpoint and integration test (#510)
glorv Jan 18, 2021
a098961
test: fix s3 integration test (#555)
glorv Jan 25, 2021
5b63605
grafana dashboards support multiple cluster (#556)
Jan 26, 2021
9f87f4e
metrics: use tidb_cluster label get variable values (#559)
Jan 27, 2021
6c382e2
restore: add importing progress and optimize the accuracy of restore …
glorv Jan 27, 2021
c69c738
backend/local: fallback retryIngest to retryWrite (#554)
glorv Jan 29, 2021
1f12021
backend: implement disk quota (#493)
kennytm Jan 29, 2021
2bdf07e
Add 'pkg/lightning/' from commit '1f120218b385d40702d7061830803cb11c1…
overvenus Feb 22, 2021
8f256be
*: replace pingcap/tidb-lightning with pingcap/br/pkg/lightning
overvenus Dec 28, 2020
b9d9b1a
tests: enable TLS by default
overvenus Dec 30, 2020
43e5393
tests: adjust success message and shrink tikv reserve space
overvenus Dec 31, 2020
9cfa18f
tests, pkg/lightning: migrate lightning test view
overvenus Jan 30, 2021
ec56142
tests: migrate lightning rest tests
overvenus Jan 30, 2021
f4349cb
lightning: move web to root directory
overvenus Jan 5, 2021
0b0b404
*: flatten lightning packages
overvenus Jan 5, 2021
14fd9e0
cmd: flatten br/cmd packages
overvenus Jan 5, 2021
bd7ed24
*: add challenge-bot.yml and address review comments
overvenus Feb 5, 2021
645d6d3
*: update go mod
overvenus Feb 18, 2021
926f421
remove tidb 5.0 specific changes
overvenus Feb 22, 2021
45831c6
fix unbound var and restore SQL without default charset
overvenus Feb 23, 2021
35bab09
Merge branch 'release-4.0' into release-4.0-merging
overvenus Mar 3, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*_generated.go linguist-generated=true
18 changes: 17 additions & 1 deletion .github/ISSUE_TEMPLATE/bug-report.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
name: "🐛 Bug Report"
about: Something isn't working as expected
title: ''
labels: 'bug'
labels: 'type/bug '
---

Please answer these questions before submitting your issue. Thanks!
Expand All @@ -23,7 +23,23 @@ If possible, provide a recipe for reproducing the error.

<!--
br -V
tidb-lightning -V
tidb-server -V
tikv-server -V
pd-server -V
-->

5. Operation logs
- Please upload `br.log` for BR if possible
- Please upload `tidb-lightning.log` for TiDB-Lightning if possible
- Please upload `tikv-importer.log` from TiKV-Importer if possible
- Other interesting logs


6. Configuration of the cluster and the task
- `tidb-lightning.toml` for TiDB-Lightning if possible
- `tikv-importer.toml` for TiKV-Importer if possible
- `topology.yml` if deployed by TiUP


7. Screenshot/exported-PDF of Grafana dashboard or metrics' graph in Prometheus if possible
2 changes: 1 addition & 1 deletion .github/ISSUE_TEMPLATE/feature-request.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
name: "🚀 Feature Request"
about: I have a suggestion
labels: enhancement
labels: 'type/feature-request'
---

## Feature Request
Expand Down
24 changes: 24 additions & 0 deletions .github/ISSUE_TEMPLATE/question.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
---
name: "\U0001F914 Question"
labels: "type/question"
about: Usage question that isn't answered in docs or discussion

---

## Question


<!--

Before asking a question, make sure you have:

- Searched existing Stack Overflow questions.
- Googled your question.
- Searched open and closed [GitHub issues](https://github.com/pingcap/br/issues?q=is%3Aissue)
- Read the documentation:
* [BR documentation](https://docs.pingcap.com/tidb/stable/backup-and-restore-tool)
* [Lightning documentation](https://docs.pingcap.com/tidb/stable/tidb-lightning-overview)
* [BR 中文文档](https://docs.pingcap.com/zh/tidb/stable/backup-and-restore-tool)
* [Lightning 中文文档](https://docs.pingcap.com/zh/tidb/stable/tidb-lightning-overview)

-->
1 change: 1 addition & 0 deletions .github/challenge-bot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
defaultSigLabel: sig/migrate
22 changes: 11 additions & 11 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
br
bin/
_tools/
.idea/
backupmeta
/br
/bin
/.idea
*.log
*.ngo
*.coverprofile
coverage.txt
docker/data/
docker/logs/
/docker/data/
/docker/logs/
*.swp
.DS_Store
go.mod
go.sum
/go.mod
/go.sum

# for the web interface
web/node_modules/
web/dist/
18 changes: 0 additions & 18 deletions CHANGELOG.md

This file was deleted.

123 changes: 102 additions & 21 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,66 +1,141 @@
PROTOC ?= $(shell which protoc)
PROTOS := $(shell find $(shell pwd) -type f -name '*.proto' -print)
CWD := $(shell pwd)
PACKAGES := go list ./...
CWD := $(shell pwd)
TOOLS := $(CWD)/tools/bin
PACKAGES := go list ./... | grep -vE 'vendor|test|proto|diff|bin|fuzz'
PACKAGE_DIRECTORIES := $(PACKAGES) | sed 's/github.com\/pingcap\/br\/*//'
PACKAGE_FILES := $$(find . -name '*.go' -type f | grep -vE 'vendor|\.pb\.go|lightning/mock|res_vfsdata')
CHECKER := awk '{ print } END { if (NR > 0) { exit 1 } }'

BR_PKG := github.com/pingcap/br

LDFLAGS += -X "$(BR_PKG)/pkg/utils.BRReleaseVersion=$(shell git describe --tags --dirty)"
VERSION := v4.0.0-dev
release_branch_regex := ^release-[0-9]\.[0-9].*$$
ifneq ($(shell git rev-parse --abbrev-ref HEAD | egrep $(release_branch_regex)),)
# If we are in release branch, use tag version.
VERSION := $(shell git describe --tags --dirty)
else ifneq ($(shell git status --porcelain),)
# Add -dirty if the working tree is dirty for non release branch.
VERSION := $(VERSION)-dirty
endif

LDFLAGS += -X "$(BR_PKG)/pkg/utils.BRReleaseVersion=$(VERSION)"
LDFLAGS += -X "$(BR_PKG)/pkg/utils.BRBuildTS=$(shell date -u '+%Y-%m-%d %I:%M:%S')"
LDFLAGS += -X "$(BR_PKG)/pkg/utils.BRGitHash=$(shell git rev-parse HEAD)"
LDFLAGS += -X "$(BR_PKG)/pkg/utils.BRGitBranch=$(shell git rev-parse --abbrev-ref HEAD)"
# TODO: unify LDFLAGS
LDFLAGS += -X "$(BR_PKG)/pkg/lightning/common.ReleaseVersion=$(VERSION)"
LDFLAGS += -X "$(BR_PKG)/pkg/lightning/common.BuildTS=$(shell date -u '+%Y-%m-%d %I:%M:%S')"
LDFLAGS += -X "$(BR_PKG)/pkg/lightning/common.GitHash=$(shell git rev-parse HEAD)"
LDFLAGS += -X "$(BR_PKG)/pkg/lightning/common.GitBranch=$(shell git rev-parse --abbrev-ref HEAD)"
LDFLAGS += -X "$(BR_PKG)/pkg/lightning/common.GoVersion=$(shell go version)"

LIGHTNING_BIN := bin/tidb-lightning
LIGHTNING_CTL_BIN := bin/tidb-lightning-ctl
BR_BIN := bin/br
VFSGENDEV_BIN := tools/bin/vfsgendev
TEST_DIR := /tmp/backup_restore_test

path_to_add := $(addsuffix /bin,$(subst :,/bin:,$(GOPATH)))
export PATH := $(path_to_add):$(PATH)

GOBUILD := CGO_ENABLED=1 GO111MODULE=on go build -trimpath -ldflags '$(LDFLAGS)'
GOTEST := CGO_ENABLED=1 GO111MODULE=on go test -ldflags '$(LDFLAGS)'
PREPARE_MOD := cp go.mod1 go.mod && cp go.sum1 go.sum
FINISH_MOD := cp go.mod go.mod1 && cp go.sum go.sum1

RACE_FLAG =
ifeq ("$(WITH_RACE)", "1")
RACEFLAG = -race
RACE_FLAG = -race
GOBUILD = CGO_ENABLED=1 GO111MODULE=on $(GO) build -ldflags '$(LDFLAGS)'
endif

all: build check test

prepare:
$(PREPARE_MOD)

build:
finish-prepare:
$(FINISH_MOD)

%_generated.go: %.rl
ragel -Z -G2 -o tmp_parser.go $<
@echo '// Code generated by ragel DO NOT EDIT.' | cat - tmp_parser.go | sed 's|//line |//.... |g' > $@
@rm tmp_parser.go

data_parsers: tools pkg/lightning/mydump/parser_generated.go web
PATH="$(GOPATH)/bin":"$(PATH)":"$(TOOLS)" protoc -I. -I"$(GOPATH)/src" pkg/lightning/checkpoints/file_checkpoints.proto --gogofaster_out=.
$(TOOLS)/vfsgendev -source='"github.com/pingcap/br/pkg/lightning/web".Res' && mv res_vfsdata.go pkg/lightning/web/

web:
cd web && npm install && npm run build

build: br lightning lightning-ctl

br:
$(PREPARE_MOD)
$(GOBUILD) $(RACEFLAG) -o $(BR_BIN) cmd/br/*.go

lightning_for_web:
$(PREPARE_MOD)
$(GOBUILD) $(RACEFLAG) -o bin/br
$(GOBUILD) $(RACE_FLAG) -tags dev -o $(LIGHTNING_BIN) cmd/tidb-lightning/main.go

lightning:
$(PREPARE_MOD)
$(GOBUILD) $(RACE_FLAG) -o $(LIGHTNING_BIN) cmd/tidb-lightning/main.go

lightning-ctl:
$(PREPARE_MOD)
$(GOBUILD) $(RACE_FLAG) -o $(LIGHTNING_CTL_BIN) cmd/tidb-lightning-ctl/main.go

build_for_integration_test:
$(PREPARE_MOD)
@make failpoint-enable
($(GOTEST) -c -cover -covermode=count \
-coverpkg=$(BR_PKG)/... \
-o bin/br.test && \
-o $(BR_BIN).test \
github.com/pingcap/br/cmd/br && \
$(GOTEST) -c -cover -covermode=count \
-coverpkg=$(BR_PKG)/... \
-o $(LIGHTNING_BIN).test \
github.com/pingcap/br/cmd/tidb-lightning && \
$(GOTEST) -c -cover -covermode=count \
-coverpkg=$(BR_PKG)/... \
-o $(LIGHTNING_CTL_BIN).test \
github.com/pingcap/br/cmd/tidb-lightning-ctl && \
$(GOBUILD) $(RACEFLAG) -o bin/locker tests/br_key_locked/*.go && \
$(GOBUILD) $(RACEFLAG) -o bin/gc tests/br_z_gc_safepoint/*.go && \
$(GOBUILD) $(RACEFLAG) -o bin/oauth tests/br_gcs/*.go && \
$(GOBUILD) $(RACEFLAG) -o bin/rawkv tests/br_rawkv/*.go) || (make failpoint-disable && exit 1)
$(GOBUILD) $(RACEFLAG) -o bin/rawkv tests/br_rawkv/*.go && \
$(GOBUILD) $(RACE_FLAG) -o bin/parquet_gen tests/lightning_checkpoint_parquet/*.go \
) || (make failpoint-disable && exit 1)
@make failpoint-disable

test:
$(PREPARE_MOD)
@make failpoint-enable
$(GOTEST) $(RACEFLAG) -tags leak ./... || ( make failpoint-disable && exit 1 )
$(GOTEST) $(RACEFLAG) -tags leak $$($(PACKAGES)) || ( make failpoint-disable && exit 1 )
@make failpoint-disable

testcover: tools
mkdir -p "$(TEST_DIR)"
$(PREPARE_MOD)
@make failpoint-enable
GO111MODULE=on tools/bin/overalls \
-project=$(BR_PKG) \
-covermode=count \
-ignore='.git,vendor,tests,_tools' \
-debug \
-- -coverpkg=./... || ( make failpoint-disable && exit 1 )
$(GOTEST) -cover -covermode=count -coverprofile="$(TEST_DIR)/cov.unit.out" $$($(PACKAGES)) || ( make failpoint-disable && exit 1 )
@make failpoint-disable

integration_test: bins build build_for_integration_test
tests/run.sh

coverage: tools
tools/bin/gocovmerge "$(TEST_DIR)"/cov.* | grep -vE ".*.pb.go|.*__failpoint_binding__.go" > "$(TEST_DIR)/all_cov.out"
ifeq ("$(JenkinsCI)", "1")
tools/bin/goveralls -coverprofile=$(TEST_DIR)/all_cov.out -service=jenkins-ci -repotoken $(COVERALLS_TOKEN)
else
go tool cover -html "$(TEST_DIR)/all_cov.out" -o "$(TEST_DIR)/all_cov.html"
grep -F '<option' "$(TEST_DIR)/all_cov.html"
endif

bins:
@which bin/tidb-server
@which bin/tikv-server
Expand All @@ -73,6 +148,7 @@ bins:
@which bin/libtiflash_proxy.so
@which bin/cdc
@which bin/fake-gcs-server
@which bin/tikv-importer
if [ ! -d bin/flash_cluster_manager ]; then echo "flash_cluster_manager not exist"; exit 1; fi

tools:
Expand All @@ -88,9 +164,11 @@ check:
static: export GO111MODULE=on
static: prepare tools
@ # Not running vet and fmt through metalinter becauase it ends up looking at vendor
tools/bin/gofumports -w -d -format-only -local $(BR_PKG) $$($(PACKAGE_DIRECTORIES)) 2>&1 | $(CHECKER)
tools/bin/govet --shadow $$($(PACKAGE_DIRECTORIES)) 2>&1 | $(CHECKER)
tools/bin/gofumports -w -d -format-only -local $(BR_PKG) $(PACKAGE_FILES) 2>&1 | $(CHECKER)
# TODO: go vet lightning packages too.
tools/bin/govet --shadow $$($(PACKAGE_DIRECTORIES) | grep -v "lightning") 2>&1 | $(CHECKER)

# TODO: lint lightning packages too.
@# why some lints are disabled?
@# gochecknoglobals - disabled because we do use quite a lot of globals
@# goimports - executed above already, gofumports
Expand Down Expand Up @@ -130,17 +208,20 @@ static: prepare tools
--disable exhaustive \
--disable godot \
--disable gosec \
$$($(PACKAGE_DIRECTORIES))
$$($(PACKAGE_DIRECTORIES) | grep -v "lightning")
# pingcap/errors APIs are mixed with multiple patterns 'pkg/errors',
# 'juju/errors' and 'pingcap/parser'. To avoid confusion and mistake,
# we only allow a subset of APIs, that's "Normalize|Annotate|Trace|Cause".
# TODO: check lightning packages.
@# TODO: allow more APIs when we need to support "workaound".
grep -Rn --exclude="*_test.go" -E "(\t| )errors\.[A-Z]" cmd pkg | \
grep -Rn --include="*.go" --exclude="*_test.go" -E "(\t| )errors\.[A-Z]" \
$$($(PACKAGE_DIRECTORIES) | grep -vE "tests|lightning") | \
grep -vE "Normalize|Annotate|Trace|Cause|RedactLogEnabled" 2>&1 | $(CHECKER)

lint: prepare tools
@echo "linting"
CGO_ENABLED=0 tools/bin/revive -formatter friendly -config revive.toml $$($(PACKAGES))
# TODO: lint lightning packages.
CGO_ENABLED=0 tools/bin/revive -formatter friendly -config revive.toml $$($(PACKAGES) | grep -v "lightning")

tidy:
@echo "go mod tidy"
Expand All @@ -160,4 +241,4 @@ failpoint-enable: tools
failpoint-disable: tools
tools/bin/failpoint-ctl disable

.PHONY: tools
.PHONY: tools web
2 changes: 1 addition & 1 deletion cmd/backup.go → cmd/br/backup.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2020 PingCAP, Inc. Licensed under Apache-2.0.

package cmd
package main

import (
"github.com/pingcap/errors"
Expand Down
3 changes: 2 additions & 1 deletion cmd/cmd.go → cmd/br/cmd.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2020 PingCAP, Inc. Licensed under Apache-2.0.

package cmd
package main

import (
"context"
Expand Down Expand Up @@ -151,6 +151,7 @@ func Init(cmd *cobra.Command) (err error) {
}

// Initialize the pprof server.
// TODO: Support TLS.
statusAddr, e := cmd.Flags().GetString(FlagStatusAddr)
if e != nil {
err = e
Expand Down
2 changes: 1 addition & 1 deletion cmd/debug.go → cmd/br/debug.go
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
// Copyright 2020 PingCAP, Inc. Licensed under Apache-2.0.

package cmd
package main

import (
"bytes"
Expand Down
15 changes: 7 additions & 8 deletions main.go → cmd/br/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,6 @@ import (
"github.com/pingcap/log"
"github.com/spf13/cobra"
"go.uber.org/zap"

"github.com/pingcap/br/cmd"
)

func main() {
Expand Down Expand Up @@ -44,19 +42,20 @@ func main() {
TraverseChildren: true,
SilenceUsage: true,
}
cmd.AddFlags(rootCmd)
cmd.SetDefaultContext(ctx)
AddFlags(rootCmd)
SetDefaultContext(ctx)
rootCmd.AddCommand(
cmd.NewDebugCommand(),
cmd.NewBackupCommand(),
cmd.NewRestoreCommand(),
NewDebugCommand(),
NewBackupCommand(),
NewRestoreCommand(),
)
// Ouputs cmd.Print to stdout.
rootCmd.SetOut(os.Stdout)

rootCmd.SetArgs(os.Args[1:])
if err := rootCmd.Execute(); err != nil {
cancel()
log.Error("br failed", zap.Error(err))
os.Exit(1)
os.Exit(1) // nolint:gocritic
}
}
File renamed without changes.
Loading