BR failed to backup stats

[3pointer]

Please answer these questions before submitting your issue. Thanks!

1. What did you do?
   If possible, provide a recipe for reproducing the error.
   Upgrade TiDB cluster from v4.0.0 to v4.0.9. then use BR v4.0.9 to backup.

2. What did you expect to see?
   backup success.

3. What did you see instead?

[ERROR] [backup.go:25] ["failed to backup"] [error="[types:1406]Data Too Long, field len 5, data len 6"] [errorVerbose="[types:1406]Data Too Long, field len 5, data len 6\n
github.com/pingcap/errors.AddStack\n
\t	github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\n
github.com/pingcap/errors.(*Error).GenWithStack\n
\t	github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:147\n
github.com/pingcap/tidb/types.ProduceStrWithSpecifiedTp\n
\t	github.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/types/datum.go:995\n
github.com/pingcap/tidb/types.(*Datum).convertToString\n
\t	github.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/types/datum.go:942\n
github.com/pingcap/tidb/types.(*Datum).ConvertTo\n
\t	github.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/types/datum.go:821\n
github.com/pingcap/tidb/statistics/handle.(*Handle).histogramFromStorage\n
\t	github.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/statistics/handle/handle.go:667\n
github.com/pingcap/tidb/statistics/handle.(*Handle).columnStatsFromStorage\n
\t	github.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/statistics/handle/handle.go:460\n
github.com/pingcap/tidb/statistics/handle.(*Handle).tableStatsFromStorage\n
\t	github.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/statistics/handle/handle.go:539\n
github.com/pingcap/tidb/statistics/handle.(*Handle).tableStatsToJSON\n
\t	github.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/statistics/handle/dump.go:89\n
github.com/pingcap/tidb/statistics/handle.(*Handle).DumpStatsToJSON\n
\t	github.com/pingcap/tidb@v1.1.0-beta.0.20201214152324-ce2f365189d3/statistics/handle/dump.go:76\n
github.com/pingcap/br/pkg/backup.BuildBackupRangeAndSchema\n
\t	github.com/pingcap/br@/pkg/backup/client.go:340\n
github.com/pingcap/br/pkg/task.RunBackup\n
\t	github.com/pingcap/br@/pkg/task/backup.go:257\n
github.com/pingcap/br/cmd.runBackupCommand\n
\t	github.com/pingcap/br@/cmd/backup.go:24\n
github.com/pingcap/br/cmd.newFullBackupCommand.func1\n
\t	github.com/pingcap/br@/cmd/backup.go:85\n
github.com/spf13/cobra.(*Command).execute\n
\t	github.com/spf13/cobra@v1.0.0/command.go:842\n
github.com/spf13/cobra.(*Command).ExecuteC\n
\t	github.com/spf13/cobra@v1.0.0/command.go:950\n
github.com/spf13/cobra.(*Command).Execute\n
\t	github.com/spf13/cobra@v1.0.0/command.go:887\n
main.main\n
\t	github.com/pingcap/br@/main.go:58\n
runtime.main\n
\t	runtime/proc.go:203\n
runtime.goexit\n
\t	runtime/asm_amd64.s:1357"] [

4. What version of BR and TiDB/TiKV/PD are you using?
   BR v4.0.9
   TiDB v4.0.9 (upgrade from v4.0.0)

[3pointer]

The failed reason is DumpsStatsToJson is not compatible when TiDB is upgraded from v4.0.0.
To workaround, user can add --ignore-stats=true to command to skip backup stats. and we need find a better way to backup stats and consider with compatibility.

[kennytm]

This is a bug in TiDB. Still, such failure probably shouldn't abort backup/restore since stats aren't essential.

also, the dump stats error should log which table is the cause.

br/pkg/backup/client.go

Lines 348 to 358 in 458d542

	var stats []byte
	if !ignoreStats {
	jsonTable, err := h.DumpStatsToJSON(dbInfo.Name.String(), tableInfo, nil)
	if err != nil {
	return nil, nil, errors.Trace(err)
	}
	stats, err = json.Marshal(jsonTable)
	if err != nil {
	return nil, nil, errors.Trace(err)
	}
	}

[IANTHEREAL]

What is the trigger mechanism of dumpStats failure?

stats shouldn't cause the backup to fail. If the stats can be backup&restored, it is good. If BR can't back up stats, it needs to output a backup summary log indicating that backup stats failed; and if there is no stats when restoring, then BR needs to be able to support running analyze table

stats information is stored to backupmeta. what If the entire cluster is backup in a large-scale cluster? how to deal with the large size backupmeta, is it sense?

[kennytm]

if the stats field is nil during restore, it will just be skipped.

br/pkg/restore/client.go

Lines 816 to 826 in 458d542

	if table.Stats != nil {
	log.Info("start loads analyze after validate checksum",
	zap.Stringer("db name", tbl.OldTable.DB.Name),
	zap.Stringer("name", tbl.OldTable.Info.Name),
	zap.Int64("old id", tbl.OldTable.Info.ID),
	zap.Int64("new id", tbl.Table.ID),
	)
	if err := rc.statsHandler.LoadStatsFromJSON(rc.dom.InfoSchema(), table.Stats); err != nil {
	log.Error("analyze table failed", zap.Any("table", table.Stats), zap.Error(err))
	}
	}

missing stats won't prevent user accessing the data, just that the queries are not optimized. ANALYZE TABLE can be performed separately.

[IANTHEREAL]

OK, it can be a workaround method.

For a better user experience, restore needs to support analyze table for tables without stats. Many users may not know analyze table, and restore should be a complete operation, the cluster should be in a good state after restoring

[kennytm]

running "analyze table" requires scanning the tables again though, which is slow (esp. when restoring from an old backup where the stats are all missing)

[IANTHEREAL]

This is true, but analyze table is necessary. So we should make sure that the backup statistics need to be successful in major situations

[IANTHEREAL]

As discussed with @kennytm, some consensuses are as follows

1. the most important thing is that BR needs to ensure the success rate of backup stats to ensure that most of the backup statistics can be backed up successfully, the success rate can reach 99.9%
2. table restored without stats can be considered unavailable, and requires human or restore program intervention
3. ignore-stats option of backup is mainly for testing. If someone does backup with ignore-stats， restore does not need to execute analyze table automatically
4. BR restore should provide an option skip-analyze whose default values is false
5. the backup that is from tidb (version < 4.0.9) can skip analyze table
6. If there are table backups that do not contain stats information, the user should be reminded that restore would take a long time
7. If restore fails to execute analyze table statement, the log will prompt the user to manually execute analyze table, and continue to restore other tables.

[kennytm]

my preference is:

4. the option should be --analyze (default is true) which could be set to false for those who really can't wait
5. no need to do version check. we do analyze even if restoring pre-4.0.9 backup archives
8. we should clearly document this analyze behavior.

[IANTHEREAL]

I have no problem with these preferences ⏩

For version <4.0.9, I have no special requirements. In order to restore quickly, we can allow the restore of version less than 4.0.9 not to execute analyze table, you can choose it. But I don’t want there are many stats backup failures after version 4.0.9 to avoid too long restore time