Problem: BR's internal TiDB version must match cluster's TiDB version.

[kennytm]

TiDB has a concept called "bootstrap version" (tidb_server_version). It represents the schemas of the system tables. If a TiDB server with a newer bootstrap version (e.g. v41) is started on a storage with older version (e.g. v40), the TiDB server will perform a "rolling upgrade" (execute some DDL statements) to migrate the schemas to v41.

This action is also performed when BR uses its own TiDB to bootstrap the session.

br/pkg/conn/conn.go

Line 133 in 56e206e

dom, err := session.BootstrapSession(storage)

This brings us to the problem:

• If BR version < cluster TiDB version, BR cannot understand system schemas and thus won't run.
• If BR version > cluster TiDB version, BR will "rolling upgrade" and TiDB can't use the cluster anymore unless user also upgrades TiDB.

Thus the only safe solution is to ensure BR version = cluster TiDB version.

How should our version management cope with this?

This was revealed in #158 (https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/br_ghpr_unit_and_integration_test/detail/br_ghpr_unit_and_integration_test/828/pipeline/) due to pingcap/tidb#14598 renaming the column password (v40) to authentication_string (v41) in the system table mysql.user.

[kennytm]

Developers from TiDB confirmed that session.BootstrapSession will not perform rolling upgrade. So incompatible cluster version would simply result in error instead of accidentally upgrading the cluster.

We still need to upgrade the tidb deps in BR.

[3pointer]

Found session.BootrapSession will perform rolling upgrade

tidb: v3.1.0.beta.1
br: master

before run br backup:

MySQL [(none)]> select * from mysql.user;
+------+------+----------+-------------+-------------+-------------+-------------+-------------+-----------+--------------+------------+-----------------+------------+--------------+------------+-----------------------+------------------+--------------+------------------+----------------+---------------------+--------------------+------------+------------------+------------+--------------+------------------+----------------+----------------+---------------+
| Host | User | Password | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Process_priv | Grant_priv | References_priv | Alter_priv | Show_db_priv | Super_priv | Create_tmp_table_priv | Lock_tables_priv | Execute_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Index_priv | Create_user_priv | Event_priv | Trigger_priv | Create_role_priv | Drop_role_priv | Account_locked | Shutdown_priv |
+------+------+----------+-------------+-------------+-------------+-------------+-------------+-----------+--------------+------------+-----------------+------------+--------------+------------+-----------------------+------------------+--------------+------------------+----------------+---------------------+--------------------+------------+------------------+------------+--------------+------------------+----------------+----------------+---------------+
| %    | root |          | Y           | Y           | Y           | Y           | Y           | Y         | Y            | Y          | Y               | Y          | Y            | Y          | Y                     | Y                | Y            | Y                | Y              | Y                   | Y                  | Y          | Y                | Y          | Y            | Y                | Y              | N              | Y             |
+------+------+----------+-------------+-------------+-------------+-------------+-------------+-----------+--------------+------------+-----------------+------------+--------------+------------+-----------------------+------------------+--------------+------------------+----------------+---------------------+--------------------+------------+------------------+------------+--------------+------------------+----------------+----------------+---------------+

after:

+------+------+-----------------------+-------------+-------------+-------------+-------------+-------------+-----------+--------------+------------+-----------------+------------+--------------+------------+-----------------------+------------------+--------------+------------------+----------------+---------------------+--------------------+------------+------------------+------------+--------------+------------------+----------------+----------------+---------------+-------------+-----------+
| Host | User | authentication_string | Select_priv | Insert_priv | Update_priv | Delete_priv | Create_priv | Drop_priv | Process_priv | Grant_priv | References_priv | Alter_priv | Show_db_priv | Super_priv | Create_tmp_table_priv | Lock_tables_priv | Execute_priv | Create_view_priv | Show_view_priv | Create_routine_priv | Alter_routine_priv | Index_priv | Create_user_priv | Event_priv | Trigger_priv | Create_role_priv | Drop_role_priv | Account_locked | Shutdown_priv | Reload_priv | File_priv |
+------+------+-----------------------+-------------+-------------+-------------+-------------+-------------+-----------+--------------+------------+-----------------+------------+--------------+------------+-----------------------+------------------+--------------+------------------+----------------+---------------------+--------------------+------------+------------------+------------+--------------+------------------+----------------+----------------+---------------+-------------+-----------+
| %    | root |                       | Y           | Y           | Y           | Y           | Y           | Y         | Y            | Y          | Y               | Y          | Y            | Y          | Y                     | Y                | Y            | Y                | Y              | Y                   | Y                  | Y          | Y                | Y          | Y            | Y                | Y              | N              | Y             | N           | N         |
+------+------+-----------------------+-------------+-------------+-------------+-------------+-------------+-----------+--------------+------------+-----------------+------------+--------------+------------+-----------------------+------------------+--------------+------------------+----------------+---------------------+--------------------+------------+------------------+------------+--------------+------------------+----------------+----------------+---------------+-------------+-----------+

password changed to authentication_string, then when use correct verion of br(v3.1.0.beta.1) will cause Error: [planner:1054]Unknown column 'password' in 'field list'