Allow BR to run without requiring read/write permission to cloud storage

[kennytm]

Feature Request

Describe your feature request related problem:

Currently the BR node requires check-exists and write permission of the backupmeta file during Backup, and read permission during Restore, in additional to the permissions from TiKV nodes. This is considered violating the "principle of least privilege" regarding access to the external storage, esp. for cloud storage.

Describe the feature you'd like:

Rather than doing the read/write directly, BR can delegate read/write of backupmeta to an arbitrary TiKV node.

TiKV's backup service would need two new gRPC message:

service Backup {
    // ...
    rpc Check(StorageBackend) returns (EmptyResponse) {}
    rpc UploadBackupMeta(UploadBackupMetaRequest) returns (EmptyResponse) {}
}

message UploadBackupMetaRequest {
    StorageBackend storage_backend = 1;
    BackupMeta content = 2;
}

message EmptyResponse {
}

and the import_sst service would need one new gRPC messages:

service ImportSST {
    // ...
    rpc FetchBackupMeta(StorageBackend) returns (BackupMeta) {}
}

BR would pick a random TiKV in the cluster for uploading and downloading the backupmeta content.

The br validate decode function should still work outside the cluster, so the Go implementation of ExternalStorage must be retained. Besides, BR should fallback to upload/download the file itself when TiKV returns "Unimplemented", so we have backward compatibility.

This interface assumes client-side encryption is not going to be supported in the short term, or can be entirely managed by the TiKV nodes.

Describe alternatives you've considered:

Don't do this.

Teachability, Documentation, Adoption, Migration Strategy:

This change is mostly compatible with existing cluster version, but for local:// storage a subtle change will happen:

• After backup, the backupmeta file is written on a random TiKV node rather than the BR node
• Before restore, the backupmeta file must be copied on to every TiKV node, while it can be omitted on BR.

[kennytm]

cc @kolbe.

[kolbe]

Thanks for considering this important detail! I think we should embrace the "principle of least privilege" wherever possible.