Backup to common cloud storage

[tennix]

BR support common cloud storage

Overview

Integrate BR with common cloud object storage (S3, GCS, Azure Blob storage etc).

Problem statement

Currently, BR supports local storage where backup files are stored on local directory. But the backup files need to be collected together and copied to every TiKV node. This is difficult to use in practice, so it's better to mount an NFS like filesystem to every TiKV node and BR node. However, mounting NFS to every node is difficult to set up and error-prone.

Alternatively, object storage is better for this scenario, especially that it's quite common to backup to S3/GCS on public cloud.

TODO list

S3 Support (2400 points)

• Extend current backup protobuf to support S3 compatible (S3, Ceph object storage, Minio, Aliyun OSS) storage (900 points / medium) @tennix backup: extend backup storage for S3 kvproto#507
• Store BR metadata to S3 compatible object storage (600 points / easy) @DanielZhangQD backup: store backupmeta to s3 #103
• Store TiKV SST files to S3 compatible object storage (900 points / medium) @yiwu-arbug external_storage: add S3 support tikv/tikv#6209

GCS Support (1800 points)

• Extend backup protobuf to support GCS (300 points / easy) @SunRunAway backup: add GCS proto kvproto#514
• Store BR metadata to GCS (600 points / easy) @SunRunAway storage: support GCS backup and restore #102
• Store TiKV SST files to GCS (900 points / medium) @yiwu-arbug external_storage: add GCS support tikv/tikv#6283

TiDB Operator integration (900 points)

• Integrate BR with tidb-operator to allow easy deployment (900 points / medium) @DanielZhangQD support backup to S3 with br tidb-operator#1280

Test (2100 points)

• Unit test coverage >= 60% (900 points / easy)
• e2e test (1200 points / medium) [WIP] GCS: add e2e test #108

Mentors

• @tennix
• @overvenus
• @kennytm

Recommended skills

• Go language
• Rust language
• Familiar with S3/GCS/Azure blob storage

[yiwu-arbug]

I want to join in this task~

[SunRunAway]

@francis0407 and I also have interest in this issue.

[gregwebs]

Another task that can be added to this is to ensure proper streaming to object storage without copying data in memory.

[tennix]

Another task that can be added to this is to ensure proper streaming to object storage without copying data in memory.

What do you mean "without copying data in memory"?

[gregwebs]

I guess if streaming is working well then perhaps it isn't necessary to state this. But we should avoid copying wherever possible. The current implementation uses read_to_end which I presume copies the data every time it re-sizes its vector.

[SunRunAway]

Another task that can be added to this is to ensure proper streaming to object storage without copying data in memory.

What do you mean "without copying data in memory"?

I think it is that uploading a file from disk without buffering all file data into memory.

[gregwebs]

I think it is that uploading a file from disk without buffering all file data into memory.

Even though it is an SST file, I think it is starting from in memory?

[kennytm]

The writer interface on TiKV is currently

    fn write(&self, name: &str, reader: &mut dyn Read) -> io::Result<()>;

so the cloud storage implementation decides how to read the SST file and upload. For local storage we use Rust's built-in std::io::copy which streams up to 8 KiB at a time.