From 5da5085ea5864e170d2b11f1f6fa1a425bf7b91b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Thu, 12 Aug 2021 12:41:58 +0200 Subject: [PATCH 01/19] encryption-at-rest: Update --- encryption-at-rest.md | 49 +++++++++++++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 11 deletions(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index c567eb15e881d..4e57cf4879def 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -3,20 +3,32 @@ title: Encryption at Rest summary: Learn how to enable encryption at rest to protect sensitive data. --- -# Encryption at Rest New in v4.0.0 +# Encryption at Rest + +> **Warning:** +> +> Using encryption-at-rest for PD is an experimental feature. + +> **Note:** +> +> When deploying on AWS it is recommened to use [EBS encryption](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html) unless you are using non-EBS storage like local NVMe storage. Encryption at rest means that data is encrypted when it is stored. For databases, this feature is also referred to as TDE (transparent data encryption). This is opposed to encryption in flight (TLS) or encryption in use (rarely used). Different things could be doing encryption at rest (SSD drive, file system, cloud vendor, etc), but by having TiKV do the encryption before storage this helps ensure that attackers must authenticate with the database to gain access to data. For example, when an attacker gains access to the physical machine, data cannot be accessed by copying files on disk. -TiKV supports encryption at rest starting from v4.0.0. The feature allows TiKV to transparently encrypt data files using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in [CTR](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation) mode. To enable encryption at rest, an encryption key must be provided by user and this key is called master key. The master key can be provided via AWS KMS (recommended), or specifying a key stored as plaintext in a file. TiKV automatically rotates data keys that it used to encrypt actual data files. Manually rotating the master key can be done occasionally. Note that encryption at rest only encrypts data at rest (namely, on disk) and not while data is transferred over network. It is advised to use TLS together with encryption at rest. +TiKV supports encryption at rest. The feature allows TiKV to transparently encrypt data files using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in [CTR](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation) mode. To enable encryption at rest, an encryption key must be provided by user and this key is called master key. The master key can be provided via AWS KMS (recommended), or specifying a key stored as plaintext in a file. TiKV automatically rotates data keys that it used to encrypt actual data files. Manually rotating the master key can be done occasionally. Note that encryption at rest only encrypts data at rest (namely, on disk) and not while data is transferred over network. It is advised to use TLS together with encryption at rest. + +TiFlash also supports encryption at rest and while it only stores some metadata PD also supports data at rest. Encryption at rest can be enabled per component. -Also from v4.0.0, BR supports S3 server-side encryption (SSE) when backing up to S3. A customer owned AWS KMS key can also be used together with S3 server-side encryption. +Using AWS KMS is also possible for on-premise deployments. + +BR supports S3 server-side encryption (SSE) when backing up to S3. A customer owned AWS KMS key can also be used together with S3 server-side encryption. ## Warnings The current version of TiKV encryption has the following drawbacks. Be aware of these drawbacks before you get started: -* When a TiDB cluster is deployed, the majority of user data is stored in TiKV nodes, and that data will be encrypted when encryption is enabled. However, a small amount of user data is stored in PD nodes as metadata (for example, secondary index keys used as TiKV region boundaries). As of v4.0.0, PD doesn't support encryption at rest. It is recommended to use storage-level encryption (for example, file system encryption) to help protect sensitive data stored in PD. -* TiFlash supports encryption at rest since v4.0.5. For details, refer to [Encryption at Rest for TiFlash](#encryption-at-rest-for-tiflash-new-in-v405). When deploying TiKV with TiFlash earlier than v4.0.5, data stored in TiFlash is not encrypted. +* When a TiDB cluster is deployed, the majority of user data is stored in TiKV and TiFlash nodes, and that data will be encrypted when encryption is enabled. However, a small amount of user data is stored in PD nodes as metadata (for example, secondary index keys used as TiKV region boundaries). PD also supports encryption at rest. +* TiFlash supports encryption at rest, for details, refer to [Encryption at Rest for TiFlash](#encryption-at-rest-for-tiflash). * TiKV currently does not exclude encryption keys and user data from core dumps. It is advised to disable core dumps for the TiKV process when using encryption at rest. This is not currently handled by TiKV itself. * TiKV tracks encrypted data files using the absolute path of the files. As a result, once encryption is turned on for a TiKV node, the user should not change data file paths configuration such as `storage.data-dir`, `raftstore.raftdb-path`, `rocksdb.wal-dir` and `raftdb.wal-dir`. * TiKV, TiDB, and PD info logs might contain user data for debugging purposes. The info log and this data in it are not encrypted. It is recommended to enable [log redaction](/log-redaction.md). @@ -38,14 +50,27 @@ Data keys are generated by TiKV and passed to the underlying storage engine (nam Regardless of data encryption method, data keys are encrypted using AES256 in GCM mode for additional authentication. This required the master key to be 256 bits (32 bytes), when passing from file instead of KMS. +### Key creation + +Go to the [AWS KMS](https://console.aws.amazon.com/kms) on the AWS console. Make sure the correct region is selected on the top right corner of your console. Make and click "Create a key". Select "Symmetric" as Key type. After this you can set an alias an description and set tags. + +It is also possible to do this with the AWS Cli: + +``` +aws --region us-west-2 kms create-key +aws --region us-west-2 kms create-alias --alias-name "alias/tidb-tde" --target-key-id 0987dcba-09fe-87dc-65ba-ab0987654321 +``` + +The `key-id` for the second step is giving in the output of the first command. + ### Configure encryption -To enable encryption, you can add the encryption section in TiKV's configuration file: +To enable encryption, you can add the encryption section in the configuration files of TiKV and PD: ``` [security.encryption] -data-encryption-method = aes128-ctr -data-key-rotation-period = 7d +data-encryption-method = "aes128-ctr" +data-key-rotation-period = "168h" # 7 days ``` Possible values for `data-encryption-method` are "aes128-ctr", "aes192-ctr", "aes256-ctr" and "plaintext". The default value is "plaintext", which means encryption is not turned on. `data-key-rotation-period` defines how often TiKV rotates the data key. Encryption can be turned on for a fresh TiKV cluster, or an existing TiKV cluster, though only data written after encryption is enabled is guaranteed to be encrypted. To disable encryption, remove `data-encryption-method` in the configuration file, or reset it to "plaintext", and restart TiKV. To change encryption method, update `data-encryption-method` in the configuration file and restart TiKV. @@ -60,7 +85,9 @@ region = "us-west-2" endpoint = "https://kms.us-west-2.amazonaws.com" ``` -The `key-id` specifies the key id for the KMS CMK. The `region` is the AWS region name for the KMS CMK. The `endpoint` is optional and doesn't need to be specified normally, unless you are using a AWS KMS compatible service from a non-AWS vendor. +The `key-id` specifies the key id for the KMS CMK. The `region` is the AWS region name for the KMS CMK. The `endpoint` is optional and doesn't need to be specified normally, unless you are using a AWS KMS compatible service from a non-AWS vendor or need to use a [VPC endpoint for KMS](https://docs.aws.amazon.com/kms/latest/developerguide/kms-vpc-endpoint.html). + +It is possible to use [multi-Region keys](https://docs.aws.amazon.com/kms/latest/developerguide/multi-region-keys-overview.html). For this you need to setup a primary key in a specific region and add replica keys in the regions you require. To specify a master key that's stored in a file, the master key configuration would look like the following: @@ -141,8 +168,8 @@ When restoring the backup, both `--s3.sse` and `--s3.sse-kms-key-id` should NOT ./br restore full --pd --storage "s3:/// --s3.region " ``` -## Encryption at rest for TiFlash New in v4.0.5 +## Encryption at rest for TiFlash -TiFlash supports encryption at rest since v4.0.5. Data keys are generated by TiFlash. All files (including data files, schema files, and temporary files) written into TiFlash (including TiFlash Proxy) are encrypted using the current data key. The encryption algorithms, the encryption configuration (in the `tiflash-learner.toml` file) supported by TiFlash, and the meanings of monitoring metrics are consistent with those of TiKV. +TiFlash supports encryption at rest. Data keys are generated by TiFlash. All files (including data files, schema files, and temporary files) written into TiFlash (including TiFlash Proxy) are encrypted using the current data key. The encryption algorithms, the encryption configuration (in the `tiflash-learner.toml` file) supported by TiFlash, and the meanings of monitoring metrics are consistent with those of TiKV. If you have deployed TiFlash with Grafana, you can check the **TiFlash-Proxy-Details** -> **Encryption** panel. From 10ee36ebbd06038eea41b44ee5b73341947ba5cd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Mon, 23 Aug 2021 07:57:29 +0200 Subject: [PATCH 02/19] refactor --- encryption-at-rest.md | 50 +++++++++++++++++++++++-------------------- 1 file changed, 27 insertions(+), 23 deletions(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 4e57cf4879def..5510f10408f91 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -5,33 +5,43 @@ summary: Learn how to enable encryption at rest to protect sensitive data. # Encryption at Rest -> **Warning:** -> -> Using encryption-at-rest for PD is an experimental feature. - > **Note:** > -> When deploying on AWS it is recommened to use [EBS encryption](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html) unless you are using non-EBS storage like local NVMe storage. +> When deploying on AWS it is recommened to consider using [EBS encryption](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html) instead unless you are using non-EBS storage like local NVMe storage. Encryption at rest means that data is encrypted when it is stored. For databases, this feature is also referred to as TDE (transparent data encryption). This is opposed to encryption in flight (TLS) or encryption in use (rarely used). Different things could be doing encryption at rest (SSD drive, file system, cloud vendor, etc), but by having TiKV do the encryption before storage this helps ensure that attackers must authenticate with the database to gain access to data. For example, when an attacker gains access to the physical machine, data cannot be accessed by copying files on disk. -TiKV supports encryption at rest. The feature allows TiKV to transparently encrypt data files using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in [CTR](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation) mode. To enable encryption at rest, an encryption key must be provided by user and this key is called master key. The master key can be provided via AWS KMS (recommended), or specifying a key stored as plaintext in a file. TiKV automatically rotates data keys that it used to encrypt actual data files. Manually rotating the master key can be done occasionally. Note that encryption at rest only encrypts data at rest (namely, on disk) and not while data is transferred over network. It is advised to use TLS together with encryption at rest. +## Per component encryption support + +When a TiDB cluster is deployed, the majority of user data is stored in TiKV and TiFlash nodes. Some metadata is stored in PD nodes (for example, secondary index keys used as TiKV region boundaries). To get the full benefits of this feature encryption needs to be enabled for all components. Backups, log files and data transmitted over the network should also be considered when implementing this. + +### TiKV + +TiKV supports encryption at rest, this allows TiKV to transparently encrypt data files using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in [CTR](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation) mode. To enable encryption at rest, an encryption key must be provided by user and this key is called master key. The master key can be provided via AWS KMS (recommended), or specifying a key stored as plaintext in a file. TiKV automatically rotates data keys that it used to encrypt actual data files. Manually rotating the master key can be done occasionally. Note that encryption at rest only encrypts data at rest (namely, on disk) and not while data is transferred over network. It is advised to use TLS together with encryption at rest -TiFlash also supports encryption at rest and while it only stores some metadata PD also supports data at rest. Encryption at rest can be enabled per component. +It is possible to use AWS KMS for on-premise deployments, but this is not required. -Using AWS KMS is also possible for on-premise deployments. +TiKV currently does not exclude encryption keys and user data from core dumps. It is advised to disable core dumps for the TiKV process when using encryption at rest. This is not currently handled by TiKV itself. -BR supports S3 server-side encryption (SSE) when backing up to S3. A customer owned AWS KMS key can also be used together with S3 server-side encryption. +TiKV tracks encrypted data files using the absolute path of the files. As a result, once encryption is turned on for a TiKV node, the user should not change data file paths configuration such as `storage.data-dir`, `raftstore.raftdb-path`, `rocksdb.wal-dir` and `raftdb.wal-dir`. + +### TiFlash + +TiFlash supports encryption at rest. Data keys are generated by TiFlash. All files (including data files, schema files, and temporary files) written into TiFlash (including TiFlash Proxy) are encrypted using the current data key. The encryption algorithms, the encryption configuration (in the `tiflash-learner.toml` file) supported by TiFlash, and the meanings of monitoring metrics are consistent with those of TiKV. + +If you have deployed TiFlash with Grafana, you can check the **TiFlash-Proxy-Details** -> **Encryption** panel. -## Warnings +### PD -The current version of TiKV encryption has the following drawbacks. Be aware of these drawbacks before you get started: +Using encryption-at-rest for PD is an experimental feature. This is configured in the same way as TiKV. -* When a TiDB cluster is deployed, the majority of user data is stored in TiKV and TiFlash nodes, and that data will be encrypted when encryption is enabled. However, a small amount of user data is stored in PD nodes as metadata (for example, secondary index keys used as TiKV region boundaries). PD also supports encryption at rest. -* TiFlash supports encryption at rest, for details, refer to [Encryption at Rest for TiFlash](#encryption-at-rest-for-tiflash). -* TiKV currently does not exclude encryption keys and user data from core dumps. It is advised to disable core dumps for the TiKV process when using encryption at rest. This is not currently handled by TiKV itself. -* TiKV tracks encrypted data files using the absolute path of the files. As a result, once encryption is turned on for a TiKV node, the user should not change data file paths configuration such as `storage.data-dir`, `raftstore.raftdb-path`, `rocksdb.wal-dir` and `raftdb.wal-dir`. -* TiKV, TiDB, and PD info logs might contain user data for debugging purposes. The info log and this data in it are not encrypted. It is recommended to enable [log redaction](/log-redaction.md). +### Backups with BR + +BR supports S3 server-side encryption (SSE) when backing up to S3. A customer owned AWS KMS key can also be used together with S3 server-side encryption. See [BR S3 server-side encryption](/encryption-at-rest.md#br-s3-server-side-encryption) for details. + +### Logging + +TiKV, TiDB, and PD info logs might contain user data for debugging purposes. The info log and this data in it are not encrypted. It is recommended to enable [log redaction](/log-redaction.md). ## TiKV encryption at rest @@ -166,10 +176,4 @@ When restoring the backup, both `--s3.sse` and `--s3.sse-kms-key-id` should NOT ``` ./br restore full --pd --storage "s3:/// --s3.region " -``` - -## Encryption at rest for TiFlash - -TiFlash supports encryption at rest. Data keys are generated by TiFlash. All files (including data files, schema files, and temporary files) written into TiFlash (including TiFlash Proxy) are encrypted using the current data key. The encryption algorithms, the encryption configuration (in the `tiflash-learner.toml` file) supported by TiFlash, and the meanings of monitoring metrics are consistent with those of TiKV. - -If you have deployed TiFlash with Grafana, you can check the **TiFlash-Proxy-Details** -> **Encryption** panel. +``` \ No newline at end of file From 1db39327b1aedd395a12bbcef42c879124bb5c9e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 1 Sep 2021 09:57:08 +0200 Subject: [PATCH 03/19] Update encryption-at-rest.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- encryption-at-rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 5510f10408f91..f14b2ab98facb 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -7,7 +7,7 @@ summary: Learn how to enable encryption at rest to protect sensitive data. > **Note:** > -> When deploying on AWS it is recommened to consider using [EBS encryption](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html) instead unless you are using non-EBS storage like local NVMe storage. +> If your cluster is deployed on AWS and uses the EBS storage, it is recommended to use the EBS encryption. See [AWS documentation - EBS Encryption](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSEncryption.html). You are using the non-EBS storage on AWS such as the local NVMe storage, it is recommended to use encryption at rest introduced in this document. Encryption at rest means that data is encrypted when it is stored. For databases, this feature is also referred to as TDE (transparent data encryption). This is opposed to encryption in flight (TLS) or encryption in use (rarely used). Different things could be doing encryption at rest (SSD drive, file system, cloud vendor, etc), but by having TiKV do the encryption before storage this helps ensure that attackers must authenticate with the database to gain access to data. For example, when an attacker gains access to the physical machine, data cannot be accessed by copying files on disk. From 14979c28feff71467737bb1804beeac80e624120 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 1 Sep 2021 09:57:47 +0200 Subject: [PATCH 04/19] Update encryption-at-rest.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- encryption-at-rest.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index f14b2ab98facb..4e8fd034eb46d 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -11,7 +11,9 @@ summary: Learn how to enable encryption at rest to protect sensitive data. Encryption at rest means that data is encrypted when it is stored. For databases, this feature is also referred to as TDE (transparent data encryption). This is opposed to encryption in flight (TLS) or encryption in use (rarely used). Different things could be doing encryption at rest (SSD drive, file system, cloud vendor, etc), but by having TiKV do the encryption before storage this helps ensure that attackers must authenticate with the database to gain access to data. For example, when an attacker gains access to the physical machine, data cannot be accessed by copying files on disk. -## Per component encryption support +## Encryption support in different TiDB components + +In a TiDB cluster, different components use different encryption methods. This section introduces the encryption supports in different TiDB components such as TiKV, TiFlash, PD, and Backup & Restore (BR). When a TiDB cluster is deployed, the majority of user data is stored in TiKV and TiFlash nodes. Some metadata is stored in PD nodes (for example, secondary index keys used as TiKV region boundaries). To get the full benefits of this feature encryption needs to be enabled for all components. Backups, log files and data transmitted over the network should also be considered when implementing this. From 868cc43243cc33d40614ad55c7649376d2bf2d3b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 1 Sep 2021 09:58:49 +0200 Subject: [PATCH 05/19] Update encryption-at-rest.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- encryption-at-rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 4e8fd034eb46d..7e722f950f76a 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -15,7 +15,7 @@ Encryption at rest means that data is encrypted when it is stored. For databases In a TiDB cluster, different components use different encryption methods. This section introduces the encryption supports in different TiDB components such as TiKV, TiFlash, PD, and Backup & Restore (BR). -When a TiDB cluster is deployed, the majority of user data is stored in TiKV and TiFlash nodes. Some metadata is stored in PD nodes (for example, secondary index keys used as TiKV region boundaries). To get the full benefits of this feature encryption needs to be enabled for all components. Backups, log files and data transmitted over the network should also be considered when implementing this. +When a TiDB cluster is deployed, the majority of user data is stored on TiKV and TiFlash nodes. Some metadata is stored on PD nodes (for example, secondary index keys used as TiKV Region boundaries). To get the full benefits of encryption at rest, you need to enable encryption for all components. Backups, log files, and data transmitted over the network should also be considered when you implement encryption. ### TiKV From 04cd757dff207353e65c34c20582065b05d8b491 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 1 Sep 2021 09:59:11 +0200 Subject: [PATCH 06/19] Update encryption-at-rest.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- encryption-at-rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 7e722f950f76a..f040617cc078f 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -66,7 +66,7 @@ Regardless of data encryption method, data keys are encrypted using AES256 in GC Go to the [AWS KMS](https://console.aws.amazon.com/kms) on the AWS console. Make sure the correct region is selected on the top right corner of your console. Make and click "Create a key". Select "Symmetric" as Key type. After this you can set an alias an description and set tags. -It is also possible to do this with the AWS Cli: +You can also perform the operations using the AWS CLI: ``` aws --region us-west-2 kms create-key From 90ff12dc65f0343bd4726452441ddf0517433599 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 1 Sep 2021 09:59:20 +0200 Subject: [PATCH 07/19] Update encryption-at-rest.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- encryption-at-rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index f040617cc078f..1b40324d5955d 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -68,7 +68,7 @@ Go to the [AWS KMS](https://console.aws.amazon.com/kms) on the AWS console. Make You can also perform the operations using the AWS CLI: -``` +```shell aws --region us-west-2 kms create-key aws --region us-west-2 kms create-alias --alias-name "alias/tidb-tde" --target-key-id 0987dcba-09fe-87dc-65ba-ab0987654321 ``` From dda207079a2270699d7c64e5f6eb30c0e0bbda18 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 1 Sep 2021 10:00:52 +0200 Subject: [PATCH 08/19] Update encryption-at-rest.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- encryption-at-rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 1b40324d5955d..a6555f6fce29e 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -73,7 +73,7 @@ aws --region us-west-2 kms create-key aws --region us-west-2 kms create-alias --alias-name "alias/tidb-tde" --target-key-id 0987dcba-09fe-87dc-65ba-ab0987654321 ``` -The `key-id` for the second step is giving in the output of the first command. +The `--target-key-id` to enter in the second command is in the output of the first command. ### Configure encryption From ae2a5d2267a87a1619c120ec3df4f537414698b1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 1 Sep 2021 10:01:45 +0200 Subject: [PATCH 09/19] Update encryption-at-rest.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- encryption-at-rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index a6555f6fce29e..1f86c1540baa7 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -97,7 +97,7 @@ region = "us-west-2" endpoint = "https://kms.us-west-2.amazonaws.com" ``` -The `key-id` specifies the key id for the KMS CMK. The `region` is the AWS region name for the KMS CMK. The `endpoint` is optional and doesn't need to be specified normally, unless you are using a AWS KMS compatible service from a non-AWS vendor or need to use a [VPC endpoint for KMS](https://docs.aws.amazon.com/kms/latest/developerguide/kms-vpc-endpoint.html). +The `key-id` specifies the key ID for the KMS CMK. The `region` is the AWS region name for the KMS CMK. The `endpoint` is optional and you do not need to specify it normally unless you are using an AWS KMS-compatible service from a non-AWS vendor or need to use a [VPC endpoint for KMS](https://docs.aws.amazon.com/kms/latest/developerguide/kms-vpc-endpoint.html). It is possible to use [multi-Region keys](https://docs.aws.amazon.com/kms/latest/developerguide/multi-region-keys-overview.html). For this you need to setup a primary key in a specific region and add replica keys in the regions you require. From a30c3befe02a48ecbf61e0141f0908a868b8d69b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Wed, 1 Sep 2021 10:02:17 +0200 Subject: [PATCH 10/19] Update encryption-at-rest.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- encryption-at-rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 1f86c1540baa7..7b8ff970e71b2 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -99,7 +99,7 @@ endpoint = "https://kms.us-west-2.amazonaws.com" The `key-id` specifies the key ID for the KMS CMK. The `region` is the AWS region name for the KMS CMK. The `endpoint` is optional and you do not need to specify it normally unless you are using an AWS KMS-compatible service from a non-AWS vendor or need to use a [VPC endpoint for KMS](https://docs.aws.amazon.com/kms/latest/developerguide/kms-vpc-endpoint.html). -It is possible to use [multi-Region keys](https://docs.aws.amazon.com/kms/latest/developerguide/multi-region-keys-overview.html). For this you need to setup a primary key in a specific region and add replica keys in the regions you require. +You can also use [multi-Region keys](https://docs.aws.amazon.com/kms/latest/developerguide/multi-region-keys-overview.html) in AWS. For this, you need to set up a primary key in a specific region and add replica keys in the regions you require. To specify a master key that's stored in a file, the master key configuration would look like the following: From 6357d4a3b8441da67b92615a451132033407fd91 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Fri, 3 Sep 2021 13:27:33 +0200 Subject: [PATCH 11/19] Respond to comments --- encryption-at-rest.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 7b8ff970e71b2..5714aee366141 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -19,9 +19,9 @@ When a TiDB cluster is deployed, the majority of user data is stored on TiKV and ### TiKV -TiKV supports encryption at rest, this allows TiKV to transparently encrypt data files using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in [CTR](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation) mode. To enable encryption at rest, an encryption key must be provided by user and this key is called master key. The master key can be provided via AWS KMS (recommended), or specifying a key stored as plaintext in a file. TiKV automatically rotates data keys that it used to encrypt actual data files. Manually rotating the master key can be done occasionally. Note that encryption at rest only encrypts data at rest (namely, on disk) and not while data is transferred over network. It is advised to use TLS together with encryption at rest +TiKV supports encryption at rest, this allows TiKV to transparently encrypt data files using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in [CTR](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation) mode. To enable encryption at rest, an encryption key must be provided by user and this key is called master key. The master key that is used to encrypt the data keys is provided by the user. TiKV automatically rotates data keys that it used to encrypt actual data files. Manually rotating the master key can be done occasionally. Note that encryption at rest only encrypts data at rest (namely, on disk) and not while data is transferred over network. It is advised to use TLS together with encryption at rest -It is possible to use AWS KMS for on-premise deployments, but this is not required. +AWS KMS can be used for both Cloud and on-premise deployments, but this is not required. It is also possible to supply the plaintext master key in a file. TiKV currently does not exclude encryption keys and user data from core dumps. It is advised to disable core dumps for the TiKV process when using encryption at rest. This is not currently handled by TiKV itself. From 35af200b9675fdc6cd1e216c4aeee7726387c1b6 Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 10 Sep 2021 14:24:53 +0800 Subject: [PATCH 12/19] Update encryption-at-rest.md --- encryption-at-rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 5714aee366141..99953ff494207 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -35,7 +35,7 @@ If you have deployed TiFlash with Grafana, you can check the **TiFlash-Proxy-Det ### PD -Using encryption-at-rest for PD is an experimental feature. This is configured in the same way as TiKV. +Encryption-at-rest for PD is an experimental feature, which is configured in the same way as in TiKV. ### Backups with BR From b515af9e76321bb8edab8c3a06fd2507800fdf7f Mon Sep 17 00:00:00 2001 From: TomShawn <41534398+TomShawn@users.noreply.github.com> Date: Fri, 10 Sep 2021 14:24:59 +0800 Subject: [PATCH 13/19] Update encryption-at-rest.md --- encryption-at-rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 99953ff494207..3e39ffbf3f08a 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -39,7 +39,7 @@ Encryption-at-rest for PD is an experimental feature, which is configured in the ### Backups with BR -BR supports S3 server-side encryption (SSE) when backing up to S3. A customer owned AWS KMS key can also be used together with S3 server-side encryption. See [BR S3 server-side encryption](/encryption-at-rest.md#br-s3-server-side-encryption) for details. +BR supports S3 server-side encryption (SSE) when backing up data to S3. A customer-owned AWS KMS key can also be used together with S3 server-side encryption. See [BR S3 server-side encryption](/encryption-at-rest.md#br-s3-server-side-encryption) for details. ### Logging From b31aa92e2e3d91160250da2c5aaf6bedd03d7b43 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Fri, 10 Sep 2021 08:31:36 +0200 Subject: [PATCH 14/19] Update encryption-at-rest.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- encryption-at-rest.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 3e39ffbf3f08a..514e06f8c1940 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -64,7 +64,13 @@ Regardless of data encryption method, data keys are encrypted using AES256 in GC ### Key creation -Go to the [AWS KMS](https://console.aws.amazon.com/kms) on the AWS console. Make sure the correct region is selected on the top right corner of your console. Make and click "Create a key". Select "Symmetric" as Key type. After this you can set an alias an description and set tags. + +To create a key on AWS, follow these steps: + +1. Go to the [AWS KMS](https://console.aws.amazon.com/kms) on the AWS console. +2. Make sure that you have selected the correct region on the top right corner of your console. +3. Click **Create a key** and select **Symmetric** as the key type. +4. Set an alias for the key and set tags. You can also perform the operations using the AWS CLI: From 9b1b46110bf12fc690d8ce6e59a11436baca8cf7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Fri, 10 Sep 2021 08:33:15 +0200 Subject: [PATCH 15/19] Update KMS things --- encryption-at-rest.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 514e06f8c1940..529ba5c044621 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -69,8 +69,8 @@ To create a key on AWS, follow these steps: 1. Go to the [AWS KMS](https://console.aws.amazon.com/kms) on the AWS console. 2. Make sure that you have selected the correct region on the top right corner of your console. -3. Click **Create a key** and select **Symmetric** as the key type. -4. Set an alias for the key and set tags. +3. Click **Create key** and select **Symmetric** as the key type. +4. Set an alias for the key. You can also perform the operations using the AWS CLI: @@ -184,4 +184,4 @@ When restoring the backup, both `--s3.sse` and `--s3.sse-kms-key-id` should NOT ``` ./br restore full --pd --storage "s3:/// --s3.region " -``` \ No newline at end of file +``` From 94219037bc29737c90271d5b124a881609ab3b35 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Fri, 10 Sep 2021 08:36:24 +0200 Subject: [PATCH 16/19] Remove some duplicated content --- encryption-at-rest.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 529ba5c044621..6c51fccae8274 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -52,13 +52,13 @@ TiKV, TiDB, and PD info logs might contain user data for debugging purposes. Th TiKV currently supports encrypting data using AES128, AES192 or AES256, in CTR mode. TiKV uses envelope encryption. As a result, two types of keys are used in TiKV when encryption is enabled. * Master key. The master key is provided by user and is used to encrypt the data keys TiKV generates. Management of master key is external to TiKV. -* Data key. The data key is generated by TiKV and is the key actually used to encrypt data. The data key is automatically rotated by TiKV. +* Data key. The data key is generated by TiKV and is the key actually used to encrypt data. The same master key can be shared by multiple instances of TiKV. The recommended way to provide a master key in production is via AWS KMS. Create a customer master key (CMK) through AWS KMS, and then provide the CMK key ID to TiKV in the configuration file. The TiKV process needs access to the KMS CMK while it is running, which can be done by using an [IAM role](https://aws.amazon.com/iam/). If TiKV fails to get access to the KMS CMK, it will fail to start or restart. Refer to AWS documentation for [KMS](https://docs.aws.amazon.com/kms/index.html) and [IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) usage. Alternatively, if using custom key is desired, supplying the master key via file is also supported. The file must contain a 256 bits (or 32 bytes) key encoded as hex string, end with a newline (namely, `\n`), and contain nothing else. Persisting the key on disk, however, leaks the key, so the key file is only suitable to be stored on the `tempfs` in RAM. -Data keys are generated by TiKV and passed to the underlying storage engine (namely, RocksDB). All files written by RocksDB, including SST files, WAL files, and the MANIFEST file, are encrypted by the current data key. Other temporary files used by TiKV that may include user data are also encrypted using the same data key. Data keys are automatically rotated by TiKV every week by default, but the period is configurable. On key rotation, TiKV does not rewrite all existing files to replace the key, but RocksDB compaction are expected to rewrite old data into new data files, with the most recent data key, if the cluster gets constant write workload. TiKV keeps track of the key and encryption method used to encrypt each of the files and use the information to decrypt the content on reads. +Data keys are passed to the underlying storage engine (namely, RocksDB). All files written by RocksDB, including SST files, WAL files, and the MANIFEST file, are encrypted by the current data key. Other temporary files used by TiKV that may include user data are also encrypted using the same data key. Data keys are automatically rotated by TiKV every week by default, but the period is configurable. On key rotation, TiKV does not rewrite all existing files to replace the key, but RocksDB compaction are expected to rewrite old data into new data files, with the most recent data key, if the cluster gets constant write workload. TiKV keeps track of the key and encryption method used to encrypt each of the files and use the information to decrypt the content on reads. Regardless of data encryption method, data keys are encrypted using AES256 in GCM mode for additional authentication. This required the master key to be 256 bits (32 bytes), when passing from file instead of KMS. From 1be9604c3863c6632569c2a15c91a68637315e47 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Fri, 10 Sep 2021 08:37:09 +0200 Subject: [PATCH 17/19] Update encryption-at-rest.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- encryption-at-rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 6c51fccae8274..fbe45a534e7d2 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -21,7 +21,7 @@ When a TiDB cluster is deployed, the majority of user data is stored on TiKV and TiKV supports encryption at rest, this allows TiKV to transparently encrypt data files using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in [CTR](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation) mode. To enable encryption at rest, an encryption key must be provided by user and this key is called master key. The master key that is used to encrypt the data keys is provided by the user. TiKV automatically rotates data keys that it used to encrypt actual data files. Manually rotating the master key can be done occasionally. Note that encryption at rest only encrypts data at rest (namely, on disk) and not while data is transferred over network. It is advised to use TLS together with encryption at rest -AWS KMS can be used for both Cloud and on-premise deployments, but this is not required. It is also possible to supply the plaintext master key in a file. +Optionally, you can use AWS KMS for both cloud and on-premises deployments. You can also supply the plaintext master key in a file. TiKV currently does not exclude encryption keys and user data from core dumps. It is advised to disable core dumps for the TiKV process when using encryption at rest. This is not currently handled by TiKV itself. From b6fc95676f07f0bd353a648d1a16a7c271683a34 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Fri, 10 Sep 2021 08:42:49 +0200 Subject: [PATCH 18/19] Fix linter issue --- encryption-at-rest.md | 1 - 1 file changed, 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index fbe45a534e7d2..89162db5d631e 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -64,7 +64,6 @@ Regardless of data encryption method, data keys are encrypted using AES256 in GC ### Key creation - To create a key on AWS, follow these steps: 1. Go to the [AWS KMS](https://console.aws.amazon.com/kms) on the AWS console. From 3d02bcff2f02df7c68cec17a285da861abc270c9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dani=C3=ABl=20van=20Eeden?= Date: Fri, 10 Sep 2021 09:29:25 +0200 Subject: [PATCH 19/19] Update encryption-at-rest.md Co-authored-by: TomShawn <41534398+TomShawn@users.noreply.github.com> --- encryption-at-rest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/encryption-at-rest.md b/encryption-at-rest.md index 89162db5d631e..67017060961e1 100644 --- a/encryption-at-rest.md +++ b/encryption-at-rest.md @@ -19,7 +19,7 @@ When a TiDB cluster is deployed, the majority of user data is stored on TiKV and ### TiKV -TiKV supports encryption at rest, this allows TiKV to transparently encrypt data files using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in [CTR](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation) mode. To enable encryption at rest, an encryption key must be provided by user and this key is called master key. The master key that is used to encrypt the data keys is provided by the user. TiKV automatically rotates data keys that it used to encrypt actual data files. Manually rotating the master key can be done occasionally. Note that encryption at rest only encrypts data at rest (namely, on disk) and not while data is transferred over network. It is advised to use TLS together with encryption at rest +TiKV supports encryption at rest. This feature allows TiKV to transparently encrypt data files using [AES](https://en.wikipedia.org/wiki/Advanced_Encryption_Standard) in [CTR](https://en.wikipedia.org/wiki/Block_cipher_mode_of_operation) mode. To enable encryption at rest, an encryption key must be provided by the user and this key is called master key. TiKV automatically rotates data keys that it used to encrypt actual data files. Manually rotating the master key can be done occasionally. Note that encryption at rest only encrypts data at rest (namely, on disk) and not while data is transferred over network. It is advised to use TLS together with encryption at rest. Optionally, you can use AWS KMS for both cloud and on-premises deployments. You can also supply the plaintext master key in a file.