From 27088e20b7c24e28264f4ffe16b2536abab36d3a Mon Sep 17 00:00:00 2001
From: Andor Molnar <andor@cloudera.com>
Date: Mon, 22 Aug 2022 18:19:26 +0200
Subject: [PATCH 1/5] HBASE-27226. Add generic TLS config + self-signed certs

---
 src/main/asciidoc/_chapters/security.adoc | 126 ++++++++++++++++++++++
 1 file changed, 126 insertions(+)

diff --git a/src/main/asciidoc/_chapters/security.adoc b/src/main/asciidoc/_chapters/security.adoc
index 744ad2ba605c..50e8134d83ef 100644
--- a/src/main/asciidoc/_chapters/security.adoc
+++ b/src/main/asciidoc/_chapters/security.adoc
@@ -675,6 +675,132 @@ For more information about ACLs, please see the <<hbase.accesscontrol.configurat
 It should be possible for clients to authenticate with the HBase cluster through the REST gateway in a pass-through manner via SPNEGO HTTP authentication.
 This is future work.
 
+== Transport Level Security (TLS) in HBase RPC communication
+
+Since version `2.6.0` HBase supports TLS encryption in server-client and Master-RegionServer communication.
+link:https://en.wikipedia.org/wiki/Transport_Layer_Security/[Transport Layer Security (TLS)] is a standard
+cryptographic protocol designed to provide communications security over a computer network. HBase TLS implementation
+works exactly how secure websites are accessed via *https* prefix in a web browser: once established all communication
+on the channel will be securely hidden from malicious access.
+
+The encryption works at the transport level which means it's independent of the configured authentication method. Secure
+client access mentioned in the previous section requires Kerberos to be configured and used in HBase authentication, while
+TLS can be configured with any other SASL mechanism or even with simple client access methods, effectively preventing
+attackers from eavesdropping the communication. No Kerberos KDC or other complicated infrastructure required.
+
+HBase TLS is based on the Netty library therefore it only works with Netty client and server RPC implementation. Netty's
+powerful SSL implementation is a great foundation for highly secure and performant communication providing the latest and
+greatest cryptographic solution at all times.
+
+Since Region Servers effectively work as clients from Master's perspective, TLS supports encrypted communication
+between cluster members too.
+
+=== Server side configuration
+
+We need to setup Java key store for the server. Key store is the list of private keys that a server can use to configure TLS
+encryption. See link:https://en.wikipedia.org/wiki/Transport_Layer_Security/[TLS wikipedia page]
+for further details of the protocol. Add the following configuration to `hbase-site.xml` on Master, Region Servers and HBase
+clients:
+
+[source,xml]
+----
+<property>
+  <name>hbase.server.netty.tls.enabled</name>
+  <value>true</value>
+</property>
+<property>
+  <name>hbase.rpc.tls.keystore.type</name>
+  <value>JKS</value>
+</property>
+<property>
+  <name>hbase.rpc.tls.keystore.location</name>
+  <value>/path/to/keystore.jks</value>
+</property>
+<property>
+  <name>hbase.rpc.tls.keystore.password</name>
+  <value>keyStor3pa$$w0rd</value>
+</property>
+----
+
+The supported store types are based on the registered security providers. If not specified, JKS will be used by default.
+
+=== Client side configuration
+
+We need to configure trust store for the client. Trust store contains the list of certificates that the client should trust
+when doing the handshake with the server. Add the following to `hbase-site.xml`.
+
+[source,xml]
+----
+<property>
+  <name>hbase.client.netty.tls.enabled</name>
+  <value>true</value>
+</property>
+<property>
+  <name>hbase.rpc.tls.truststore.type</name>
+  <value>JKS</value>
+</property>
+<property>
+  <name>hbase.rpc.tls.truststore.location</name>
+  <value>/path/to/truststore.jks</value>
+</property>
+<property>
+  <name>hbase.rpc.tls.truststore.password</name>
+  <value>trustStor3pa$$w0rd</value>
+</property>
+----
+
+However specifying a trust store is not always required. Standard JDK implementations are shipped with a standard list
+of trusted certificates (the certificates of Certificate Authorities) and if your private key is provided by one of them,
+you don't need to configure your clients to trust it. Similarly to an internet browser, you don't need to setup the
+certificates of every single website you're planning to visit. Later in this documentation we'll walk through the steps of
+creating self-signed certificates which requires a trust store setup.
+
+You can check the list of public certificate authorities shipped with your JDK implementation:
+
+----
+keytool -keystore $JAVA_HOME/jre/lib/security/cacerts -list
+----
+
+Password is empty by default.
+
+=== Creating self-signed certificates
+
+While obtaining globally trusted certificates from Certificate Authorities is convenient is perfect valid to generate
+our own private/public keypairs and set them up specifically for the HBase cluster. Especially if we don't want to enable
+public access to the cluster, paying money for a certificate doesn't make sense.
+
+Follow the following steps to generate self-signed certificates.
+
+. Create SSL key store JKS to store local credentials
+
+Please note that the alias (-alias) and the distinguished name (-dname) must match the hostname of the machine that is
+associated with, otherwise hostname verification won't work.
+
+----
+keytool -genkeypair -alias $(hostname -f) -keyalg RSA -keysize 2048 -dname "cn=$(hostname -f)" -keypass password -keystore keystore.jks -storepass password
+----
+
+At the end of this operation you'll have as many key store files as many servers you have in your cluster. Each cluster member
+will have its own key store.
+
+[start=2]
+. Extract the signed public key (certificate) from each key store
+
+----
+keytool -exportcert -alias $(hostname -f) -keystore keystore.jks -file $(hostname -f).cer -rfc
+----
+
+[start=3]
+. Create SSL trust store JKS containing certificates for the clients
+
+The same truststore (storing all accepted certs) should be shared on participants of the cluster. You need to use different
+aliases to store multiple certificates in the same truststore. Name of the aliases doesn't matter.
+
+----
+keytool -importcert -alias [host1..3] -file [host1..3].cer -keystore truststore.jks -storepass password
+----
+
+
 == Securing Access to HDFS and ZooKeeper
 Secure HBase requires secure ZooKeeper and HDFS so that users cannot access and/or modify the metadata and data from under HBase. HBase uses HDFS (or configured file system) to keep its data files as well as write ahead logs (WALs) and other data. HBase uses ZooKeeper to store some metadata for operations (master address, table locks, recovery state, etc).
 

From d53c05d50b2db3afc8566ffd3fc973314e6f1791 Mon Sep 17 00:00:00 2001
From: Andor Molnar <andor@cloudera.com>
Date: Mon, 22 Aug 2022 19:01:07 +0200
Subject: [PATCH 2/5] HBASE-27226. Add upgrade process and additional props

---
 src/main/asciidoc/_chapters/security.adoc | 96 +++++++++++++++++++++++
 1 file changed, 96 insertions(+)

diff --git a/src/main/asciidoc/_chapters/security.adoc b/src/main/asciidoc/_chapters/security.adoc
index 50e8134d83ef..d9b6862d29c9 100644
--- a/src/main/asciidoc/_chapters/security.adoc
+++ b/src/main/asciidoc/_chapters/security.adoc
@@ -800,6 +800,102 @@ aliases to store multiple certificates in the same truststore. Name of the alias
 keytool -importcert -alias [host1..3] -file [host1..3].cer -keystore truststore.jks -storepass password
 ----
 
+=== Upgrading existing non-TLS cluster with no downtime
+
+Here are the steps needed to upgrade an already running HBase cluster to TLS without downtime by taking advantage of
+port unification functionality. There's a property on server side called `hbase.server.netty.tls.supportplaintext`
+which makes possible to accept TLS and plaintext connections on the same socket port.
+
+. Create the necessary key stores and trust stores for all server participants as described in the previous section.
+
+. Enable secure communication on the Master node with `supportplaintext=True`. Restart the Master.
+
+. Enable secure communication on the Region Servers with `supportplaintext=True`. Restart RSs in rolling restart fashion.
+
+. Enable secure communication on the clients.
+
+. Disable plaintext communication on the Master by removing `supportplaintext` property. Restart Master.
+
+. Disable plaintext communication on the Region Servers by removing `supportplaintext` property. Restart RSs in rolling
+restart fashion.
+
+=== Additional configuration
+
+==== Enabled protocols
+
+Comma-separated list of TLS protocol versions to enable. Default is empty.
+
+[source,xml]
+----
+<property>
+  <name>hbase.client.netty.tls.enabledProtocols</name>
+  <value>TLSv1.2,TLSv1.3</value>
+</property>
+----
+
+==== Default protocol
+
+Set the default TLS protocol version to use. Default is TLSv1.2. Use this protocol if enabled protocols is not defined.
+
+[source,xml]
+----
+<property>
+  <name>hbase.client.netty.tls.protocol</name>
+  <value>TLSv1.2</value>
+</property>
+----
+
+==== Enabled cipher suites
+
+List of enabled cipher suites in TLS protocol. Useful when you want to disable certain cipher suites due to recent
+security concerns. Default value is a mix of CBC and GCM ciphers. Due to performance reasons we prefer CBC ciphers for
+Java 8 and GCM ciphers for Java 9+.
+
+[source,xml]
+----
+<property>
+  <name>hbase.client.netty.tls.ciphersuites</name>
+  <value>TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256</value>
+</property>
+----
+
+==== Certificate Revocation Checking
+
+There's a built-in mechanism in JDK's TrustManager which automatically checks certificates for revocation. See
+link:https://docs.oracle.com/cd/E19263-01/817-5215/ssl.html#wp19807[Managing Server Certificates]. Disabled by default.
+
+[source,xml]
+----
+<property>
+  <name>hbase.client.netty.tls.clr</name>
+  <value>false</value>
+</property>
+----
+
+==== Online Certificate Status Protocol
+
+Enables link:https://en.wikipedia.org/wiki/OCSP_stapling[OCSP] stapling. Please note that not all `SSLProvider`
+implementations support OCSP stapling and an exception will be thrown upon. Disabled by default.
+
+[source,xml]
+----
+<property>
+  <name>hbase.client.netty.tls.ocsp</name>
+  <value>false</value>
+</property>
+----
+
+==== Client handshake timeout
+
+Set the TLS client handshake timeout is milliseconds. Default is 5 seconds.
+
+[source,xml]
+----
+<property>
+  <name>hbase.client.netty.tls.handshaketimeout</name>
+  <value>5000</value>
+</property>
+----
 
 == Securing Access to HDFS and ZooKeeper
 Secure HBase requires secure ZooKeeper and HDFS so that users cannot access and/or modify the metadata and data from under HBase. HBase uses HDFS (or configured file system) to keep its data files as well as write ahead logs (WALs) and other data. HBase uses ZooKeeper to store some metadata for operations (master address, table locks, recovery state, etc).

From 994af41bb5bd3ae39a9a9d0beb0beb81cc67834e Mon Sep 17 00:00:00 2001
From: Andor Molnar <andor@cloudera.com>
Date: Tue, 23 Aug 2022 15:22:50 +0200
Subject: [PATCH 3/5] HBASE-27226. Amend upgrade steps + fix some typos

---
 src/main/asciidoc/_chapters/security.adoc | 80 ++++++++++++++++++++---
 1 file changed, 71 insertions(+), 9 deletions(-)

diff --git a/src/main/asciidoc/_chapters/security.adoc b/src/main/asciidoc/_chapters/security.adoc
index d9b6862d29c9..c1563fe33c35 100644
--- a/src/main/asciidoc/_chapters/security.adoc
+++ b/src/main/asciidoc/_chapters/security.adoc
@@ -688,7 +688,7 @@ client access mentioned in the previous section requires Kerberos to be configur
 TLS can be configured with any other SASL mechanism or even with simple client access methods, effectively preventing
 attackers from eavesdropping the communication. No Kerberos KDC or other complicated infrastructure required.
 
-HBase TLS is based on the Netty library therefore it only works with Netty client and server RPC implementation. Netty's
+HBase TLS is based on the Netty library therefore it only works with Netty client and server RPC implementations. Netty's
 powerful SSL implementation is a great foundation for highly secure and performant communication providing the latest and
 greatest cryptographic solution at all times.
 
@@ -697,7 +697,7 @@ between cluster members too.
 
 === Server side configuration
 
-We need to setup Java key store for the server. Key store is the list of private keys that a server can use to configure TLS
+We need to set up Java key store for the server. Key store is the list of private keys that a server can use to configure TLS
 encryption. See link:https://en.wikipedia.org/wiki/Transport_Layer_Security/[TLS wikipedia page]
 for further details of the protocol. Add the following configuration to `hbase-site.xml` on Master, Region Servers and HBase
 clients:
@@ -749,9 +749,9 @@ when doing the handshake with the server. Add the following to `hbase-site.xml`.
 </property>
 ----
 
-However specifying a trust store is not always required. Standard JDK implementations are shipped with a standard list
+However, specifying a trust store is not always required. Standard JDK implementations are shipped with a standard list
 of trusted certificates (the certificates of Certificate Authorities) and if your private key is provided by one of them,
-you don't need to configure your clients to trust it. Similarly to an internet browser, you don't need to setup the
+you don't need to configure your clients to trust it. Similarly to an internet browser, you don't need to set up the
 certificates of every single website you're planning to visit. Later in this documentation we'll walk through the steps of
 creating self-signed certificates which requires a trust store setup.
 
@@ -765,8 +765,8 @@ Password is empty by default.
 
 === Creating self-signed certificates
 
-While obtaining globally trusted certificates from Certificate Authorities is convenient is perfect valid to generate
-our own private/public keypairs and set them up specifically for the HBase cluster. Especially if we don't want to enable
+While obtaining globally trusted certificates from Certificate Authorities is convenient, it's perfectly valid to generate
+your own private/public keypairs and set them up specifically for the HBase cluster. Especially if you don't want to enable
 public access to the cluster, paying money for a certificate doesn't make sense.
 
 Follow the following steps to generate self-signed certificates.
@@ -808,17 +808,79 @@ which makes possible to accept TLS and plaintext connections on the same socket
 
 . Create the necessary key stores and trust stores for all server participants as described in the previous section.
 
-. Enable secure communication on the Master node with `supportplaintext=True`. Restart the Master.
+. Enable secure communication on the Master node in _server-only_ mode with plaintext support.
 
-. Enable secure communication on the Region Servers with `supportplaintext=True`. Restart RSs in rolling restart fashion.
+[source,xml]
+----
+<property>
+  <name>hbase.client.netty.tls.enabled</name>
+  <value>false</value>
+</property>
+<property>
+  <name>hbase.server.netty.tls.enabled</name>
+  <value>true</value>
+</property>
+<property>
+  <name>hbase.server.netty.tls.supportplaintext</name>
+  <value>true</value>
+</property>
+...keystore / truststore setup ...
+----
+
+Restart the Master. Master now accepts both TLS/non-TLS connections and works with non-TLS in client mode.
+
+[start=3]
+. Enable secure communication on the Region Servers in both _server and client_ mode with plaintext support.
+
+[source,xml]
+----
+<property>
+  <name>hbase.client.netty.tls.enabled</name>
+  <value>true</value>
+</property>
+<property>
+  <name>hbase.server.netty.tls.enabled</name>
+  <value>true</value>
+</property>
+<property>
+  <name>hbase.server.netty.tls.supportplaintext</name>
+  <value>true</value>
+</property>
+...keystore / truststore setup ...
+----
 
+Restart RSs in rolling restart fashion. RSs accepts both TLS/non-TLS communication.
+
+[start=4]
 . Enable secure communication on the clients.
 
-. Disable plaintext communication on the Master by removing `supportplaintext` property. Restart Master.
+. Enable client-mode TLS on master and disable plaintext mode.
 
+[source,xml]
+----
+<property>
+  <name>hbase.client.netty.tls.enabled</name>
+  <value>true</value>
+</property>
+<property>
+  <name>hbase.server.netty.tls.enabled</name>
+  <value>true</value>
+</property>
+<property>
+  <name>hbase.server.netty.tls.supportplaintext</name>
+  <value>false</value>
+</property>
+----
+
+Restart Master.
+
+[start=6]
 . Disable plaintext communication on the Region Servers by removing `supportplaintext` property. Restart RSs in rolling
 restart fashion.
 
+WARNING: Once `hbase.client.netty.tls.enabled` is enabled on the server side, the cluster will only be able to communicate
+with other clusters which have TLS enabled. For example, this would impact intra-cluster replication.
+
 === Additional configuration
 
 ==== Enabled protocols

From 33c15161b48a4e28b736b0b28cc893ce9c02c0da Mon Sep 17 00:00:00 2001
From: Andor Molnar <andor@cloudera.com>
Date: Thu, 25 Aug 2022 15:05:33 +0200
Subject: [PATCH 4/5] HBASE-27226. Add warning block for replication, fix some
 other review feedback

---
 src/main/asciidoc/_chapters/security.adoc | 26 +++++++++++++++++++++--
 1 file changed, 24 insertions(+), 2 deletions(-)

diff --git a/src/main/asciidoc/_chapters/security.adoc b/src/main/asciidoc/_chapters/security.adoc
index c1563fe33c35..c0a48c3b865a 100644
--- a/src/main/asciidoc/_chapters/security.adoc
+++ b/src/main/asciidoc/_chapters/security.adoc
@@ -831,6 +831,18 @@ Restart the Master. Master now accepts both TLS/non-TLS connections and works wi
 
 [start=3]
 . Enable secure communication on the Region Servers in both _server and client_ mode with plaintext support.
+Client mode here will ensure that RegionServer's communication to Master is encrypted.
+
+[WARNING]
+====
+*Replication*
+
+If you have read replicas enabled in your cluster or replication between two different clusters, you
+have to break this into two steps. Secure communication has to be enabled on the _server side_ first with plaintext
+support and once all Region Servers are upgraded you can repeat the upgrade by enabling _client side_ as well.
+
+You have to prepare all Region Servers for secure communication before upgrading the client side.
+====
 
 [source,xml]
 ----
@@ -849,11 +861,21 @@ Restart the Master. Master now accepts both TLS/non-TLS connections and works wi
 ...keystore / truststore setup ...
 ----
 
-Restart RSs in rolling restart fashion. RSs accepts both TLS/non-TLS communication.
+Restart Region Servers in rolling restart fashion. They send requests with TLS and accept both TLS and non-TLS communication.
 
 [start=4]
 . Enable secure communication on the clients.
 
+[source,xml]
+----
+<property>
+  <name>hbase.client.netty.tls.enabled</name>
+  <value>true</value>
+</property>
+...truststore setup ...
+----
+
+[start=5]
 . Enable client-mode TLS on master and disable plaintext mode.
 
 [source,xml]
@@ -879,7 +901,7 @@ Restart Master.
 restart fashion.
 
 WARNING: Once `hbase.client.netty.tls.enabled` is enabled on the server side, the cluster will only be able to communicate
-with other clusters which have TLS enabled. For example, this would impact intra-cluster replication.
+with other clusters which have TLS enabled. For example, this would impact inter-cluster replication.
 
 === Additional configuration
 

From 51785a3a7d5d31d473b816db27838cc94a06ca0e Mon Sep 17 00:00:00 2001
From: Andor Molnar <andor@cloudera.com>
Date: Thu, 1 Sep 2022 14:53:25 +0200
Subject: [PATCH 5/5] HBASE-27226. Add Hadoop credential api support and
 filetype autodetection

---
 src/main/asciidoc/_chapters/security.adoc | 37 ++++++++++++-----------
 1 file changed, 20 insertions(+), 17 deletions(-)

diff --git a/src/main/asciidoc/_chapters/security.adoc b/src/main/asciidoc/_chapters/security.adoc
index c0a48c3b865a..12d34d5c0d14 100644
--- a/src/main/asciidoc/_chapters/security.adoc
+++ b/src/main/asciidoc/_chapters/security.adoc
@@ -695,6 +695,17 @@ greatest cryptographic solution at all times.
 Since Region Servers effectively work as clients from Master's perspective, TLS supports encrypted communication
 between cluster members too.
 
+[NOTE]
+====
+From version 2.6.0 HBase supports the
+link:https://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-common/CredentialProviderAPI.html[Hadoop CredentialProvider API]
+to avoid storing sensitive information in HBase configuration files. The recommended way of storing keystore / truststore passwords
+is to use one of the supported credential providers e.g. the local jceks file provider. You can find more information
+about how to setup credential providers in the Hadoop documentation.
+
+The CLI interface for accessing the Hadoop Credential Shell is also available in HBase CLI. Type `hbase credential` to get help.
+====
+
 === Server side configuration
 
 We need to set up Java key store for the server. Key store is the list of private keys that a server can use to configure TLS
@@ -708,21 +719,16 @@ clients:
   <name>hbase.server.netty.tls.enabled</name>
   <value>true</value>
 </property>
-<property>
-  <name>hbase.rpc.tls.keystore.type</name>
-  <value>JKS</value>
-</property>
 <property>
   <name>hbase.rpc.tls.keystore.location</name>
   <value>/path/to/keystore.jks</value>
 </property>
-<property>
-  <name>hbase.rpc.tls.keystore.password</name>
-  <value>keyStor3pa$$w0rd</value>
-</property>
 ----
 
-The supported store types are based on the registered security providers. If not specified, JKS will be used by default.
+Use `hbase.rpc.tls.keystore.password` alias to retrieve key store password from Hadoop credential provider.
+
+NOTE: The supported storefile formats are based on the registered security providers and the loader can be autodetected from
+the file extension. If needed, the file format can be explicitly specified with the `hbase.rpc.tls.keystore.type` property.
 
 === Client side configuration
 
@@ -735,20 +741,17 @@ when doing the handshake with the server. Add the following to `hbase-site.xml`.
   <name>hbase.client.netty.tls.enabled</name>
   <value>true</value>
 </property>
-<property>
-  <name>hbase.rpc.tls.truststore.type</name>
-  <value>JKS</value>
-</property>
 <property>
   <name>hbase.rpc.tls.truststore.location</name>
   <value>/path/to/truststore.jks</value>
 </property>
-<property>
-  <name>hbase.rpc.tls.truststore.password</name>
-  <value>trustStor3pa$$w0rd</value>
-</property>
 ----
 
+Use `hbase.rpc.tls.truststore.password` alias to retrieve trust store password from Hadoop credential provider.
+
+NOTE: The supported storefile formats are based on the registered security providers and the loader can be autodetected from
+the file extension. If needed, the file format can be explicitly specified with the `hbase.rpc.tls.truststore.type` property.
+
 However, specifying a trust store is not always required. Standard JDK implementations are shipped with a standard list
 of trusted certificates (the certificates of Certificate Authorities) and if your private key is provided by one of them,
 you don't need to configure your clients to trust it. Similarly to an internet browser, you don't need to set up the