Add optional callback for user-defined TLS 1.2 key derivation function

[lars-du]

Use Case

We use Botan for TLS-PSK on embedded devices equipped with a hardware-protected environment (HPE), such as a Hardware Security Module (HSM) or Trusted Execution Environment (TEE). The pre-shared key (PSK) is securely provisioned to the devices and stored within the HPE. For security reasons, the PSK must not leave the HPE at runtime. This presents a challenge: Botan runs in the normal user environment (Rich Execution Environment), while the PSK is only accessible inside the HPE, meaning Botan cannot access the PSK directly.
To address this issue, this PR introduces an optional callback that allows the TLS key derivation function (KDF) to be delegated to the application. In our scenario, this callback is used to offload the computation of the TLS master secret to the HPE during the TLS 1.2 handshake. This enables Botan to use the master secret derived from the PSK without ever requiring direct access to the PSK itself.

Usage example

Callback implementation in user code

class tls_callbacks_example : public Botan::TLS::Callbacks
{
    // mandatory botan callbacks:
    // - void tls_emit_data(std::span<const uint8_t> data) final ...
    // - void tls_record_received(uint64_t seq_no, std::span<const uint8_t> data) final ...
    // - void tls_alert(Botan::TLS::Alert alert) final ...
 
    // new additional optional callback:
    std::unique_ptr<Botan::KDF>
    tls12_protocol_specific_kdf(std::string_view prf_algo) const final
    {
        if(prf_algo == "MD5" || prf_algo == "SHA-1") {     
            return std::make_unique<tls_12_prf_hpe>("SHA-256");
        }
 
        return std::make_unique<tls_12_prf_hpe>(prf_algo);
    }
};

Implementation of user-defined TLS 1.2 pseudo random function

class tls_12_prf_hpe final : public Botan::KDF
{
public:
    explicit tls_12_prf_hpe(std::string_view hash_function)
      : m_hash_function{hash_function}
      , m_original_prf{Botan::KDF::create_or_throw(name())}
    {
    }
 
    std::string name() const final;
 
    std::unique_ptr<Botan::KDF> new_object() const final;
 
    void perform_kdf(std::span<uint8_t> key,
                     std::span<const uint8_t> secret,
                     std::span<const uint8_t> salt,
                     std::span<const uint8_t> label) const override final;
 
private:
    std::string const           m_hash_function;
    std::unique_ptr<Botan::KDF> m_original_prf;
 
    bool kdf_internal(std::span<std::uint8_t>       out_derived_key,
                      std::span<std::uint8_t const> secret,
                      std::span<std::uint8_t const> salt,
                      std::span<std::uint8_t const> label) const;
};
 
std::string
tls_12_prf_hpe::name() const
{
    return "TLS-12-PRF(" + m_hash_function + ")";
}
 
std::unique_ptr<Botan::KDF>
tls_12_prf_hpe::new_object() const
{
    return std::make_unique<tls_12_prf_hpe>(m_hash_function);
}
 
bool
tls_12_prf_hpe::kdf_internal(std::span<std::uint8_t>       out_derived_key,
                             std::span<std::uint8_t const> secret,
                             std::span<std::uint8_t const> salt,
                             std::span<std::uint8_t const> label) const
{
    // The TLS 1.2 PRF is called for multiple purposes (with different labels), but we are only interested in the case
    // where the label ends with "master secret" and the hash function is "SHA-256".
    // In this case, we calculate the master secret inside the HPE.
    //
    // The following labels are expected:
    // - "master secret"          -> calculate master secret in HPE
    // - "extended master secret" -> calculate master secret in HPE
    // - "key expansion"          -> derive using original KDF from botan
    // - "client finished"        -> derive using original KDF from botan
    // - "server finished"        -> derive using original KDF from botan
    // - ...                      -> derive using original KDF from botan
 
    const std::string label_str = {label.begin(), label.end()};
    if(not label_str.ends_with("master secret") || (m_hash_function != "SHA-256")) {
        m_original_prf->derive_key(out_derived_key, secret, salt, label);
        return true;
    }

    // ignore secret since it contains a dummy pre-master secret only
    auto const master_secret = hpe_derive_tls_master_secret(salt, label);  // CALL TO HPE
    if (master_secret.size() != out_derived_key.size()) {
        return false;
    }
 
    std::copy(master_secret.begin(), master_secret.end(), out_derived_key.begin());
    return true;
}
 
void tls_12_prf_hpe::perform_kdf(std::span<uint8_t> key,
                                 std::span<const uint8_t> secret,
                                 std::span<const uint8_t> salt,
                                 std::span<const uint8_t> label) const {
{
    if(not kdf_internal(key, secret, salt, label)) {
        // In case the key could not be derived, return a random key that will provoke a TLS alert due to an invalid
        // record MAC. This ensures that no information about invalid key identifiers is revealed to the remote peer.
        fill_with_random_bytes(key);
    }
}

[coveralls]

coverage: 90.666% (-0.003%) from 90.669%
when pulling 19c70c5 on cariad-tech:add-external-tls12-master-secret-calculation
into 46f6fc3 on randombit:master.

[reneme]

Interesting use case! Thanks for the polished PR. In fact I always wondered if it would be useful to enable the application to swap out cryptographic primitives for things like this.

I think it would be valuable to generalize this to TLS 1.3 though. And that shouldn't be too hard either, given that TLS 1.3's key schedule is also just based on KDFs. See the constructor of Cipher_State in tls_cipher_state.cpp (roughly line 442). We should be able to replace those hard-coded constructions of HKDF_Extract and HKDF_Expand with calls into your new callback.

The only caveat I see immediately: in your example you're using the "label" to decide when to delegate to your HPE to obtain the PSK. In TLS 1.3 the invocation to HKDF-Extract doesn't have use-case specific labels. So that might be a show stopper.

[frederik-do]

I think it would be valuable to generalize this to TLS 1.3 though. And that shouldn't be too hard either, given that TLS 1.3's key schedule is also just based on KDFs. See the constructor of Cipher_State in tls_cipher_state.cpp (roughly line 442). We should be able to replace those hard-coded constructions of HKDF_Extract and HKDF_Expand with calls into your new callback.

The only caveat I see immediately: in your example you're using the "label" to decide when to delegate to your HPE to obtain the PSK. In TLS 1.3 the invocation to HKDF-Extract doesn't have use-case specific labels. So that might be a show stopper.

Yes, we also thought about extending this to TLS 1.3 and we did some analysis a while ago.
Unfortunately, it turned out not to be so easy as in TLS 1.2.

The issue with TLS 1.3 is, that it derives a lot of keys and some of them already quite early before any random/connection specific data is involved.

TLS 1.3 derives the Early Secret from the PSK via HKDF-Extract(0*HASHLEN, PSK).
Means, the Early Secret remains the same for all TLS connections the use the same PSK.

If the Early Secret would be derived from the PSK inside a hardware protected environment (HPE) and then passed to the
userland, an attacker who gains access to the Early Secret could use it to establish new TLS connections.
Therefore, we must keep the Early Secret in the HPE as well.

From the Early Secret TLS derives the binder_key:
binder_key = HKDF-Expand(Early Secret, [Len]["tls13 ext binder"][HASH("")], HASH-LEN)

The binder_key is also the same for all new connections but at least bound to the purpose label "tls13 ext binder".
So we could keep the PSK and the Early Secret inside the HPE and pass all following keys to the userland.
That would include these keys:

client_early_traffic_secret  = HKDF-Expand(EarlySecret, [Len]["tls13 c e traffic"][HASH(ClientHello)], HASH-LEN)
early_exporter_master_secret = HKDF-Expand(EarlySecret, [Len]["tls13 e exp master"][HASH(ClientHello)], HASH-LEN)
Derived Early Secret         = HKDF-Expand(EarlySecret, [Len]["tls13 derived"][HASH("")], HASH-LEN)

The Derived Early Secret is also the same for all connections using the same PSK.
The client_early_traffic_secret and early_exporter_master_secret are connection specific since they are based on
the hash of the ClientHello which includes a random value.
But the last two keys are not relevant for all types of connections as far as I know.

So if an attacker gets the binder_key and the Derived Early Secret it might be enough to establish new connections if
they do not employ early traffic or make use of the early_exporter_master_secret.

This might still be some gain in security compared to having the PSK in the userland, but not much.

Since the binder_key is used to calculate the HMAC for the psk identities inside the ClientHello we would also need to
delegate the corresponding HMAC operation to the HPE.

This would then maybe provide sufficient security as the binder_key is intended to bind the PSK to one particular handshake only.

So all in all, we think it would be possible to achieve something similar with TLS 1.3 but it is definitely way more complex.
We suggest to keep this out of this PR and maybe address this at a later point of time.

[lars-du]

Hi @reneme @randombit, do you need more input from our side for this PR?

[reneme]

Thanks for your patience on this. This looks good to me, but I'm leaving a second review and the merge to @randombit.

[reneme]

So all in all, we think it would be possible to achieve something similar with TLS 1.3 but it is definitely way more complex.
We suggest to keep this out of this PR and maybe address this at a later point of time.

I agree with your observations. Essentially, in Botan we would need to make some of the TLS 1.3 key schedule implementation (currently in tls_cipher_state.h) customizable and optionally expose it to the application. Currently, objects of this class derive and hold all the relevant secrets of a TLS 1.3 connection and implement their usages.

For your use case of keeping the PSK in a protected environment, most of the key derivation would therefore need to be implemented inside the HPE (at least until the ServerHello is received and the ephemeral key exchange is done). Therefore, in Botan, we would need to at least make the state transition methods customizable by the application. See tls_cipher_state.cpp for some more documentation.

Al least those methods would need to become customizable:

• For PSK-based setups and session resumptions

  • init_with_psk - to calculate the binder_key using the PSK
  • advance_with_client_hello - to allow sending early data (which is currently not supported by Botan, though)
  • advance_with_server_hello - incorporating the ephemeral shared secret after which all derived secrets are truly session specific

• For handshakes that don't use a PSK

  • init_with_server_hello - key derivation starts only once the ephemeral shared secret is available and all derived keys are always session specific

But I think, if we ever extend the TLS 1.3 implementation in that way, it would probably make sense to make the entire cipher state class customizable. That would give an application maximal flexibility when and even if key material leaves the HPE. In the extreme case no such key material would ever have to leave the HPE including for the bulk traffic encryption if this is feasible or desired.