Add hooks for outbound TLS start and close.

[shinrich]

Adds new hook points for plugins at the start and the close of an outbound TLS connection. Also adding two new Plugin APIs so the plugin can fetch the invoking Continuation or Transaction from the new outbound TLS connection.

The commit includes tests and documentation.

As this PR includes new APIs, I will send an API review email around.

[randall]

[approve ci autest]

[SolidWallOfCode]

Does it matter if the close hook callback calls TSVConnReenable?

[SolidWallOfCode]

Is this for TSNetConnect or TSNetInvokingContGet?

[shinrich]

Oops, copy error. Will update.

[SolidWallOfCode]

"default SSL connections"? Do you mean "connection options"?

[shinrich]

You are correct. Will update.

[SolidWallOfCode]

If the SSL handshake fails to complete, can it be the case that the START hook is called but not the CLOSE hook?

[shinrich]

Yes, as it stands that can be the case. Probably not a great thing. Will look into adding the hook invocation in the pre-handshake complete error cases.

[CrendKing]

I see the testing of whether the hooks are invoked, but it is not clear to me if we could alter the logic flow of the TLS handshake by manipulating the continuation.

In #4013, I wanted to achieve this: AFTER OpenSSL verifies the server certificate against CA, inspect the server certificate using plugin-supplied function, and continue/terminate the TLS handshake based on the return value of the function.

Can this implementation achieve the same?

[shinrich]

@CrendKing I think you want to look at PR #4414 and PR #4385. In that PR, you can set server.verify.policy to ENFORCED and server.verify.propoerties to SIGNATURE (ignoring the standard name requirements) and then register your plugin on the TS_SSL_VERIFY_SERVER_HOOK hook. If your plugin is invoked and is not pleased with the certificate, it can terminate the connection.

I'll add a test for this case tomorrow in PR #4414.

[CrendKing]

If your plugin is invoked and is not pleased with the certificate, it can terminate the connection.

I read #4414, and I found this code is used to handle the hook

    // If the previous configured checks passed, give the hook a try
+   netvc->callHooks(TS_EVENT_SSL_SERVER_VERIFY_HOOK);
  }
  return signature_ok;

I do not quite get how does the hook function from my plugin could alter the outcome of the OpenSSL verify function. The return value is not used, and I assume OpenSSL does not take ATS continuation into consideration in any way. Could you elaborate?

[shinrich]

Updated to address @SolidWallOfCode's comments

[SolidWallOfCode]

Is this different from test 13? It has the same comment.

[repodude]

TS_VCONN_OUTBOUND_START_HOOK - The use case I need to address requires that the plugin set the certificate used in the origin side TLS mutual auth, based on information that came in on client side request. This hook looks like it will be perfectly timed. 👍🏻

[shinrich]

Yes. that was our motivating use case as well.

[shinrich]

Addressed @CredKing's issue in PR #4414

[bryancall]

A backport PR was created for 8.1.0 because of conflicts: #5237