Skip to content

Fix double count ssl connection metrics#9594

Merged
clintropolis merged 4 commits intoapache:masterfrom
clintropolis:fix-double-count-ssl-connection-metrics
Apr 4, 2020
Merged

Fix double count ssl connection metrics#9594
clintropolis merged 4 commits intoapache:masterfrom
clintropolis:fix-double-count-ssl-connection-metrics

Conversation

@clintropolis
Copy link
Copy Markdown
Member

@clintropolis clintropolis commented Mar 31, 2020

Description

This PR fixes an issue where the jetty/numOpenConnections metric is double counted for SSL connections due to the way all ConnectionFactory instances are wrapped in JettyMonitoringConnectionFactory in JettyServerModule initialization, instead of what I believe to be the correct usage of only monitoring the default ConnectionFactory for the ServerConnector.

The TLS ServerConnector is constructed with 2 ConnectionFactory instances:

...
      final ServerConnector connector = new ServerConnector(
          server,
          new SslConnectionFactory(sslContextFactory, HTTP_1_1_STRING),
          new HttpConnectionFactory(httpsConfiguration)
      );
...

It is my understanding (and observation using the debugger) that both create a connection, the SslConnectionFactory to un-encrypt the request connection, and the HttpConnectionFactory to do HTTP stuff to the now un-encrypted connection. However, since both of these were prior to this PR wrapped in JettyMonitoringConnectionFactory, a single TLS connection would count as 2 active connections, despite only 1 request being made. This PR modifies JettyServerModule to now only monitor the ConnectionFactory with the same protocol as the default ServerConnector protocol to correctly single count the connection.

The added test JettyTest.testNumConnectionsMetricHttps fails without the modifications to JettyServerModule. The keystore/truststore in the test resources were generated in the same manner as used in the integration tests.


This PR has:

  • been self-reviewed.
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths.
  • been tested in a test Druid cluster.

Key changed/added classes in this PR
  • JettyServerModule

Comment thread examples/bin/dsql-main
parser_fmt.add_argument('--header', action='store_true', help='Include header row for formats "csv" and "tsv"')
parser_fmt.add_argument('--tsv-delimiter', type=str, default='\t', help='Delimiter for format "tsv"')
parser_oth.add_argument('--context-option', '-c', type=str, action='append', help='Set context option for this connection, see https://docs.imply.io/on-prem/query-data/sql for options')
parser_oth.add_argument('--context-option', '-c', type=str, action='append', help='Set context option for this connection, see https://druid.apache.org/docs/latest/querying/sql.html#connection-context for options')
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

new InputStreamResponseHandler()
);
// sad
Thread.sleep(100);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe there is some better way for waiting here rather than sleeping.. Maybe we can do like

while (jsm.getActiveConnections() == 0) {
  Thread.sleep(100);
}
Assert.assertEquals(1, jsm.getActiveConnections());
go.get();

Copy link
Copy Markdown
Contributor

@jihoonson jihoonson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@clintropolis
Copy link
Copy Markdown
Member Author

Thanks for review @jihoonson!

@clintropolis clintropolis merged commit 4d277db into apache:master Apr 4, 2020
@clintropolis clintropolis deleted the fix-double-count-ssl-connection-metrics branch April 4, 2020 06:29
clintropolis added a commit to clintropolis/druid that referenced this pull request Apr 4, 2020
* fix double counted jetty/numOpenConnections metric for ssl connections

* tests

* more better

* style
himanshug pushed a commit that referenced this pull request Apr 5, 2020
* fix double counted jetty/numOpenConnections metric for ssl connections

* tests

* more better

* style
@jihoonson jihoonson added this to the 0.18.0 milestone Apr 21, 2020
JulianJaffePinterest pushed a commit to JulianJaffePinterest/druid that referenced this pull request Jun 12, 2020
* fix double counted jetty/numOpenConnections metric for ssl connections

* tests

* more better

* style
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants