Skip to content

Add identity to query metrics, logs.#4862

Merged
gianm merged 2 commits intoapache:masterfrom
gianm:security-fix-metrics-logging
Sep 28, 2017
Merged

Add identity to query metrics, logs.#4862
gianm merged 2 commits intoapache:masterfrom
gianm:security-fix-metrics-logging

Conversation

@gianm
Copy link
Copy Markdown
Contributor

@gianm gianm commented Sep 26, 2017

Adds "identity" to metrics and logs generated by QueryLifecycle.

Also fix a bug where unauthorized requests would not emit any logs or metrics,
and instead would log a "Tried to emit logs and metrics twice" warning.

Also rename QueryResource's "getServer" to "cancelQuery", because that's what
it does.

@leventov FYI in this patch I added identity as a dimension that is always part of
query metrics. I did it for three reasons.

  1. In 0.11 Druid now always has a concept of an authenticated user for each query. By default the authenticated user is a system superuser, but the concept is still there.
  2. Reporting user identity is a nearly universal feature of similar software (other databases).
  3. If a site isn't using any authorization beyond the default, then this dimension will not hurt aggregation of metrics at all, since it will always have the same value.

Also fix a bug where unauthorized requests would not emit any logs or metrics,
and instead would log a "Tried to emit logs and metrics twice" warning.

Also rename QueryResource's "getServer" to "cancelQuery", because that's what
it does.

@Override
public void identity(String identity)
{
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm still against adding any new dimensions, emitted by default, see #4570 (comment) (and later comments in that thread). Also I committed to contribute something like "full" metrics, with big disclaimer of no backward compatibility: #4570 (comment).

But if you still want to emit it by default, I don't have much motivation to fight against this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought the potential usefulness of the dimension outweighed the concerns you raised in #4570 (comment), especially since unless you go and specifically set up custom authenticators, the cardinality of the dimension will be 1.

What do you feel about a compromise where this dimension is emitted by default only if the user has authentication configured?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gianm

emitted by default only if the user has authentication configured?

What did you have in mind for the "has authentication configured" check?

Copy link
Copy Markdown
Contributor Author

@gianm gianm Sep 27, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jon-wei I was thinking checking if the authenticators list is non-default.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gianm my main concert is inability to take out or change the dimension in the future, not inefficiency in this case

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gianm that sounds good to me

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@leventov @jon-wei I changed it to not emit identity by default. Maybe it will change in a future patch but I will keep this one uncontroversial.

@jon-wei
Copy link
Copy Markdown
Contributor

jon-wei commented Sep 27, 2017

changes LGTM

@gianm
Copy link
Copy Markdown
Contributor Author

gianm commented Sep 28, 2017

Thanks @jon-wei for reviewing. I just added a patch to not emit identity by default; everything else is the same.

@gianm
Copy link
Copy Markdown
Contributor Author

gianm commented Sep 28, 2017

Thanks @leventov.

@gianm gianm merged commit a19f22b into apache:master Sep 28, 2017
@gianm gianm deleted the security-fix-metrics-logging branch September 28, 2017 18:45
gianm added a commit to implydata/druid-public that referenced this pull request Oct 30, 2017
* Add identity to query metrics, logs.

Also fix a bug where unauthorized requests would not emit any logs or metrics,
and instead would log a "Tried to emit logs and metrics twice" warning.

Also rename QueryResource's "getServer" to "cancelQuery", because that's what
it does.

* Do not emit identity by default.
gianm added a commit to implydata/druid-public that referenced this pull request Nov 14, 2017
* Add identity to query metrics, logs.

Also fix a bug where unauthorized requests would not emit any logs or metrics,
and instead would log a "Tried to emit logs and metrics twice" warning.

Also rename QueryResource's "getServer" to "cancelQuery", because that's what
it does.

* Do not emit identity by default.
gianm added a commit to implydata/druid-public that referenced this pull request Dec 5, 2017
* Add identity to query metrics, logs.

Also fix a bug where unauthorized requests would not emit any logs or metrics,
and instead would log a "Tried to emit logs and metrics twice" warning.

Also rename QueryResource's "getServer" to "cancelQuery", because that's what
it does.

* Do not emit identity by default.
@jon-wei jon-wei added this to the 0.12.0 milestone Jan 5, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants