Skip to content

More methods in QueryMetrics and TopNQueryMetrics (the last part of #3798)#4284

Merged
drcrallen merged 3 commits intoapache:masterfrom
metamx:more-queryMetrics-methods
Jun 7, 2017
Merged

More methods in QueryMetrics and TopNQueryMetrics (the last part of #3798)#4284
drcrallen merged 3 commits intoapache:masterfrom
metamx:more-queryMetrics-methods

Conversation

@leventov
Copy link
Copy Markdown
Member

  • Added various hook methods to QueryMetrics and TopNQueryMetrics and call them from TopNQueryEngine. When somebody needs the hooks added to QueryMetrics for other query types, corresponding query engine(s) could be updated to call the hooks, in another PR.
  • Added BitmapResultFactory, an abstraction that wraps BitmapFactory and provides ability to record the bitmap construction "form" (hierarchical tree of bitmap unions, intersections, and complements) and report it via a query metric dimension. It was mentioned here: Don't union with empty bitmap in the end in Filters.matchPredicate() #3754 (comment)

This PR is the last part of #3798.

@leventov leventov added this to the 0.10.1 milestone May 17, 2017
@leventov leventov requested a review from drcrallen May 17, 2017 01:03
Granularity gran,
boolean descending
boolean descending,
@Nullable QueryMetrics<?> queryMetrics
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

instead of putting null everywhere, can you make a backwards compatible method signature and just @Deprecated it?

Copy link
Copy Markdown
Member Author

@leventov leventov May 24, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@drcrallen null queryMetrics is allowed not for backward compatibility, it is valid, means that the caller doesn't want or couldn't record metrics. OTOH updated signatures are not in the interfaces/classes which are extension points, so adding "backwards compatible" deprecated methods is pointless.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

/**
* {@link #getBitmapIndex} and {@link #getBitmapResult} methods both have default implementations, delegating to each
* other. Every implementation of {@link Filter} should override {@link #getBitmapResult}, currently it has a default
* implementation for compatibility with Filters in extensions. In Druid 0.11 {@link #getBitmapResult} is going to
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm... I'm not sure if we have a good way to "not forget about" this intention, can you file an issue related to this with a 0.11 tag?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created #4323

return getBitmapResult(selector, new DefaultBitmapResultFactory(selector.getBitmapFactory()));
}

default <T> T getBitmapResult(BitmapIndexSelector selector, BitmapResultFactory<T> bitmapResultFactory)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it appropriate to mark this one @Deprecated so that any usage gets the notice about the intended 0.11 change?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@drcrallen probably you meant this comment on getBitmapIndex(), but even getBitmapIndex() is not deprecated for usage, for those who don't need BitmapResult thing. getBitmapIndex() is deprecated for overriding.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok

@drcrallen
Copy link
Copy Markdown
Contributor

so if I understand this correctly, the key change here is the change from using different creation methods for bitmap results to using a unified BitmapResultFactory interface in order to facilitate better metrics collection of the bitmap operations. I think that makes some good sense overall, but I'm a bit unclear on what the future intentions of BitmapResultFactory are. Should there be specialty methods for every type of bitmap operation someone might want to do? How do you want to handle future additions to that factory? Is there a way someone can have a custom BitmapResultFactory and not have it break in arbitrary future versions?

@leventov
Copy link
Copy Markdown
Member Author

@drcrallen

Should there be specialty methods for every type of bitmap operation someone might want to do?

Yes.

How do you want to handle future additions to that factory?

Just add methods, let custom BitmapResultFactory impls break. BitmapResultFactory is a part of the QueryMetrics subsystem, which is designed to break often. It is described in QueryMetrics Javadoc comment, and rationale is described here: #3954 (comment). Also added a note in BitmapResultFactory Javadoc.

@jon-wei
Copy link
Copy Markdown
Contributor

jon-wei commented Jun 2, 2017

Design/code looks good to me, 👍

@drcrallen did you have any other feedback on this?

I also noticed a couple of conflicts with #4288 (in DefaultQueryMetrics and DefaultTopNQueryMetrics) when applying this patch locally, not sure why they don't show up here...

@leventov
Copy link
Copy Markdown
Member Author

leventov commented Jun 2, 2017

Thanks @jon-wei. The master branch merged cleanly.

@drcrallen drcrallen merged commit b487fa3 into apache:master Jun 7, 2017
@drcrallen drcrallen deleted the more-queryMetrics-methods branch June 7, 2017 16:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants