This repository was archived by the owner on Jun 14, 2024. It is now read-only.
Support refresh quick mode by using Hybrid Scan #238
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
imback82
reviewed
Nov 2, 2020
src/main/scala/com/microsoft/hyperspace/index/rules/RuleUtils.scala
Outdated
Show resolved
Hide resolved
pirz
reviewed
Nov 5, 2020
src/main/scala/com/microsoft/hyperspace/actions/RefreshQuickAction.scala
Outdated
Show resolved
Hide resolved
src/main/scala/com/microsoft/hyperspace/actions/RefreshQuickAction.scala
Outdated
Show resolved
Hide resolved
src/test/scala/com/microsoft/hyperspace/index/E2EHyperspaceRulesTests.scala
Show resolved
Hide resolved
This was
linked to
issues
Nov 18, 2020
pirz
reviewed
Nov 23, 2020
| // In case the index has appended files and/or deleted files which are updated by quick | ||
| // refresh mode, we should handle them in query time by using Hybrid Scan. | ||
| // Because quick refresh doesn't update index data, but just metadata in IndexLogEntry. | ||
| lazy val quickRefreshedIndex = index.hasSourceUpdate |
Contributor
There was a problem hiding this comment.
If we are using it only once, do we need to create a separate (lazy) val for it?
Collaborator
Author
There was a problem hiding this comment.
It's mainly for readability and comments. + It's only required when HybridScan is enabled, so it's lazy val.
Contributor
There was a problem hiding this comment.
I would add a comment if you added lazy to avoid creating sets for appended files, etc.
src/main/scala/com/microsoft/hyperspace/index/rules/RuleUtils.scala
Outdated
Show resolved
Hide resolved
src/test/scala/com/microsoft/hyperspace/index/RefreshIndexTests.scala
Outdated
Show resolved
Hide resolved
Collaborator
Author
|
@imback82 @pirz @apoorvedave1 Could you review this change? Thanks! |
pirz
previously approved these changes
Dec 2, 2020
Contributor
|
LGTM, thanks @sezruby |
imback82
reviewed
Dec 3, 2020
src/main/scala/com/microsoft/hyperspace/actions/RefreshQuickAction.scala
Outdated
Show resolved
Hide resolved
src/main/scala/com/microsoft/hyperspace/index/rules/RuleUtils.scala
Outdated
Show resolved
Hide resolved
| // In case the index has appended files and/or deleted files which are updated by quick | ||
| // refresh mode, we should handle them in query time by using Hybrid Scan. | ||
| // Because quick refresh doesn't update index data, but just metadata in IndexLogEntry. | ||
| lazy val quickRefreshedIndex = index.hasSourceUpdate |
Contributor
There was a problem hiding this comment.
I would add a comment if you added lazy to avoid creating sets for appended files, etc.
src/main/scala/com/microsoft/hyperspace/index/rules/RuleUtils.scala
Outdated
Show resolved
Hide resolved
imback82
previously approved these changes
Dec 3, 2020
src/main/scala/com/microsoft/hyperspace/actions/RefreshQuickAction.scala
Outdated
Show resolved
Hide resolved
| (index.sourceFileInfoSet -- exist, filesAppended) | ||
| if (!HyperspaceConf.hybridScanEnabled(spark) && index.hasSourceUpdate) { | ||
| // If the index contains the source update info, we need to handle | ||
| // appendedFiles and deletedFiles in IndexLogEntry. |
Contributor
There was a problem hiding this comment.
Can you add back the comment that signature validation was done by this time?
Contributor
imback82
approved these changes
Dec 3, 2020
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the context for this pull request?
Fixes #174
Fixes #169
What changes were proposed in this pull request?
Support quick refresh of indexes which only updates the metadata -
Updatedata - of the index.After the refresh, the latest signature will be used for index selection process (getCandidateIndexes) and the appended files and deleted files of
Updatewill be handled by Hybrid Scan.Does this PR introduce any user-facing change?
Yes
How was this patch tested?
Unit test