This repository was archived by the owner on Jun 14, 2024. It is now read-only.
Pluggable source provider #227
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Oct 29, 2020
Contributor
Author
imback82
commented
Nov 23, 2020
src/main/scala/com/microsoft/hyperspace/index/sources/interfaces.scala
Outdated
Show resolved
Hide resolved
| * @tparam A Type of the initial value. | ||
| * @tparam B Type of the final value. | ||
| */ | ||
| class CacheWithTransform[A, B](val init: () => A, val transform: A => B) { |
Contributor
Author
There was a problem hiding this comment.
I need to add a test for this class.
| * - The relation is [[HadoopFsRelation]] with [[PartitioningAwareFileIndex]] as file index. | ||
| * - Its file format implements [[DataSourceRegister]]. | ||
| */ | ||
| class DefaultFileBasedSource(private val spark: SparkSession) extends FileBasedSourceProvider { |
Contributor
Author
There was a problem hiding this comment.
I need to create a test for this class.
| * | ||
| * @param spark Spark session. | ||
| */ | ||
| class FileBasedSourceProviderManager(spark: SparkSession) { |
Contributor
Author
There was a problem hiding this comment.
I need to create a test for this.
sezruby
reviewed
Nov 23, 2020
| spark.sessionState.conf | ||
| .getConfString( | ||
| "spark.hyperspace.index.sources.fileBasedBuilders", | ||
| "com.microsoft.hyperspace.index.sources.default.DefaultFileBasedSourceBuilder") |
Collaborator
There was a problem hiding this comment.
Why this config is necessary? Can we just create all builders?
Contributor
Author
There was a problem hiding this comment.
To control what builders are being loaded similar to how plugins, listeners, etc. are loaded in SparkContext. What's the benefit of creating all builders available? The only benefit I can think of is not having to specify this config, but having a tight control seems better to me.
imback82
commented
Nov 25, 2020
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the context for this pull request?
Introduce pluggable source provider.
What changes were proposed in this pull request?
This PR proposes to introduce pluggable source provider APIs so that new source such as Delta Lake can be easily plugged in.
Does this PR introduce any user-facing change?
Yes. This PR exposes a new way to plug in a source from which an index can be created.
How was this patch tested?
Existing tests