feat: support _rowid meta column for spark connector in java#3194
Merged
wjones127 merged 4 commits intolance-format:mainfrom Dec 6, 2024
Merged
feat: support _rowid meta column for spark connector in java#3194wjones127 merged 4 commits intolance-format:mainfrom
wjones127 merged 4 commits intolance-format:mainfrom
Conversation
SaintBacchus
force-pushed
the
SupportRowIdInJava
branch
from
def8623 to
f62523a
Compare
Collaborator
Author
eddyxu
requested review from
LuQQiu,
westonpace and
wjones127
and removed request for
westonpace and
wjones127
wjones127
requested changes
Dec 4, 2024
Contributor
wjones127
left a comment
wjones127
left a comment
There was a problem hiding this comment.
The code looks good to me. My only serious worry is the licensing.
SaintBacchus
force-pushed
the
SupportRowIdInJava
branch
2 times, most recently
from
b329f52 to
d324d62
Compare
Collaborator
Author
|
@wjones127 I had added the license file in lancedb. Plz review it |
wjones127
requested changes
Dec 5, 2024
Comment on lines
+438
to
+437
| ------------------------------------------------------------------------------------ | ||
| This product bundles various third-party components under other open source licenses. | ||
| This section summarizes those components and their licenses. See licenses/ | ||
| for text of these licenses. | ||
|
|
||
|
|
||
| Apache Software Foundation License 2.0 | ||
| -------------------------------------- | ||
|
|
||
| common/network-common/src/main/java/org/apache/spark/network/util/LimitedInputStream.java | ||
| core/src/main/java/org/apache/spark/util/collection/TimSort.java | ||
| core/src/main/resources/org/apache/spark/ui/static/bootstrap* | ||
| core/src/main/resources/org/apache/spark/ui/static/vis* | ||
| docs/js/vendor/bootstrap.js | ||
| connector/spark-ganglia-lgpl/src/main/java/com/codahale/metrics/ganglia/GangliaReporter.java | ||
| core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.min.js | ||
| core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.css | ||
|
|
||
| Python Software Foundation License | ||
| ---------------------------------- | ||
|
|
||
| python/pyspark/loose_version.py | ||
|
|
||
| BSD 3-Clause | ||
| ------------ | ||
|
|
||
| python/lib/py4j-*-src.zip | ||
| python/pyspark/cloudpickle/*.py | ||
| python/pyspark/join.py | ||
|
|
||
| The CSS style for the navigation sidebar of the documentation was originally | ||
| submitted by Óscar Nájera for the scikit-learn project. The scikit-learn project | ||
| is distributed under the 3-Clause BSD license. | ||
|
|
||
|
|
||
| MIT License | ||
| ----------- | ||
|
|
||
| core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js | ||
| core/src/main/resources/org/apache/spark/ui/static/*dataTables* | ||
| core/src/main/resources/org/apache/spark/ui/static/graphlib-dot.min.js | ||
| core/src/main/resources/org/apache/spark/ui/static/jquery* | ||
| core/src/main/resources/org/apache/spark/ui/static/sorttable.js | ||
| docs/js/vendor/anchor.min.js | ||
| docs/js/vendor/jquery* | ||
| docs/js/vendor/modernizer* | ||
|
|
||
| ISC License | ||
| ----------- | ||
|
|
||
| core/src/main/resources/org/apache/spark/ui/static/d3.min.js | ||
|
|
||
|
|
||
| Creative Commons CC0 1.0 Universal Public Domain Dedication | ||
| ----------------------------------------------------------- | ||
| (see LICENSE-CC0.txt) | ||
|
|
||
| data/mllib/images/kittens/29.5.a_b_EGDP022204.jpg | ||
| data/mllib/images/kittens/54893.jpg | ||
| data/mllib/images/kittens/DP153539.jpg | ||
| data/mllib/images/kittens/DP802813.jpg | ||
| data/mllib/images/multi-channel/chr30.4.184.jpg | ||
|
|
||
| https://github.com/apache/spark/blob/master/LICENSE No newline at end of file |
Contributor
There was a problem hiding this comment.
I think you can remove all of this, since none of this applies to the files you brought over.
Suggested change
| ------------------------------------------------------------------------------------ | |
| This product bundles various third-party components under other open source licenses. | |
| This section summarizes those components and their licenses. See licenses/ | |
| for text of these licenses. | |
| Apache Software Foundation License 2.0 | |
| -------------------------------------- | |
| common/network-common/src/main/java/org/apache/spark/network/util/LimitedInputStream.java | |
| core/src/main/java/org/apache/spark/util/collection/TimSort.java | |
| core/src/main/resources/org/apache/spark/ui/static/bootstrap* | |
| core/src/main/resources/org/apache/spark/ui/static/vis* | |
| docs/js/vendor/bootstrap.js | |
| connector/spark-ganglia-lgpl/src/main/java/com/codahale/metrics/ganglia/GangliaReporter.java | |
| core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.min.js | |
| core/src/main/resources/org/apache/spark/ui/static/d3-flamegraph.css | |
| Python Software Foundation License | |
| ---------------------------------- | |
| python/pyspark/loose_version.py | |
| BSD 3-Clause | |
| ------------ | |
| python/lib/py4j-*-src.zip | |
| python/pyspark/cloudpickle/*.py | |
| python/pyspark/join.py | |
| The CSS style for the navigation sidebar of the documentation was originally | |
| submitted by Óscar Nájera for the scikit-learn project. The scikit-learn project | |
| is distributed under the 3-Clause BSD license. | |
| MIT License | |
| ----------- | |
| core/src/main/resources/org/apache/spark/ui/static/dagre-d3.min.js | |
| core/src/main/resources/org/apache/spark/ui/static/*dataTables* | |
| core/src/main/resources/org/apache/spark/ui/static/graphlib-dot.min.js | |
| core/src/main/resources/org/apache/spark/ui/static/jquery* | |
| core/src/main/resources/org/apache/spark/ui/static/sorttable.js | |
| docs/js/vendor/anchor.min.js | |
| docs/js/vendor/jquery* | |
| docs/js/vendor/modernizer* | |
| ISC License | |
| ----------- | |
| core/src/main/resources/org/apache/spark/ui/static/d3.min.js | |
| Creative Commons CC0 1.0 Universal Public Domain Dedication | |
| ----------------------------------------------------------- | |
| (see LICENSE-CC0.txt) | |
| data/mllib/images/kittens/29.5.a_b_EGDP022204.jpg | |
| data/mllib/images/kittens/54893.jpg | |
| data/mllib/images/kittens/DP153539.jpg | |
| data/mllib/images/kittens/DP802813.jpg | |
| data/mllib/images/multi-channel/chr30.4.184.jpg | |
| https://github.com/apache/spark/blob/master/LICENSE |
Collaborator
Author
There was a problem hiding this comment.
OK, I had removed it
Comment on lines
+25
to
+36
| /** | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| SPDX-FileCopyrightText: Copyright The Lance Authors | ||
|
|
||
| The following code is originally from https://github.com/apache/spark/blob/master/sql/catalyst/src/test/scala/org/apache/spark/sql/util/ArrowUtilsSuite.scala | ||
| and is licensed under the Apache license: | ||
|
|
||
| License: Apache License 2.0, Copyright 2014 and onwards The Apache Software Foundation. | ||
| https://github.com/apache/spark/blob/master/LICENSE | ||
|
|
||
| It has been modified by the Lance developers to fit the needs of the Lance project. | ||
| */ |
Contributor
There was a problem hiding this comment.
Could you move this up? It's confusing with the other license above.
Collaborator
Author
There was a problem hiding this comment.
I just put it in the apache license header.
Comment on lines
+25
to
+36
| /** | ||
| SPDX-License-Identifier: Apache-2.0 | ||
| SPDX-FileCopyrightText: Copyright The Lance Authors | ||
|
|
||
| The following code is originally from https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/vectorized/ArrowColumnVectorSuite.scala | ||
| and is licensed under the Apache license: | ||
|
|
||
| License: Apache License 2.0, Copyright 2014 and onwards The Apache Software Foundation. | ||
| https://github.com/apache/spark/blob/master/LICENSE | ||
|
|
||
| It has been modified by the Lance developers to fit the needs of the Lance project. | ||
| */ |
SaintBacchus
force-pushed
the
SupportRowIdInJava
branch
from
df3ca8e to
bbf165f
Compare
wjones127
approved these changes
Dec 6, 2024
Contributor
wjones127
left a comment
wjones127
left a comment
There was a problem hiding this comment.
Thanks for working with me on the license stuff. Looks good now :)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As discussion in PR, I had implement the _rowid meta column just in java package.