docs: add spark r/w lance demo#3574
Conversation
github-actions
Bot
added
the
documentation
| Training Multi-Modal models using a Lance dataset <./clip_training.rst> | ||
| Deep Learning Artefact Management using Lance <./artefact_management.rst> No newline at end of file | ||
| Deep Learning Artefact Management using Lance <./artefact_management.rst> | ||
| Reading and writing a Lance dataset in Spark <./spark_example.rst> No newline at end of file |
There was a problem hiding this comment.
I think this is a spark data source example. It will have another spark catalog connector example when the catalog is ready.
So this doc should rename as spark_datasource_example.rst
| * lance-core JAR: Core Rust Spark binding exposing Lance features to Java (available `here <https://repo1.maven.org/maven2/com/lancedb/lance-core/0.23.0/lance-core-0.23.0.jar>`_) | ||
| * lance-spark JAR: Spark connector for reading/writing Lance format (available `here <https://repo1.maven.org/maven2/com/lancedb/lance-spark/0.23.0/lance-spark-0.23.0.jar>`_) | ||
|
|
||
| Place these JARs in the ``${SPARK_HOME}/jars`` directory, then run: |
There was a problem hiding this comment.
These two jars are not enough to run lance connector. The arrow and JNI are needed.
But I think how to set up the spark env should be another doc under integrations.
|
hello, curious whether you could also write a basic example for pyspark that would be compatible with spark3.5 and python3.10? |
@dowjones226 Of course, we can add a pyspark demo, but to reduce this PR's review context, I'd rather open another PR to add the pyspark demo, WDYT? |
|
@eddyxu Can we push this work to land as soon as possible? |
|
@yanghua yes that would be great! please cc me on the PR. Thank you. |
4 participants
No description provided.