Question: Intergration Plan/Solution for Lance with Iceberg

[littleDrew]

Query engine

Spark engine

Question

hi @pvary, it seems that you are working intergrate lance into iceberg, can you show the detail implementation plan/solution.

Here i want to have a discuss or know previous analysis about select which way to intergrate with iceberg:

1st way: use lance file format to intergrate with iceberg, thus like following implementation by @westonpace
2nd way: use lance table format to intergrate with iceberg
Here i thought, considering about performance issue, it seems the 2st way(use lance table format) will be better for good performance

Thus want to have a dissuss as for implementation way. if use 1st way, can we achive good performance, or have we evalute the pros and cons of both implementation way.

public enum FileFormat {
PUFFIN("puffin", false),
ORC("orc", true),
PARQUET("parquet", true),
AVRO("avro", true),
LANCE("lance", true) // add lance as an file format of iceberg instead of table format
}
westonpace@84bf5c5

Hope we have a disscus for this together, as i also have noticed lance have a good performance in proccessing/reading multi-dimentional data. @electrum @martint @massdosage @sullis

[pvary]

@littleDrew: We are in the very early steps on the integration, and I see a long road ahead before it could work out of the box.

We are working on the early steps to enable plugging in new File Formats into Iceberg (#12298). Lance is only one of the possible candidates, and the Iceberg community needs some more discussion on the next steps.

I'm not sure what you mean by the 2nd way (use lance table format to integrate with Iceberg)

[manuzhang]

Maybe the 2nd way is like delta integration?

[manuzhang]

Meanwhile, I'd love to hear thoughts from @jackye1995 and @Xuanwo, who've been involving in both projects.

[littleDrew]

@littleDrew: We are in the very early steps on the integration, and I see a long road ahead before it could work out of the box.

We are working on the early steps to enable plugging in new File Formats into Iceberg (#12298). Lance is only one of the possible candidates, and the Iceberg community needs some more discussion on the next steps.

I'm not sure what you mean by the 2nd way (use lance table format to integrate with Iceberg)

• Hi @pvary, can you explain in detail as for above long road, what feature and work is needed in ahead. If you have detail design doc as for integrating Iceberg with lance file format, hope you can have a share and discuss together.

• Another question, i am consider what's kinds of benefit can we achive to integrate lceberg with lance file format(may be the random access performance will not be good as in native lance), what's your idea as this question.

• As for the 2nd way, here may refers integrate with lance fragment(treat this as one data file in iceberg), corresponding doc is provided by lance developer "jack ye", this may may can use fragment level metadata and index, and also use lance file level index(chunk index) esaliy, but here may have some effort, how do you like about this.
  https://docs.google.com/document/u/0/d/1AUAO2M_kmVvoCA2GYCBFvAcaM8zieU2fJmHVYuiew9o/edit?tab=t.0&pli=1&authuser=0#heading=h.3fnc1e5nn72y

[pvary]

Having a detailed document at this point would be premature.

The steps which are needed:

1. Create an API layer for readers and writers - Core: Interface based DataFile reader and writer API #12298
2. Create a test framework to verify that the readers and writers are working by the specification
   • After this point if a file format (like Lance) decides that it wants to support Iceberg tables, it can create a test implementation which could be used for functional and performance testing
3. The community needs to decide if/how it wants to support new file formats. I see the following (not exclusive) possibilities myself:
   a. Add the new certified format to the FileFormat enum
   b. Change the File Format enum to a String, and enable "any" file formats
   c. Add an intermediate layer to allow testing of file formats which implement this intermediate layer

The benefit of integrating a new file format could be:

1. Performance benefits, if the reading/writing of the format is more effective than Parquet/ORC/Avro
2. Compatibility benefit, as more engine could read the data. This is more pronounced with 3.c, since there is no need implement the readers and writers for every FileFormat+Engine combination
3. Catalog integration benefit, as tables could be organized by a single catalog.

About @jackye1995's document:
I like the integration solution proposed by him. There is one point, where we need to enhance the current File Format API proposal for this. Currently the readers and a writers are working on a single data stream. To enable the integration mentioned in the proposal, we need to push the FileIO to the readers and writers so they can open additional data/index files.

[littleDrew]

Hi @pvary, these 3 needed steps, it seems that: step 1 you have done, and step 3 seems need to be discuss by iceberg community member, does there existed a planed discussion meeting to decide this. and step 2 looks more like do more test to verify.

and as for the performance benefit to integrate with lance compare with parquet, i also have do some test, here i mainly consider full scan performance and random access performance.

• The test using 1 millon rows, it shows that, the scan performance may better (1.5~3x faster) than parquet in condition the column not so many(should less than 50 column), but if column are two much scan performance will not be better than parquet.
• and the random access performance, my current test like lance official test case, it show that have 40x faster, when only random take 20~50 rows from the file. I still keep doubt as for 40x faster as only test lance file instead of lance table.
• disk space issue: i found use lance will take 50% more disk space than parquet. may be we also should consider how to deal with this well.

Thus, do you think some other work like implement much more native lance reader is needed to improve performance(like random access).

[pvary]

@littleDrew: We will definitely need these steps, but I try to focus on finishing the first ones before rushing forward.
These changes are quite big, and need a community approval. The community currently focuses on wrapping up V3, probably we need can progress with more speed after that

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'