[Feature] Don't store redundant primary key columns

[zhongyujiang]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

I found that Paimon will redundantly store the primary key columns in files with a name prefixed with _KEY_, this increase the query overhead and storage size. Especially when there are several primary key fields, like 5 or even more.

Is it possible to directly use the original column values to be projected as the primary key during query? I think that can reduce the load of reading and writing, as well as the storage volume.

Solution

Generate primary key values directly from original columns instead of use redundant primary key columns.

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[JingsongLi]

Hi @zhongyujiang
Yes, of course, there is a draft: #1825 but he have no time to move forward.

[zhongyujiang]

Thanks for the reply, good to know that we’re already working on this!

[JingsongLi]

#4651

[JingsongLi]

#4666

[JingsongLi]

We are introducing this optimization!