[Feature] Reduce the access of filesystem when reading paimon table

[wg1026688210]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently, the computation engine frequently reads the schema file and the size of the data files from filesystem when reading the Paimon table.This lead to additional access to the file system. The purpose of this issue is to reduce access to the file system by caching the schema ,paimon table and so on.

Solution

• Hive:
  • Cache the table into Split to avoid loading Paimon table when create reader [Hive]Pass Table into Split to Avoid Loading from FileSystem when createRecordReader #2927
  • Cache TableSchema into Configuration to avoid loading read scheme file in PaimonSerDe [Hive] Cache TableSchema into Configuration to avoid loading read scheme file in PaimonSerDe #2946
• Core:
  • Obtain the size of the data file from the metadata. [core] orc/parquet reader obtain the fileSize from metadata #2918
  • cache the schema for SchemaManager. [Core]add table schema cache for SchemaManager #2939

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[zhangjun0x01]

I submit an pr to cache table in catalog , #2971
you could help me review it

[zhangjun0x01]

we can also to cache manifest file to improve query speed, after all, plan() is also a very heavy operation

https://docs.cloudera.com/cdw-runtime/cloud/iceberg-how-to/topics/iceberg-manifest-caching.html

[JingsongLi]

I submit an pr to cache table in catalog , #2971 you could help me review it

@zhangjun0x01 This is a separate issue. In theory, we can cache all the files.

[JingsongLi]

@wg1026688210 I think we have finished this issue, thanks!

[zhangjun0x01]

I submit an pr to cache table in catalog , #2971 you could help me review it

@zhangjun0x01 This is a separate issue. In theory, we can cache all the files.

yes ，we can open other pr to cache manifest , like this apache/iceberg#4518

[Kyofin]

I submit an pr to cache table in catalog , #2971 you could help me review it

@zhangjun0x01 This is a separate issue. In theory, we can cache all the files.

yes ，we can open other pr to cache manifest , like this apache/iceberg#4518

Is there any progress? This is a very important requirement.