Implement incremental update using commit stats (SnapshotSummary)

[ajantha-bhat]

Feature Request / Improvement

This could be an experimental direction and can be controlled by a flag.

This might bloat up the table metadata file size when millions of partitions are present.

Query engine

None

[ajantha-bhat]

Lower priority and will be working on this once all other planned activities from #8450 is done.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'