This repository was archived by the owner on Jun 14, 2024. It is now read-only.
This repository was archived by the owner on Jun 14, 2024. It is now read-only.
Incremental Indexing: Support for incremental changes to index when new data is added #29
Description
Describe the issue
Currently when new data is added, the indexes become obsolete. The only way to make them usable is through the refresh() api which recreates complete index from scratch. Add functionality to support creation of indexes only on additional new data instead of recreating from scratch.
Expected behavior
New additional data to old data doesn't require rebuilding of indexes on complete data. Only additional data gets indexed
Intermediate issues and PRs
- Capture source file info to detect the change of the dataset Capture source file info to detect the change of the dataset #120
- The root paths captured in index log entry should always be absolute paths The root paths captured in index log entry should always be absolute paths #126
- Store index file names and file sizes in operation log Store index file names and file sizes in operation log #107
- Add full list of index files to index log entry Add full list of index files to index log entry #103
- Index metadata should always store absolute paths (alternate solution) Index metadata should always store absolute paths (alternate solution) #128, Index metadata should always store absolute paths #127
- Store file name, size and modified time of index files in metadata Store file name, size and modified time of index files in metadata #122
- New Tree based Structure for index metadata New Tree based Structure for index metadata #139
- Index merge functionality PR: Add merge functionality on Directory object #162
- Incremental indexing support for Append-only data Add RefreshIncrementalAction class to support index creation on newly appended data #110 , PR: Add RefreshIncrementalAction to support index creation on newly appended data #163