feat!: support index progress reporting via callbacks#5910
feat!: support index progress reporting via callbacks#5910wjones127 merged 6 commits intolance-format:mainfrom
Conversation
|
ACTION NEEDED The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
|
I used this to create the chart in #5907 |
|
this should be marked breaking, sorry for not marking that right. I can't edit the title so someone will need to add it. This adds a progress parameter to some of the index building functions. |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
| /// Called at stage boundaries during index construction. Stage names are | ||
| /// index-type-specific (e.g. "train_ivf", "shuffle", "build_partitions" for | ||
| /// vector indices; "load_data", "build_pages" for scalar indices). |
There was a problem hiding this comment.
Is it an error for multiple stages to be active at the same time?
There was a problem hiding this comment.
good thought. As currently structured stages are sequential, so if you need to do two things you would need to fold it into one stage. We could possibly figure out a way to relax that or evolve this to enable multiple stages at once.
Do we have index types today that would benefit from that? This only breaks out stages for IVF; I haven't given consideration to the others yet.
wjones127
changed the title
wkalt
changed the title
wkalt
force-pushed
the
task/index-build-progress-reporting
branch
from
dc04053 to
d5a898e
Compare
|
Happy to merge this once conflicts and lint errors are resolved. |
This adds support for progress reporting on index builds, allowing callers to determine whether a build is progressing, what stage it is in, and in some cases how far into the stage.
track iterations across the whole stage
wkalt
force-pushed
the
task/index-build-progress-reporting
branch
from
d5a898e to
d374d41
Compare
|
@wjones127 thanks, this should be good to go. |
2 participants
This adds support for progress reporting on index builds, allowing callers to determine whether a build is progressing, what stage it is in, and in some cases how far into the stage.
Breaking: updates some index builder function signatures to include a progress callback implementation. A noop implementation is included in the patch.