Top-level indices part 2: empty vector indices

[wjones127]

Follow up to #3940

We are trying to make it possible to keep indices even though we don't have enough data to actually train them. In an earlier PR, we are adding a train: bool parameter to let the user choose whether they want to defer training an index. If train=False, then we just create an empty index and it will show num_indexed_rows: 0 in stats. They can later call optimize_indices() to do the actual training.

Vector indices are more complex because of IVF partitions, since we need a minimum amount of data to create them.

Here is the behavior I think we want:

1. Can pass train=False to create an empty index. No IVF or PQ will be trained. Will save metadata as empty array.
2. If you call create_index(..., train=True) or optimize_indices():
   • < 256 non-null vectors -> same as train=False
   • 256 <= number of non-null vectors < num_partitions * 256 => train on smaller number of partitions
   • number of non-null vectors >= num_partitions * 256 -> train full index

[BubbleCal]

some concerns:

1. if there are <256 valid vectors
   • if we don't actually train the index, it may cause confusion, the flat KNN would cause IO, if the users pass fast_search=True, they will see no data
   • FYI if a vector index has been created, and the user append rows less than 256, then optimize_indices for appending a delta is possible, because it doesn't retrain IVF/PQ
2. changing the num_partitions param if there are not enough vectors, still it sounds confusing, will this change the num_partitions back to what the users specified when it has enough vectors?

[wjones127]

if we don't actually train the index, it may cause confusion, the flat KNN would cause IO, if the users pass fast_search=True, they will see no data

We still have flat KNN whenever there are unindexed rows. There just will be some points where even if you call optimize indices, num_unindexed_rows will never go to 0. Although that seems solveable long term: we could add support for small single partitions later, if we find that's a good idea.

if the users pass fast_search=True, they will see no data

That is the other disadvantage. Again, probably solveable by allowing creating a small single partition index on < 256 vectors. Maybe at that scale we skip PQ training and just write raw vectors as single partition or something.

still it sounds confusing, will this change the num_partitions back to what the users specified when it has enough vectors?

Yes, it should update partitions once there is enough vectors. This can happen when optimize_indices is called.

[wjones127]

To motivate this a little, here is the problem:

1. A dataset can have any number of rows, and this can change lot. A table might start with 0 rows. Get 100,000 rows. Go back to 0. Get 1000 rows at a time, until it reaches 40,000 rows.
2. Users shouldn't have to reconfigure their index every time they write. They should be able to configure an index in the beginning, and then call optimize_indices throughout the lifecycle of the table to get an appropriately built vector index.

On some level, I'm not even sure if users should be setting num_partitions at all, most of the time. It would be nice to default to auto or something, and we just use the current row count to figure out how many partitions are needed. A better parameter might be target_rows_per_partition or for large datasets max_partitions.

[wjones127]

I've been thinking about this and to make this work we really need to make optimize indices able to control the number of partitions. I think we can think of these as a sort of two dimensional plane right with num partitions and num rows as the axes. We need the user to be able to define some ideal relationship between num rows and num of partitions. Then optimize_indices() job becomes to check whether we're in the ideal zone and if we're not to change the number of partitions to bring it back to the ideal state.