Conversation
achirkin
left a comment
achirkin
left a comment
There was a problem hiding this comment.
Thank you for the contribution. I see the PR changes the accumulation type from fp16 to fp32 to avoid fp overflow. However I'm not sure if this is desirable in general and whether the speed drop is acceptable for the cases when the overflow doesn't happen.
Maybe we'd better just advise the user to switch to fp32 variant of the algorithm?
Please support the PR with the benchmark results (using cuvs ann-bench) before and after the PR for fp16 and fp32 if you decide to proceed with this approach.
| template <> | ||
| struct config<half> { | ||
| using value_t = half; | ||
| using value_t = float; |
There was a problem hiding this comment.
Please check whether this is used outside the IVF-Flat. Changing the accumulation type like this can have a drastic impact on performance.
There was a problem hiding this comment.
I did simple benchmark and it showed no significant differences. I'll use cuvs ann-bench to get a more detailed benchmark results later
There was a problem hiding this comment.
@hhy3 any updates here? We're about to begin burndown for 25.08 release. Should we consider this for 25.08 or push to 25.10 (October)?
There was a problem hiding this comment.
@cjnolet sorry for my late reply. I just benchmark the performance, and it shows some regression on performance, especially when nprobe is small, but significantly improves recall:
IVF-Flat FP16 AccT=float vs AccT=half Benchmark Results
- Dataset: cohere-768-angular-fp16 (1M vectors, 768 dims, inner_product)
- GPU: NVIDIA A100-PCIE-40GB
- nlist=4096, ratio=10, niter=20, k=100
Latency (bs=64)
| nprobe | half latency (ms) | float latency (ms) | change | half recall | float recall |
|---|---|---|---|---|---|
| 32 | 4.02 | 5.04 | +25% | 0.815 | 0.849 |
| 64 | 6.22 | 6.89 | +11% | 0.867 | 0.913 |
| 128 | 9.87 | 10.2 | +3% | 0.897 | 0.955 |
Throughput (bs=1000, threads:1)
| nprobe | half QPS | float QPS | change | half recall | float recall |
|---|---|---|---|---|---|
| 32 | 24.7k | 22.8k | -8% | 0.816 | 0.850 |
| 64 | 13.8k | 13.2k | -4% | 0.868 | 0.913 |
| 128 | 7.87k | 7.84k | -0.4% | 0.898 | 0.956 |
cjnolet
added
bug
Cherry-picked from upstream PR rapidsai#1084. Uses float accumulator for FP16 distance computation to prevent overflow when distance values exceed FP16 range. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
3 participants
This PR fixes issue #914 that accumulation using fp16 causes overflow