Adding a simple program to measure speed of dot products

[ikawrakow]

I was surprised by the belief that the dot product x * y, where x holds quantized model weights and y contains floating point values, it is faster to quantize y, and to perform the dot product using the quantized y values (accepting the associated loss in precision), then to just directly compute x * y. So, I had to try it myself. This PR adds a simple program that measures the time it takes to perform the dot product between vectors holding 1 << 18 values. I picked a relatively large vector size to not have to get involved with the science of accurately measuring elapsed time for short lasting operations.

Basically, we fill two vectors x and y with random values and quantize x into q. We then measure the time for

1. Computing d = q * y directly
2. Computing y' = quantize(y); d = q * y'

For 2. we use the vectorized (SIMD-ified) functions from ggml (or, if requested by a command line argument, the corresponding scalar functions from ggml).

On my Mac, 1. is faster than 2 (~55 us vs ~75 us). On the x86_64 CPU that I have available (Ryzen 7950X), 1. is somewhat slower compared to the AVX2 implementation (~50 us vs ~35 us).

On both CPUs the direct product 1. as implemented in the dot() function in this POC is much faster than the scalar version of 2 from ggml. (~15X faster on the Ryzen 7950X and ~6X faster on the Mac).

I think that with some ARM_NEON or AVX2 magic one should be able to further speed up 1.

To use it, make -j and then e.g. ./vdot 100 to measure 100 dot products with the SIMD-ified ggml functions, or ./vdot 100 1 to measure the scalar ggml functions instead.

Added a comparison for Q4_1 quantization. Here, the direct product 1. is faster than 2. for ARM_NEON and AVX2. On my Mac I get ~69 us for 1 and ~121 us for 2. On the Ryzen 7950X I measured ~60 us for 1. and ~62 us for 2. In any case, implemented as in this POC, the dot product of Q4_1 quantized values is only marginally slower (~20%) than Q4_0.

[sw]

On my Core i3-8100 (AVX2):

$ ./vdot 100
<dot> = -74.2272, -73.9193
time = 128.407 +/- 4.38483 us. maxt = 150.281 us
timeq = 106.679 +/- 4.16175 us. maxt = 126.484 us

Please consider putting it in examples/benchmark instead of creating a new folder.

[ikawrakow]

@sw Thank you for the measurement. Yes, of course, I can move to examples. My thinking was that this is a POC, so it is better to have a folder for POCs for this (and possibly future POCs) before some of these POCs become "examples".

[ggerganov]

Either adding poc or moving to examples is fine. Just leave it as it is if you haven't started moving it

I'll try to plug this approach in the actual mul mat routines and see how is the performance.

P.S. Squash your commits on merge

[sw]

Yes, examples is maybe not a great name, but it already contains various bits and pieces like your program.