ggml : alternative Q4_3 implementation using modified Q8_0

[ggerganov]

This one looks promising - it does not change the Q4_3 format from master and only modifies slightly Q8_0 by adding low and high sums. The results should be identical, but now the Q4_3 dot product evaluates much faster:

#define QK8_0 32
typedef struct {
    float   d;          // delta
    float   s0;         // d * sum(qs[i]) low
    float   s1;         // d * sum(qs[i]) high
    int8_t  qs[QK8_0];  // quants
} block_q8_0;

llama_print_timings:      sample time =    47.11 ms /    64 runs   (    0.74 ms per run)
llama_print_timings: prompt eval time =   482.44 ms /     8 tokens (   60.30 ms per token)
llama_print_timings:        eval time =  3419.36 ms /    63 runs   (   54.28 ms per run)
llama_print_timings:       total time =  3959.05 ms

I think this is the way to go. But, let's see the ppl results from the Q4_3a #1108 approach first

[ggerganov]

Will fix the AVX2 implementation tomorrow and merge it

[sw]

As mentioned in #1099 where I intend to fix this, the #if condition is wrong here, causing the code below to be executed for AVX2, essentially duplicating the work. Just a thing to keep in mind or fix when measuring performance.