PERF: Increase threashold for using binary search in IndexEngine

[l3robot]

Checklist

• closes PERF: Unreliable performance of .loc with big non-unique index dataframe #54550 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Explanation

So after a bit of research for #54550, I found that the source of the behavior is coming from PR #22826 at this line and particularly this comment here.

In the comment, @rtlee9 makes an excellent explanation for the reason behind the threshold. The default behavior is a linear search (O(n)) in the pandas index and #22826 added a binary search optimization for monotonically increasing index which is in O(m*log(n)) where m is the number of target indexes.

Proposed Solution

I'm proposing we increase the limit, which was set conservatively to 5. We could at least increase it to n / (2 * log2[n]) which is derivable from m*log2[n]<n and dividing per 2 for keeping a certain margin. I'll open a PR for that.

New results of benchmarking + homemade script

Here is a screenshot of asv continuous -f 1.1 -E virtualenv upstream/main HEAD -b ^indexing.*Index* output (done on Macbook Air m1, inside the docker container provided by pandas):

Here is also a new output for the script use to demonstrate the problem in #54700:

With all at once: num_indexes=1 => 0.00048s
With all at once: num_indexes=2 => 0.00045s
With all at once: num_indexes=3 => 0.00038s
With all at once: num_indexes=4 => 0.00035s
With all at once: num_indexes=5 => 0.00035s
With all at once: num_indexes=6 => 0.00045s
With all at once: num_indexes=7 => 0.00042s
With all at once: num_indexes=8 => 0.00066s
With all at once: num_indexes=9 => 0.00049s
With all at once: num_indexes=10 => 0.00048s

Thanks in advance for the review!

[l3robot]

bit_lenght is just a fast way of computing the ceiling of log2.

[WillAyd]

nice! I think this is good

[mroeschke]

Suggested change

	- Increase the threshold to use binary search instead of a linear search in IndexEngine :issue:`54550`
	- Performance improvement when indexing with more than 4 keys (:issue:`54550`)

[l3robot]

Nice, thanks a lot for the review.
done!

[mroeschke]

Thanks @l3robot

[l3robot]

Thanks @l3robot

Glad to contribute!