[CB] Better parametrization for compile

[remi-or]

Summary

This PR adds three attributes to the compile config, to have granularity over how varlen (handles mixed prefil and decode batches) and decode (only decode batches) are compiled. We want to have this kind of granularity because varlen has 2 dynamic axis (number of query tokens and KV tokens) so the shape of varlen batches varies a lot, ad it greatly benefits from dynamic=True for compile. On the contrary, decode batches have only 1 dynamic axis (number of query tokens) so dynamic=False is tolerable and brings some speedup (roughly 5 to 10%).
This is a lot to handle for a non power user, so all of this is handled if use_default_compile_configs=True in the config

Performance

Arguments	Duration (s)	Generated tokens	Throughput (tok/s)
--samples 32 --max-new-tokens 2048	24.65	65536	2658.25
--samples 32 --max-new-tokens 2048 --compile	19.45	65536	3368.95
--samples 32 --max-new-tokens 4096	55.19	131072	2374.86
--samples 32 --max-new-tokens 4096 --compile	43.66	131072	3002.35
--samples 32 --max-new-tokens 8192	131.01	262144	2000.97
--samples 32 --max-new-tokens 8192 --compile	109.9	262144	2385.21
--samples 100 --max-new-tokens 4096	90.02	409600	4549.98
--samples 100 --max-new-tokens 4096 --compile	77.44	409600	5289.48

The script ran is the continuous batching example with the arguments listed above and --use-async --block-table 64 --log-level WARNING --seed 0 --force-max-length.

Tests

Added a test for the feature, which passes. All tests pass.
Generations look good.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

NIce! 🤗