[docs] continuous batching

[stevhliu]

updates the continuous batching docs

• new page for the API reference
• adds sections for new features like CUDA graphs, async batching, prefix caching, logprobs (depending on when its merged)
• clearer example of generation with varying loads using continuous_batching_context_manager
• new page explaining how the system works underneath. it explains the scheduling half of the system, but doesn't cover the memory side yet. i'm thinking it may make more sense to move the paged attention doc in here to fill the gap. what do you think @remi-or ?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[remi-or]

LGTM! I think the memory sides needs expanding I agree, but we can leave that for another PR. This is a very good starting point to keep the doc more up to date. Thanks!

[remi-or]

i'm thinking it may make more sense to move the paged attention doc in here to fill the gap. what do you think @remi-or ?

Not really, this is more about the kernel than the actual page-management system. I can draft something up for that.