feat: Add n support for TRT-LLM (Stacked PR on top of #8744)

[indrajit96]

Overview:

Add TRT-LLM backend support for the OpenAI-compatible n field on top of PR #8744.
This PR is intentionally scoped to TRT-LLM only. The shared OpenAI/Rust response contract and vLLM plumbing are handled in the parent PR.

WHY:

PR #8744 adds the shared Dynamo contract for carrying multiple OpenAI choices by preserving each backend output index.

TensorRT-LLM supports n in SamplingParams as the number of sequences to generate, and trtllm-serve exposes an OpenAI-compatible /v1/chat/completions endpoint. Dynamo needed the TRT-LLM-specific plumbing to pass n through and keep streamed choices separated.

References:

• TRT-LLM OpenAI-compatible server: https://nvidia.github.io/TensorRT-LLM/commands/trtllm-serve.html
• TRT-LLM SamplingParams.n: https://nvidia.github.io/TensorRT-LLM/llm-api/reference.html

Details:

• components/src/dynamo/trtllm/llm_engine.py

  • Preserves TRT-LLM output choice indexes.
  • Tracks cumulative token offsets per choice so each Dynamo chunk emits only the new token delta for that choice.
  • Reports completion usage using all returned choices.

• components/src/dynamo/trtllm/request_handlers/handler_base.py

  • Passes n through to TRT-LLM sampling params.
  • Preserves output index in streamed chunks.
  • Tracks token and logprob offsets per choice index for interleaved n > 1 output streams.
  • Keeps TRT-LLM’s internal best_of field aligned with n when needed, because TRT-LLM validates best_of >= n.

• components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py

  • Adds unit coverage that n is applied to sampling params.
  • Verifies the internal best_of compatibility adjustment for TRT-LLM validation.

• docs/backends/trtllm/trtllm-reference-guide.md

  • Documents the TRT-LLM n > 1 behavior relevant to this backend path.

• tests/serve/test_trtllm.py

  • Adds pre-merge serve coverage validating that a chat request with n=2 returns two choices.

Where should the reviewer start?

Start with:

• components/src/dynamo/trtllm/request_handlers/handler_base.py

That file contains the main TRT-LLM streaming path and the per-choice cursor logic for n > 1.

Then review:

• components/src/dynamo/trtllm/llm_engine.py
• components/src/dynamo/trtllm/tests/test_trtllm_handler_base.py
• tests/serve/test_trtllm.py
• docs/backends/trtllm/trtllm-reference-guide.md

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx

Summary by CodeRabbit

Release Notes

• New Features

  • Added full support for requesting multiple response choices using the n parameter with TensorRT-LLM backend
  • Implemented independent token tracking per output choice for accurate streaming and token accounting

• Bug Fixes

  • Fixed token delta computation to correctly process all output choices instead of only the first
  • Corrected completion token counting to aggregate properly across all generated choices

• Documentation

  • Added configuration guide for enabling multiple response choices with TensorRT-LLM, including required environment variable setup

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-e1741288-895f-498f-8998-4ab1ab83c957.docs.buildwithfern.com/dynamo/dev