Skip to content

show exactly what framework (vllm,trtllm,sglang) legend #127

@functionstackx

Description

@functionstackx

See parent issue for context.

Image Image

As shown in the diagram above, there are some caveats to this one that might make this a bit trickier than originally thought (there may be some refactoring involved).

My initial thinking is this: for each selection combination of Model + ISL/OSL + Precision, you can "calculate" the set of all possible calculations of GPU (hw field which is truncated as described here) + Framework (framework field after mapping applied, as described here + additional software stuff (such as mtp field)1. Once you have the set of all these combinations for a particular selection, you use these as selectable configurations on the legend and on the "Select a GPU for comparison" dropdown. This means that the "Select a GPU for comparison" dropdown can no longer be static and will depend on the selected Model + ISL/OSL + Precision.

Footnotes

  1. Another annoying detail, with this PR feat: multinode first class integration #251, the mtp: on/off field has been changed to spec-decoding: [str]. Therefore, we will have to support both the mtp field and the newer spec-decoding field.

Metadata

Metadata

Projects

Status

Done

Relationships

None yet

Development

No branches or pull requests

Issue actions