Skip to content

Create Eagle3DraftModels which use the full verifier vocab #200

@fynnsu

Description

@fynnsu

Currently Eagle3DraftModel has t2d and dt2 (vocab mapping) tensors as required inputs to it's init function. This means if a model uses the full verifier vocab, users must create "no-op" mapping tensors and pass them to the drafter. Furthermore, these mapping tensors will still be saved in the model checkpoints during training and then subsequently used by vllm when running the model.

It would be better if we made these optional, and used the full verifier vocab when they aren't provided.

Todo:

  • Make t2d and dt2 tensors optional inputs to Eagle3DraftModel
  • Verify that if one is provided, the other is as well
  • Make sure all existing uses of the tensors in Eagle3DraftModel can handle non-existent/None values
  • Update scripts/train.py so that t2d and d2t are not required inputs
  • Update scripts/gen_and_train.py if needed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions