llama : set attrs of mislabelled EOT/EOM tokens

[bakkot]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

Some models (such as the official phi-3-mini-4k-instruct-gguf) have an <|end|> or similar token which is detected as being EOT or EOM but which is not marked as control.

This causes issues where the token will be included in the output when it shouldn't. A complete reproduction is to install the phi-3-mini-4k-instruct-gguf linked above and then run

./llama-cli -m ./models/Phi-3-mini-4k-instruct-q4.gguf --prompt "What kind of thing is a llama? Response: " --no-display-prompt --temp 0 --grammar 'root ::= "animal" | "object"'

which will print animal<|end|>.

In these cases, manually set the token's attr to LLAMA_TOKEN_ATTR_CONTROL so that we know not to print it. That fixes the above repro to correctly print just animal.

[compilade]

Note that reconverting the Phi-3 models since #8228 should mark <|end|> as CONTROL.

But we have no control over the "official" models.