model: add hunyuan dense

[stevenkuang-tencent]

Update:

• Support hunyuan_dense
• fix hunyuan_moe chat template

[CISC]

@stevenkuang-tencent gentle ping

[stevenkuang-tencent]

Politely asking, can this pull request be merged now? @CISC

[CISC]

@stevenkuang-tencent Yes, but the chat template gives me pause, please follow up once model is released if there are any problems.

[jacekpoplawski]

Is this for upcoming models or old ones?
Because https://huggingface.co/tencent/Hunyuan-4B-Instruct is not accessible and in the vllm I see https://huggingface.co/tencent/Hunyuan-7B-Instruct-0124 is mentioned

[stevenkuang-tencent]

Is this for upcoming models or old ones? Because https://huggingface.co/tencent/Hunyuan-4B-Instruct is not accessible and in the vllm I see https://huggingface.co/tencent/Hunyuan-7B-Instruct-0124 is mentioned

It is for upcoming models. Those models will come soon.

[jacekpoplawski]

It is for upcoming models. Those models will come soon.

that's fantastic news, thanks!

[pwilkin]

Just wanted to chime in, tested IQ4NL quants and the output is completely incoherent.

[arch-btw]

Same issue here, tried it with the different flags but it still doesn't work:

-cnv --jinja

-cnv --chat-template hunyuan-dense

-cnv --chat-template hunyuan-moe

Example output:

hello

，不对，这样处理的话，比如对于输入序列“1,

@stevenkuang-tencent

[pwilkin]

My 3 attempts were:

• Lots of Chinese text after "Hello"
• Started "<think" then completely froze (but generation wasn't finished)
• Kept repeating "answer:" in new session
  Something is completely broken.

[stevenkuang-tencent]

The chat-template has been updated before the model is open sourced, and we are updating it synchronously

[arch-btw]

@stevenkuang-tencent thank you

@pwilkin I put this together and this seems to work for now, although it's not an official solution:

Save as hunyuan4b.jinja
Then run with --jinja --chat-template-file hunyuan4b.jinja
The model defaults to /no_think but putting /think before the prompt works.

hunyuan4b.jinja:

{%- if 'add_generation_prompt' not in context %}
    {%- set add_generation_prompt = false %}
{%- endif %}

{%- set ns = namespace(is_first=false) %}

{%- for message in messages if message['role'] == 'system' %}
    {%- if ns.is_first %}
        {%- set ns.is_first = false %}
        {{- bos_token -}}
        {{- message['content'] }}
    {%- else %}
        {{- '\n\n' + message['content'] }}
    {%- endif %}
{%- endfor %}

{%- for message in messages %}
    {% if message['role'] == 'user' %}
        <｜hy_User｜>{{ message['content'] }}<｜hy_Assistant｜>
    {% endif %}
    
    {% if message['role'] == 'assistant' %}
        {{ message['content'] }}{{ eos_token }}
    {% endif %}
{%- endfor %}

{%- if add_generation_prompt and not ns.is_last_user %}
    <｜hy_Assistant｜>
{%- endif %}

{%- if enable_thinking is defined and not enable_thinking %}
    ...
{%- endif %}

[pwilkin]

@stevenkuang-tencent thank you

@pwilkin I put this together and this seems to work for now, although it's not an official solution:

Save as hunyuan4b.jinja Then run with --jinja --chat-template-file hunyuan4b.jinja The model defaults to /no_think but putting /think before the prompt works.

hunyuan4b.jinja:

{%- if 'add_generation_prompt' not in context %}
    {%- set add_generation_prompt = false %}
{%- endif %}

{%- set ns = namespace(is_first=false) %}

{%- for message in messages if message['role'] == 'system' %}
    {%- if ns.is_first %}
        {%- set ns.is_first = false %}
        {{- bos_token -}}
        {{- message['content'] }}
    {%- else %}
        {{- '\n\n' + message['content'] }}
    {%- endif %}
{%- endfor %}

{%- for message in messages %}
    {% if message['role'] == 'user' %}
        <｜hy_User｜>{{ message['content'] }}<｜hy_Assistant｜>
    {% endif %}
    
    {% if message['role'] == 'assistant' %}
        {{ message['content'] }}{{ eos_token }}
    {% endif %}
{%- endfor %}

{%- if add_generation_prompt and not ns.is_last_user %}
    <｜hy_Assistant｜>
{%- endif %}

{%- if enable_thinking is defined and not enable_thinking %}
    ...
{%- endif %}

What's in the "..." part? The current contents?

[arch-btw]

I think so, when I remove it (with thinking enabled) it starts talking in Chinese again.

[pwilkin]

Nope, on Hunyuan 7B still garbage. Tried the fixed prompt from their tokenizer config, but still doesn't work.

[pwilkin]

I guess it might have something to do with this:
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect

[arch-btw]

I think 7b uses a different tokenizer.

[pwilkin]

Yes, but it's been incorrectly uploaded from what I've seen.