Question: llama.cpp server support

[RLDiary]

Prerequisites

• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new useful question to share that cannot be answered within Discussions.

Background Description

Hi

I am trying to load this using Llama.CPP HTTP server. Is this supported please? And if yes, please could you give me an example of how to do this?

thanks

Possible Answer

No response

[tc-mb]

Prerequisites

• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new useful question to share that cannot be answered within Discussions.

Background Description

Hi

I am trying to load this using Llama.CPP HTTP server. Is this supported please? And if yes, please could you give me an example of how to do this?

thanks

Possible Answer

No response

It should not be supported yet. I have not modified this call.

[SunYabin0329]

Is there any plan to support it?

[fatihmtlm]

I am also trying to run this model using llama-server.exe but couldn't managed to. So do I have to use transformers ? I also couldn't manage to build ollama for windows (It kinda seemed hard so haven't tried much)

[tc-mb]

@ALL
Hi, I don't always pay attention to the issue area in this fork code repo.

1. If this issue still needs to be answered, please raise an issue in the main repo with "llamacpp" label. I will respond very quickly.
2. If this issue no longer needs to be answered, I will close it this week.