Add transparent conversion for "tools" to "functions" in v1/chat/completions endpoint

[stippi2]

Is your feature request related to a problem? Please describe.

Yes. I got LocalAI running, both with Docker and building on my Mac. I tested with the Mistral model, but noticed that it didn't use functions. The reason is that 'functions' is actually already deprecated and there is now a 'tools' array instead. And my client is already using that.

Describe the solution you'd like
The tools array is a layer above functions. Each tool has a 'type', and the only supported type is 'function'. So if the type is function, there is a second field named function, which is then the Function object that used to be directly contained in the old functions array. So I imaging a simple conversion can be done, and I've actually tried to implement that:

diff --git a/api/schema/openai.go b/api/schema/openai.go
index 6355ff6..abf5dff 100644
--- a/api/schema/openai.go
+++ b/api/schema/openai.go
@@ -2,6 +2,7 @@ package schema
 
 import (
        "context"
+       "encoding/json"
 
        config "github.com/go-skynet/LocalAI/api/config"
 
@@ -114,8 +115,8 @@ type OpenAIRequest struct {
        Messages []Message `json:"messages" yaml:"messages"`
 
        // A list of available functions to call
-       Functions    []grammar.Function `json:"functions" yaml:"functions"`
-       FunctionCall interface{}        `json:"function_call" yaml:"function_call"` // might be a string or an object
+       Functions    []grammar.Function `json:"functions,omitempty" yaml:"functions"`
+       FunctionCall interface{}        `json:"function_call,omitempty" yaml:"function_call"` // might be a string or an object
 
        Stream bool `json:"stream"`
 
@@ -133,3 +134,34 @@ type OpenAIRequest struct {
        // AutoGPTQ
        ModelBaseName string `json:"model_base_name" yaml:"model_base_name"`
 }
+
+// UnmarshalJSON implements the json.Unmarshaler interface and maps "tools" and "tool_call" to "functions" and "function_call".
+func (r *OpenAIRequest) UnmarshalJSON(data []byte) error {
+       type Alias OpenAIRequest
+
+       // Define an auxiliary struct with additional fields to avoid infinite recursion when using the standard JSON unmarshalling.
+       var aux struct {
+               Alias
+               Tools    []grammar.Tool `json:"tools,omitempty" yaml:"tools"`
+               ToolCall interface{}    `json:"tool_call,omitempty" yaml:"tool_call"`
+       }
+
+       if err := json.Unmarshal(data, &aux); err != nil {
+               return err
+       }
+
+       // Copy the deserialized data back into the original request.
+       *r = OpenAIRequest(aux.Alias)
+
+       // Map "tools" and "tool_call".
+       if len(aux.Tools) > 0 {
+               for _, tool := range aux.Tools {
+                       r.Functions = append(r.Functions, tool.Function)
+               }
+       }
+       if aux.ToolCall != nil {
+               r.FunctionCall = aux.ToolCall
+       }
+
+       return nil
+}
diff --git a/pkg/grammar/tools.go b/pkg/grammar/tools.go
index ef56662..5b3e497 100644
--- a/pkg/grammar/tools.go
+++ b/pkg/grammar/tools.go
+ package grammar
+ 
+type Tool struct {
+       Type     string   `json:"type"`
+       Function Function `json:"function,omitempty"`
 }
+type Tools []Tool

... but it crashes. Unfortunately, I only get a 500 and a nil panic in the log, but not actually where it happens. With some guidance, maybe I can provide a PR.

[mudler]

Hey @stippi2 thanks for the heads up and the initial work!

To get more debugging informations you could comment out this line:

LocalAI/api/api.go

Line 149 in 5e155fb

app.Use(recover.New())

(even if we should gate that with the --debug flag, currently it's not the case).

It will avoid the server to recovery so you should be able to see the full stack trace of the error

[stippi2]

Thanks for the hint!

[stippi2]

The panic is because at some point the ctx context.Context is nil. If I do not provide the custom UnmarshalJSON() method, that problem goes away.
I then tried to just convert my local version to support "tools" unconditionally, and later stumbled across the fact, that function calls are not supported in streaming mode. Turning off streaming mode would require a bunch more changes in my client, so I put this on hold for now.

[mudler]

@stippi2 going to help here, see #1715 - still missing streaming support

[mudler]

closing as handled in #1715