[API Proposal]: A Code Generator For Function Calling in M.E.A.I.

[gunpal5]

Background and motivation

Hello,

Based on my analysis of the M.E.A.I. source code, the current function calling mechanism utilizes Reflection to generate method schemas and invoke functions. While AOT works with Reflection, it presents potential future compatibility risks as the .NET ecosystem evolves. Therefore, I propose a code generator for AI Functions. This would create JSON schemas and invocation code at compile time, leading to more explicit and robust code. Developers would simply annotate their functions, classes, or interfaces with a designated attribute.

The underlying AI specification and mechanism will remain the same. The code generator will generate a separate class inherited from AIFunction.

Mr. Konstantin S. and I have successfully employed a similar code generation approach in our repositories for LangChain.net and Google_GenerativeAI SDK. Therefore, we possess the necessary experience and background to develop this code generator.

I would like to know about your views and opinions about this.

API Proposal

//Single Function
[GenAIFunction(Description = "Single function Description")]
public async Task<MyResponse> SingleAIFunction( [Description("My param1 description")] int param1, Description[("My param2 description")] MyParamClass param2)
{
}

//Interface with multiple functions
[GenAITool]
public interface IMyToolInterface
{
        [Description("My function 1 Description")]
        public async Task<MyResponse> MyToolFunction( [Description("My param1 description")] int param1, Description[("My param2 description")] MyParamClass param2)
        {
        }

        [Description("My function 2 Description")]
        public async Task<MyResponse> MyToolFunction2( [Description("My param1 description")] int param1, Description[("My param2 description")] MyParamClass param2)
        {
        }
}

API Usage

// Single Function AsTool() Generated by Code Generator
var tool = SingleAIFunction.AsTool()

//multiple functions, AsTools() generated by code generator for interface implementations
var tools = MyToolImpl.AsTools();

Alternative Designs

No response

Risks

No response

[gunpal5]

Hello M.E.A.I. Team,

I know you are really busy with this big project. I really appreciate your hard work.

I had some free time so I just improved upon our code generator for LangChain SDK, and implemented a NativeAOT compatible function tools code generator which doesn't explicitly require to define the JsonSerializerContext and the syntaxes are much simpler than Official SDK.

For M.E.A.I. our syntax looks like this:

For interfaces

[GenerateJsonSchema(MeaiFunction = true)]
public interface IWeatherFunctions
{
//methods
}

//chat options
options.Tools = service.AsMeaiTools();

For methods

[FunctionTool(MeaiFunction=true)]
[System.ComponentModel.Description("Get book page content")]
public static Task<string> GetBookPageContentAsync(string bookName, int bookPageNumber, CancellationToken cancellationToken = default)
{
    return Task.FromResult("there is a nice weather outside, and I am stuck at home.");
}

//Tools class auto generated by Code Generator
var tools = new Tools([GetBookPageContentAsync]);
chatOptions.Tools = tools.AsMeaiTools();

I just integrated this new code generation into my Google_GenerativeAI SDK and did a proper testing its working fine. for detailed implementation and syntax overview: please check out this Wiki Page

P.S. Although the new code generator works fine for it's purpose. there are still room for improvements such as we would still need to do code refactoring for code quality and we are using System.Text.Json code generator codes for JsonSerializerContext until we find some elegant solution to it. (because code generator doesn't allow chaining and It would have taken time to implement our own code generator for JsonSerializerContext)

Regards,
Gunpal Jain

[stephentoub]

Thanks, and sorry for the delay in responding.

Adding a source generator in some form has been on our radar, but it hasn't been a priority for an initial release, as it hasn't been necessary to enable the initial set of desired experiences, and Native AOT is supported even without it.

But I can imagine we might want to add a source generator in the future, and we can leave this open tracking that.

What do you see as the main benefits of a source generator here?

[gunpal5]

@stephentoub

There are many benefits especially with the interface type of tools as mentioned above.

• You would only need to decorate your interface once and code generator will generate the necessary codes. Then you can build several reusable plugin/tools e.g. JavascriptExecutorTool, PythonExecutorTool or better combine all of the languages into one tool and call it CodeExecutor. or a dedicated MathTool to provide all the mathematical functions, or create a tool for finance related mathmatical functions, and so on.
• Using individual Methods as Function Tool is not practical for big projects. you would need to some how keep track of individual methods. by using interface type of tool you will have more organization.

I can list out many more benefits. like no need of defining serialization context for each method arguments.

I tested the M.E.A.I AIFunctionFactory it just fails if you don't pass the serialization context with class type arguments with AOT.

also, if you have multiple methods you would need to keep writing same codes over again and again. for example let's say you have 3 methods: MyMethod1, MyMethod2, MyMethod3. you would need to define the AIFunction like this:

var tool1 = AIFunctionFactory.Create(MyMethod1);
var tool2 = AIFunctionFactory.Create(MyMethod2);
var tool3 = AIFunctionFactory.Create(MyMethod3);

options.Tools =new List<string>{tool1, tool2, tool3};

with code generation you can simply do this like this:

var tools = new Tools([MyMethod1,MyMethod2,MyMethod3]);
chatOptions.Tools = tools.AsMeaiTools();

personally, I don't like the idea of using the individual method as function tool.

[stephentoub]

I tested the M.E.A.I AIFunctionFactory it just fails if you don't pass the serialization context with class type arguments with AOT.

I'm not sure how you avoid that with a source generator, given that source generators don't layer. The source generator for M.E.AI wouldn't currently be able to emit code that the JSON source generator picked up. So even with a source generator, you'd still need to ensure that you'd registered for JSON serialization all the types used as parameters (just like you do, for the same reason, with ASP.NET minimal APIs), and you'd need to make sure that the generator was then using the relevant context.

also, if you have multiple methods you would need to keep writing same codes over again and again.
var tools = new Tools([MyMethod1,MyMethod2,MyMethod3]);

That doesn't require a code generator, e.g.

class Tools(params Delegate[] delegates)
{
    private readonly List<AITool> _tools = delegates.Select(d => (AITool)AIFunctionFactory.Create(d)).ToList();
    public List<AITool> AsMeaiTools() => _tools;
}

You would only need to decorate your interface once and code generator will generate the necessary codes.

I'm not following. Can you share an example? If you're saying you would attribute all of the methods in a type, then again, that doesn't require a source generator. It's just a few lines of reflection-based code. What experience would a "code generator for function calling in M.E.AI" actually provide that requires a source generator to do it? Typically source generators at this level end up being not about new experiences but about performance, as otherwise reflection can be used, e.g.

static List<AIFunction> GetAIFunctions<T>(T t) =>
    (from m in t.GetType().GetMethods()
     where m.GetCustomAttribute<GenAIFunctionAttribute>() is not null
     select AIFunctionFactory.Create(m, t)).ToList();

I'm not arguing against source generators. And like I said, we might want to add one in the future. But it would need to be for the right reasons and the right experience.

[gunpal5]

I really admire how you managed to accomplish AOT without using a code generator. As far as I understand reflection is kind of forbidden in case of AOT and I really don't know which part of reflection is safe to use in case of AOT. I'll try to study about this next.

You are right about code generator layering issue. I tried to figure out a solution for that, but couldn't find any elegant solution to it.

with decorating interface, I meant we just need to annotate an interface once then we can define as many methods into it e.g.

[GenerateJsonSchema]
public interface IMathFunction
{
   double Add(double number1, double number2);
   double Substract(double number1, double number2);
  double Multiply(double number1, double number2);
....
....
}

Then you can simply implement this interface, with this you'll have reusable tool which you can use in any project, or anywhere in the solution.

I think, there are few benefits of code generator, Performance, NativeAOT, and possibly auto generated JsonSerializerContext.

if we look on the other side at the scale of this project. code generator would not provide any substantial contribution to this project. It can be implemented later.

Utility/helper classes can be coded with existing implementation to make it more developer friendly.

Thanks for your time.

[stephentoub]

I tried to figure out a solution for that, but couldn't find any elegant solution to it.

Unfortunately, there isn't a great one right now.

As far as I understand reflection is kind of forbidden in case of AOT and I really don't know which part of reflection is safe to use in case of AOT.

You can configure the tools to help you know both know when you still have issues and also get tips on how to fix them. I'd also recommend this blog post and video:
https://devblogs.microsoft.com/dotnet/creating-aot-compatible-libraries/
https://www.youtube.com/watch?v=N-MrQeZ1enY&list=PLdo4fOcmZ0oX8eqDkSw4hH9cSehrGgdr1

Reflection emit is effectively forbidden, because there's no JIT to compile the emitted IL into asm. But reflection itself is fine, you just need to ensure things are appropriately annotated so that the necessary information is kept around. For example, in my earlier example with:

static List<AIFunction> GetAIFunctions<T>(T t)

you'd actually want to annotate the T with an attribute that tells the linker it shouldn't trim away any public methods on any type used for T:

static List<AIFunction> GetAIFunctions<[DynamicallyAccessedMembers(DynamicallyAccessedMemberTypes.PublicMethods)] T>(T t)

Thanks for your time.

Thanks for your interest and suggestions.

Obviously, as well, such a source generator doesn't have to be part of the core M.E.AI nuget packages, either. You're welcome to ship your own source generator. And enough interest in that could help motivate us to want to ship something "built in".