.Net: Bug: Semantic Kernel inflates token usage of input prompt by 20% due to Unicode escaping and URL encoding of the prompt

[Evyatar108]

Describe the bug
the semantic kernel package in c# is inflating the usage tokens by around 20% for my prompts because it escapes Unicode characters and URL encodes the string of the content field of the messages array.

I tested it on a prompt that has 49k input tokens based on the metadata information from the Azure OpenAI api, and when using semantic kernel it inflates the input tokens to 63k

It's very easy to see the prompt differences when it has JSON in it

Platform

• OS: Windows
• IDE: Visual Studio
• Language: C#
• Source: NuGet package version 1.15

[blinchi]

And it's not just the consumption of tokens, it's that it costs more for the LLM to process it. When receiving encrypted data, the responses are sent to you encoded, when it should not be that way. In my case using Gemini.

[atiq-bs23]

@markwallace-microsoft can you please check #7052

[markwallace-microsoft]

Fixed via #7098

[blinchi]

This is not fixed in dotnet version 1.16.2.
When the prompt is generated it still sends it with escaped characters.

This occurs when a prompt is passed:

[sonphnt]

Do we have a right way to see a prompt is actually sent to the azure open ai instead of checking chat history message? I have the same issue here but with function calling, somehow uses it a lot of tokens even though the function returns a very simple data and end up with thousands of tokens used