[BUG] LLM is calling tools with undeclared parameters

[aniamisiorek]

Basic checks

• I searched existing issues - this hasn't been reported
• I can reproduce this consistently
• This is a RubyLLM bug, not my application code

What's broken?

Tool.call() does not perform any sort of parameter validation:
result = execute(**args.transform_keys(&:to_sym))
Our team has run into an issue where the LLM will call tools with unsupported parameters, resulting in unknown keyword error.

How to reproduce

1. Define a tool with some set of parameters
2. Add the tool to a chat, and prompt the LLM to use the tool with a parameter you did not define.
3. Sometimes it will mention that the parameter is not accepted, but other times it will attempt to call the tool with fake parameters.

Expected behavior

Tool.call handles hallucinated parameters before a keyword error is triggered.

What actually happened

from tool_call table:

--clients--update_client

"{""last_name"": ""Brown"", ""first_name"": ""Goose"", ""new_advisor_code"": ""AWM-2000-106""}"

new_advisor_code is not a supported parameter for UpdateClient, so application throws an error, Failed to send message: unknown keyword: :new_advisor_code"

Environment

• Ruby: 3.4.2
• RubyLLM: 1.9.1
• Provider: AWS Bedrock -- Anthropic Sonnet

[smerickson]

I've experienced something similar and what's interesting is that I'm also using AWS bedrock. Although tool calls through Anthropic Sonnet have been more reliable for me. I've experienced this same issue but more frequently with Haiku 4.5

[smerickson]

Here's a few additional data points from my experience...

1. I'm using streaming
2. I'm using an agentic loop with multiple tool calls
3. It seems more likely to happen when using a tool call and one of the parameters to the tool will be a larger payload.
4. I wasn't having this experience when I was using the AWS Bedrock client library directly. It has been happening since trying to switch to RubyLLM. As a result, it feels different than the random hallucinations LLMs can sometimes make. In general my experience with Anthropic models is that they tend to be extremely reliable at tool calls. But the frequency of these issues with RubyLLM and Bedrock makes it seem like maybe more of a library issue. But i haven't been able to trace down any further technical details yet. Still try to debug what might be happening at a lower level.

UPDATE

I found that I was running into a max tokens error. Which would result in incomplete tool call signature and cause the error. it was happening more frequently with Hakiu because Hakiu has a smaller default max tokens (4K)

UPDATE 2

One more update on my experience with this issue. I'm still getting this error even after increasing my max tokens. After a little research using the stream debug config, I've found that it seems more likely an issue with the Stream Accumulator or the way that the Bedrock Provider is parsing the stream and making it available to the accumulator. I have a gross monkey patch that seems to be working. But I'm still trying to come up with a more consice explanation of what I think is happening.

[aniamisiorek]

@smerickson interesting, thanks for the write-up and updates! We haven't done much research on our end (we chalked it up to a hallucination), but we aren't using streaming. Do you mind sharing the monkey-patch? Would be curious to see the fix you've found.