[CI/CD] Introduce testing for the clients and agents

[srmanda-cs]

We need a way to establish tests for all of the clients and agents within AIOpsLab. We also need a way to include these tests with the CI pipeline running that was introduced in #19

How do we test the clients and agents without introducing API costs per run is the BIG question. We primarily need these tests to make sure that, when we are refactoring the code as we planned to in #146 and to support #49 in the future, we don't introduce regressions into the benchmark and break functionality for everyone.

[gaganso]

Hi @srmanda-cs, we should be testing the APIs/tools that are exposed to the agents since the users are expected to build their agents over these. If those don't regress/change (their responses, formats, the docstrings of these tools, etc), I don't think the agents would regress. So, we could test these APIs/tools using an agent with mocked LLM calls (the chat completion APIs).

For the agents in the clients/ directory, we could run an integration test after the refactoring is done. We could also benchmark on more recent models. This would also provide new comparison data since it has been a while since we have benchmarked them. This could be a different follow up PR.

What do you think? @srmanda-cs @HacksonClark

[HacksonClark]

I think mocking the tools will be fine, or having a mock agent of sorts that replays some actual agent actions from a prior run with a real agent).

[srmanda-cs]

When we do the clients/ directory refactoring, the APIs and tools exposed to the agent will probably not be touched. I was referring specifically to the clients there because once we start consolidating the code, for example say someone who has been using the DeepSeek agent might find it unusable because one function didn't inherit properly. We would want some automated tests to make sure that post refactoring these clients function exactly as before.

We need to test the token trimming (I think for OpenRouter) + init_context + get_action for all agents. We would have to use MagicMock or a similar solution to mock the responses of each client. Once, we are done with these unit tests, we can consolidate the code. We could probably reduce it by 50%

Then, we start testing the session APIs exposed to these clients, and then try to get the context passed which we weren't able to do before. If we do it the other way around, we would have to make modifications in every single client file.

That was the plan in my head at least

[HacksonClark]

I think this is already a bit too complicated, we should stick to testing just core benchmark functionality instead of trying to test each individual agent, I feel there is too much work to be done there.

[srmanda-cs]

It's not that complicated or that much work, maybe a few weeks from my side at most. But, if you really want to leave the clients directory alone and want help in other parts of the project, I'm totally up for it!

[HacksonClark]

If you're willing to spend the time I have no issues with you doing it! I appreciate your contributions.