Improve Robustness of CSV Dataset Loader

[aviralgarg05]

Description

load_local_csv in src/agentunit/datasets/base.py splits tool lists by a hardcoded ; and context by ||.
This is brittle and does not support standard CSV quoting/escaping for these delimiters.

Task

• Use csv module's advanced features or a more robust parsing logic.
• Allow configuring delimiters via metadata or arguments.
• Be robust to malformed fields.

Source

https://github.com/aviralgarg05/agentunit/blob/main/src/agentunit/datasets/base.py#L96