Skip to content

User prompt handling for multi-dataset testing#1010

Merged
chakravarthik27 merged 2 commits intorelease/2.2.0from
Enchaments/implement-the-prompt-handling-for-different-datasets
Apr 15, 2024
Merged

User prompt handling for multi-dataset testing#1010
chakravarthik27 merged 2 commits intorelease/2.2.0from
Enchaments/implement-the-prompt-handling-for-different-datasets

Conversation

@chakravarthik27
Copy link
Copy Markdown
Collaborator

The following way to configure the multi-prompt for multi-dataset testing

harness = Harness(
    task="question-answering",
    model={"model": "http://localhost:1234/v1/chat/completions", "hub": "lm-studio"},
    data=[
        {"data_source": "BoolQ", "split": "test-tiny"},
        {"data_source": "NQ-open", "split": "test-tiny"},
        {"data_source": "MedQA", "split": "test-tiny"},
        {"data_source": "LogiQA", "split": "test-tiny"},
    ],
    config={
        "model_parameters": {
            "max_tokens": 64,
            "user_prompt": {
                "BoolQ": "Answer the following question with a yes or no: {question}",
                "NQ-open": "Answer the following question with a short answer: {question}",
                "MedQA": "Answer the following medical question: {question} {options}",
                "LogiQA": "Answer the following logic question: {question} {options}"
            }
        },
        "tests": {
            "defaults": {
                "min_pass_rate": 1.0
            },
            "robustness": {
                "add_typo": {
                    "min_pass_rate": 0.7
                },
                "lowercase": {
                    "min_pass_rate": 0.7
                }
            }
        }
    }
)

@chakravarthik27 chakravarthik27 added 💡Enhancements Something can be improved v2.2.0 Issue or request to be done in current release labels Apr 12, 2024
@chakravarthik27 chakravarthik27 self-assigned this Apr 12, 2024
@chakravarthik27 chakravarthik27 linked an issue Apr 12, 2024 that may be closed by this pull request
@chakravarthik27 chakravarthik27 merged commit 75453d0 into release/2.2.0 Apr 15, 2024
@chakravarthik27 chakravarthik27 deleted the Enchaments/implement-the-prompt-handling-for-different-datasets branch August 30, 2024 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

💡Enhancements Something can be improved v2.2.0 Issue or request to be done in current release

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement the prompt handling for different datasets

2 participants