Feautre/integration with web api

[chakravarthik27]

Enhancing LangTest for Generic API Support

Description:

In response to user needs for flexibility in utilizing various language models hosted on different platforms, we're excited to introduce a major enhancement to LangTest: Generic API Support. This new feature empowers users to seamlessly integrate any API system into LangTest, enabling them to leverage their preferred language models with ease.

Key Features:

1. Generic API Integration: LangTest now supports connecting to any API system, allowing users to specify the API URL, parameters, and result parsing function.

2. Customizable Parameters: Users can define the URL and parameters of their API, along with a function to parse the results according to their specific requirements.

3. Enhanced Flexibility: This update provides users unparalleled flexibility in utilizing their preferred language models, whether hosted on OpenAI, Hugging Face, or other custom platforms like vLLM.

Parameters:

Define the input_processer function for creating a payload and the output_parser function is used to extract the output from the response.

url = "https://your-custom-api-endpoint.com"

# headers
headers = {
    "Content-Type": "application/json",
}

# lambda function to create payload
input_processor= lambda content: {
     "contents": [{"role": "user", "parts": [{"text": content}]}]}

# lambda function to parse the result from the response
output_praser = lambda response: response['candidates'][0]['content']['parts'][0]['text']

Setup Code:

To take advantage of this feature, users can utilize the following setup code:

from langtest import Harness

# Initialize Harness with API parameters
harness = Harness(
    task="question-answering",
    model={
        "model": {
            "url": url,
            "headers": headers,
            "input_processor": input_processor,
            "output_parser": output_parser,
        },
        "hub": "web",
    },
    data={
        "data_source": "OpenBookQA",
        "split": "test-tiny",
    }
)
harness.generate().run().report()

Usage:

Following this setup, users can seamlessly integrate their custom API systems into LangTest, enabling them to perform language testing and evaluation using their preferred language models.

Feedback and Contribution:

We encourage the community to explore this new feature, provide feedback, and contribute to further improvements. Your input is invaluable in shaping the future of LangTest and ensuring it meets the diverse needs of language testing professionals and enthusiasts.

Conclusion:

With Generic API Support, LangTest continues to evolve as a versatile tool for language testing and evaluation, offering users unprecedented flexibility in harnessing the power of language models hosted across different platforms. Start integrating your custom APIs today and unlock a new level of language testing capabilities with LangTest.