\""]},{"cell_type":"markdown","metadata":{"id":"tCXcKn_9BXEa"},"source":["### Multi Dataset Testing\n","\n","In order to evaluate the model's performance on multiple datasets, we can utilize a Jupyter notebook and provide a list of dictionaries to the `data` parameter. Each dictionary within the list should contain the following keys:\n","\n","```\n","data=[\n"," {\"data_source\": \"BoolQ\", \"split\": \"test-tiny\"},\n"," {\"data_source\": \"NQ-open\", \"split\": \"test-tiny\"},\n"," {\"data_source\": \"MedQA\", \"split\": \"test-tiny\"},\n"," {\"data_source\": \"LogiQA\", \"split\": \"test-tiny\"},\n","],\n","```\n","\n","Here, we specify different data sources and their corresponding splits for testing. This allows for a comprehensive evaluation of the model's performance across diverse datasets. The notebook can then be executed to assess how well the model generalizes to various types of questions and contexts presented in these datasets."]},{"cell_type":"code","execution_count":3,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":45,"status":"ok","timestamp":1692371630216,"user":{"displayName":"Prikshit sharma","userId":"07819241395213139913"},"user_tz":-330},"id":"ASv9E02sBXrp","outputId":"fb19b9ec-3bd9-416e-f2fc-dc3190b8a861"},"outputs":[{"name":"stdout","output_type":"stream","text":["Test Configuration : \n"," {\n"," \"model_parameters\": {\n"," \"max_tokens\": 64\n"," },\n"," \"tests\": {\n"," \"defaults\": {\n"," \"min_pass_rate\": 1.0\n"," },\n"," \"robustness\": {\n"," \"add_typo\": {\n"," \"min_pass_rate\": 0.7\n"," },\n"," \"lowercase\": {\n"," \"min_pass_rate\": 0.7\n"," }\n"," }\n"," }\n","}\n"]}],"source":["harness = Harness(\n"," task=\"question-answering\",\n"," model={\"model\": \"gpt-3.5-turbo-instruct\", \"hub\": \"openai\"},\n"," data=[\n"," {\"data_source\": \"BoolQ\", \"split\": \"dev-tiny\"},\n"," {\"data_source\": \"NQ-open\", \"split\": \"test-tiny\"}\n"," ],\n",")"]},{"cell_type":"markdown","metadata":{"id":"_wvVHxeSDWLV"},"source":["## Robustness\n","\n","For tests we used uppercase, Dyslexia Word Swap, Add Slangs, Insert Abbreviations and Speech to Text typos . Other available robustness tests for QA task are:\n","* `add_context`\n","* `add_contraction`\n","* `add_punctuation`\n","* `add_typo`\n","* `add_ocr_typo`\n","* `american_to_british`\n","* `british_to_american`\n","* `lowercase`\n","* `strip_punctuation`\n","* `titlecase`\n","* `uppercase`\n","* `number_to_word`\n","* `add_abbreviation`\n","* `add_speech_to_text_typo`\n","* `add_slangs`\n","* `dyslexia_word_swap`\n","* `multiple_perturbations`\n","* `adjective_synonym_swap`\n","* `adjective_antonym_swap`\n","* `strip_all_punctuation`"]},{"cell_type":"markdown","metadata":{"id":"HYExqs-pDbvz"},"source":["You can also set prompts and other model parameters in config. Possible parameters are:\n","* `user_prompt:` Prompt to be given to the model.\n","* `temperature:` Temperature of the model.\n","* `max_tokens:` Maximum number of output tokens allowed for model."]},{"cell_type":"markdown","metadata":{},"source":["To configure prompts for different datasets, you can use the `user_prompt` dictionary. Here's how it works:\n","\n","- Each key in the dictionary represents a dataset name or task (e.g., \"BoolQ\", \"NQ-open\").\n","- The corresponding value is a string template that defines the user prompt for the dataset.\n","- The template can include placeholders:\n"," - `{context}`: This will be replaced with the actual context (passage) relevant to the question from the specific dataset.\n"," - `{question}`: This will be replaced with the actual question from the dataset.\n","- The newline character `\\n` can be used to separate the context and question in the final prompt.\n","\n","Here the example:\n","```python\n","harness.configure(\n"," {\n"," \"model_parameters\": {\n"," \"user_prompt\": {\n"," \"BoolQ\": \"Answer the following question with a True or False. {context}\\nQuestion {question}\",\n"," \"NQ-open\": \"Answer the following question. Question {question}\",\n"," }\n"," },\n"," ....\n"," })\n","```"]},{"cell_type":"code","execution_count":4,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":42,"status":"ok","timestamp":1692371630218,"user":{"displayName":"Prikshit sharma","userId":"07819241395213139913"},"user_tz":-330},"id":"EzzlV0u4DbN9","outputId":"2a3926cd-9c23-45a6-a0b8-b31b29692be3"},"outputs":[{"data":{"text/plain":["{'model_parameters': {'user_prompt': {'BoolQ': 'Answer the following question with a True or False. {context}\\nQuestion {question}',\n"," 'NQ-open': 'Answer the following question. Question {question}'}},\n"," 'tests': {'defaults': {'min_pass_rate': 0.65},\n"," 'robustness': {'uppercase': {'min_pass_rate': 0.66},\n"," 'dyslexia_word_swap': {'min_pass_rate': 0.6},\n"," 'add_abbreviation': {'min_pass_rate': 0.6},\n"," 'add_slangs': {'min_pass_rate': 0.6},\n"," 'add_speech_to_text_typo': {'min_pass_rate': 0.6}}}}"]},"execution_count":4,"metadata":{},"output_type":"execute_result"}],"source":["harness.configure(\n"," {\n"," \"model_parameters\": {\n"," \"user_prompt\": {\n"," \"BoolQ\": \"Answer the following question with a True or False. {context}\\nQuestion {question}\",\n"," \"NQ-open\": \"Answer the following question. Question {question}\",\n"," }\n"," },\n"," \"tests\": {\n"," \"defaults\": {\"min_pass_rate\": 0.65},\n"," \"robustness\": {\n"," \"uppercase\": {\"min_pass_rate\": 0.66},\n"," \"dyslexia_word_swap\": {\"min_pass_rate\": 0.60},\n"," \"add_abbreviation\": {\"min_pass_rate\": 0.60},\n"," \"add_slangs\": {\"min_pass_rate\": 0.60},\n"," \"add_speech_to_text_typo\": {\"min_pass_rate\": 0.60},\n"," },\n"," }\n"," }\n",")"]},{"cell_type":"markdown","metadata":{"id":"P7TKPJd3Dft1"},"source":["➤ You can adjust the level of transformation in the sentence by using the \"`prob`\" parameter, which controls the proportion of words to be changed during robustness tests.\n","\n","➤ **NOTE** : \"`prob`\" defaults to 1.0, which means all words will be transformed.\n","```\n","harness.configure(\n","{\n"," 'tests': {\n"," 'defaults': {'min_pass_rate': 0.65},\n"," 'robustness': {\n"," 'uppercase': {'min_pass_rate': 0.66, 'prob': 0.50},\n"," 'dyslexia_word_swap':{'min_pass_rate': 0.60, 'prob': 0.70},\n"," }\n"," }\n","})\n","\n","```"]},{"cell_type":"markdown","metadata":{"id":"SW71UKHfDi2q"},"source":["Here we have configured the harness to perform Five robustness tests and defined the minimum pass rate for each test."]},{"cell_type":"code","execution_count":5,"metadata":{"id":"a9Q8i7-KDgR5"},"outputs":[],"source":["#slice the data\n","harness.data = {k: v[:10] for k, v in harness.data.items()}"]},{"cell_type":"markdown","metadata":{"id":"GlBMu35ODm77"},"source":["### Generating the test cases."]},{"cell_type":"code","execution_count":6,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":58028,"status":"ok","timestamp":1692371688215,"user":{"displayName":"Prikshit sharma","userId":"07819241395213139913"},"user_tz":-330},"id":"L1NQcBCHDomc","outputId":"e3df8f16-fadd-4fbb-e479-2f098f07ba5a"},"outputs":[{"name":"stdout","output_type":"stream","text":["================================================================================\n"," BoolQ \n","================================================================================\n"]},{"name":"stderr","output_type":"stream","text":["Generating testcases...: 100%|██████████| 1/1 [00:00\n","\n","\n"," \n"," \n"," | \n"," category | \n"," dataset_name | \n"," test_type | \n"," original_context | \n"," original_question | \n"," perturbed_context | \n"," perturbed_question | \n","
\n"," \n"," \n"," \n"," | 0 | \n"," robustness | \n"," BoolQ | \n"," uppercase | \n"," All biomass goes through at least some of thes... | \n"," does ethanol take more energy make that produces | \n"," ALL BIOMASS GOES THROUGH AT LEAST SOME OF THES... | \n"," DOES ETHANOL TAKE MORE ENERGY MAKE THAT PRODUCES | \n","
\n"," \n"," | 1 | \n"," robustness | \n"," BoolQ | \n"," uppercase | \n"," Property tax or 'house tax' is a local tax on ... | \n"," is house tax and property tax are same | \n"," PROPERTY TAX OR 'HOUSE TAX' IS A LOCAL TAX ON ... | \n"," IS HOUSE TAX AND PROPERTY TAX ARE SAME | \n","
\n"," \n"," | 2 | \n"," robustness | \n"," BoolQ | \n"," uppercase | \n"," Phantom pain sensations are described as perce... | \n"," is pain experienced in a missing body part or ... | \n"," PHANTOM PAIN SENSATIONS ARE DESCRIBED AS PERCE... | \n"," IS PAIN EXPERIENCED IN A MISSING BODY PART OR ... | \n","
\n"," \n"," | 3 | \n"," robustness | \n"," BoolQ | \n"," uppercase | \n"," Harry Potter and the Escape from Gringotts is ... | \n"," is harry potter and the escape from gringotts ... | \n"," HARRY POTTER AND THE ESCAPE FROM GRINGOTTS IS ... | \n"," IS HARRY POTTER AND THE ESCAPE FROM GRINGOTTS ... | \n","
\n"," \n"," | 4 | \n"," robustness | \n"," BoolQ | \n"," uppercase | \n"," Hydroxyzine preparations require a doctor's pr... | \n"," is there a difference between hydroxyzine hcl ... | \n"," HYDROXYZINE PREPARATIONS REQUIRE A DOCTOR'S PR... | \n"," IS THERE A DIFFERENCE BETWEEN HYDROXYZINE HCL ... | \n","
\n"," \n"," | ... | \n"," ... | \n"," ... | \n"," ... | \n"," ... | \n"," ... | \n"," ... | \n"," ... | \n","
\n"," \n"," | 80 | \n"," robustness | \n"," NQ-open | \n"," add_speech_to_text_typo | \n"," - | \n"," who played grand moff tarkin in rogue one | \n"," - | \n"," Hoo played grand moff tarkin in rogue one | \n","
\n"," \n"," | 81 | \n"," robustness | \n"," NQ-open | \n"," add_speech_to_text_typo | \n"," - | \n"," youngest current member of the house of repres... | \n"," - | \n"," youngest current member of the Hause of repres... | \n","
\n"," \n"," | 82 | \n"," robustness | \n"," NQ-open | \n"," add_speech_to_text_typo | \n"," - | \n"," who wrote the miraculous journey of edward tulane | \n"," - | \n"," Houx wrote the miraculous journey of edward tu... | \n","
\n"," \n"," | 83 | \n"," robustness | \n"," NQ-open | \n"," add_speech_to_text_typo | \n"," - | \n"," when did the night mare before christmas come out | \n"," - | \n"," when did the night Mehr before christmas come out | \n","
\n"," \n"," | 84 | \n"," robustness | \n"," NQ-open | \n"," add_speech_to_text_typo | \n"," - | \n"," when does the green book come out in theaters | \n"," - | \n"," when does the green book come out Inn theaters | \n","
\n"," \n","
\n","85 rows × 7 columns
\n",""],"text/plain":[" category dataset_name test_type \\\n","0 robustness BoolQ uppercase \n","1 robustness BoolQ uppercase \n","2 robustness BoolQ uppercase \n","3 robustness BoolQ uppercase \n","4 robustness BoolQ uppercase \n",".. ... ... ... \n","80 robustness NQ-open add_speech_to_text_typo \n","81 robustness NQ-open add_speech_to_text_typo \n","82 robustness NQ-open add_speech_to_text_typo \n","83 robustness NQ-open add_speech_to_text_typo \n","84 robustness NQ-open add_speech_to_text_typo \n","\n"," original_context \\\n","0 All biomass goes through at least some of thes... \n","1 Property tax or 'house tax' is a local tax on ... \n","2 Phantom pain sensations are described as perce... \n","3 Harry Potter and the Escape from Gringotts is ... \n","4 Hydroxyzine preparations require a doctor's pr... \n",".. ... \n","80 - \n","81 - \n","82 - \n","83 - \n","84 - \n","\n"," original_question \\\n","0 does ethanol take more energy make that produces \n","1 is house tax and property tax are same \n","2 is pain experienced in a missing body part or ... \n","3 is harry potter and the escape from gringotts ... \n","4 is there a difference between hydroxyzine hcl ... \n",".. ... \n","80 who played grand moff tarkin in rogue one \n","81 youngest current member of the house of repres... \n","82 who wrote the miraculous journey of edward tulane \n","83 when did the night mare before christmas come out \n","84 when does the green book come out in theaters \n","\n"," perturbed_context \\\n","0 ALL BIOMASS GOES THROUGH AT LEAST SOME OF THES... \n","1 PROPERTY TAX OR 'HOUSE TAX' IS A LOCAL TAX ON ... \n","2 PHANTOM PAIN SENSATIONS ARE DESCRIBED AS PERCE... \n","3 HARRY POTTER AND THE ESCAPE FROM GRINGOTTS IS ... \n","4 HYDROXYZINE PREPARATIONS REQUIRE A DOCTOR'S PR... \n",".. ... \n","80 - \n","81 - \n","82 - \n","83 - \n","84 - \n","\n"," perturbed_question \n","0 DOES ETHANOL TAKE MORE ENERGY MAKE THAT PRODUCES \n","1 IS HOUSE TAX AND PROPERTY TAX ARE SAME \n","2 IS PAIN EXPERIENCED IN A MISSING BODY PART OR ... \n","3 IS HARRY POTTER AND THE ESCAPE FROM GRINGOTTS ... \n","4 IS THERE A DIFFERENCE BETWEEN HYDROXYZINE HCL ... \n",".. ... \n","80 Hoo played grand moff tarkin in rogue one \n","81 youngest current member of the Hause of repres... \n","82 Houx wrote the miraculous journey of edward tu... \n","83 when did the night Mehr before christmas come out \n","84 when does the green book come out Inn theaters \n","\n","[85 rows x 7 columns]"]},"execution_count":7,"metadata":{},"output_type":"execute_result"}],"source":["harness.testcases()"]},{"cell_type":"markdown","metadata":{"id":"akSniLOoDxOp"},"source":["harness.generate() method automatically generates the test cases (based on the provided configuration)"]},{"cell_type":"markdown","metadata":{"id":"wk_cgK2BDzcM"},"source":["### Running the tests"]},{"cell_type":"code","execution_count":8,"metadata":{"colab":{"base_uri":"https://localhost:8080/"},"executionInfo":{"elapsed":48720,"status":"ok","timestamp":1692371736914,"user":{"displayName":"Prikshit sharma","userId":"07819241395213139913"},"user_tz":-330},"id":"nje7KWD9Dx3Y","outputId":"5ac4304a-0078-49ad-84b0-c5b6c2f58155"},"outputs":[{"name":"stdout","output_type":"stream","text":["================================================================================\n"," BoolQ \n","================================================================================\n"]},{"name":"stderr","output_type":"stream","text":["Running testcases... : 100%|██████████| 48/48 [00:52<00:00, 1.10s/it]\n"]},{"name":"stdout","output_type":"stream","text":["--------------------------------------------------------------------------------\n","\n","================================================================================\n"," NQ-open \n","================================================================================\n"]},{"name":"stderr","output_type":"stream","text":["Running testcases... : 100%|██████████| 37/37 [00:48<00:00, 1.31s/it]"]},{"name":"stdout","output_type":"stream","text":["--------------------------------------------------------------------------------\n","\n"]},{"name":"stderr","output_type":"stream","text":["\n"]},{"data":{"text/plain":[]},"execution_count":8,"metadata":{},"output_type":"execute_result"}],"source":["harness.run()"]},{"cell_type":"markdown","metadata":{"id":"7GnDWiU6D2S4"},"source":["Called after harness.generate() and is to used to run all the tests. Returns a pass/fail flag for each test."]},{"cell_type":"markdown","metadata":{"id":"q17wkdZcD4T8"},"source":["### Generated Results"]},{"cell_type":"code","execution_count":9,"metadata":{"colab":{"base_uri":"https://localhost:8080/","height":805},"executionInfo":{"elapsed":18550,"status":"ok","timestamp":1692371755410,"user":{"displayName":"Prikshit sharma","userId":"07819241395213139913"},"user_tz":-330},"id":"yJta_DvJD3xh","outputId":"91be0a8f-f014-4e04-81bd-8eaa521c84c9"},"outputs":[{"data":{"text/html":["\n","\n","
\n"," \n"," \n"," | \n"," category | \n"," dataset_name | \n"," test_type | \n"," original_context | \n"," original_question | \n"," perturbed_context | \n"," perturbed_question | \n"," expected_result | \n"," actual_result | \n"," pass | \n","
\n"," \n"," \n"," \n"," | 0 | \n"," robustness | \n"," BoolQ | \n"," uppercase | \n"," All biomass goes through at least some of thes... | \n"," does ethanol take more energy make that produces | \n"," ALL BIOMASS GOES THROUGH AT LEAST SOME OF THES... | \n"," DOES ETHANOL TAKE MORE ENERGY MAKE THAT PRODUCES | \n"," \\n\\n\\nTrue | \n"," TRUE | \n"," True | \n","
\n"," \n"," | 1 | \n"," robustness | \n"," BoolQ | \n"," uppercase | \n"," Property tax or 'house tax' is a local tax on ... | \n"," is house tax and property tax are same | \n"," PROPERTY TAX OR 'HOUSE TAX' IS A LOCAL TAX ON ... | \n"," IS HOUSE TAX AND PROPERTY TAX ARE SAME | \n"," \\n\\nTrue | \n"," \\n\\nTrue | \n"," True | \n","
\n"," \n"," | 2 | \n"," robustness | \n"," BoolQ | \n"," uppercase | \n"," Phantom pain sensations are described as perce... | \n"," is pain experienced in a missing body part or ... | \n"," PHANTOM PAIN SENSATIONS ARE DESCRIBED AS PERCE... | \n"," IS PAIN EXPERIENCED IN A MISSING BODY PART OR ... | \n"," ?\\n\\n\\nTrue | \n"," \\nTrue | \n"," True | \n","
\n"," \n"," | 3 | \n"," robustness | \n"," BoolQ | \n"," uppercase | \n"," Harry Potter and the Escape from Gringotts is ... | \n"," is harry potter and the escape from gringotts ... | \n"," HARRY POTTER AND THE ESCAPE FROM GRINGOTTS IS ... | \n"," IS HARRY POTTER AND THE ESCAPE FROM GRINGOTTS ... | \n"," \\n\\nTrue | \n"," ?\\n\\nTrue | \n"," True | \n","
\n"," \n"," | 4 | \n"," robustness | \n"," BoolQ | \n"," uppercase | \n"," Hydroxyzine preparations require a doctor's pr... | \n"," is there a difference between hydroxyzine hcl ... | \n"," HYDROXYZINE PREPARATIONS REQUIRE A DOCTOR'S PR... | \n"," IS THERE A DIFFERENCE BETWEEN HYDROXYZINE HCL ... | \n"," \\n\\nTrue | \n"," OATE\\n\\nTrue | \n"," False | \n","
\n"," \n"," | ... | \n"," ... | \n"," ... | \n"," ... | \n"," ... | \n"," ... | \n"," ... | \n"," ... | \n"," ... | \n"," ... | \n"," ... | \n","
\n"," \n"," | 80 | \n"," robustness | \n"," NQ-open | \n"," add_speech_to_text_typo | \n"," - | \n"," who played grand moff tarkin in rogue one | \n"," - | \n"," Hoo played grand moff tarkin in rogue one | \n"," \\n\\nPeter Cushing | \n"," \\n\\nPeter Cushing played Grand Moff Tarkin in ... | \n"," True | \n","
\n"," \n"," | 81 | \n"," robustness | \n"," NQ-open | \n"," add_speech_to_text_typo | \n"," - | \n"," youngest current member of the house of repres... | \n"," - | \n"," youngest current member of the Hause of repres... | \n"," \\n\\nAs of 2021, the youngest current member of... | \n"," \\n\\nAs of 2021, the youngest current member of... | \n"," True | \n","
\n"," \n"," | 82 | \n"," robustness | \n"," NQ-open | \n"," add_speech_to_text_typo | \n"," - | \n"," who wrote the miraculous journey of edward tulane | \n"," - | \n"," Houx wrote the miraculous journey of edward tu... | \n"," \\n\\nThe Miraculous Journey of Edward Tulane wa... | \n"," \\n\\nWho wrote \"The Miraculous Journey of Edwar... | \n"," False | \n","
\n"," \n"," | 83 | \n"," robustness | \n"," NQ-open | \n"," add_speech_to_text_typo | \n"," - | \n"," when did the night mare before christmas come out | \n"," - | \n"," when did the night Mehr before christmas come out | \n"," \\n\\nThe Nightmare Before Christmas was release... | \n"," \\n\\nThe Nightmare Before Christmas was release... | \n"," True | \n","
\n"," \n"," | 84 | \n"," robustness | \n"," NQ-open | \n"," add_speech_to_text_typo | \n"," - | \n"," when does the green book come out in theaters | \n"," - | \n"," when does the green book come out Inn theaters | \n"," \\n\\nThe green book was released in theaters on... | \n"," \\n\\nThe release date for the green book in the... | \n"," False | \n","
\n"," \n","
\n","
85 rows × 10 columns
\n","
\n","\n","
\n"," \n"," \n"," | \n"," | \n"," | \n"," Benchmarking Results: gpt-3.5-turbo-instruct | \n","
\n"," \n"," | \n"," | \n"," | \n"," fail_count | \n"," pass_count | \n"," pass_rate | \n"," minimum_pass_rate | \n"," pass | \n","
\n"," \n"," | dataset_name | \n"," category | \n"," test_type | \n"," | \n"," | \n"," | \n"," | \n"," | \n","
\n"," \n"," \n"," \n"," | BoolQ | \n"," robustness | \n"," uppercase | \n"," 2 | \n"," 8 | \n"," 80% | \n"," 66% | \n"," True | \n","
\n"," \n"," | dyslexia_word_swap | \n"," 2 | \n"," 8 | \n"," 80% | \n"," 60% | \n"," True | \n","
\n"," \n"," | add_abbreviation | \n"," 1 | \n"," 9 | \n"," 90% | \n"," 60% | \n"," True | \n","
\n"," \n"," | add_slangs | \n"," 4 | \n"," 4 | \n"," 50% | \n"," 60% | \n"," False | \n","
\n"," \n"," | add_speech_to_text_typo | \n"," 3 | \n"," 7 | \n"," 70% | \n"," 60% | \n"," True | \n","
\n"," \n"," | NQ-open | \n"," robustness | \n"," uppercase | \n"," 3 | \n"," 7 | \n"," 70% | \n"," 66% | \n"," True | \n","
\n"," \n"," | dyslexia_word_swap | \n"," 2 | \n"," 5 | \n"," 71% | \n"," 60% | \n"," True | \n","
\n"," \n"," | add_abbreviation | \n"," 5 | \n"," 4 | \n"," 44% | \n"," 60% | \n"," False | \n","
\n"," \n"," | add_slangs | \n"," 1 | \n"," 1 | \n"," 50% | \n"," 60% | \n"," False | \n","
\n"," \n"," | add_speech_to_text_typo | \n"," 5 | \n"," 4 | \n"," 44% | \n"," 60% | \n"," False | \n","
\n"," \n","
\n","
-## 2.1.0
+## 2.2.0
------------------
## 📢 Highlights
-John Snow Labs is thrilled to announce the release of LangTest 2.1.0! This update brings exciting new features and improvements designed to streamline your language model testing workflows and provide deeper insights.
+John Snow Labs is excited to announce the release of LangTest 2.2.0! This update introduces powerful new features and enhancements to elevate your language model testing experience and deliver even greater insights.
-- **🔗 Enhanced API-based LLM Integration:** LangTest now supports testing API-based Large Language Models (LLMs). This allows you to seamlessly integrate diverse LLM models with LangTest and conduct performance evaluations across various datasets.
+- 🏆 **Model Ranking & Leaderboard**: LangTest introduces a comprehensive model ranking system. Use harness.get_leaderboard() to rank models based on various test metrics and retain previous rankings for historical comparison.
-- **📂 Expanded File Format Support:** LangTest 2.1.0 introduces support for additional file formats, further increasing its flexibility in handling different data structures used in LLM testing.
+- 🔍 **Few-Shot Model Evaluation:** Optimize and evaluate your models using few-shot prompt techniques. This feature enables you to assess model performance with minimal data, providing valuable insights into model capabilities with limited examples.
-- **📊 Improved Multi-Dataset Handling:** We've made significant improvements in how LangTest manages multiple datasets. This simplifies workflows and allows for more efficient testing across a wider range of data sources.
+- 📊 **Evaluating NER in LLMs:** This release extends support for Named Entity Recognition (NER) tasks specifically for Large Language Models (LLMs). Evaluate and benchmark LLMs on their NER performance with ease.
-- **🖥️ New Benchmarking Commands**: LangTest now boasts a set of new commands specifically designed for benchmarking language models. These commands provide a structured approach to evaluating model performance and comparing results across different models and datasets.
+- 🚀 **Enhanced Data Augmentation:** The new DataAugmenter module allows for streamlined and harness-free data augmentation, making it simpler to enhance your datasets and improve model robustness.
+
+- 🎯 **Multi-Dataset Prompts:** LangTest now offers optimized prompt handling for multiple datasets, allowing users to add custom prompts for each dataset, enabling seamless integration and efficient testing.
## 🔥 Key Enhancements:
-### **🔗 Streamlined Integration and Enhanced Functionality for API-Based Large Language Models:**
-[](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Generic_API-Based_Model_Testing_Demo.ipynb)
-
-This feature empowers you to seamlessly integrate virtually any language model hosted on an external API platform. Whether you prefer OpenAI, Hugging Face, or even custom vLLM solutions, LangTest now adapts to your workflow. `input_processor` and `output_parser` functions are not required for openai api compatible server.
-
-#### Key Features:
-
-- **Effortless API Integration:** Connect to any API system by specifying the API URL, parameters, and a custom function for parsing the returned results. This intuitive approach allows you to leverage your preferred language models with minimal configuration.
+### **🏆 Comprehensive Model Ranking & Leaderboard**
+[](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/benchmarks/Benchmarking_with_Harness.ipynb)
+The new Model Ranking & Leaderboard system offers a comprehensive way to evaluate and compare model performance based on various metrics across different datasets. This feature allows users to rank models, retain historical rankings, and analyze performance trends.
-- **Customizable Parameters:** Define the URL, parameters specific to your chosen API, and a parsing function tailored to extract the desired output. This level of control ensures compatibility with diverse API structures.
+**Key Features:**
+- **Comprehensive Ranking**: Rank models based on various performance metrics across multiple datasets.
+- **Historical Comparison**: Retain and compare previous rankings for consistent performance tracking.
+- **Dataset-Specific Insights**: Evaluate model performance on different datasets to gain deeper insights.
-- **Unparalleled Flexibility:** Generic API Support removes platform limitations. Now, you can seamlessly integrate language models from various sources, including OpenAI, Hugging Face, and even custom vLLM solutions hosted on private platforms.
-
-#### How it Works:
-
-**Parameters:**
-Define the `input_processer` function for creating a payload and the `output_parser` function is used to extract the output from the response.
-
-```python
-GOOGLE_API_KEY = ""
-model_url = f"https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent?key={GOOGLE_API_KEY}"
+**How It Works:**
-# headers
-headers = {
- "Content-Type": "application/json",
-}
+The following are steps to do model ranking and visualize the leaderboard for `google/flan-t5-base` and `google/flan-t5-large` models.
+**1.** Setup and configuration of the Harness are as follows:
-# function to create a payload
-def input_processor(content):
- return {"contents": [
- {
- "role": "user",
- "parts": [
- {
- "text": content
- }
- ]
- }
- ]}
-
-# function to extract output from model response
-def output_parser(response):
- try:
- return response['candidates'][0]['content']['parts'][0]['text']
- except:
- return ""
+```yaml
+# config.yaml
+model_parameters:
+ max_tokens: 64
+ device: 0
+ task: text2text-generation
+tests:
+ defaults:
+ min_pass_rate: 0.65
+ robustness:
+ add_typo:
+ min_pass_rate: 0.7
+ lowercase:
+ min_pass_rate: 0.7
```
-
-To take advantage of this feature, users can utilize the following setup code:
-
```python
from langtest import Harness
-# Initialize Harness with API parameters
harness = Harness(
task="question-answering",
model={
- "model": {
- "url": url,
- "headers": headers,
- "input_processor": input_processor,
- "output_parser": output_parser,
- },
- "hub": "web",
+ "model": "google/flan-t5-base",
+ "hub": "huggingface"
},
- data={
- "data_source": "OpenBookQA",
- "split": "test-tiny",
+ data=[
+ {
+ "data_source": "MedMCQA"
+ },
+ {
+ "data_source": "PubMedQA"
+ },
+ {
+ "data_source": "MMLU"
+ },
+ {
+ "data_source": "MedQA"
+ }
+ ],
+ config="config.yml",
+ benchmarking={
+ "save_dir":"~/.langtest/leaderboard/" # required for benchmarking
}
)
-# Generate, Run and get Report
-harness.generate().run().report()
```
-
+**2**. generate the test cases, run on the model, and get the report as follows:
+```python
+harness.generate().run().report()
+```
+
-### 📂 Streamlined Data Handling and Evaluation
+**3**. Similarly, do the same steps for the `google/flan-t5-large` model with the same `save_dir` path for benchmarking and the same `config.yaml`
-This feature streamlines your testing workflows by enabling LangTest to process a wider range of file formats directly.
+**4**. Finally, the leaderboard can show the model rank by calling the below code.
+```python
+harness.get_leaderboard()
+```
+
-#### Key Features:
+**Conclusion:**
+The Model Ranking & Leaderboard system provides a robust and structured method for evaluating and comparing models across multiple datasets, enabling users to make data-driven decisions and continuously improve model performance.
-- **Effortless File Format Handling:** LangTest now seamlessly ingests data from various file formats, including pickles (.pkl) in addition to previously supported formats. Simply provide the data source path in your harness configuration, and LangTest takes care of the rest.
-- **Simplified Data Source Management**: LangTest intelligently recognizes the file extension and automatically selects the appropriate processing method. This eliminates the need for manual configuration, saving you time and effort.
+### **🔍 Efficient Few-Shot Model Evaluation**
+[](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Fewshot_QA_Notebook.ipynb)
+Few-Shot Model Evaluation optimizes and evaluates model performance using minimal data. This feature provides rapid insights into model capabilities, enabling efficient assessment and optimization with limited examples.
-- **Enhanced Maintainability**: The underlying code structure is optimized for flexibility. Adding support for new file formats in the future requires minimal effort, ensuring LangTest stays compatible with evolving data storage practices.
+**Key Features:**
+- **Few-Shot Techniques**: Evaluate models with minimal data to gauge performance quickly.
+- **Optimized Performance**: Improve model outputs using targeted few-shot prompts.
+- **Efficient Evaluation**: Streamlined process for rapid and effective model assessment.
-#### How it works:
+**How It Works:**
+**1.** Set up few-shot prompts tailored to specific evaluation needs.
+```yaml
+# config.yaml
+prompt_config:
+ "BoolQ":
+ instructions: >
+ You are an intelligent bot and it is your responsibility to make sure
+ to give a concise answer. Answer should be `true` or `false`.
+ prompt_type: "instruct" # instruct for completion and chat for conversation(chat models)
+ examples:
+ - user:
+ context: >
+ The Good Fight -- A second 13-episode season premiered on March 4, 2018.
+ On May 2, 2018, the series was renewed for a third season.
+ question: "is there a third series of the good fight?"
+ ai:
+ answer: "True"
+ - user:
+ context: >
+ Lost in Space -- The fate of the castaways is never resolved,
+ as the series was unexpectedly canceled at the end of season 3.
+ question: "did the robinsons ever get back to earth"
+ ai:
+ answer: "True"
+ "NQ-open":
+ instructions: >
+ You are an intelligent bot and it is your responsibility to make sure
+ to give a short concise answer.
+ prompt_type: "instruct" # completion
+ examples:
+ - user:
+ question: "where does the electron come from in beta decay?"
+ ai:
+ answer: "an atomic nucleus"
+ - user:
+ question: "who wrote you're a grand ol flag?"
+ ai:
+ answer: "George M. Cohan"
+
+tests:
+ defaults:
+ min_pass_rate: 0.8
+ robustness:
+ uppercase:
+ min_pass_rate: 0.8
+ add_typo:
+ min_pass_rate: 0.8
+```
+**2.** Initialize the Harness with `config.yaml` file as below code
+```python
+harness = Harness(
+ task="question-answering",
+ model={"model": "gpt-3.5-turbo-instruct","hub":"openai"},
+ data=[{"data_source" :"BoolQ",
+ "split":"test-tiny"},
+ {"data_source" :"NQ-open",
+ "split":"test-tiny"}],
+ config="config.yaml"
+ )
+```
+**3.** Generate the test cases, run them on the model, and then generate the report.
```python
-from langtest import Harness
+harness.generate().run().report()
+```
+
-harness = Harness(
- task="question-answering",
- model={
- "model": "http://localhost:1234/v1/chat/completions",
- "hub": "lm-studio",
- },
- data={
- "data_source": "path/to/file.pkl", #
- },
-)
-# generate, run and report
+**Conclusion:**
+Few-Shot Model Evaluation provides valuable insights into model capabilities with minimal data, allowing for rapid and effective performance optimization. This feature ensures that models can be assessed and improved efficiently, even with limited examples.
+
+
+### **📊 Evaluating NER in LLMs**
+[](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/NER%20Casual%20LLM.ipynb)
+Evaluating NER in LLMs enables precise extraction and evaluation of entities using Large Language Models (LLMs). This feature enhances the capability to assess LLM performance on Named Entity Recognition tasks.
+
+**Key Features:**
+- **LLM-Specific Support**: Tailored for evaluating NER tasks using LLMs.
+- **Accurate Entity Extraction**: Improved techniques for precise entity extraction.
+- **Comprehensive Evaluation**: Detailed assessment of entity extraction performance.
+
+**How It Works:**
+**1.** Set up NER tasks for specific LLM evaluation.
+```python
+# Create a Harness object
+harness = Harness(task="ner",
+ model={
+ "model": "gpt-3.5-turbo-instruct",
+ "hub": "openai", },
+ data={
+ "data_source": 'path/to/conll03.conll'
+ },
+ config={
+ "model_parameters": {
+ "temperature": 0,
+ },
+ "tests": {
+ "defaults": {
+ "min_pass_rate": 1.0
+ },
+ "robustness": {
+ "lowercase": {
+ "min_pass_rate": 0.7
+ }
+ },
+ "accuracy": {
+ "min_f1_score": {
+ "min_score": 0.7,
+ },
+ }
+ }
+ }
+ )
+```
+**2.** Generate the test cases based on the configuration in the Harness, run them on the model, and get the report.
+```python
harness.generate().run().report()
```
-### 📊 Multi-Dataset Handling and Evaluation
-[](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/Multiple_dataset.ipynb)
+
+
+Examples:
+
+
+**Conclusion:**
+Evaluating NER in LLMs allows for accurate entity extraction and performance assessment using LangTest's comprehensive evaluation methods. This feature ensures thorough and reliable evaluation of LLMs on Named Entity Recognition tasks.
-This feature empowers you to efficiently benchmark your language models across a wider range of datasets.
-#### Key Features:
+### **🚀 Enhanced Data Augmentation**
+[](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/Data_Augmenter_Notebook.ipynb)
+Enhanced Data Augmentation introduces a new `DataAugmenter` class, enabling streamlined and harness-free data augmentation. This feature simplifies the process of enriching datasets to improve model robustness and performance.
-- **Effortless Multi-Dataset Testing:** LangTest now seamlessly integrates and executes tests on multiple datasets within a single harness configuration. This streamlined approach eliminates the need for repetitive setups, saving you time and resources.
+**Key Features:**
+- **Harness-Free Augmentation**: Perform data augmentation without the need for harness testing.
+- **Improved Workflow**: Simplified processes for enhancing datasets efficiently.
+- **Robust Models**: Increase model robustness through effective data augmentation techniques.
-- **Enhanced Fairness Evaluation**: By testing models across diverse datasets, LangTest helps identify and mitigate potential biases. This ensures your models perform fairly and accurately on a broader spectrum of data, promoting ethical and responsible AI development.
+**How It Works:**
+The following are steps to import the `DataAugmenter` class from LangTest.
+**1.** Create a config.yaml for the data augmentation.
+```yaml
+# config.yaml
+parameters:
+ type: proportion
+ style: new
+tests:
+ robustness:
+ uppercase:
+ max_proportion: 0.2
+ lowercase:
+ max_proportion: 0.2
+
+```
+**2.** Initialize the `DataAugmenter` class and apply various tests for augmentation to your datasets.
+```python
+from langtest.augmentation import DataAugmenter
+from langtest.tasks.task import TaskManager
+
+data_augmenter = DataAugmenter(
+ task=TaskManager("ner"), # use the ner, text-classification, question-answering...
+ config="config.yaml",
+)
+```
+**3.** Provide the training dataset to `data_augmenter`.
+```python
+data_augmenter.augment(data={
+ 'data_source': 'path/to/conll03.conll'
+})
+```
+**4.** Then, save the augmented dataset.
+```
+data_augmenter.save("augmented.conll")
+```
+**Conclusion:**
+Enhanced Data Augmentation capabilities in LangTest ensure that your models are more robust and capable of handling diverse data scenarios. This feature simplifies the augmentation process, leading to improved model performance and reliability.
-- **Robust Accuracy Assessment:** Multi-dataset support empowers you to conduct more rigorous accuracy testing. By evaluating models on various datasets, you gain a deeper understanding of their strengths and weaknesses across different data distributions. This comprehensive analysis strengthens your confidence in the model's real-world performance.
-#### How it works:
+### **🎯Multi-Dataset Prompts**
+[](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/MultiPrompt_MultiDataset.ipynb)
+Multi-Dataset Prompts streamline the process of integrating and testing various data sources by allowing users to define custom prompts for each dataset. This enhancement ensures efficient prompt handling across multiple datasets, enabling comprehensive performance evaluations.
-Initiate the Harness class
+**Key Features:**
+
+- **Custom Prompts:** Add tailored prompts for each dataset to enhance testing accuracy.
+- **Seamless Integration:** Easily incorporate multiple datasets into your testing environment.
+- **Improved Efficiency:** Simplified workflows for handling diverse data sources.
+
+**How It Works:**
+**1.** Initiate the Harness with `BoolQ` and `NQ-open` datasets.
```python
+# Import Harness from the LangTest library
+from langtest import Harness
+
harness = Harness(
task="question-answering",
model={"model": "gpt-3.5-turbo-instruct", "hub": "openai"},
data=[
- {"data_source": "NQ-open", "split": "test-tiny",},
- {"data_source": "MedQA", "split": "test-tiny"},
- {"data_source": "LogiQA", "split": "test-tiny"},
+ {"data_source": "BoolQ", "split": "dev-tiny"},
+ {"data_source": "NQ-open", "split": "test-tiny"}
],
)
```
-Configure the accuracy tests in Harness class
+**2.** Configure prompts specific to each dataset, allowing tailored evaluations.
```python
harness.configure(
{
+ "model_parameters": {
+ "user_prompt": {
+ "BoolQ": "Answer the following question with a True or False. {context}\nQuestion {question}",
+ "NQ-open": "Answer the following question. Question {question}",
+ }
+ },
"tests": {
"defaults": {"min_pass_rate": 0.65},
-
- "accuracy": {
- "llm_eval": {"min_score": 0.60},
- "min_rouge1_score": {"min_score": 0.60},
- "min_rouge2_score": {"min_score": 0.60},
- "min_rougeL_score": {"min_score": 0.60},
- "min_rougeLsum_score": {"min_score": 0.60},
+ "robustness": {
+ "uppercase": {"min_pass_rate": 0.66},
+ "dyslexia_word_swap": {"min_pass_rate": 0.60},
+ "add_abbreviation": {"min_pass_rate": 0.60},
+ "add_slangs": {"min_pass_rate": 0.60},
+ "add_speech_to_text_typo": {"min_pass_rate": 0.60},
},
}
}
)
```
-harness.generate() generates testcases, .run() executes them, and .report() compiles results.
+**3.** Generate the test cases, run them on the model, and get the report.
```python
harness.generate().run().report()
```
-
-
-### 🖥️ Streamlined Evaluation Workflows with Enhanced CLI Commands
-[](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/benchmarks/Langtest_Cli_Eval_Command.ipynb)
-
-LangTest's evaluation capabilities, focusing on report management and leaderboards. These enhancements empower you to:
-
-- **Streamlined Reporting and Tracking:** Effortlessly save and load detailed evaluation reports directly from the command line using `langtest eval`, enabling efficient performance tracking and comparative analysis over time, with manual file review options in the `~/.langtest` or `./.langtest` folder.
-
-- **Enhanced Leaderboards:** Gain valuable insights with the new langtest show-leaderboard command. This command displays existing leaderboards, providing a centralized view of ranked model performance across evaluations.
-
-- **Average Model Ranking:** Leaderboard now include the average ranking for each evaluated model. This metric provides a comprehensive understanding of model performance across various datasets and tests.
-
-### How it works:
-
-First, create the `parameter.json` or `parameter.yaml` in the working directory
-
-**JSON Format**
-```json
-{
- "task": "question-answering",
- "model": {
- "model": "google/flan-t5-base",
- "hub": "huggingface"
- },
- "data": [
- {
- "data_source": "MedMCQA"
- },
- {
- "data_source": "PubMedQA"
- },
- {
- "data_source": "MMLU"
- },
- {
- "data_source": "MedQA"
- }
- ],
- "config": {
- "model_parameters": {
- "max_tokens": 64,
- "device": 0,
- "task": "text2text-generation"
- },
- "tests": {
- "defaults": {
- "min_pass_rate": 0.70
- },
- "robustness": {
- "add_typo": {
- "min_pass_rate": 0.70
- }
- }
- }
- }
-}
-```
-**Yaml Format**
-```yaml
-task: question-answering
-model:
- model: google/flan-t5-base
- hub: huggingface
-data:
-- data_source: MedMCQA
-- data_source: PubMedQA
-- data_source: MMLU
-- data_source: MedQA
-config:
- model_parameters:
- max_tokens: 64
- device: 0
- task: text2text-generation
- tests:
- defaults:
- min_pass_rate: 0.70
- robustness:
- add_typo:
- min_pass_rate: 0.7
-
-```
-And open the terminal or cmd in your system
-```bash
-langtest eval --model \
- --hub \
- -c < your configuration file like parameter.json or parameter.yaml>
-```
-Finally, we can know the leaderboard and rank of the model.
-
+
-----
-
-To visualize the leaderboard anytime using the CLI command
-```bash
-langtest show-leaderboard
-```
-
+**Conclusion:**
+Multi-dataset prompts in LangTest empower users to efficiently manage and test multiple data sources, resulting in more effective and comprehensive language model evaluations.
## 📒 New Notebooks
+{:.table2}
| Notebooks | Colab Link |
|--------------------|-------------|
-| Generic API-based Model Testing | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Generic_API-Based_Model_Testing_Demo.ipynb)|
-| Multi-Dataset | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/Multiple_dataset.ipynb) |
-| Langtest Eval Cli Command | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/benchmarks/Langtest_Cli_Eval_Command.ipynb) |
-----------------
+| Model Ranking & Leaderboard | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/benchmarks/Benchmarking_with_Harness.ipynb)|
+| Fewshot Model Evaluation | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Fewshot_QA_Notebook.ipynb) |
+| Evaluating NER in LLMs | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/NER%20Casual%20LLM.ipynb) |
+| Data Augmenter | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/Data_Augmenter_Notebook.ipynb) |
+| Multi-Dataset Prompts | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/MultiPrompt_MultiDataset.ipynb) |
+
{%- include docs-langtest-pagination.html -%}
diff --git a/docs/pages/docs/langtest_versions/release_notes_2_1_0.md b/docs/pages/docs/langtest_versions/release_notes_2_1_0.md
new file mode 100644
index 000000000..926aab701
--- /dev/null
+++ b/docs/pages/docs/langtest_versions/release_notes_2_1_0.md
@@ -0,0 +1,327 @@
+---
+layout: docs
+header: true
+seotitle: LangTest - Deliver Safe and Effective Language Models | John Snow Labs
+title: LangTest Release Notes
+permalink: /docs/pages/docs/langtest_versions/release_notes_2_1_0
+key: docs-release-notes
+modify_date: 2024-04-02
+---
+
+
+
+## 2.1.0
+
+## 📢 Highlights
+
+John Snow Labs is thrilled to announce the release of LangTest 2.1.0! This update brings exciting new features and improvements designed to streamline your language model testing workflows and provide deeper insights.
+
+- **🔗 Enhanced API-based LLM Integration:** LangTest now supports testing API-based Large Language Models (LLMs). This allows you to seamlessly integrate diverse LLM models with LangTest and conduct performance evaluations across various datasets.
+
+- **📂 Expanded File Format Support:** LangTest 2.1.0 introduces support for additional file formats, further increasing its flexibility in handling different data structures used in LLM testing.
+
+- **📊 Improved Multi-Dataset Handling:** We've made significant improvements in how LangTest manages multiple datasets. This simplifies workflows and allows for more efficient testing across a wider range of data sources.
+
+- **🖥️ New Benchmarking Commands**: LangTest now boasts a set of new commands specifically designed for benchmarking language models. These commands provide a structured approach to evaluating model performance and comparing results across different models and datasets.
+
+
+
+## 🔥 Key Enhancements:
+
+### **🔗 Streamlined Integration and Enhanced Functionality for API-Based Large Language Models:**
+[](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Generic_API-Based_Model_Testing_Demo.ipynb)
+
+This feature empowers you to seamlessly integrate virtually any language model hosted on an external API platform. Whether you prefer OpenAI, Hugging Face, or even custom vLLM solutions, LangTest now adapts to your workflow. `input_processor` and `output_parser` functions are not required for openai api compatible server.
+
+#### Key Features:
+
+- **Effortless API Integration:** Connect to any API system by specifying the API URL, parameters, and a custom function for parsing the returned results. This intuitive approach allows you to leverage your preferred language models with minimal configuration.
+
+- **Customizable Parameters:** Define the URL, parameters specific to your chosen API, and a parsing function tailored to extract the desired output. This level of control ensures compatibility with diverse API structures.
+
+- **Unparalleled Flexibility:** Generic API Support removes platform limitations. Now, you can seamlessly integrate language models from various sources, including OpenAI, Hugging Face, and even custom vLLM solutions hosted on private platforms.
+
+#### How it Works:
+
+**Parameters:**
+Define the `input_processer` function for creating a payload and the `output_parser` function is used to extract the output from the response.
+
+```python
+GOOGLE_API_KEY = ""
+model_url = f"https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent?key={GOOGLE_API_KEY}"
+
+# headers
+headers = {
+ "Content-Type": "application/json",
+}
+
+# function to create a payload
+def input_processor(content):
+ return {"contents": [
+ {
+ "role": "user",
+ "parts": [
+ {
+ "text": content
+ }
+ ]
+ }
+ ]}
+
+# function to extract output from model response
+def output_parser(response):
+ try:
+ return response['candidates'][0]['content']['parts'][0]['text']
+ except:
+ return ""
+```
+
+To take advantage of this feature, users can utilize the following setup code:
+
+```python
+from langtest import Harness
+
+# Initialize Harness with API parameters
+harness = Harness(
+ task="question-answering",
+ model={
+ "model": {
+ "url": url,
+ "headers": headers,
+ "input_processor": input_processor,
+ "output_parser": output_parser,
+ },
+ "hub": "web",
+ },
+ data={
+ "data_source": "OpenBookQA",
+ "split": "test-tiny",
+ }
+)
+# Generate, Run and get Report
+harness.generate().run().report()
+```
+
+
+
+### 📂 Streamlined Data Handling and Evaluation
+
+This feature streamlines your testing workflows by enabling LangTest to process a wider range of file formats directly.
+
+#### Key Features:
+
+- **Effortless File Format Handling:** LangTest now seamlessly ingests data from various file formats, including pickles (.pkl) in addition to previously supported formats. Simply provide the data source path in your harness configuration, and LangTest takes care of the rest.
+
+- **Simplified Data Source Management**: LangTest intelligently recognizes the file extension and automatically selects the appropriate processing method. This eliminates the need for manual configuration, saving you time and effort.
+
+- **Enhanced Maintainability**: The underlying code structure is optimized for flexibility. Adding support for new file formats in the future requires minimal effort, ensuring LangTest stays compatible with evolving data storage practices.
+
+#### How it works:
+
+```python
+from langtest import Harness
+
+harness = Harness(
+ task="question-answering",
+ model={
+ "model": "http://localhost:1234/v1/chat/completions",
+ "hub": "lm-studio",
+ },
+ data={
+ "data_source": "path/to/file.pkl", #
+ },
+)
+# generate, run and report
+harness.generate().run().report()
+```
+### 📊 Multi-Dataset Handling and Evaluation
+[](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/Multiple_dataset.ipynb)
+
+This feature empowers you to efficiently benchmark your language models across a wider range of datasets.
+
+#### Key Features:
+
+- **Effortless Multi-Dataset Testing:** LangTest now seamlessly integrates and executes tests on multiple datasets within a single harness configuration. This streamlined approach eliminates the need for repetitive setups, saving you time and resources.
+
+- **Enhanced Fairness Evaluation**: By testing models across diverse datasets, LangTest helps identify and mitigate potential biases. This ensures your models perform fairly and accurately on a broader spectrum of data, promoting ethical and responsible AI development.
+
+- **Robust Accuracy Assessment:** Multi-dataset support empowers you to conduct more rigorous accuracy testing. By evaluating models on various datasets, you gain a deeper understanding of their strengths and weaknesses across different data distributions. This comprehensive analysis strengthens your confidence in the model's real-world performance.
+
+#### How it works:
+
+Initiate the Harness class
+```python
+harness = Harness(
+ task="question-answering",
+ model={"model": "gpt-3.5-turbo-instruct", "hub": "openai"},
+ data=[
+ {"data_source": "NQ-open", "split": "test-tiny",},
+ {"data_source": "MedQA", "split": "test-tiny"},
+ {"data_source": "LogiQA", "split": "test-tiny"},
+ ],
+)
+```
+Configure the accuracy tests in Harness class
+```python
+harness.configure(
+ {
+ "tests": {
+ "defaults": {"min_pass_rate": 0.65},
+
+ "accuracy": {
+ "llm_eval": {"min_score": 0.60},
+ "min_rouge1_score": {"min_score": 0.60},
+ "min_rouge2_score": {"min_score": 0.60},
+ "min_rougeL_score": {"min_score": 0.60},
+ "min_rougeLsum_score": {"min_score": 0.60},
+ },
+ }
+ }
+)
+```
+harness.generate() generates testcases, .run() executes them, and .report() compiles results.
+```python
+harness.generate().run().report()
+```
+
+
+### 🖥️ Streamlined Evaluation Workflows with Enhanced CLI Commands
+[](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/benchmarks/Langtest_Cli_Eval_Command.ipynb)
+
+LangTest's evaluation capabilities, focusing on report management and leaderboards. These enhancements empower you to:
+
+- **Streamlined Reporting and Tracking:** Effortlessly save and load detailed evaluation reports directly from the command line using `langtest eval`, enabling efficient performance tracking and comparative analysis over time, with manual file review options in the `~/.langtest` or `./.langtest` folder.
+
+- **Enhanced Leaderboards:** Gain valuable insights with the new langtest show-leaderboard command. This command displays existing leaderboards, providing a centralized view of ranked model performance across evaluations.
+
+- **Average Model Ranking:** Leaderboard now include the average ranking for each evaluated model. This metric provides a comprehensive understanding of model performance across various datasets and tests.
+
+### How it works:
+
+First, create the `parameter.json` or `parameter.yaml` in the working directory
+
+**JSON Format**
+```json
+{
+ "task": "question-answering",
+ "model": {
+ "model": "google/flan-t5-base",
+ "hub": "huggingface"
+ },
+ "data": [
+ {
+ "data_source": "MedMCQA"
+ },
+ {
+ "data_source": "PubMedQA"
+ },
+ {
+ "data_source": "MMLU"
+ },
+ {
+ "data_source": "MedQA"
+ }
+ ],
+ "config": {
+ "model_parameters": {
+ "max_tokens": 64,
+ "device": 0,
+ "task": "text2text-generation"
+ },
+ "tests": {
+ "defaults": {
+ "min_pass_rate": 0.70
+ },
+ "robustness": {
+ "add_typo": {
+ "min_pass_rate": 0.70
+ }
+ }
+ }
+ }
+}
+```
+**Yaml Format**
+```yaml
+task: question-answering
+model:
+ model: google/flan-t5-base
+ hub: huggingface
+data:
+- data_source: MedMCQA
+- data_source: PubMedQA
+- data_source: MMLU
+- data_source: MedQA
+config:
+ model_parameters:
+ max_tokens: 64
+ device: 0
+ task: text2text-generation
+ tests:
+ defaults:
+ min_pass_rate: 0.70
+ robustness:
+ add_typo:
+ min_pass_rate: 0.7
+
+```
+And open the terminal or cmd in your system
+```bash
+langtest eval --model \
+ --hub \
+ -c < your configuration file like parameter.json or parameter.yaml>
+```
+Finally, we can know the leaderboard and rank of the model.
+
+
+----
+
+To visualize the leaderboard anytime using the CLI command
+```bash
+langtest show-leaderboard
+```
+
+
+## 📒 New Notebooks
+
+{:.table2}
+| Notebooks | Colab Link |
+|--------------------|-------------|
+| Generic API-based Model Testing | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Generic_API-Based_Model_Testing_Demo.ipynb)|
+| Multi-Dataset | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/Multiple_dataset.ipynb) |
+| Langtest Eval Cli Command | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/benchmarks/Langtest_Cli_Eval_Command.ipynb) |
+
+
+## 🐛 Fixes
+
+- Fixed multi-dataset support for accuracy task [#998]
+- Fixed bugs in langtest package [#1003][#1004]
+
+
+## ⚡ Enhancements
+- Improved the error handling in Harness run method [#990]
+- Websites Updates [#1001]
+- Updated new version for dependencies [#992]
+- Improved the data augmentation for Question-Answering task [#991]
+
+## What's Changed
+
+* Feautre/integration with web api by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/986
+* Refactor TestFactory class to handle exceptions in async tests by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/990
+* data augmentation support for question-answering task by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/991
+* Updated dependencies by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/992
+* Fix/implement the multiple dataset support for accuracy tests by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/998
+* Feature/add support for other file formats by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/993
+* Bug Fix: Generated results are none by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/1000
+* Feature/implement load & save for benchmark reports by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/999
+* Fix/bug fixes langtest 2 1 0 rc1 by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/1003
+* website updates by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/1001
+* Fix/bug fixes langtest 2 1 0 rc1 by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/1004
+* Release/2.0.1 by @chakravarthik27 in https://github.com/JohnSnowLabs/langtest/pull/1005
+
+
+**Full Changelog**: https://github.com/JohnSnowLabs/langtest/compare/2.0.0...2.1.0
+
+## ⚒️ Previous Versions
+
+{%- include docs-langtest-pagination.html -%}
diff --git a/docs/pages/tutorials/LLM_testing_Notebooks/llm_testing_notebooks.md b/docs/pages/tutorials/LLM_testing_Notebooks/llm_testing_notebooks.md
index aa0b67247..dcb763825 100644
--- a/docs/pages/tutorials/LLM_testing_Notebooks/llm_testing_notebooks.md
+++ b/docs/pages/tutorials/LLM_testing_Notebooks/llm_testing_notebooks.md
@@ -39,4 +39,6 @@ The following table gives an overview of the different tutorial notebooks to tes
| [**Sycophancy**](sycophancy) : It is an undesirable behavior in which models tailor their responses to align with a human user's view, even when that view is not objectively correct. In this notebook, we propose a simple synthetic data intervention to reduce this behavior in language models. | OpenAI | Question-Answering | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Sycophancy.ipynb) |
| [**Stereotype**](stereotype): In this tutorial, we assess the model on gender occupational stereotype statements. | OpenAI/AI21 | Question-Answering | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Wino_Bias_LLM.ipynb) |
| [**LM Studio**](lm_studio): Running Hugging Face quantized models through LM-Studio and testing these models for a Question Answering task. | LM Studio | Question-Answering | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/LM-Studio-Demo.ipynb) |
-| [**Question Answering Benchmarking**](question_answering_benchmarking): This notebook provides a demo on benchmarking Language Models (LLMs) for Question-Answering tasks. | Hugging Face Inference API | Question-Answering | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/benchmarks/Question-Answering.ipynb) |
\ No newline at end of file
+| [**Question Answering Benchmarking**](question_answering_benchmarking): This notebook provides a demo on benchmarking Language Models (LLMs) for Question-Answering tasks. | Hugging Face Inference API | Question-Answering | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/benchmarks/Question-Answering.ipynb) |
+| **Fewshot Model Evaluation**: This notebook provides a demo on Optimize and evaluate your models using few-shot prompt techniques | OpenAI | Question-Answering | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Fewshot_QA_Notebook.ipynb) |
+| **Evaluating NER in LLMs**:In this tutorial, we assess the support for Named Entity Recognition (NER) tasks specifically for Large Language Models (LLMs) | OpenAI | Question-Answering | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/NER%20Casual%20LLM.ipynb) |
\ No newline at end of file
diff --git a/docs/pages/tutorials/miscellaneous_notebooks/miscellaneous_notebooks.md b/docs/pages/tutorials/miscellaneous_notebooks/miscellaneous_notebooks.md
index 13c26c50e..530ffd633 100644
--- a/docs/pages/tutorials/miscellaneous_notebooks/miscellaneous_notebooks.md
+++ b/docs/pages/tutorials/miscellaneous_notebooks/miscellaneous_notebooks.md
@@ -39,3 +39,5 @@ The following table gives an overview of the different tutorial notebooks. In th
| **LangTestCallback**: In this section, we discussed how to utilize the LangTestCallback funtion while training an Text Classification transformers model. | Hugging Face | Text-Classification | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/HF_Callback_Text_Classification.ipynb) |
| **Multiple_dataset**: In this section, we discussed how to evaluate multiple datasets for a particular model. | OpenAI |Question-Answering | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/Multiple_dataset.ipynb) |
| **Generic API-Based Model**: In this section, we discussed how to test API-based models hosted using Ollama, vLLM, and other tools. | Web |Question-Answering | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/llm_notebooks/Generic_API-Based_Model_Testing_Demo.ipynb) |
+| **Data Augmenter**: In this Notebook, we can allows for streamlined and harness-free data augmentation, making it simpler to enhance your datasets and improve model robustness. | - |NER | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/Data_Augmenter_Notebook.ipynb) |
+| **Multi-Dataset Prompt Configs**: In this Notebook, we discussed about optimized prompt handling for multiple datasets, allowing users to add custom prompts for each dataset, enabling seamless integration and efficient testing. | OpenAI |Question-Answering | [](https://colab.research.google.com/github/JohnSnowLabs/langtest/blob/main/demo/tutorials/misc/MultiPrompt_MultiDataset.ipynb) |