Release/1.3.0

.github/workflows/llm_tests_build.yml

@@ -0,0 +1,45 @@
+name: Manual Tests
+
+# This allows you to run the workflow manually from the GitHub Actions UI. We use it for running llm-tests.
+on: 
+  workflow_dispatch:
+
+jobs:
+  manual_tests:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: [ "3.8", "3.9", "3.10" ]
+
+    steps:
+      - uses: actions/checkout@v3
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v3
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - uses: snok/install-poetry@v1
+        with:
+          version: 1.3.1
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+          installer-parallel: true
+
+      - name: Load cached venv
+        id: cached-poetry-dependencies
+        uses: actions/cache@v3
+        with:
+          path: .venv
+          key: venv-${{ runner.os }}-${{ steps.setup-python.outputs.python-version }}-${{ hashFiles('**/poetry.lock') }}
+
+      - name: Install dependencies
+        if: steps.cached-poetry-dependencies.outputs.cache-hit != 'true'
+        run: |
+          poetry install --with dev --all-extras
+
+      - name: Run manual_tests
+        env: 
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+        run: |
+          poetry run pytest manual_tests/

CONTRIBUTING.md

@@ -0,0 +1,167 @@
+# Contributing to LangTest
+
+Thank you for your interest in contributing to LangTest! We value the contributions of our community members and are excited to have you on board. . To make the development journey smooth and collaborative, please adhere to the following steps:
+- Thoroughly review the guidelines provided below.
+- If you're eager to add a new feature to the LangTest library, navigate to the [documentation](https://langtest.org/docs/pages/docs/contribute) for a clear understanding.
+- To illustrate the process, we've included a simple example in the documentation to guide you.
+
+## Finding an Issue
+
+1. **Explore Our Issues:**
+Browse through our list of open issues on the [Issues tab](https://github.com/YourUsername/LangTest/issues). These issues cover a wide range of topics and difficulties. You might find something that aligns with your skills and interests.
+
+2. **Filter and Search:**
+Use the provided labels and filters to narrow down the list of issues. You can filter by tags like `feature`, `bug`, `fix`, etc. Also, utilize the GitHub search bar to look for keywords related to your expertise.
+
+3. **Read and Understand:**
+Once you've found an issue that captures your attention, take a moment to thoroughly read and understand it. Make sure you comprehend the problem, the expected solution, and any existing discussion around it.
+
+4. **Clarify Doubts:**
+If you have any questions or need further clarification, don't hesitate to comment on the issue. Our community is friendly and always willing to help. It's better to seek clarity before diving in.
+
+5. **Claim the Issue:**
+If you feel confident about tackling the issue, leave a comment expressing your intention to work on it. This prevents duplication of efforts and allows us to provide guidance if needed.
+
+6. **Understand Contribution Guidelines:**
+Familiarize yourself with our [Contribution Guidelines](https://github.com/JohnSnowLabs/langtest/wiki). This document outlines coding standards, pull request procedures, and other important details.
+
+## Contribution Process
+
+<img align="right" width="300" src="https://github.com/RakshitKhajuria/first-contributions/assets/71117423/be22d54d-5b62-4a23-b213-0268ed195021" alt="fork this repository" />
+
+1. ### Fork this repository
+
+Fork this repository by clicking on the fork button on the top of this page.
+This will create a copy of this repository in your account.
+
+2. ### Clone the repository
+
+<img align="right" width="300" src="https://github.com/RakshitKhajuria/test/assets/71117423/b07c7d60-ab80-4b7f-b972-58fcd1f50741" alt="clone this repository" />
+
+Now clone the forked repository to your machine. Go to your GitHub account, open the forked repository, click on the code button and then click the _copy to clipboard_ icon.
+
+Open a terminal and run the following git command:
+
+```
+git clone "url you just copied"
+```
+
+where "url you just copied" (without the quotation marks) is the url to this repository (your fork of this project). See the previous steps to obtain the url.
+
+<img align="right" width="300" src="https://github.com/RakshitKhajuria/test/assets/71117423/8eef9f61-283d-4bad-9e7f-8a526e33615e" alt="copy URL to clipboard" />
+
+For example:
+
+```
+git clone https://github.com/username/langtest.git
+```
+
+where `username` is your GitHub username.
+
+3. ### Create a branch
+
+Change to the repository directory on your computer (if you are not already there):
+
+```
+cd langtest
+```
+
+Now create a branch using the `git checkout -b` command:
+
+```
+git checkout -b your-branch-name
+```
+
+4. **Set Up Environment**: Create a virtual environment for LangTest using your preferred method. For example, you can use `venv` or `conda`.
+
+5. ### Install Poetry
+
+Poetry is the package manager used for this project. If you don't have Poetry installed, run the following command:
+
+   ```bash
+   pip install poetry
+   ```
+
+6. ### Install Dependencies
+
+Use Poetry to install the project dependencies:
+
+   ```bash
+   poetry install --with dev
+   ```
+
+7. ### Make necessary changes and commit those changes 
+
+Add those changes to the branch you just created using the `git add` command:
+
+```
+git add .
+```
+Now commit those changes using the `git commit` command with a meaningful message:
+
+```
+git commit -m "Add feature: feature-name" 
+```
+
+8. ### Push changes to GitHub
+
+Push your changes using the command `git push`:
+
+```
+git push -u origin your-branch-name
+```
+
+replacing `your-branch-name` with the name of the branch you created earlier.
+
+9. ### Submit your changes for review
+
+If you go to your repository on GitHub, you'll see a `Compare & pull request` button. Click on that button.
+
+<img style="float: right;" src="https://github.com/RakshitKhajuria/test/assets/71117423/86cb56af-1dad-467e-afc1-f4594615783b" alt="create a pull request" />
+
+10. ### Create a pull request (PR)
+
+Create a pull request (PR) from your forked repository to the main LangTest repository.
+- Clearly describe the changes you've made and the issue it addresses in the PR description.
+- If the PR is related to an existing issue, mention it using the syntax "Fixes #issue_number."
+> For pull request template click [here](https://github.com/JohnSnowLabs/langtest/blob/main/PULL_REQUEST_TEMPLATE.md)
+
+<img style="float: right;" src="https://github.com/RakshitKhajuria/test/assets/71117423/5c629508-53a7-4444-b036-15f694df675c" alt="submit pull request" />
+
+Our team will review your pull request, provide feedback, and work with you to ensure that your contribution meets the project's guidelines and quality standards
+
+## Proposing a New Issue
+
+1. If you have an idea for a new feature, improvement, or bug fix, you can propose a new issue.
+2. Go to the [Issue Tracker](https://github.com/JohnSnowLabs/langtest/issues) and click on "New Issue" or just create a new issue from [here](https://github.com/JohnSnowLabs/langtest/issues/new) 
+3. Choose the appropriate issue lables (feature, fix, bug, etc.).
+4. Provide a clear and concise title and description of the proposed issue.
+5. Add labels and assignees, if applicable, to help categorize and track the issue.
+
+We're excited to have you on board with us! Together, we can make NLP models safer, more reliable, and fairer for everyone. 
+
+### **Happy contributing!** 💫
+
+## Where to go from here?
+
+Congrats! You just completed the standard fork -> clone -> edit -> pull request workflow that you'll often encounter as a contributor!
+
+Celebrate your contribution and share it with your friends and followers by going to [LinkedIn](www.linkedin.com).
+
+You can join our community team if you need any help or have any questions. 
+
+- [Slack](https://www.johnsnowlabs.com/slack-redirect/) For live discussion with the LangTest community, join the `#langtest` channel
+- [GitHub](https://github.com/JohnSnowLabs/langtest/tree/main) For bug reports, feature requests, and contributions
+- [Discussions](https://github.com/JohnSnowLabs/langtest/discussions) To engage with other community members, share ideas, and show off how you use LangTest!
+
+## Contributors
+
+We would like to acknowledge all contributors of this open-source community project. 
+
+<a href="https://github.com/johnsnowlabs/langtest/graphs/contributors">
+  <img src="https://contrib.rocks/image?repo=johnsnowlabs/langtest" />
+</a>
+
+## License
+
+LangTest is released under the [Apache License 2.0](https://github.com/JohnSnowLabs/langtest/blob/main/LICENSE), which guarantees commercial use, modification, distribution, patent use, private use and sets limitations on trademark use, liability and warranty.

demo/tutorials/end-to-end-notebooks/HuggingFace_Real_World_Notebook.ipynb

@@ -111,13 +111,14 @@
         "<br/>\n",
         "\n",
         "\n",
-        "| Parameter  | Description |  \n",
-        "| - | - | \n",
-        "|**task**     |Task for which the model is to be evaluated (text-classification or ner)|\n",
-        "|**model**     |PipelineModel or path to a saved model or pretrained pipeline/model from hub.\n",
-        "|**data**       |Path to the data that is to be used for evaluation. Can be .csv or .conll file in the CoNLL format \n",
-        "|**config**     |Configuration for the tests to be performed, specified in form of a YAML file.\n",
-        "|**hub**       |model hub to load from the path. Required if model param is passed as path.|\n",
+        "\n",
+        "| Parameter     | Description |\n",
+        "| ------------- | ----------- |\n",
+        "| **task**      | Task for which the model is to be evaluated (text-classification or ner) |\n",
+        "| **model**     | Specifies the model(s) to be evaluated. Can be a dictionary or a list of dictionaries. Each dictionary should contain 'model' and 'hub' keys. If a path is specified, the dictionary must contain 'model' and 'hub' keys. |\n",
+        "| **data**      | The data to be used for evaluation. A dictionary providing flexibility and options for data sources. It should include the following keys: <ul><li>data_source (mandatory): The source of the data.</li><li>subset (optional): The subset of the data.</li><li>feature_column (optional): The column containing the features.</li><li>target_column (optional): The column containing the target labels.</li><li>split (optional): The data split to be used.</li></ul> |\n",
+        "| **config**    | Configuration for the tests to be performed, specified in the form of a YAML file. |\n",
+        "\n",
         "\n",
         "<br/>\n",
         "<br/>"
@@ -549,7 +550,8 @@
       },
       "outputs": [],
       "source": [
-        "h = Harness(task=\"ner\", model=\"trained_model\", hub=\"huggingface\", data=\"sample.conll\")"
+        "\n",
+        "h = Harness(task=\"ner\", model={\"model\": \"trained_model\", \"hub\": \"huggingface\"}, data={\"data_source\" :\"sample.conll\"})"
       ]
     },
     {

demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Custom_Pipeline_Notebook.ipynb

@@ -109,13 +109,14 @@
         "<br/>\n",
         "\n",
         "\n",
-        "| Parameter  | Description |\n",
-        "| - | - |\n",
-        "|**task**     |Task for which the model is to be evaluated (text-classification or ner)|\n",
-        "|**model**     |PipelineModel or path to a saved model or pretrained pipeline/model from hub.\n",
-        "|**data**       |Path to the data that is to be used for evaluation. Can be .csv or .conll file in the CoNLL format\n",
-        "|**config**     |Configuration for the tests to be performed, specified in form of a YAML file.\n",
-        "|**hub**       |model hub to load from the path. Required if model param is passed as path.|\n",
+        "\n",
+        "| Parameter     | Description |\n",
+        "| ------------- | ----------- |\n",
+        "| **task**      | Task for which the model is to be evaluated (text-classification or ner) |\n",
+        "| **model**     | Specifies the model(s) to be evaluated. Can be a dictionary or a list of dictionaries. Each dictionary should contain 'model' and 'hub' keys. If a path is specified, the dictionary must contain 'model' and 'hub' keys. |\n",
+        "| **data**      | The data to be used for evaluation. A dictionary providing flexibility and options for data sources. It should include the following keys: <ul><li>data_source (mandatory): The source of the data.</li><li>subset (optional): The subset of the data.</li><li>feature_column (optional): The column containing the features.</li><li>target_column (optional): The column containing the target labels.</li><li>split (optional): The data split to be used.</li></ul> |\n",
+        "| **config**    | Configuration for the tests to be performed, specified in the form of a YAML file. |\n",
+        "\n",
         "\n",
         "<br/>\n",
         "<br/>"
@@ -409,7 +410,7 @@
       },
       "outputs": [],
       "source": [
-        "harness = Harness(task=\"ner\", model=ner_model, data=\"sample.conll\")"
+        "harness = Harness(task=\"ner\", model={\"model\": ner_model, \"hub\": \"johnsnowlabs\"}, data={\"data_source\" :\"sample.conll\"})"
       ]
     },
     {

demo/tutorials/end-to-end-notebooks/JohnSnowLabs_RealWorld_Notebook.ipynb

@@ -109,13 +109,14 @@
         "<br/>\n",
         "\n",
         "\n",
-        "| Parameter  | Description |  \n",
-        "| - | - | \n",
-        "|**task**     |Task for which the model is to be evaluated (text-classification or ner)|\n",
-        "|**model**     |PipelineModel or path to a saved model or pretrained pipeline/model from hub.\n",
-        "|**data**       |Path to the data that is to be used for evaluation. Can be .csv or .conll file in the CoNLL format \n",
-        "|**config**     |Configuration for the tests to be performed, specified in form of a YAML file.\n",
-        "|**hub**       |model hub to load from the path. Required if model param is passed as path.|\n",
+        "\n",
+        "| Parameter     | Description |\n",
+        "| ------------- | ----------- |\n",
+        "| **task**      | Task for which the model is to be evaluated (text-classification or ner) |\n",
+        "| **model**     | Specifies the model(s) to be evaluated. Can be a dictionary or a list of dictionaries. Each dictionary should contain 'model' and 'hub' keys. If a path is specified, the dictionary must contain 'model' and 'hub' keys. |\n",
+        "| **data**      | The data to be used for evaluation. A dictionary providing flexibility and options for data sources. It should include the following keys: <ul><li>data_source (mandatory): The source of the data.</li><li>subset (optional): The subset of the data.</li><li>feature_column (optional): The column containing the features.</li><li>target_column (optional): The column containing the target labels.</li><li>split (optional): The data split to be used.</li></ul> |\n",
+        "| **config**    | Configuration for the tests to be performed, specified in the form of a YAML file. |\n",
+        "\n",
         "\n",
         "<br/>\n",
         "<br/>"
@@ -242,7 +243,7 @@
       },
       "outputs": [],
       "source": [
-        "harness = Harness(task=\"ner\", model=ner_model, data=\"sample.conll\", hub=\"johnsnowlabs\")"
+        "harness = Harness(task=\"ner\", model={\"model\": ner_model, \"hub\": \"johnsnowlabs\"}, data={\"data_source\" :\"sample.conll\"})"
       ]
     },
     {
@@ -1284,7 +1285,7 @@
         }
       ],
       "source": [
-        "harness = Harness.load(\"saved_test_configurations\",model=augmented_ner_model)"
+        "harness = Harness.load(\"saved_test_configurations\",model=augmented_ner_model,task=\"ner\")"
       ]
     },
     {

demo/tutorials/end-to-end-notebooks/Spacy_Real_World_Notebook.ipynb

@@ -89,13 +89,14 @@
         "<br/>\n",
         "\n",
         "\n",
-        "| Parameter  | Description |  \n",
-        "| - | - | \n",
-        "|**task**     |Task for which the model is to be evaluated (text-classification or ner)|\n",
-        "|**model**     |PipelineModel or path to a saved model or pretrained pipeline/model from hub.\n",
-        "|**data**       |Path to the data that is to be used for evaluation. Can be .csv or .conll file in the CoNLL format \n",
-        "|**config**     |Configuration for the tests to be performed, specified in form of a YAML file.\n",
-        "|**hub**       |model hub to load from the path. Required if model param is passed as path.|\n",
+        "\n",
+        "| Parameter     | Description |\n",
+        "| ------------- | ----------- |\n",
+        "| **task**      | Task for which the model is to be evaluated (text-classification or ner) |\n",
+        "| **model**     | Specifies the model(s) to be evaluated. Can be a dictionary or a list of dictionaries. Each dictionary should contain 'model' and 'hub' keys. If a path is specified, the dictionary must contain 'model' and 'hub' keys. |\n",
+        "| **data**      | The data to be used for evaluation. A dictionary providing flexibility and options for data sources. It should include the following keys: <ul><li>data_source (mandatory): The source of the data.</li><li>subset (optional): The subset of the data.</li><li>feature_column (optional): The column containing the features.</li><li>target_column (optional): The column containing the target labels.</li><li>split (optional): The data split to be used.</li></ul> |\n",
+        "| **config**    | Configuration for the tests to be performed, specified in the form of a YAML file. |\n",
+        "\n",
         "\n",
         "<br/>\n",
         "<br/>"
@@ -416,7 +417,7 @@
       },
       "outputs": [],
       "source": [
-        "h = Harness(task=\"ner\",model=spacy_model, data=\"/content/sample.conll\")"
+        "h = Harness(task=\"ner\",model={\"model\": spacy_model, \"hub\": \"spacy\"}, data={\"data_source\": \"/content/sample.conll\"})"
       ]
     },
     {