feat: Add Cleanlab's AI Reliability Bundle to Langflow

docs/docs/Integrations/Cleanlab/integrations-cleanlab.md

@@ -0,0 +1,145 @@
+---
+title: Integrate Cleanlab Evaluations with Langflow
+slug: /integrations-cleanlab
+---
+
+Unlock trustworthy Agentic, RAG, and LLM pipelines with Cleanlab's evaluation and remediation suite.
+
+[Cleanlab](https://www.cleanlab.ai/) adds automation and trust to every data point going in and every prediction coming out of AI and RAG solutions.
+
+This Langflow integration provides 3 modular components that assess and improve the **trustworthiness** of any LLM or RAG pipeline output, enabling critical oversight for safety-sensitive, enterprise, and production GenAI applications.
+
+Use this bundle to:
+- Quantify trustworthiness of ANY LLM response with a **0-1 score**
+- Explain why a response may be good or bad
+- Evaluate **context sufficiency**, **groundedness**, **helpfulness**, and **query clarity** with quantitative scores (for RAG/Agentic pipelines with context)
+- Remediate low-trust responses with warnings or fallback answers
+
+## Prerequisites
+
+Before using these components, you'll need:
+
+- A [Cleanlab API key](https://tlm.cleanlab.ai/)
+
+
+## Components
+
+### `CleanlabEvaluator`
+
+**Purpose:** Evaluate and explain the trustworthiness of a prompt + response pair using Cleanlab. More details on how the score works [here](https://help.cleanlab.ai/tlm/).
+
+#### Inputs
+
+| Name                  | Type       | Description                                                         |
+|-----------------------|------------|---------------------------------------------------------------------|
+| system_prompt         | Message    | (Optional) System message prepended to the prompt                   |
+| prompt                | Message    | The user-facing input to the LLM                                    |
+| response              | Message    | OpenAI's, Claude, etc. model's response to evaluate                 |
+| cleanlab_api_key               | Secret     | Your Cleanlab API key                                               |
+| cleanlab_evaluation_model                 | Dropdown   | Evaluation model used by Cleanlab (GPT-4, Claude, etc.) This does not need to be the same model that generated the response.                               |
+| quality_preset        | Dropdown   | Tradeoff between evaluation speed and accuracy                      |
+
+#### Outputs
+
+| Name                  | Type       | Description                                                         |
+|-----------------------|------------|---------------------------------------------------------------------|
+| score                 | number     | Trust score between 0–1                                             |
+| explanation           | Message    | Explanation of the trust score                                      |
+| response              | Message    | Returns the original response for easy chaining to `CleanlabRemediator` component                          |
+
+---
+
+### `CleanlabRemediator`
+
+**Purpose:** Use the trust score from the `CleanlabEvaluator` component to determine whether to show, warn about, or replace an LLM response. This component has configurables for the score threshold, warning text, and fallback message which you can customize as needed.
+
+#### Inputs
+
+| Name                        | Type       | Description                                                                 |
+|-----------------------------|------------|-----------------------------------------------------------------------------|
+| response                    | Message    | The response to potentially remediate                                      |
+| score                       | number     | Trust score from `CleanlabEvaluator`                                       |
+| explanation                 | Message    | (Optional) Explanation to append if warning is shown                       |
+| threshold                   | float      | Minimum trust score to pass response unchanged                             |
+| show_untrustworthy_response| bool       | Show original response with warning if untrustworthy                       |
+| untrustworthy_warning_text | Prompt     | Warning text for untrustworthy responses                                   |
+| fallback_text              | Prompt     | Fallback message if response is hidden                                     |
+
+#### Output
+
+| Name                  | Type       | Description                                                                 |
+|-----------------------|------------|-----------------------------------------------------------------------------|
+| remediated_response   | Message    | Final message shown to user after remediation logic                         |
+
+
+See example outputs below!
+
+---
+
+### `CleanlabRAGEvaluator`
+
+**Purpose:** Comprehensively evaluate RAG and LLM pipeline outputs by analyzing the context, query, and response quality using Cleanlab. This component assesses trustworthiness, context sufficiency, response groundedness, helpfulness, and query ease. Learn more about Cleanlab's evaluation metrics [here](https://help.cleanlab.ai/tlm/use-cases/tlm_rag/). You can also use the `CleanlabRemediator` component with this one to remediate low-trust responses coming from the RAG pipeline.
+
+#### Inputs
+
+| Name                     | Type      | Description                                                                |
+|--------------------------|-----------|----------------------------------------------------------------------------|
+| cleanlab_api_key                  | Secret    | Your Cleanlab API key                                                      |
+| cleanlab_evaluation_model                    | Dropdown  | Evaluation model used by Cleanlab (GPT-4, Claude, etc.) This does not need to be the same model that generated the response.                               |
+| quality_preset           | Dropdown  | Tradeoff between evaluation speed and accuracy                                                |
+| context                  | Message   | Retrieved context from your RAG system                                     |
+| query                    | Message   | The original user query                                                    |
+| response                 | Message   | OpenAI's, Claude, etc. model's response based on the context and query                              |
+| run_context_sufficiency  | bool      | Evaluate whether context supports answering the query                      |
+| run_response_groundedness| bool      | Evaluate whether the response is grounded in the context                   |
+| run_response_helpfulness | bool      | Evaluate how helpful the response is                                       |
+| run_query_ease           | bool      | Evaluate if the query is vague, complex, or adversarial                    |
+
+#### Outputs
+
+| Name                  | Type       | Description                                                                 |
+|-----------------------|------------|-----------------------------------------------------------------------------|
+| trust_score           | number     | Overall trust score                                                         |
+| trust_explanation     | Message    | Explanation for trust score                                                 |
+| other_scores          | dict       | Dictionary of optional enabled RAG evaluation metrics                       |
+| evaluation_summary    | Message    | Markdown summary of query, context, response, and evaluation results        |
+
+---
+
+## Example Flows
+
+The following example flows show how to use the `CleanlabEvaluator` and `CleanlabRemediator` components to evaluate and remediate responses from any LLM, and how to use the `CleanlabRAGEvaluator` component to evaluate RAG pipeline outputs.
+
+### Evaluate and remediate responses from any LLM
+
+[Download](./eval_and_remediate_cleanlab.json) the flow to follow along!
+
+This flow evaluates and remediates the trustworthiness of a response from any LLM using the `CleanlabEvaluator` and `CleanlabRemediator` components.
+
+![Evaluate response trustworthiness](./eval_response.png)
+
+Simply connect the `Message` output from any LLM component (like OpenAI, Anthropic, or Google) to the `response` input of the `CleanlabEvaluator` component, along with connecting your prompt to its `prompt` input.
+
+That's it! The `CleanlabEvaluator` component will return a trust score and explanation which you can use however you'd like.
+
+The `CleanlabRemediator` component uses this trust score and user configurable settings to determine whether to output the original response, warn about it, or replace it with a fallback answer.
+
+The example below shows a response that was determined to be untrustworthy (score of .09) and flagged with a warning by the `CleanlabRemediator` component.
+
+![CleanlabRemediator Example](./cleanlab_remediator_example.png)
+
+If you don't want to show untrustworthy responses, you can also configure the `CleanlabRemediator` to replace the response with a fallback message.
+
+![CleanlabRemediator Example](./cleanlab_remediator_example_fallback.png)
+
+### Evaluate RAG pipeline
+
+The below flow is the `Vector Store RAG` example template, with the `CleanlabRAGEvaluator` component added to evaluate the context, query, and response. You can use the `CleanlabRAGEvaluator` with any flow that has a context, query, and response. Simply connect the `context`, `query`, and `response` outputs from any RAG pipeline to the `CleanlabRAGEvaluator` component.
+
+![Evaluate RAG pipeline](./eval_rag.png)
+
+Here is an example of the `Evaluation Summary` output from the `CleanlabRAGEvaluator` component.
+
+![Evaluate RAG pipeline](./eval_summary_rag.png)
+
+Notice how the `Evaluation Summary` includes the query, context, response, and all the evaluation results! In this example, the `Context Sufficiency` and `Response Groundedness` scores are low (0.002) because the context doesn't contain information about the query and the response is not grounded in the context.

docs/sidebars.js

@@ -237,6 +237,11 @@ module.exports = {
           id: "Integrations/Composio/integrations-composio",
           label: "Composio",
         },
+        {
+          type: "doc",
+          id: "Integrations/Cleanlab/integrations-cleanlab",
+          label: "Cleanlab",
+        },
         {
           type: 'category',
           label: 'Google',

pyproject.toml

@@ -124,6 +124,7 @@ dependencies = [
     "langchain-ibm>=0.3.8",
     "opik>=1.6.3",
     "openai>=1.68.2",
+    "cleanlab-tlm>=1.1.2",
     'gassist>=0.0.1; sys_platform == "win32"',
     "twelvelabs>=0.4.7",
 ]

src/backend/base/langflow/components/cleanlab/__init__.py

@@ -0,0 +1,5 @@
+from .cleanlab_evaluator import CleanlabEvaluator
+from .cleanlab_rag_evaluator import CleanlabRAGEvaluator
+from .cleanlab_remediator import CleanlabRemediator
+
+__all__ = ["CleanlabEvaluator", "CleanlabRAGEvaluator", "CleanlabRemediator"]

src/backend/base/langflow/components/cleanlab/cleanlab_evaluator.py

@@ -0,0 +1,145 @@
+from cleanlab_tlm import TLM
+
+from langflow.custom import Component
+from langflow.io import (
+    DropdownInput,
+    MessageTextInput,
+    Output,
+    SecretStrInput,
+)
+from langflow.schema.message import Message
+
+
+class CleanlabEvaluator(Component):
+    """A component that evaluates the trustworthiness of LLM responses using Cleanlab.
+
+    This component takes a prompt and response pair, along with optional system instructions,
+    and uses Cleanlab's evaluation algorithms to generate a trust score and explanation.
+
+    Inputs:
+        - system_prompt (MessageTextInput): Optional system-level instructions prepended to the user prompt.
+        - prompt (MessageTextInput): The user's prompt or query sent to the LLM.
+        - response (MessageTextInput): The response generated by the LLM to be evaluated. This should come from the
+          LLM component, i.e. OpenAI, Gemini, etc.
+        - api_key (SecretStrInput): Your Cleanlab API key.
+        - model (DropdownInput): The model used by Cleanlab to evaluate the response (can differ from the
+          generation model).
+        - quality_preset (DropdownInput): Tradeoff setting for accuracy vs. speed and cost. Higher presets are
+          slower but more accurate.
+
+    Outputs:
+        - response_passthrough (Message): The original response, passed through for downstream use.
+        - score (number): A float between 0 and 1 indicating Cleanlab's trustworthiness score for the response.
+        - explanation (Message): A textual explanation of why the response received its score.
+
+    This component works well in conjunction with the CleanlabRemediator to create a complete trust evaluation
+    and remediation pipeline.
+
+    More details on the evaluation metrics can be found here: https://help.cleanlab.ai/tlm/tutorials/tlm/
+    """
+
+    display_name = "Cleanlab Evaluator"
+    description = "Evaluates any LLM response using Cleanlab and outputs trust score and explanation."
+    icon = "Cleanlab"
+    name = "CleanlabEvaluator"
+
+    inputs = [
+        MessageTextInput(
+            name="system_prompt",
+            display_name="System Message",
+            info="System-level instructions prepended to the user query.",
+            value="",
+        ),
+        MessageTextInput(
+            name="prompt",
+            display_name="Prompt",
+            info="The user's query to the model.",
+            required=True,
+        ),
+        MessageTextInput(
+            name="response",
+            display_name="Response",
+            info="The response to the user's query.",
+            required=True,
+        ),
+        SecretStrInput(
+            name="api_key",
+            display_name="Cleanlab API Key",
+            info="Your Cleanlab API key.",
+            required=True,
+        ),
+        DropdownInput(
+            name="model",
+            display_name="Cleanlab Evaluation Model",
+            options=[
+                "gpt-4.1",
+                "gpt-4.1-mini",
+                "gpt-4.1-nano",
+                "o4-mini",
+                "o3",
+                "gpt-4.5-preview",
+                "gpt-4o-mini",
+                "gpt-4o",
+                "o3-mini",
+                "o1",
+                "o1-mini",
+                "gpt-4",
+                "gpt-3.5-turbo-16k",
+                "claude-3.7-sonnet",
+                "claude-3.5-sonnet-v2",
+                "claude-3.5-sonnet",
+                "claude-3.5-haiku",
+                "claude-3-haiku",
+                "nova-micro",
+                "nova-lite",
+                "nova-pro",
+            ],
+            info="The model Cleanlab uses to evaluate the response. This does NOT need to be the same model that "
+            "generated the response.",
+            value="gpt-4o-mini",
+            required=True,
+            advanced=True,
+        ),
+        DropdownInput(
+            name="quality_preset",
+            display_name="Quality Preset",
+            options=["base", "low", "medium", "high", "best"],
+            value="medium",
+            info="This determines the accuracy, latency, and cost of the evaluation. Higher quality is generally "
+            "slower but more accurate.",
+            required=True,
+            advanced=True,
+        ),
+    ]
+
+    outputs = [
+        Output(display_name="Response", name="response_passthrough", method="pass_response", types=["Message"]),
+        Output(display_name="Trust Score", name="score", method="get_score", types=["number"]),
+        Output(display_name="Explanation", name="explanation", method="get_explanation", types=["Message"]),
+    ]
+
+    def _evaluate_once(self):
+        if not hasattr(self, "_cached_result"):
+            full_prompt = f"{self.system_prompt}\n\n{self.prompt}" if self.system_prompt else self.prompt
+            tlm = TLM(
+                api_key=self.api_key,
+                options={"log": ["explanation"], "model": self.model},
+                quality_preset=self.quality_preset,
+            )
+            self._cached_result = tlm.get_trustworthiness_score(full_prompt, self.response)
+        return self._cached_result
+
+    def get_score(self) -> float:
+        result = self._evaluate_once()
+        score = result.get("trustworthiness_score", 0.0)
+        self.status = f"Trust score: {score:.2f}"
+        return score
+
+    def get_explanation(self) -> Message:
+        result = self._evaluate_once()
+        explanation = result.get("log", {}).get("explanation", "No explanation returned.")
+        return Message(text=explanation)
+
+    def pass_response(self) -> Message:
+        self.status = "Passing through response."
+        return Message(text=self.response)