feat: Add template tests by Yukiyukiyeah · Pull Request #9083 · langflow-ai/langflow

Yukiyukiyeah · 2025-07-17T04:53:56Z

This pull request introduces comprehensive testing and validation for starter project templates in the Langflow codebase. The changes include new workflows, validation utilities, and test cases to ensure the integrity and functionality of templates. Additionally, the validate_code function has been enhanced to include a custom execution context for Langflow-specific imports.

CI Workflow Enhancements:

Added a new test-templates job to the CI workflow to test starter project templates. This job validates templates for proper structure, security, and functionality using pytest. [1] [2]
Introduced a standalone template-tests.yml workflow triggered by changes to relevant files or via manual dispatch.

Validation Utilities:

Created src/backend/base/langflow/utils/template_validation.py to provide functions for validating template structure, flow building, code correctness, and execution.
Enhanced the validate_code function to use a custom execution context with Langflow-specific imports, improving its ability to evaluate template-related code. [1] [2]

Testing Improvements:

Added src/backend/tests/unit/template/test_starter_projects.py to perform comprehensive tests on all starter project templates, including JSON validity, structure, flow building, and execution.
Updated the Makefile with a new template_tests target to run all template tests.

Pre-commit Hook:

Introduced a new pre-commit hook to validate starter project templates for structural and security issues.This pull request introduces a comprehensive testing framework for validating starter project templates in the Langflow project. It includes automated workflows, pre-commit hooks, Makefile commands, and unit tests to ensure templates are valid, have the required structure, and are free from basic security issues.

Workflow and Automation Enhancements:

.github/workflows/template-tests.yml: Added a new GitHub Actions workflow to automatically test starter project templates on code pushes and pull requests. It sets up Python 3.12, installs dependencies, and runs pytest on the template test file.

Pre-commit Hook Updates:

.pre-commit-config.yaml: Added a new pre-commit hook (validate-starter-projects) to validate JSON templates in the starter_projects directory for correctness and security.

Makefile Integration:

Makefile: Added a template_tests target to run all starter project template tests using pytest.

Unit Test Implementation:

src/backend/tests/unit/template/test_starter_projects.py: Created a new test suite to validate starter project templates. It checks for valid JSON, required structure (nodes and edges), and basic security issues.

Documentation:

src/backend/tests/unit/template/__init__.py: Added a module-level docstring describing the purpose of the template testing module.

Summary by CodeRabbit

New Features
- Introduced comprehensive validation utilities and tests for starter project templates, ensuring structural integrity, buildability, and runtime execution.
- Added asynchronous validation against a remote API endpoint for enhanced template checking.
- Integrated a new Makefile target and pre-commit hook for automated template validation.
- Set up a dedicated GitHub Actions workflow to run template tests on relevant changes.
Tests
- Added extensive automated tests to verify the correctness and security of starter project templates.

coderabbitai · 2025-07-17T04:54:03Z

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Walkthrough

This update introduces comprehensive validation and testing for Langflow starter project templates. It adds a new validation utility module, a dedicated test suite for templates, a Makefile target, a pre-commit hook, and a GitHub Actions workflow. Existing code validation logic is improved with a custom execution context for function definitions.

Changes

File(s)	Change Summary
.github/workflows/template-tests.yml	Added a GitHub Actions workflow to run template tests on relevant changes.
.pre-commit-config.yaml	Added a pre-commit hook to validate starter project templates on JSON file changes.
Makefile	Added `template_tests` target to run starter project template tests.
src/backend/base/langflow/utils/template_validation.py	New module providing layered validation utilities for starter project templates (structure, build, endpoint, execution).
src/backend/base/langflow/utils/validate.py	Updated `validate_code` to use a custom execution context with Langflow and typing imports.
src/backend/tests/unit/template/init.py	Added module docstring for the new template test module.
src/backend/tests/unit/template/test_starter_projects.py	New test suite for validating, building, and executing starter project templates.

Sequence Diagram(s)

sequenceDiagram
    participant Dev as Developer
    participant PreCommit as Pre-commit Hook
    participant TestRunner as Test Suite
    participant Validation as Template Validation Utils
    participant API as Langflow API

    Dev->>PreCommit: Commit JSON template
    PreCommit->>Validation: validate_template_structure
    PreCommit->>Validation: validate_flow_can_build
    PreCommit->>Validation: validate_template_comprehensive_async
    PreCommit->>API: (async) validate_flow_endpoint
    PreCommit->>API: (async) validate_flow_execution
    PreCommit-->>Dev: Report errors if any

    Dev->>GitHub: Push/PR triggers workflow
    GitHub->>TestRunner: Run template tests
    TestRunner->>Validation: Run structure/build/endpoint/execution validations
    TestRunner->>API: (async) validate via API endpoints
    TestRunner-->>GitHub: Report test results

Suggested labels

lgtm

Suggested reviewers

erichare

✨ Finishing Touches

🧪 Generate unit tests

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch add-template-test

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
- I pushed a fix in commit <commit_id>, please review it.
- Explain this complex logic.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
- @coderabbitai explain this code block.
- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
- @coderabbitai read src/utils.ts and explain its main purpose.
- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
- @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai generate docstrings to generate docstrings for this PR.
@coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
@coderabbitai generate unit tests to generate unit tests for this PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

coderabbitai

Actionable comments posted: 2

🧹 Nitpick comments (4)

src/backend/tests/unit/template/test_starter_projects.py (1)
26-29: Consider making the starter projects path configurable.

The hardcoded path could break if the directory structure changes. Consider using environment variables or configuration files for better maintainability.
+import os
+
 def get_starter_projects_path() -> Path:
     """Get path to starter projects directory."""
-    return Path("src/backend/base/langflow/initial_setup/starter_projects")
+    base_path = os.environ.get(
+        "LANGFLOW_STARTER_PROJECTS_PATH",
+        "src/backend/base/langflow/initial_setup/starter_projects"
+    )
+    return Path(base_path)
src/backend/base/langflow/utils/template_validation.py (3)
15-49: Consider adding edge validation for completeness.

The function thoroughly validates nodes but doesn't validate edge structure. Edges should have source and target fields.

Add edge validation after line 40:
# Check edges have required fields
for i, edge in enumerate(data.get("edges", [])):
    if "source" not in edge:
        errors.append(f"{filename}: Edge {i} missing 'source'")
    if "target" not in edge:
        errors.append(f"{filename}: Edge {i} missing 'target'")
108-168: Well-implemented endpoint validation with thorough error checking!

The function properly validates code snippets via API with comprehensive error extraction. Consider using constants for HTTP status codes instead of magic numbers.

227-293: Consider making the event stream timeout configurable.

The hardcoded 10-second timeout might be insufficient for complex flows. Consider making it configurable.
-async def _validate_event_stream(response, job_id: str, filename: str, errors: list[str]) -> None:
+async def _validate_event_stream(
+    response, job_id: str, filename: str, errors: list[str], timeout: float = 10.0
+) -> None:
     """Validate the event stream from flow execution.
 
     Args:
         response: The response object with event stream
         job_id: The job ID to verify in events
         filename: Name of the template file for error reporting
         errors: List to append errors to
+        timeout: Maximum time to wait for event stream completion
     """
And update the call in validate_flow_execution:
-            await _validate_event_stream(events_response, job_id, filename, errors)
+            await _validate_event_stream(events_response, job_id, filename, errors, timeout=30.0)

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d9f3ced and bafdc9a.

📒 Files selected for processing (7)

.github/workflows/template-tests.yml (1 hunks)
.pre-commit-config.yaml (1 hunks)
Makefile (1 hunks)
src/backend/base/langflow/utils/template_validation.py (1 hunks)
src/backend/base/langflow/utils/validate.py (2 hunks)
src/backend/tests/unit/template/__init__.py (1 hunks)
src/backend/tests/unit/template/test_starter_projects.py (1 hunks)

🧰 Additional context used

📓 Path-based instructions (6)

src/backend/**/*.py

Instructions used from: