feat: add static components index to avoid rebuild on startup#10181
Conversation
This script generates a prebuilt index of all built-in components in the lfx.components package, saving it as a JSON file for quick loading at runtime. It includes versioning and integrity verification through SHA256 hashing.
- Bump revision to 3 in uv.lock. - Update dependency markers for several packages to improve compatibility with Python versions and platforms. - Increment versions for langflow (1.6.4) and langflow-base (0.6.4). - Adjust dependency markers for packages related to darwin platform to enhance specificity.
- Added entry for user-specific component index cache directory to .gitignore. - Included member_servers.json in the ignore list for better file management.
- Introduced functions to detect development mode and read a custom component index from a specified path or URL. - Added caching mechanism for dynamically generated component indices to improve performance. - Updated `import_langflow_components` to utilize the new index reading and caching logic, allowing for faster startup in production mode. - Added `components_index_path` to settings for user-defined index configuration.
…ndex - Introduced a new workflow that triggers on pull requests and manual dispatch to update the component index. - The workflow checks for changes in the component index and commits updates if necessary. - Added a comment feature to notify users when the component index is updated.
ogabrielluiz
requested review from
Adam-Aghili,
edwinjosechittilappilly and
jordanrfrazier
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughAdds a GitHub Actions workflow to build and push a prebuilt component index. Introduces a Python script to generate the index JSON with hashing. Extends components loading to prefer a prebuilt/cached index with fallbacks and a new settings field for custom index paths. Updates .gitignore and bumps dependencies in starter project JSONs. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant App as Application
participant IF as ComponentsInterface
participant SS as SettingsService
participant IDX as Prebuilt Index (JSON)
participant C as Cache (~/.cache/lfx)
participant DL as Dynamic Loader
participant CC as Custom Components
App->>IF: import_langflow_components(settings_service?)
IF->>SS: get components_index_path
alt Non-dev & index path provided
IF->>IDX: Read index (file/URL)
IDX-->>IF: Entries + sha256 + version
IF->>IF: Validate sha256/version
else Non-dev & no path
IF->>IDX: Read built-in index
IDX-->>IF: Entries + sha256 + version
IF->>IF: Validate sha256/version
else Dev mode or index missing/invalid
IF->>C: Try load cached index
alt Cache hit & valid
C-->>IF: Cached entries
else Cache miss/invalid
IF->>DL: Discover & import modules
DL-->>IF: Built entries
IF->>C: Save generated index to cache
end
end
IF->>CC: Load/merge custom components
IF-->>App: Combined components map
sequenceDiagram
autonumber
actor Dev as Developer
participant GH as GitHub Actions
participant Repo as Repository
participant UV as uv (package manager)
participant Script as build_component_index.py
Dev->>Repo: Open PR (changes in components or script)
GH->>Repo: checkout
GH->>UV: setup-uv + uv sync
GH->>Script: uv run scripts/build_component_index.py
Script-->>Repo: Write component_index.json
GH->>Repo: Detect changes to index
alt Changes detected
GH->>Repo: Commit "update component_index.json [skip ci]" and push
GH->>Dev: Comment on PR (index updated)
else No changes
GH-->>Dev: No action
end
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Suggested labels
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 error, 1 warning, 1 inconclusive)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
enhancement
Codecov Report❌ Patch coverage is
❌ Your project status has failed because the head coverage (48.60%) is below the target coverage (55.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #10181 +/- ##
==========================================
- Coverage 24.76% 24.73% -0.04%
==========================================
Files 1090 1090
Lines 40108 40117 +9
Branches 5550 5550
==========================================
- Hits 9934 9921 -13
- Misses 30003 30025 +22
Partials 171 171
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/lfx/src/lfx/interface/components.py (1)
407-417: Schema mismatch breaks downstream lookups.
component_cache.all_types_dictis set to a flat type->components map, but other functions (e.g., get_type_dict, ensure_component_loaded) expect a top-level "components" wrapper. This causes incorrect merges and lookups.Normalize to a single shape with "components" at the top, and flatten custom dicts if needed:
- langflow_components = await import_langflow_components(settings_service) - custom_components_dict = await _determine_loading_strategy(settings_service) - - # merge the dicts - component_cache.all_types_dict = { - **langflow_components["components"], - **custom_components_dict, - } - component_count = sum(len(comps) for comps in component_cache.all_types_dict.values()) - await logger.adebug(f"Loaded {component_count} components") + langflow_components = await import_langflow_components(settings_service) + custom_components_dict = await _determine_loading_strategy(settings_service) + + # Flatten custom dict if it already has a "components" wrapper + custom_flat = custom_components_dict.get("components", custom_components_dict) or {} + + merged = { + **langflow_components["components"], + **custom_flat, + } + + component_cache.all_types_dict = {"components": merged} + component_count = sum(len(comps) for comps in merged.values()) + await logger.adebug(f"Loaded {component_count} components")This keeps existing callers that read via ["components"] working and prevents accidental nesting.
🧹 Nitpick comments (6)
.github/workflows/update-component-index.yml (2)
18-23: Checkoutrefis unsafe for non-PR events.
${{ github.head_ref }}is empty forworkflow_dispatch, which can lead to ambiguous checkout. Prefer conditional checkout or omitrefoutside PRs.Use two steps:
- For PRs:
- name: Checkout repository (PR) if: github.event_name == 'pull_request' uses: actions/checkout@v4 with: ref: ${{ github.head_ref }} token: ${{ secrets.GITHUB_TOKEN }}
- For others:
- name: Checkout repository if: github.event_name != 'pull_request' uses: actions/checkout@v4
32-34: Force dynamic rebuild during CI.Ensure the script doesn’t load an existing index by setting dev mode for the build step.
- - name: Build component index - run: uv run python scripts/build_component_index.py + - name: Build component index + env: + LFX_DEV: "1" + run: uv run python scripts/build_component_index.pysrc/lfx/src/lfx/interface/components.py (2)
312-319: Docstring/package name mismatch.Comments reference
langflow.components...while the package prefix islfx.components. Update to avoid confusion.- # Extract the top-level subpackage name after "langflow.components." - # e.g., "langflow.components.Notion.add_content_to_page" -> "Notion" + # Extract the top-level subpackage name after "lfx.components." + # e.g., "lfx.components.Notion.add_content_to_page" -> "Notion"
64-133: Optional: make remote index fetching resilient when httpx isn’t installed.If users set a remote URL but don’t have httpx installed, this silently returns None. Provide a clear error or fallback to stdlib urllib.
- Catch ModuleNotFoundError for httpx and log a precise message.
- Optionally use urllib.request as a fallback to read the URL.
- Consider relaxing strict version matching for custom paths (warn instead of discard), if admins intentionally pin cross-version indices.
scripts/build_component_index.py (2)
41-47: Ensure a fresh build (not from prebuilt index).
import_langflow_components()will load the prebuilt/cached index unless dev mode is active. To guarantee a dynamic rebuild here, setLFX_DEV=1in the workflow (recommended) or set a default in the script.If you prefer enforcing in-script:
def build_component_index(): """Build the component index by scanning all modules in lfx.components. @@ - try: + try: + import os + os.environ.setdefault("LFX_DEV", "1") import asyncioWe already suggested setting
LFX_DEVin the workflow step. Choose one source of truth to avoid confusion.
78-87: Consider deriving output path from the installed package.Hardcoding
scripts/../src/lfx/src/lfx/_assets/...ties the script to repo layout. Resolving via the package location is more robust.Example:
import inspect, lfx pkg_dir = Path(inspect.getfile(lfx)).parent output_path = pkg_dir / "_assets" / "component_index.json"Keeps behavior identical but decouples from repo structure.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (22)
.github/workflows/update-component-index.yml(1 hunks).gitignore(1 hunks)scripts/build_component_index.py(1 hunks)src/backend/base/langflow/initial_setup/starter_projects/Hybrid Search RAG.json(1 hunks)src/backend/base/langflow/initial_setup/starter_projects/Instagram Copywriter.json(1 hunks)src/backend/base/langflow/initial_setup/starter_projects/Invoice Summarizer.json(1 hunks)src/backend/base/langflow/initial_setup/starter_projects/Market Research.json(1 hunks)src/backend/base/langflow/initial_setup/starter_projects/News Aggregator.json(2 hunks)src/backend/base/langflow/initial_setup/starter_projects/Nvidia Remix.json(2 hunks)src/backend/base/langflow/initial_setup/starter_projects/Pokédex Agent.json(1 hunks)src/backend/base/langflow/initial_setup/starter_projects/Price Deal Finder.json(1 hunks)src/backend/base/langflow/initial_setup/starter_projects/Research Agent.json(1 hunks)src/backend/base/langflow/initial_setup/starter_projects/SaaS Pricing.json(1 hunks)src/backend/base/langflow/initial_setup/starter_projects/Search agent.json(2 hunks)src/backend/base/langflow/initial_setup/starter_projects/Sequential Tasks Agents.json(4 hunks)src/backend/base/langflow/initial_setup/starter_projects/Simple Agent.json(1 hunks)src/backend/base/langflow/initial_setup/starter_projects/Social Media Agent.json(3 hunks)src/backend/base/langflow/initial_setup/starter_projects/Travel Planning Agents.json(3 hunks)src/backend/base/langflow/initial_setup/starter_projects/Vector Store RAG.json(2 hunks)src/backend/base/langflow/initial_setup/starter_projects/Youtube Analysis.json(2 hunks)src/lfx/src/lfx/interface/components.py(4 hunks)src/lfx/src/lfx/services/settings/base.py(1 hunks)
🧰 Additional context used
🧬 Code graph analysis (2)
scripts/build_component_index.py (1)
src/lfx/src/lfx/interface/components.py (1)
import_langflow_components(183-293)
src/lfx/src/lfx/interface/components.py (1)
src/backend/tests/unit/api/v2/test_mcp_servers_file.py (1)
settings_service(96-97)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (47)
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 40/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 23/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 38/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 35/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 28/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 29/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 36/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 31/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 22/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 34/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 39/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 24/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 32/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 26/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 20/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 37/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 30/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 18/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 25/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 21/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 27/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 33/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 9/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 2/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 17/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 16/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 13/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 19/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 1/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 15/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 7/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 4/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 12/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 6/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 5/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 11/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 14/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 3/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 10/40
- GitHub Check: Run Frontend Tests / Playwright Tests - Shard 8/40
- GitHub Check: Run Backend Tests / Unit Tests - Python 3.10 - Group 4
- GitHub Check: Run Backend Tests / Unit Tests - Python 3.10 - Group 3
- GitHub Check: Run Backend Tests / Integration Tests - Python 3.10
- GitHub Check: Test Starter Templates
- GitHub Check: Run Backend Tests / Unit Tests - Python 3.10 - Group 2
- GitHub Check: Run Backend Tests / Unit Tests - Python 3.10 - Group 1
- GitHub Check: Run Backend Tests / Unit Tests - Python 3.10 - Group 5
🔇 Additional comments (16)
src/backend/base/langflow/initial_setup/starter_projects/Simple Agent.json (1)
892-893: Dependency bump looks good.Patch upgrade to
langchain_core0.3.78 is consistent with related starter updates and should be safe.src/backend/base/langflow/initial_setup/starter_projects/SaaS Pricing.json (1)
847-853: langchain_core 0.3.78 is live and installs successfully Verified via pip dry-run.src/backend/base/langflow/initial_setup/starter_projects/Search agent.json (2)
114-120: scrapegraph_py 1.33.0 is available on PyPI
Pip dry-run install succeeded without errors.
902-908: langchain_core 0.3.78 published and compatible Verified viapip install --dry-run: installable; pydantic 2.10.6 meets its>=2.7.4,<3.0.0requirement and no dependency conflicts found.src/backend/base/langflow/initial_setup/starter_projects/Pokédex Agent.json (1)
1197-1203: Confirm langchain_core 0.3.78 release
langchain_core==0.3.78 installs successfully with all required wheels; review its changelog/release notes for any breaking changes before merging.src/backend/base/langflow/initial_setup/starter_projects/News Aggregator.json (1)
1136-1138: Dependency metadata bump LGTMThe starter template now aligns with the latest component metadata; no further changes needed.
Also applies to: 1792-1794
src/backend/base/langflow/initial_setup/starter_projects/Sequential Tasks Agents.json (1)
365-366: Patch bump looks goodThanks for aligning all of the starter dependency metadata with
langchain_core==0.3.78; the update is consistent across the templates and I don’t see any conflicts.Also applies to: 1013-1014, 2409-2410, 3046-3047
src/backend/base/langflow/initial_setup/starter_projects/Youtube Analysis.json (1)
301-302: Version bump aligns with repo-wide upgrade.Both
langchain_coreentries now reference 0.3.78, matching the coordinated dependency update across starter projects. No further action needed.Also applies to: 767-768
.gitignore (1)
285-287: Ignore additions look correct.Adding
member_servers.jsonand the user-specific**/.cache/lfx/cache folder prevents accidental commits of generated component indexes while keeping the packaged asset untouched.src/backend/base/langflow/initial_setup/starter_projects/Social Media Agent.json (1)
159-161: Dependency bump looks good.The langchain_core update is consistent across the project metadata; nothing else needed.
Also applies to: 390-392
src/backend/base/langflow/initial_setup/starter_projects/Invoice Summarizer.json (1)
1113-1114: LGTM on the version sync.The langchain_core bump aligns with the rest of the component configuration.
src/backend/base/langflow/initial_setup/starter_projects/Research Agent.json (1)
2537-2539: Dependency bump looks good.The
langchain_coreupdate to 0.3.78 aligns with the rest of the starter metadata—no issues spotted.src/backend/base/langflow/initial_setup/starter_projects/Market Research.json (1)
1509-1510: Dependency bump looks good.The langchain_core upgrade to 0.3.78 aligns with the coordinated starter-project updates and has no conflicts here.
src/backend/base/langflow/initial_setup/starter_projects/Price Deal Finder.json (1)
1565-1566: Consistent version alignment.The langchain_core bump to 0.3.78 keeps this starter in sync with the others and raises no issues.
src/lfx/src/lfx/services/settings/base.py (1)
161-167: New setting looks solid.Adding
components_index_pathwith clear documentation cleanly exposes the prebuilt index override without disrupting existing defaults.src/lfx/src/lfx/interface/components.py (1)
639-645: Verify import path alias.This imports from
langflow.services.depswhile the module useslfx.*elsewhere. Confirm the correct package name/alias; otherwise this will raise ModuleNotFoundError in environments without thelangflowalias.Suggested fix if alias isn’t present:
- from langflow.services.deps import get_settings_service + from lfx.services.deps import get_settings_service
edwinjosechittilappilly
left a comment
edwinjosechittilappilly
left a comment
There was a problem hiding this comment.
LGTM approving with suggested changes. Not a blocker
- Updated the `_dev_mode` function to improve clarity and functionality in detecting development mode. - Refined environment variable checks to explicitly handle "1"/"true"/"yes" for development and "0"/"false"/"no" for production. - Maintained the editable install heuristic as a fallback for determining the mode when the environment variable is not set.
- Revised the `_dev_mode` function to clarify the detection of development mode. - Removed the editable install heuristic, making the environment variable `LFX_DEV` the sole determinant for development mode. - Updated documentation to reflect the new behavior and ensure accurate understanding of the mode switching.
- Added tips for enabling dynamic component loading with `LFX_DEV=1` for faster development. - Emphasized the importance of using `LFX_DEV=1` for live reloading of components during development. - Included instructions for manually rebuilding the component index for testing purposes.
github-actions
Bot
added
enhancement
- Introduced comprehensive unit tests for the component index system, covering functions such as _dev_mode, _read_component_index, _save_generated_index, and import_langflow_components. - Tests include various scenarios for development mode detection, reading and saving component indices, and handling custom paths and URLs. - Enhanced test coverage to ensure robustness and reliability of the component index features.
- Modified the workflow to include separate checkout steps for pull requests and manual dispatch events. - Added an environment variable `LFX_DEV` to the build step for enhanced development mode support. - Improved clarity in the workflow structure to accommodate different triggering events.
…d entries - Added new timezone options including America/Boise, Australia/North, and Etc/GMT-2. - Removed outdated timezone entries to streamline the selection process. - Updated the component index structure to enhance clarity and maintainability.
- Updated comment to specify that the output path is relative to the script location and intended for development/CI use, not from the installed package. - Enhanced clarity for future developers regarding the script's execution context.
- Changed the comment to reflect the correct module path for extracting subpackage names. - Flattened the custom components dictionary if it has a "components" wrapper for consistency. - Merged built-in and custom components into a single structure, ensuring the output maintains a "components" wrapper. - Updated the component count calculation to reflect the new merged structure.
|
@mpawlow added the |
… components for improved indexing information
…ics on modules and components, and enhancing performance tracking
github-actions
Bot
added
enhancement
github-actions
Bot
added
enhancement
github-actions
Bot
added
enhancement
|
✅ Component index has been automatically updated due to changes in |
github-actions
Bot
added
enhancement
…ge aliased and dotted imports
github-actions
Bot
added
enhancement
|
✅ Component index has been automatically updated due to changes in |
…alysis and summary
github-actions
Bot
added
enhancement
|
2 participants
Component Index System
Overview
The component index system provides instant startup performance by prebuilding a static index of all built-in components. This reduces component loading from 15-20 seconds to ~10ms in production.
Architecture
Three-Tier Loading Strategy
Production (Built-in Index): ~10ms
src/lfx/src/lfx/_assets/component_index.jsonProduction (Fallback Cache): ~15ms after first run
~/.cache/lfx/component_index.jsonDevelopment Mode: ~15-20s (always builds)
LFX_DEV=1Building the Index Manually
Environment Variables
New Feature: Selective Module Loading
When developing components, you can now load only the modules you're working on:
Benefits:
LFX_DEV=1still loads all componentsImplementation:
mistral,openai,anthropic)CI Integration
The GitHub Actions workflow (
.github/workflows/update-component-index.yml) automatically:src/lfx/src/lfx/components/**This ensures the index stays in sync with component changes without manual intervention.
How It Works
Index Structure
{ "version": "0.1.12", "sha256": "cb7560c7...", "entries": [ ["CategoryName", { "ComponentName": { "template": {...}, "display_name": "...", ... } }] ] }Loading Flow
Changes Made
_parse_dev_mode()to return(enabled, module_list)tupleimport_langflow_components()Summary by CodeRabbit
New Features
Starter Projects
Chores