refactor: load components by importing them directly#8395
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThis update introduces extensive refactoring and enhancements across Langflow's backend, focusing on component architecture, asynchronous loading, and improved configuration. Several retriever and utility components are restructured, with new input schemas and error handling. The built-in component loading system is overhauled for asynchronous, parallel discovery. Numerous starter project JSONs are updated to refine metadata, input requirements, and UI hints. Legacy and deactivated components are modularized, and new comprehensive tests are added for component loading performance and correctness. Changes
Sequence Diagram(s)Asynchronous Built-in Component LoadingsequenceDiagram
participant Interface as interface/components.py
participant Asyncio as asyncio
participant Importer as importlib
participant Component as langflow.components.<subpackage>.<module>
participant Cache as ComponentCache
Interface->>Asyncio: to_thread(discover modules in langflow.components)
Asyncio->>Importer: import each module in parallel
loop For each module
Importer->>Component: import module
Component-->>Importer: provide classes with Langflow attributes
Importer-->>Asyncio: return component classes
end
Asyncio-->>Interface: aggregated component templates
Interface->>Cache: store built-in components in cache
Custom Component Template Creation (Updated Logic)sequenceDiagram
participant Utils as custom/utils.py
participant Component as CustomComponent/Component
participant HTTP as HTTPException
Utils->>Component: get_component_instance(custom_component)
alt Not a "Component" or "CustomComponent"
Component-->>Utils: return input directly
else
Utils->>Component: validate code type and evaluate code
alt Missing/invalid code
Utils->>HTTP: raise HTTP 400 error
else
Component-->>Utils: instantiate component
end
end
Utils->>Component: run_build_config
alt Subclass of Component
Component-->>Utils: return build config directly
else
Utils->>Component: evaluate code, build config
alt Error
Utils->>HTTP: raise HTTP 400 error
end
end
Utils-->>Caller: return template and instance
Starter Project "URL" Component: Old vs. New FlowOld Flow (Simple Loader)sequenceDiagram
participant User
participant URLComponent
participant Loader as AsyncHtmlLoader/WebBaseLoader
User->>URLComponent: provide URLs and format
URLComponent->>Loader: fetch content (async)
Loader-->>URLComponent: return content (Text/HTML/JSON)
URLComponent-->>User: output data/text/dataframe
New Flow (Recursive Web Crawler)sequenceDiagram
participant User
participant URLComponent
participant Loader as RecursiveUrlLoader
User->>URLComponent: provide URLs, max_depth, options
loop For each URL
URLComponent->>Loader: create loader with options
Loader->>Loader: recursively crawl (up to max_depth)
Loader-->>URLComponent: return documents (Text/HTML)
end
URLComponent->>User: output dataframe/message
These diagrams illustrate the new asynchronous component loading, updated custom component template creation logic, and the transformation of the "URL" component from a simple loader to a recursive web crawler. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
ogabrielluiz
force-pushed
the
fix-component-loading
branch
from
7239751 to
5d3fa5b
Compare
github-actions
Bot
added
refactor
github-actions
Bot
added
refactor
ogabrielluiz
force-pushed
the
fix-component-loading
branch
from
5d3fa5b to
d06206f
Compare
github-actions
Bot
added
refactor
dosubot
Bot
added
the
size:XXL
github-actions
Bot
added
refactor
|
@coderabbitai review |
✅ Actions performedReview triggered.
|
Docstrings generation was requested by @ogabrielluiz. * #8395 (comment) The following files were modified: * `src/backend/base/langflow/custom/utils.py` * `src/backend/base/langflow/interface/components.py` * `src/backend/base/langflow/services/settings/base.py` * `src/backend/tests/unit/test_endpoints.py`
|
Note Generated docstrings for this pull request at #8397 |
github-actions
Bot
added
refactor
|
|
||
| def get_component_instance(custom_component: CustomComponent, user_id: str | UUID | None = None): | ||
| if isinstance(custom_component, Component | CustomComponent): | ||
| return custom_component |
There was a problem hiding this comment.
Should this conditional be removed? iiuc, this is used to create a custom component via the path of a user creating and saving one from the FE and calling the /custom_component endpoint, in which case we do want to build
There was a problem hiding this comment.
When importing we already have the instance so we don't need to do anything, although I do think this is wrong.
| # Final setup | ||
| self._set_output_types(list(self._outputs_map.values())) | ||
| self.set_class_code() | ||
| self._set_output_required_inputs() |
There was a problem hiding this comment.
@ogabrielluiz May I know the process on removal of _set_output_required_inputs?
There was a problem hiding this comment.
It isn't used (i.e. we get the warning constantly but see no problems) and the implementation is very flaky. We should replace it with another, more reliable, implementation instead of using inspect.getsource.
|
@CodeRabbit please find what triggered the change in the advanced fields of templates? |
|
@CodeRabbit can you explain why self._set_output_required_inputs() was removed? |
|
Tip For best results, initiate chat on the files or code changes.
During the refactor the team moved to a simpler, declarative scheme: • Starter-project JSONs now declare every output without a Removing the call therefore:
If a component really needs to express a hard dependency between an output and a subset of inputs, it can now do so explicitly in its template or by validating inside the output method itself, without relying on implicit reflection. |
|
Tip For best results, initiate chat on the files or code changes. After tracing the commit history and the code paths that build the JSON templates, here’s what I found: • No new automatic rule was added in the Python template-builders ( So the change was manual housekeeping aimed at decluttering the default UI and tucking rarely-used knobs under the “advanced” section—nothing in the new component-loading refactor triggers it programmatically. |
…oreRetrieverComponent
…ate_from_inputs for improved error handling and component instantiation
…(`fix-component-loading`) Here is an optimized version of your program. **Reasoning for changes:** - The check `isinstance(custom_component, Component)` followed by `type(custom_component) is not Component` causes the interpreter to potentially look up the type and MRO twice per call. - By storing `type(custom_component)` in `klass` and using `issubclass(klass, Component)`, you avoid having Python walk the MRO twice for the same object, which is subtly more efficient especially in tight loops and heavy use scenarios. - Using `issubclass()` on the object's type is semantically equivalent to `isinstance()`, except it also works for custom metaclass scenarios and is very slightly faster when type is already known. **All program logic and comments are preserved, only the relevant portion is optimized.**
…ow to enhance test coverage and improve test reliability
…cessfully" text to appear to improve test reliability
…ater than 0 before calling addFlowToTestOnEmptyLangflow 🔧 (generalBugs-shard-9.spec.ts): update tags in test case to include @workspace and @components ♻️ (generalBugs-shard-9.spec.ts): refactor code to remove unnecessary steps related to sidebar search and node handling 🔧 (store-shard-0.spec.ts): update test cases to be skipped and improve readability by using async arrow functions
|
Currently, after pulling the latest image Docker pull langflowai/langflow: 1.5.0.post1, the startup is blocked, and the debug log finally prints "DEBUG - components - Building components cache". From the source code, it appears to be blocked in the loading component section of this submission. However, on the Mac machine, installing langflow==1.5.0.post1 through UV pip does not cause any problems. Who knows what the reason is? |
|
@grebug I was able to start up Langflow on Mac with the below image tag, with default env vars. Perhaps that's the difference - can you share your environment variables and docker start command? |
|
@jordanrfrazier It's okay, it may be a problem with machine resources. Replace it with another machine. Although it may experience some lag in the same location, the service will eventually be able to function normally |
5 participants
This pull request introduces several changes across multiple files, focusing on improving documentation, enhancing functionality, and refining starter project configurations. The most significant updates include removing unused components, adding detailed docstrings to utility functions, and modifying starter project templates to improve usability and clarity.
Codebase Cleanup and Documentation Enhancements:
src/backend/base/langflow/components/retrievers/__init__.py: RemovedNeedleRetrieverfrom imports and__all__, indicating it is no longer used.src/backend/base/langflow/custom/utils.py: Added comprehensive docstrings to utility functions such asget_component_instance,run_build_config,build_custom_component_template_from_inputs, andcreate_component_template, improving clarity and usability for future developers. [1] [2] [3] [4] [5]Starter Project Updates:
General Improvements:
Basic Prompt Chaining.json,Basic Prompting.json,Blog Writer.json, andDocument Q&A.jsonto remove the requirement for anapi_keyinrequired_inputsforLanguageModelcomponents, simplifying setup for users. [1] [2] [3] [4]infofields forModel Nameinputs across multiple starter projects to provide clearer instructions on selecting a provider and refreshing model names. [1] [2] [3] [4]Temperatureinputs as advanced in several starter projects, aligning with their intended use for fine-tuning model behavior. [1] [2] [3] [4]Specific Updates:
Diet Analysis.json: Addedapi_keyas a required input forLanguageModel, reversing the removal seen in other projects to align with specific requirements.Financial Agent.json: Updated the description for theURLcomponent to clarify its functionality and replacedclean_extra_whitespacewithautoset_encodingin the template, enhancing usability. [1] [2]Summary by CodeRabbit
New Features
Bug Fixes
Refactor
Chores
Tests