feat(lfx): Bundle Separation and LFX Extension Framework#13043
Conversation
…n install (LE-1022) Adds the read-only production install path for Modes A, B, and C of the Bundle Separation iteration. Manifest-shipping pip-installed distributions and seed-directory subdirectories are discovered at server startup and registered as Extensions at @official. * discovery.py: walks importlib.metadata.distributions() + the $LANGFLOW_SEED_DIR / /opt/langflow/bundles seed root; produces DiscoveredExtension records and typed errors for malformed manifests / configured-but-missing seed dirs. * registry.py: ExtensionRegistry service with the immutability invariant for installed and seed entries. Mutation verbs (uninstall, disable, enable, install, update_entry) all raise ExtensionImmutableError carrying the typed installed-extension-immutable / seed-directory-immutable code so the invariant is testable today; the CLI uninstall surface ships in B4. * lfx extension list: read-only inspector with text and JSON output for operators inspecting Mode B/C images. * Errors: four new typed codes (installed-extension-immutable, seed-directory-immutable, seed-directory-not-found, duplicate-extension-id) plus snapshot coverage in tests/unit/extension/test_errors.py. * Tests: 165 extension tests pass, including the LE-1022 acceptance cases -- three pip-installed wheels visible at @official, three seed bundles visible at @official, and the parametrized service-layer immutability check across every mutation verb. * Docs: docs/Deployment/deployment-extensions-production.mdx covers the Dockerfile template, k8s deployment notes, the bundle packaging convention (extension.json shipped via package-data), and troubleshooting for the typed error codes.
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughIntroduces an Extension System: discovery/loader/migration/registry/reload; adds DuckDuckGo and ArXiv bundles; exposes backend reload API and frontend UI gating; updates CI guard scripts, Dockerfiles, docs, and extensive unit/integration tests. ChangesExtension System v0 integration
Sequence Diagram(s)sequenceDiagram
autonumber
participant UI as Frontend BundleHeader
participant API as FastAPI /extensions
participant Reg as BundleRegistry
participant Ldr as Extension Loader
participant Cfg as Settings
UI->>API: POST /api/v1/extensions/{ext}/bundles/{bundle}/reload
API->>Cfg: read enable_extension_reload
API-->>UI: 404 detail (disabled) alt when disabled
API->>Reg: begin_reload(bundle)
API->>Ldr: load bundle in staging namespace
Ldr-->>API: LoadResult (components/errors)
API->>Reg: atomic swap -> install BundleRecord
API-->>UI: 200 ReloadResult (ok/errors/warnings)
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes ✨ Finishing Touches🧪 Generate unit tests (beta)
|
This comment has been minimized.
This comment has been minimized.
1 similar comment
This comment has been minimized.
This comment has been minimized.
Codecov Report❌ Patch coverage is ❌ Your project check has failed because the head coverage (51.27%) is below the target coverage (60.00%). You can increase the head coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## release-1.10.0 #13043 +/- ##
==================================================
+ Coverage 54.83% 55.22% +0.39%
==================================================
Files 2150 2173 +23
Lines 200679 203453 +2774
Branches 28543 30675 +2132
==================================================
+ Hits 110034 112353 +2319
- Misses 89457 89815 +358
- Partials 1188 1285 +97
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
…overy (#12967) * feat(lfx): add extension manifest schema, validate CLI, and error formatter (LE-1014) Foundation for the Bundle Separation iteration. Defines what a valid extension.json looks like (Pydantic models + Draft 2020-12 JSON Schema), ships the offline `lfx extension validate` command, and ships the single `format_extension_error` function that every other extension-system module will use to render structured errors. What's in lfx.extension: - `ExtensionManifest` / `BundleRef` / `LangflowCompat` Pydantic models with `extra="forbid"` so unknown fields fail loudly. Deferred fields (`services`, `routes`, `hooks`, `starter_projects`, `userConfig`) are reserved as None-only so non-null values produce a dedicated `field-deferred-in-this-milestone` error instead of a generic schema wall. `bundles` accepts a list but rejects length > 1 with `multi-bundle-deferred-in-this-milestone` (validator-enforced; the loader re-checks at install time in LE-1015). - `schema.build_schema()` + `build_schema_json()` produce the publishable artifact at schemas.langflow.org/extension/v1.json. - `ExtensionError` typed envelope and `format_extension_error` -- one branch per discriminant. Codes are registered in `ERROR_CODES`; an `ExtensionError` constructed with an unknown code raises at construction time, preventing producers from shipping without a matching renderer. - `validate_extension` runs four passes: manifest discovery + schema, path-safety (no `..`, no absolute paths, no symlink escape), AST inspection of every `.py` (syntax, Component subclass present, build() declared, top-level `import *`, top-level I/O primitives), and an opt-in `--execute-imports` that runs each module in a subprocess with a temporary HOME / TMPDIR / LANGFLOW_CONFIG_DIR and LANGFLOW_*/LFX_* env vars stripped. - `lfx extension validate` and `lfx extension schema` typer subcommands. Acceptance-criteria coverage in tests/unit/extension/: - Round-trips every v0 manifest field; deferred fields rejected; multi-bundle rejected with the dedicated discriminant. - JSON Schema validates the v0 example and rejects 12 malformed manifests with distinct error paths (>= 10 required by the ticket). - Median default-validate runtime < 100ms on the basic template. - Crafted side-effect bundle: default validate does NOT execute it (canary file is never written); `--execute-imports` DOES execute it and the canary appears, while LANGFLOW_* env vars are NOT inherited by the subprocess. - Snapshot tests for every code in ERROR_CODES; a guard test verifies ERROR_CODES and the snapshot table are in lockstep so future additions cannot ship without a format branch and a snapshot. Wiring: `lfx extension` is a sub-app under the Authoring help panel so future tickets (LE-1016 init/dev, LE-1018 reload) can attach without colliding with the existing `lfx validate` (which validates flow JSON, not extensions). * fix(lfx): fall back to tomli on Python 3.10 in extension manifest loader tomllib is stdlib only on 3.11+, but lfx supports 3.10-3.13. Use the existing tomli runtime dependency as the 3.10 fallback (same API, so the import alias keeps the rest of the module unchanged). Fixes ModuleNotFoundError seen in CI on the 3.10 job. * Update src/lfx/tests/unit/extension/test_schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/extension/schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/cli/_extension_commands.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (LE-1015) Introduces lfx.extension.loader: the runtime that turns an Extension on disk into LoadedComponent records keyed by ext:<bundle>:<Class>@<slot>. Two paths in: - load_extension(root): one manifest, one Bundle, registered at @official. Re-checks multi-bundle at runtime (defense-in-depth vs. the schema). - discover_inline_bundles(paths): each subfolder of LANGFLOW_COMPONENTS_PATH is a Bundle at @extra. Walk order is platform-independent (sorted dirs, user-declared path order). First-wins on duplicate names; second emits duplicate-inline-bundle warning that names both paths. Manifest-first precedence helpers (installed_extension_roots, manifest_owning_distributions, filter_plugin_entry_points) let callers of the legacy langflow.plugins entry-point loader skip distributions that ship a manifest, so component entry-points are not double-registered. New typed error codes: module-import-failed, duplicate-component-name, duplicate-distribution, duplicate-inline-bundle, inline-bundle-name-invalid. Each ships with a format branch and a snapshot test. Tests cover the AC: single-bundle happy path, multi-bundle rejection, missing/empty/no-Component bundle, duplicate class names, deterministic walk order, recursive discovery, inline-bundle first-wins + dot-dir skip + bundle.json metadata, manifest-first precedence partition, PEP-503 distribution-name canonicalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1015 review feedback - Drop the unproduced duplicate-distribution error code; LE-1022 will add it once startup-time discovery has a place to surface it. Restores the invariant that every code in ERROR_CODES has a producer. - Clarify that intra-bundle relative imports are NOT supported in v0; only absolute references between bundle modules work in this milestone. - Use strict=False when re-resolving bundle_root in the walker; the path was already existence-checked, and a concurrent removal in the narrow window should not raise across the loader's public boundary. - Hoist the json import out of _read_inline_bundle_json's body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lfx): split extension loader into a small subpackage (LE-1015) Addresses the file-size feedback from the LE-1015 review. The single 870-line loader.py becomes a flat package keyed off the four section banners that already existed inline: loader/ __init__.py # re-exports the public surface _types.py # SLOT constants, LoadedComponent, LoadResult _discovery.py # filesystem walk + importlib.util orchestration _detection.py # Component subclass identification (MRO heuristic) _orchestrator.py # load_extension, discover_inline_bundles _plugins.py # manifest-first precedence over langflow.plugins Largest file is now _orchestrator.py at 440 LOC (was 870); every file is well under the 800-LOC project guideline. No behavior change: the public import paths from lfx.extension are unchanged, all 38 loader tests still pass. ``_canonicalize_distribution`` is now exported as ``canonicalize_distribution`` from ``loader._plugins`` (it's a stable PEP-503 helper that downstream modules will reach for, so it loses the private underscore). The test suite is split to mirror the package: tests/unit/extension/loader/ conftest.py # shared fixtures, FakeDist, autouse scrub test_load_extension.py # @official slot, identity, failure modes test_inline_bundles.py # LANGFLOW_COMPONENTS_PATH, @extra slot test_plugins.py # manifest-first precedence helpers test_types.py # LoadedComponent, LoadResult, code parity Each test file imports its own slice of the public API and pulls fixtures from conftest, so a reader looking at one banner can read it in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: LangflowCompat -> LfxCompat * fix: Remove references to ticket * fix: encode maxItems constraint in schema * fix: deferred fields and schema version * Fix ruff errors * fix: align loader tests with renamed manifest API and strip ticket refs The merge of feat/extension-validate brought in the LfxCompat / compat rename, but the loader test fixtures still used LangflowCompat / bundle_api and failed at the manifest layer. Update conftest.py + test_load_extension.py to the post-rename API. Strip 17 LE-XXXX ticket references from loader source files, errors.py loader-specific comments, and the four loader test docstrings, matching the convention applied to LE-1014. Descriptive prose preserved. Promote _BUNDLE_NAME_RE to BUNDLE_NAME_RE on the manifest module so the loader's orchestrator no longer reaches across modules into a private name. * refactor(lfx): clarify broad except, log malformed bundle.json, expand test docstring - Document why _discovery.import_bundle_module catches BaseException (startup-time loader, must surface bad bundles as typed errors rather than abort). - Add debug-level logging when bundle.json is malformed or non-object so a stale-cache footgun is at least observable. - Expand test_skips_re_imported_class docstring so future maintainers don't accidentally weaken the __module__-equality guard if package-style relative imports get added later. Also drops the stale duplicate-distribution claim from the PR description; that code is correctly deferred to LE-1022 along with /all integration. * feat(lfx): wire Extension System into /all, pathsep-split LANGFLOW_COMPONENTS_PATH, emit duplicate-distribution Closes the four AC gaps the previous reviewer flagged on PR #12967: 1. /all integration: get_and_cache_all_types_dict now also calls a new import_extension_components() that loads installed Extensions via load_installed_extensions, loads inline bundles via discover_inline_bundles, and builds frontend-node templates with extension/bundle/extension_version fields stamped on. Failures are logged and skipped per bundle. 2. LANGFLOW_COMPONENTS_PATH is now split on os.pathsep so multi-entry env vars (e.g. /a:/b on POSIX) produce multiple components-path entries instead of one literal non-existent path. Empty segments and missing paths are skipped. 3. duplicate-distribution is a real producer: load_installed_extensions surfaces a typed warning on the winner LoadResult when two distributions share a canonical name, naming every involved manifest path. 4. Manifest-first precedence runtime wiring: new filter_component_entry_points loads each entry-point and only skips ones that resolve to a Component subclass on a manifest-shipping distribution. plugin_routes.load_plugin_routes now applies it so non-component entry-points (route registrars) keep loading per the AC's 'unaffected' promise. Added the previously-missing AC test for same-distribution component+non-component partition. Also makes _distribution_canonical_name defensive against MagicMock test seams. * fix(lfx): register extension components under namespaced ID; promote duplicate-distribution to error Addresses the latest review of PR #12967: P1: /all integration now keys the cache inner dict by LoadedComponent.namespaced_id (ext:<bundle>:<Class>@<slot>) rather than the bare class name. Templates also carry the namespaced_id as an explicit field so consumers that look at the value (not the key) still see the canonical address. This is the form the LE-1020 migration table will rewrite legacy class-name references to. P2: load_installed_extensions now appends duplicate-distribution to result.errors instead of result.warnings, so LoadResult.ok=False when two distributions share a canonical name. The winner's components still appear in result.components so flows already pinned to them keep working; only the conflict status changes. Updated test to assert errors + ok=False; added explicit assertion that the winner's components are still present. * fix(lfx): installed-distribution discovery accepts pyproject.toml manifest form Closes the latest review finding on PR #12967: the installed-distribution scan only looked for extension.json, ignoring distributions whose manifest lives in [tool.langflow.extension] inside pyproject.toml. The AC explicitly treats both as valid manifest forms. _distribution_manifest_path now: - Returns extension.json immediately when present (preserves precedence matching load_manifest's discovery order). - Falls back to pyproject.toml only when extension.json is absent AND the pyproject's [tool.langflow.extension] section is parseable. Validation reuses load_manifest itself so the rule lives in exactly one place; a stray pyproject.toml without the section is correctly ignored. Tests cover: pyproject-only discovery, pyproject-without-section ignored, extension.json wins on collision, end-to-end pyproject load at @official, and pyproject-form manifest-first entry-point suppression. * refactor(lfx): tighten loader invariants, surface silent skips, harden tests Addresses the latest review feedback on PR #12967: Type-level invariants (_types.py): - LoadedComponent.__post_init__ enforces that @extra components must NOT carry a distribution. The reverse (@official without distribution) is permitted because load_extension is also used for dev-mode loads against a working tree before pip install. - LoadResult docstring documents the partial-success contract: components may be non-empty when errors is non-empty (some files imported, others failed). Callers branching on ok get strict success. Silent-failure fixes (_orchestrator.py + settings/base.py): - inline-path-missing: a non-existent / non-dir LANGFLOW_COMPONENTS_PATH entry now produces a typed warning per skipped path so a typo no longer yields zero diagnostics. Settings-layer skip bumped from debug to warning for the same reason. - bundle-json-invalid: a malformed or non-object bundle.json now surfaces a typed warning instead of silently rewriting the user-declared id/version to derived values under the same bundle name. - no-component-subclass gating uses a call-local counter instead of result.errors so the diagnostic stays accurate when a future caller reuses a LoadResult (multi-bundle / batch wrapper scenarios). Test hardening: - test_re_imported_class_is_skipped_via_module_filter rewritten to actually exercise the __module__-equality check via sys.modules injection; previously passed via module-import-failed (relative-import failure), which would silently weaken if package registration changes. - test_user_declared_path_order_is_preserved: AC #8's multi-path order case (distinct bundles in [path_b, path_a]) was unasserted; added. - test_inline_module_import_failure_attributes_identity: AC #10's identity-on-partial-failure was covered for @official but not @extra; added. - test_uses_real_distributions_by_default tightened to assert ep placement (in kept, not in skipped) instead of exact-list equality, so a future Langflow-shipped manifest doesn't silently flip the assertion. bumped 64 -> 175 passing extension tests; 20 backend integration tests still pass. * fix(lfx): malformed pyproject manifests surface manifest-invalid instead of disappearing Closes the latest review finding on PR #12967: a pyproject.toml with a [tool.langflow.extension] section that has missing/invalid required fields was silently dropped because _pyproject_has_extension_section ran full schema validation via load_manifest and returned False on ValueError/TypeError. That conflated 'no section' with 'section malformed'. Fix: detect section presence only. _pyproject_has_extension_section now calls _read_pyproject_extension (TOML parse + key lookup, no schema check). Behavior: - Section absent or pyproject TOML unparseable -> False (treat as regular non-manifest package). - Section present and is a table (valid OR schema-invalid) -> True. - Section present but is not a table -> True; the author intended to declare an extension and load_extension will surface the typed error. This way a typo'd pyproject Extension produces a typed manifest-invalid LoadResult with extension_id attribution, and manifest-first precedence still suppresses its legacy component entry-points -- matching the 'typed load results on success/failure' contract for the supported pyproject manifest form. Tests: two new cases pin the behavior. test_malformed_pyproject_section_ surfaces_manifest_invalid asserts a typed load-failure result with distribution attribution; the second test pins manifest-first suppression for malformed pyproject distributions. * refactor(lfx): emit inline-path-unreadable, document reload contract, trim rot Closes the remaining nits on PR #12967: - inline-path-unreadable (new typed error code): a configured LANGFLOW_COMPONENTS_PATH entry that raises OSError on iterdir (typically permission-denied) now produces a typed LoadResult error carrying str(exc) instead of silently swallowing the message. - duplicate-inline-bundle hint trimmed: removed forward promise about hard-error-in-a-later-release; same actionability, no expiration. - discover_inline_bundles docstring trimmed from three paragraphs to two sentences (per CLAUDE.md anti-multi-paragraph rule). - _distribution_manifest_path docstring trimmed to one-liner. - installed_extension_roots dropped Used-by caller-narration list; manifest_owning_distributions docstring rewritten to call out the shadow-load risk for direct callers and point to load_installed_ extensions for the typed warning surface. - _discovery.import_bundle_module: replaced 'until that lands in a later milestone' rot with a single line ('Absolute imports only between bundle modules; relative imports unsupported.') AND added the single-load-per-process contract note documenting why LE-1018 reload must scrub registry/sys.modules before re-invoking the loader. - load_extension docstring grew a 'Single-load-per-process contract' block telling direct callers not to rely on this function for refresh. Tests: new test_unreadable_path_emits_inline_path_unreadable pins the OSError -> typed-error path. --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
…18) (#12979) * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (LE-1015) Introduces lfx.extension.loader: the runtime that turns an Extension on disk into LoadedComponent records keyed by ext:<bundle>:<Class>@<slot>. Two paths in: - load_extension(root): one manifest, one Bundle, registered at @official. Re-checks multi-bundle at runtime (defense-in-depth vs. the schema). - discover_inline_bundles(paths): each subfolder of LANGFLOW_COMPONENTS_PATH is a Bundle at @extra. Walk order is platform-independent (sorted dirs, user-declared path order). First-wins on duplicate names; second emits duplicate-inline-bundle warning that names both paths. Manifest-first precedence helpers (installed_extension_roots, manifest_owning_distributions, filter_plugin_entry_points) let callers of the legacy langflow.plugins entry-point loader skip distributions that ship a manifest, so component entry-points are not double-registered. New typed error codes: module-import-failed, duplicate-component-name, duplicate-distribution, duplicate-inline-bundle, inline-bundle-name-invalid. Each ships with a format branch and a snapshot test. Tests cover the AC: single-bundle happy path, multi-bundle rejection, missing/empty/no-Component bundle, duplicate class names, deterministic walk order, recursive discovery, inline-bundle first-wins + dot-dir skip + bundle.json metadata, manifest-first precedence partition, PEP-503 distribution-name canonicalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1015 review feedback - Drop the unproduced duplicate-distribution error code; LE-1022 will add it once startup-time discovery has a place to surface it. Restores the invariant that every code in ERROR_CODES has a producer. - Clarify that intra-bundle relative imports are NOT supported in v0; only absolute references between bundle modules work in this milestone. - Use strict=False when re-resolving bundle_root in the walker; the path was already existence-checked, and a concurrent removal in the narrow window should not raise across the loader's public boundary. - Hoist the json import out of _read_inline_bundle_json's body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lfx): split extension loader into a small subpackage (LE-1015) Addresses the file-size feedback from the LE-1015 review. The single 870-line loader.py becomes a flat package keyed off the four section banners that already existed inline: loader/ __init__.py # re-exports the public surface _types.py # SLOT constants, LoadedComponent, LoadResult _discovery.py # filesystem walk + importlib.util orchestration _detection.py # Component subclass identification (MRO heuristic) _orchestrator.py # load_extension, discover_inline_bundles _plugins.py # manifest-first precedence over langflow.plugins Largest file is now _orchestrator.py at 440 LOC (was 870); every file is well under the 800-LOC project guideline. No behavior change: the public import paths from lfx.extension are unchanged, all 38 loader tests still pass. ``_canonicalize_distribution`` is now exported as ``canonicalize_distribution`` from ``loader._plugins`` (it's a stable PEP-503 helper that downstream modules will reach for, so it loses the private underscore). The test suite is split to mirror the package: tests/unit/extension/loader/ conftest.py # shared fixtures, FakeDist, autouse scrub test_load_extension.py # @official slot, identity, failure modes test_inline_bundles.py # LANGFLOW_COMPONENTS_PATH, @extra slot test_plugins.py # manifest-first precedence helpers test_types.py # LoadedComponent, LoadResult, code parity Each test file imports its own slice of the public API and pulls fixtures from conftest, so a reader looking at one banner can read it in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lfx): atomic-swap Bundle reload pipeline + endpoint + CLI (LE-1018) Five-stage reload (parallel staging load -> validate -> swap under write lock -> cleanup -> emit) for installed Bundles in Mode A. In-flight flows keep the pre-swap class via existing references; new flows pick up the post-swap class atomically; concurrent reloads on the same Bundle are rejected with reload-in-progress. Adds: * lfx/extension/registry.py -- BundleRegistry with per-bundle reload-in-progress guard and components_index.json writer * lfx/extension/reload.py -- the five-stage pipeline; events emission is stubbed (TODO LE-1017) so the swap mechanics can ship before the events service lands * loader: optional module_namespace param so Stage 1 lands in __reload_staging__.<id> instead of the live _lfx_ext.* namespace * errors: four new typed reload codes (reload-in-progress, reload-bundle-not-installed, reload-bundle-name-mismatch, reload-source-missing) with branch templates and snapshot tests * HTTP: POST /api/v1/extensions/{id}/bundles/{name}/reload, gated by the existing get_current_active_user dependency, returns 409 with a typed body for the in-progress collision case * CLI: lfx extension reload <id> [--bundle <name>] -- HTTP client against the dev server with text/json output and proper exit codes (--all is gated until LE-1019 lands the list endpoint) * tests: 16 reload-pipeline tests covering the AC matrix (rename round-trip, broken-bundle isolation, concurrent readers, in-flight flow, double-reload guard, bundle-name mismatch) plus 9 CLI client tests Mode A only. In Mode B/C bundle changes require a Docker image rebuild and the reload path is not exercised. The events emission in Stage 5 is intentionally stubbed -- the LE-1017 ticket will swap the body of _emit_bundle_reload_event in one place without touching the pipeline core. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…12968) * feat(lfx): add extension manifest schema, validate CLI, and error formatter (LE-1014) Foundation for the Bundle Separation iteration. Defines what a valid extension.json looks like (Pydantic models + Draft 2020-12 JSON Schema), ships the offline `lfx extension validate` command, and ships the single `format_extension_error` function that every other extension-system module will use to render structured errors. What's in lfx.extension: - `ExtensionManifest` / `BundleRef` / `LangflowCompat` Pydantic models with `extra="forbid"` so unknown fields fail loudly. Deferred fields (`services`, `routes`, `hooks`, `starter_projects`, `userConfig`) are reserved as None-only so non-null values produce a dedicated `field-deferred-in-this-milestone` error instead of a generic schema wall. `bundles` accepts a list but rejects length > 1 with `multi-bundle-deferred-in-this-milestone` (validator-enforced; the loader re-checks at install time in LE-1015). - `schema.build_schema()` + `build_schema_json()` produce the publishable artifact at schemas.langflow.org/extension/v1.json. - `ExtensionError` typed envelope and `format_extension_error` -- one branch per discriminant. Codes are registered in `ERROR_CODES`; an `ExtensionError` constructed with an unknown code raises at construction time, preventing producers from shipping without a matching renderer. - `validate_extension` runs four passes: manifest discovery + schema, path-safety (no `..`, no absolute paths, no symlink escape), AST inspection of every `.py` (syntax, Component subclass present, build() declared, top-level `import *`, top-level I/O primitives), and an opt-in `--execute-imports` that runs each module in a subprocess with a temporary HOME / TMPDIR / LANGFLOW_CONFIG_DIR and LANGFLOW_*/LFX_* env vars stripped. - `lfx extension validate` and `lfx extension schema` typer subcommands. Acceptance-criteria coverage in tests/unit/extension/: - Round-trips every v0 manifest field; deferred fields rejected; multi-bundle rejected with the dedicated discriminant. - JSON Schema validates the v0 example and rejects 12 malformed manifests with distinct error paths (>= 10 required by the ticket). - Median default-validate runtime < 100ms on the basic template. - Crafted side-effect bundle: default validate does NOT execute it (canary file is never written); `--execute-imports` DOES execute it and the canary appears, while LANGFLOW_* env vars are NOT inherited by the subprocess. - Snapshot tests for every code in ERROR_CODES; a guard test verifies ERROR_CODES and the snapshot table are in lockstep so future additions cannot ship without a format branch and a snapshot. Wiring: `lfx extension` is a sub-app under the Authoring help panel so future tickets (LE-1016 init/dev, LE-1018 reload) can attach without colliding with the existing `lfx validate` (which validates flow JSON, not extensions). * fix(lfx): fall back to tomli on Python 3.10 in extension manifest loader tomllib is stdlib only on 3.11+, but lfx supports 3.10-3.13. Use the existing tomli runtime dependency as the 3.10 fallback (same API, so the import alias keeps the rest of the module unchanged). Fixes ModuleNotFoundError seen in CI on the 3.10 job. * Update src/lfx/tests/unit/extension/test_schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/extension/schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/cli/_extension_commands.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (LE-1015) Introduces lfx.extension.loader: the runtime that turns an Extension on disk into LoadedComponent records keyed by ext:<bundle>:<Class>@<slot>. Two paths in: - load_extension(root): one manifest, one Bundle, registered at @official. Re-checks multi-bundle at runtime (defense-in-depth vs. the schema). - discover_inline_bundles(paths): each subfolder of LANGFLOW_COMPONENTS_PATH is a Bundle at @extra. Walk order is platform-independent (sorted dirs, user-declared path order). First-wins on duplicate names; second emits duplicate-inline-bundle warning that names both paths. Manifest-first precedence helpers (installed_extension_roots, manifest_owning_distributions, filter_plugin_entry_points) let callers of the legacy langflow.plugins entry-point loader skip distributions that ship a manifest, so component entry-points are not double-registered. New typed error codes: module-import-failed, duplicate-component-name, duplicate-distribution, duplicate-inline-bundle, inline-bundle-name-invalid. Each ships with a format branch and a snapshot test. Tests cover the AC: single-bundle happy path, multi-bundle rejection, missing/empty/no-Component bundle, duplicate class names, deterministic walk order, recursive discovery, inline-bundle first-wins + dot-dir skip + bundle.json metadata, manifest-first precedence partition, PEP-503 distribution-name canonicalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1015 review feedback - Drop the unproduced duplicate-distribution error code; LE-1022 will add it once startup-time discovery has a place to surface it. Restores the invariant that every code in ERROR_CODES has a producer. - Clarify that intra-bundle relative imports are NOT supported in v0; only absolute references between bundle modules work in this milestone. - Use strict=False when re-resolving bundle_root in the walker; the path was already existence-checked, and a concurrent removal in the narrow window should not raise across the loader's public boundary. - Hoist the json import out of _read_inline_bundle_json's body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lfx): split extension loader into a small subpackage (LE-1015) Addresses the file-size feedback from the LE-1015 review. The single 870-line loader.py becomes a flat package keyed off the four section banners that already existed inline: loader/ __init__.py # re-exports the public surface _types.py # SLOT constants, LoadedComponent, LoadResult _discovery.py # filesystem walk + importlib.util orchestration _detection.py # Component subclass identification (MRO heuristic) _orchestrator.py # load_extension, discover_inline_bundles _plugins.py # manifest-first precedence over langflow.plugins Largest file is now _orchestrator.py at 440 LOC (was 870); every file is well under the 800-LOC project guideline. No behavior change: the public import paths from lfx.extension are unchanged, all 38 loader tests still pass. ``_canonicalize_distribution`` is now exported as ``canonicalize_distribution`` from ``loader._plugins`` (it's a stable PEP-503 helper that downstream modules will reach for, so it loses the private underscore). The test suite is split to mirror the package: tests/unit/extension/loader/ conftest.py # shared fixtures, FakeDist, autouse scrub test_load_extension.py # @official slot, identity, failure modes test_inline_bundles.py # LANGFLOW_COMPONENTS_PATH, @extra slot test_plugins.py # manifest-first precedence helpers test_types.py # LoadedComponent, LoadResult, code parity Each test file imports its own slice of the public API and pulls fixtures from conftest, so a reader looking at one banner can read it in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lfx): add `extension init` and `extension dev` CLIs (LE-1016) The two scaffolding CLIs an Extension author types: - `lfx extension init <target>` writes the basic single-Bundle template (manifest with $schema, README, .gitignore, one Component subclass + a pytest smoke test). AC #1: the generated extension validates clean against LE-1014. AC #2: the generated test file is a valid pytest module that exercises the component's build() method. AC #3: any --template other than 'basic' fails with a typed template-deferred-in-this-milestone error and a non-zero exit. Refuses to scaffold over a non-empty target dir. - `lfx extension dev <target>` validates the local extension, records its absolute path in <config_dir>/extensions/dev_extensions.json, prints reload instructions, and execs `langflow run` (or `python -m langflow` when langflow isn't on PATH). --skip-launch registers without launching (used by tests + external dev-server scripts); --skip-validate lets authors register a known-broken manifest to debug it under the loader. Stack: - Wave 0: error codes (extension-target-exists, extension-target-invalid, local-extension-missing) with format branches and snapshot tests. - Wave 1: lfx.extension.init_template -- pure-data scaffolder, no Typer dependency so the CLI is a thin shell over it. - Wave 1: lfx.extension.dev_registry -- atomic JSON state file under the langflow user-cache dir; helpers for register / list / unregister / load_dev_extensions / dev_extension_component_paths. - Wave 2: lfx.cli._extension_commands gains init/dev subcommands. - Wave 2: langflow.main lifespan hook reads the dev registry after bundle loading and extends components_path with each registered bundle dir, so the existing palette discovery picks up dev extensions. Missing paths surface as local-extension-missing warnings (AC #5) without aborting startup. Tests (84 new, 197 total in tests/unit/extension/): - test_init_template.py: AC scenarios + identifier derivation + deterministic file shape. - test_dev_registry.py: register/list/unregister round-trip, idempotent re-register refreshes timestamp, malformed state file treated as empty, missing-path warning, recovery when path reappears, env-var override precedence. - test_cli.py: AC #1 init->validate, AC #3 deferred templates, --skip-launch registers without launching, --skip-validate short-circuits the pre-flight pass. - test_errors.py: snapshot rows for all three new codes; the every-known-code-has-a-snapshot test enforces parity. LE-1018 (reload) reuses load_dev_extensions; LE-1022 (installed-pkg discovery) shares the components_path extension pattern. AC #4 ("boots Langflow with the extension visible in the palette within 5s") is delivered jointly by `extension dev` (registers + execs) and the lifespan hook (loads on startup). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1016 review feedback Fixes the four HIGH issues + the elevated LOW from the review pass: 1. dev_extension_component_paths now forwards EVERY warning, not just local-extension-missing. Previously a duplicate-component-name (or any future warning code) was silently dropped, hiding real signal from the lifespan hook's logs. 2. Defensive emit when a LoadResult has components but source_path is None. The current loader always sets source_path, but a future hand-built LoadResult could violate that contract; we now surface a typed local-extension-missing error rather than dropping the extension silently. 3. Replaced the fragile ``min(len(parts))`` bundle-root selection with a relative-to-source-path measurement. Handles deep-vs-shallow sibling extensions correctly without depending on absolute path depth. 4. Generated README now documents the langflow/lfx prerequisite under the Develop section so authors know `pytest` requires the lfx environment, not just Python. 5. Forced LANGFLOW_LAZY_LOAD_COMPONENTS=false unconditionally in the `extension dev` exec env (was setdefault, which let a developer's global lazy-loading export silently hide their dev components from the palette and miss AC #4's 5s budget). Plus three MEDIUMs: - sys.modules cleanup in test_generated_test_file_runs_against_generated_component so a later test importing the same dotted path doesn't pick up a stale module from a deleted tmp_path. Component instantiation moved inside the try block because Component.__init__ uses inspect.getsourcefile against self.__class__'s still-live module. - Added regex-drift test that pins the init_template patterns to match manifest.py's so a schema regex change can't quietly produce invalid scaffolded manifests. - Added two new dev_registry tests covering the forward-all-warnings contract and the source_path=None defense. Tests: 201 passing (4 new); ruff check + format clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: LangflowCompat -> LfxCompat * fix: Remove references to ticket * fix: encode maxItems constraint in schema * fix: deferred fields and schema version * Fix ruff errors * fix: align loader tests with renamed manifest API and strip ticket refs The merge of feat/extension-validate brought in the LfxCompat / compat rename, but the loader test fixtures still used LangflowCompat / bundle_api and failed at the manifest layer. Update conftest.py + test_load_extension.py to the post-rename API. Strip 17 LE-XXXX ticket references from loader source files, errors.py loader-specific comments, and the four loader test docstrings, matching the convention applied to LE-1014. Descriptive prose preserved. Promote _BUNDLE_NAME_RE to BUNDLE_NAME_RE on the manifest module so the loader's orchestrator no longer reaches across modules into a private name. * fix(lfx): align init template with renamed manifest API After merging feat/extension-loader into this branch, two surfaces fell out of sync with the renamed manifest schema: - init_template generates extension.json with `lfx: {bundle_api: [1]}`, but the validator now requires `lfx: {compat: ["1"]}` per LfxCompat. This made `test_basic_template_validates_clean` fail with manifest-invalid (`lfx.compat: Field required; lfx.bundle_api: Extra inputs are not permitted`). - test_init_template_regexes_match_manifest_schema reads `manifest_mod._BUNDLE_NAME_RE`, but that symbol was promoted to public `BUNDLE_NAME_RE` in c63f84a. Update the test to reference the public name; init_template's local copy is still private since it also covers `_EXTENSION_ID_RE`, which remains private upstream. --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
…ok (#13024) * feat(lfx): add extension manifest schema, validate CLI, and error formatter (LE-1014) Foundation for the Bundle Separation iteration. Defines what a valid extension.json looks like (Pydantic models + Draft 2020-12 JSON Schema), ships the offline `lfx extension validate` command, and ships the single `format_extension_error` function that every other extension-system module will use to render structured errors. What's in lfx.extension: - `ExtensionManifest` / `BundleRef` / `LangflowCompat` Pydantic models with `extra="forbid"` so unknown fields fail loudly. Deferred fields (`services`, `routes`, `hooks`, `starter_projects`, `userConfig`) are reserved as None-only so non-null values produce a dedicated `field-deferred-in-this-milestone` error instead of a generic schema wall. `bundles` accepts a list but rejects length > 1 with `multi-bundle-deferred-in-this-milestone` (validator-enforced; the loader re-checks at install time in LE-1015). - `schema.build_schema()` + `build_schema_json()` produce the publishable artifact at schemas.langflow.org/extension/v1.json. - `ExtensionError` typed envelope and `format_extension_error` -- one branch per discriminant. Codes are registered in `ERROR_CODES`; an `ExtensionError` constructed with an unknown code raises at construction time, preventing producers from shipping without a matching renderer. - `validate_extension` runs four passes: manifest discovery + schema, path-safety (no `..`, no absolute paths, no symlink escape), AST inspection of every `.py` (syntax, Component subclass present, build() declared, top-level `import *`, top-level I/O primitives), and an opt-in `--execute-imports` that runs each module in a subprocess with a temporary HOME / TMPDIR / LANGFLOW_CONFIG_DIR and LANGFLOW_*/LFX_* env vars stripped. - `lfx extension validate` and `lfx extension schema` typer subcommands. Acceptance-criteria coverage in tests/unit/extension/: - Round-trips every v0 manifest field; deferred fields rejected; multi-bundle rejected with the dedicated discriminant. - JSON Schema validates the v0 example and rejects 12 malformed manifests with distinct error paths (>= 10 required by the ticket). - Median default-validate runtime < 100ms on the basic template. - Crafted side-effect bundle: default validate does NOT execute it (canary file is never written); `--execute-imports` DOES execute it and the canary appears, while LANGFLOW_* env vars are NOT inherited by the subprocess. - Snapshot tests for every code in ERROR_CODES; a guard test verifies ERROR_CODES and the snapshot table are in lockstep so future additions cannot ship without a format branch and a snapshot. Wiring: `lfx extension` is a sub-app under the Authoring help panel so future tickets (LE-1016 init/dev, LE-1018 reload) can attach without colliding with the existing `lfx validate` (which validates flow JSON, not extensions). * fix(lfx): fall back to tomli on Python 3.10 in extension manifest loader tomllib is stdlib only on 3.11+, but lfx supports 3.10-3.13. Use the existing tomli runtime dependency as the 3.10 fallback (same API, so the import alias keeps the rest of the module unchanged). Fixes ModuleNotFoundError seen in CI on the 3.10 job. * Update src/lfx/tests/unit/extension/test_schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/extension/schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/cli/_extension_commands.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (LE-1015) Introduces lfx.extension.loader: the runtime that turns an Extension on disk into LoadedComponent records keyed by ext:<bundle>:<Class>@<slot>. Two paths in: - load_extension(root): one manifest, one Bundle, registered at @official. Re-checks multi-bundle at runtime (defense-in-depth vs. the schema). - discover_inline_bundles(paths): each subfolder of LANGFLOW_COMPONENTS_PATH is a Bundle at @extra. Walk order is platform-independent (sorted dirs, user-declared path order). First-wins on duplicate names; second emits duplicate-inline-bundle warning that names both paths. Manifest-first precedence helpers (installed_extension_roots, manifest_owning_distributions, filter_plugin_entry_points) let callers of the legacy langflow.plugins entry-point loader skip distributions that ship a manifest, so component entry-points are not double-registered. New typed error codes: module-import-failed, duplicate-component-name, duplicate-distribution, duplicate-inline-bundle, inline-bundle-name-invalid. Each ships with a format branch and a snapshot test. Tests cover the AC: single-bundle happy path, multi-bundle rejection, missing/empty/no-Component bundle, duplicate class names, deterministic walk order, recursive discovery, inline-bundle first-wins + dot-dir skip + bundle.json metadata, manifest-first precedence partition, PEP-503 distribution-name canonicalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1015 review feedback - Drop the unproduced duplicate-distribution error code; LE-1022 will add it once startup-time discovery has a place to surface it. Restores the invariant that every code in ERROR_CODES has a producer. - Clarify that intra-bundle relative imports are NOT supported in v0; only absolute references between bundle modules work in this milestone. - Use strict=False when re-resolving bundle_root in the walker; the path was already existence-checked, and a concurrent removal in the narrow window should not raise across the loader's public boundary. - Hoist the json import out of _read_inline_bundle_json's body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lfx): split extension loader into a small subpackage (LE-1015) Addresses the file-size feedback from the LE-1015 review. The single 870-line loader.py becomes a flat package keyed off the four section banners that already existed inline: loader/ __init__.py # re-exports the public surface _types.py # SLOT constants, LoadedComponent, LoadResult _discovery.py # filesystem walk + importlib.util orchestration _detection.py # Component subclass identification (MRO heuristic) _orchestrator.py # load_extension, discover_inline_bundles _plugins.py # manifest-first precedence over langflow.plugins Largest file is now _orchestrator.py at 440 LOC (was 870); every file is well under the 800-LOC project guideline. No behavior change: the public import paths from lfx.extension are unchanged, all 38 loader tests still pass. ``_canonicalize_distribution`` is now exported as ``canonicalize_distribution`` from ``loader._plugins`` (it's a stable PEP-503 helper that downstream modules will reach for, so it loses the private underscore). The test suite is split to mirror the package: tests/unit/extension/loader/ conftest.py # shared fixtures, FakeDist, autouse scrub test_load_extension.py # @official slot, identity, failure modes test_inline_bundles.py # LANGFLOW_COMPONENTS_PATH, @extra slot test_plugins.py # manifest-first precedence helpers test_types.py # LoadedComponent, LoadResult, code parity Each test file imports its own slice of the public API and pulls fixtures from conftest, so a reader looking at one banner can read it in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: LangflowCompat -> LfxCompat * fix: Remove references to ticket * fix: encode maxItems constraint in schema * fix: deferred fields and schema version * Fix ruff errors * fix: align loader tests with renamed manifest API and strip ticket refs The merge of feat/extension-validate brought in the LfxCompat / compat rename, but the loader test fixtures still used LangflowCompat / bundle_api and failed at the manifest layer. Update conftest.py + test_load_extension.py to the post-rename API. Strip 17 LE-XXXX ticket references from loader source files, errors.py loader-specific comments, and the four loader test docstrings, matching the convention applied to LE-1014. Descriptive prose preserved. Promote _BUNDLE_NAME_RE to BUNDLE_NAME_RE on the manifest module so the loader's orchestrator no longer reaches across modules into a private name. * feat(lfx): append-only migration table + flow deserializer rewrite hook Adds the migration layer of the Extension System: an append-only JSON table that maps three legacy component reference shapes (bare class name, old import path, pre-Phase-A namespaced slot) to the post-Phase-A canonical ext:<bundle>:<Class>@<slot> identifier, plus a deserializer hook that rewrites a saved-flow payload in place against that table on load. What landed: * lfx.extension.migration.schema -- Pydantic models for MigrationEntry + MigrationTable with per-entry validators (exactly one of bare/import/slot populated; canonical target shape) and table-level uniqueness check. * lfx.extension.migration.loader -- canonical in-repo path, threadsafe process-lifetime cache, typed errors on every failure mode. * lfx.extension.migration.rewrite -- node-by-node rewrite, idempotent on canonical refs, difflib-backed closest-match suggestion for unmapped references, cross-bucket ambiguity surfaces component-name-ambiguous instead of silently loading into the wrong bundle. * Wired into Graph.from_payload before validate_flow_for_current_settings so every saved-flow load goes through migration first. * scripts/migrate/check_migration_append_only.py -- CI guard that diffs the working-tree table against origin/main and rejects removals or target mutations; reordering and additions are allowed. * 34 new unit tests covering rewrite paths, loader failure modes, schema invariants, and the CI script behavior. What is deliberately deferred: * flow-migrated event emission. The events pipeline is unavailable in this iteration; the wiring point in Graph.from_payload is marked TODO and the MigrationReport already carries every field a future emitter needs. The shipped migration_table.json starts empty; entries land alongside the pilot bundle extraction in a follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This comment has been minimized.
This comment has been minimized.
* feat(lfx): add extension manifest schema, validate CLI, and error formatter (LE-1014) Foundation for the Bundle Separation iteration. Defines what a valid extension.json looks like (Pydantic models + Draft 2020-12 JSON Schema), ships the offline `lfx extension validate` command, and ships the single `format_extension_error` function that every other extension-system module will use to render structured errors. What's in lfx.extension: - `ExtensionManifest` / `BundleRef` / `LangflowCompat` Pydantic models with `extra="forbid"` so unknown fields fail loudly. Deferred fields (`services`, `routes`, `hooks`, `starter_projects`, `userConfig`) are reserved as None-only so non-null values produce a dedicated `field-deferred-in-this-milestone` error instead of a generic schema wall. `bundles` accepts a list but rejects length > 1 with `multi-bundle-deferred-in-this-milestone` (validator-enforced; the loader re-checks at install time in LE-1015). - `schema.build_schema()` + `build_schema_json()` produce the publishable artifact at schemas.langflow.org/extension/v1.json. - `ExtensionError` typed envelope and `format_extension_error` -- one branch per discriminant. Codes are registered in `ERROR_CODES`; an `ExtensionError` constructed with an unknown code raises at construction time, preventing producers from shipping without a matching renderer. - `validate_extension` runs four passes: manifest discovery + schema, path-safety (no `..`, no absolute paths, no symlink escape), AST inspection of every `.py` (syntax, Component subclass present, build() declared, top-level `import *`, top-level I/O primitives), and an opt-in `--execute-imports` that runs each module in a subprocess with a temporary HOME / TMPDIR / LANGFLOW_CONFIG_DIR and LANGFLOW_*/LFX_* env vars stripped. - `lfx extension validate` and `lfx extension schema` typer subcommands. Acceptance-criteria coverage in tests/unit/extension/: - Round-trips every v0 manifest field; deferred fields rejected; multi-bundle rejected with the dedicated discriminant. - JSON Schema validates the v0 example and rejects 12 malformed manifests with distinct error paths (>= 10 required by the ticket). - Median default-validate runtime < 100ms on the basic template. - Crafted side-effect bundle: default validate does NOT execute it (canary file is never written); `--execute-imports` DOES execute it and the canary appears, while LANGFLOW_* env vars are NOT inherited by the subprocess. - Snapshot tests for every code in ERROR_CODES; a guard test verifies ERROR_CODES and the snapshot table are in lockstep so future additions cannot ship without a format branch and a snapshot. Wiring: `lfx extension` is a sub-app under the Authoring help panel so future tickets (LE-1016 init/dev, LE-1018 reload) can attach without colliding with the existing `lfx validate` (which validates flow JSON, not extensions). * fix(lfx): fall back to tomli on Python 3.10 in extension manifest loader tomllib is stdlib only on 3.11+, but lfx supports 3.10-3.13. Use the existing tomli runtime dependency as the 3.10 fallback (same API, so the import alias keeps the rest of the module unchanged). Fixes ModuleNotFoundError seen in CI on the 3.10 job. * Update src/lfx/tests/unit/extension/test_schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/extension/schema.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * Update src/lfx/src/lfx/cli/_extension_commands.py Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> * feat(lfx): add single-Bundle loader and LANGFLOW_COMPONENTS_PATH discovery (LE-1015) Introduces lfx.extension.loader: the runtime that turns an Extension on disk into LoadedComponent records keyed by ext:<bundle>:<Class>@<slot>. Two paths in: - load_extension(root): one manifest, one Bundle, registered at @official. Re-checks multi-bundle at runtime (defense-in-depth vs. the schema). - discover_inline_bundles(paths): each subfolder of LANGFLOW_COMPONENTS_PATH is a Bundle at @extra. Walk order is platform-independent (sorted dirs, user-declared path order). First-wins on duplicate names; second emits duplicate-inline-bundle warning that names both paths. Manifest-first precedence helpers (installed_extension_roots, manifest_owning_distributions, filter_plugin_entry_points) let callers of the legacy langflow.plugins entry-point loader skip distributions that ship a manifest, so component entry-points are not double-registered. New typed error codes: module-import-failed, duplicate-component-name, duplicate-distribution, duplicate-inline-bundle, inline-bundle-name-invalid. Each ships with a format branch and a snapshot test. Tests cover the AC: single-bundle happy path, multi-bundle rejection, missing/empty/no-Component bundle, duplicate class names, deterministic walk order, recursive discovery, inline-bundle first-wins + dot-dir skip + bundle.json metadata, manifest-first precedence partition, PEP-503 distribution-name canonicalization. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(lfx): address LE-1015 review feedback - Drop the unproduced duplicate-distribution error code; LE-1022 will add it once startup-time discovery has a place to surface it. Restores the invariant that every code in ERROR_CODES has a producer. - Clarify that intra-bundle relative imports are NOT supported in v0; only absolute references between bundle modules work in this milestone. - Use strict=False when re-resolving bundle_root in the walker; the path was already existence-checked, and a concurrent removal in the narrow window should not raise across the loader's public boundary. - Hoist the json import out of _read_inline_bundle_json's body. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(lfx): split extension loader into a small subpackage (LE-1015) Addresses the file-size feedback from the LE-1015 review. The single 870-line loader.py becomes a flat package keyed off the four section banners that already existed inline: loader/ __init__.py # re-exports the public surface _types.py # SLOT constants, LoadedComponent, LoadResult _discovery.py # filesystem walk + importlib.util orchestration _detection.py # Component subclass identification (MRO heuristic) _orchestrator.py # load_extension, discover_inline_bundles _plugins.py # manifest-first precedence over langflow.plugins Largest file is now _orchestrator.py at 440 LOC (was 870); every file is well under the 800-LOC project guideline. No behavior change: the public import paths from lfx.extension are unchanged, all 38 loader tests still pass. ``_canonicalize_distribution`` is now exported as ``canonicalize_distribution`` from ``loader._plugins`` (it's a stable PEP-503 helper that downstream modules will reach for, so it loses the private underscore). The test suite is split to mirror the package: tests/unit/extension/loader/ conftest.py # shared fixtures, FakeDist, autouse scrub test_load_extension.py # @official slot, identity, failure modes test_inline_bundles.py # LANGFLOW_COMPONENTS_PATH, @extra slot test_plugins.py # manifest-first precedence helpers test_types.py # LoadedComponent, LoadResult, code parity Each test file imports its own slice of the public API and pulls fixtures from conftest, so a reader looking at one banner can read it in isolation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(lfx): atomic-swap Bundle reload pipeline + endpoint + CLI (LE-1018) Five-stage reload (parallel staging load -> validate -> swap under write lock -> cleanup -> emit) for installed Bundles in Mode A. In-flight flows keep the pre-swap class via existing references; new flows pick up the post-swap class atomically; concurrent reloads on the same Bundle are rejected with reload-in-progress. Adds: * lfx/extension/registry.py -- BundleRegistry with per-bundle reload-in-progress guard and components_index.json writer * lfx/extension/reload.py -- the five-stage pipeline; events emission is stubbed (TODO LE-1017) so the swap mechanics can ship before the events service lands * loader: optional module_namespace param so Stage 1 lands in __reload_staging__.<id> instead of the live _lfx_ext.* namespace * errors: four new typed reload codes (reload-in-progress, reload-bundle-not-installed, reload-bundle-name-mismatch, reload-source-missing) with branch templates and snapshot tests * HTTP: POST /api/v1/extensions/{id}/bundles/{name}/reload, gated by the existing get_current_active_user dependency, returns 409 with a typed body for the in-progress collision case * CLI: lfx extension reload <id> [--bundle <name>] -- HTTP client against the dev server with text/json output and proper exit codes (--all is gated until LE-1019 lands the list endpoint) * tests: 16 reload-pipeline tests covering the AC matrix (rename round-trip, broken-bundle isolation, concurrent readers, in-flight flow, double-reload guard, bundle-name mismatch) plus 9 CLI client tests Mode A only. In Mode B/C bundle changes require a Docker image rebuild and the reload path is not exercised. The events emission in Stage 5 is intentionally stubbed -- the LE-1017 ticket will swap the body of _emit_bundle_reload_event in one place without touching the pipeline core. * feat(frontend): palette Bundle reload action + loading + toasts Adds the frontend half of the bundle reload flow. When the Bundle header is right-clicked or its overflow ("⋮") icon clicked, a Reload action fires POST /api/v1/extensions/{id}/bundles/{name}/reload and surfaces the result via the existing alert-store toast system. What landed (frontend only): * src/controllers/API/queries/extensions/ -- typed wire-format models (ReloadBundleResponse, ExtensionErrorPayload, ReloadInProgressDetail) and the useReloadBundle mutation hook. The hook unwraps the 409 `reload-in-progress` detail into a stable, parseable Error message so the UI can branch without reading status codes. * components/bundleHeaderActions.tsx -- new Select-based overflow menu next to the Bundle header chevron. Three toast paths: success (green, with components +/- delta), structural failure (red, with typed errors and inline hints), reload-in-progress (notice). Loading state swaps the kebab icon for a spinning Loader2 while the request is in flight. Renders nothing when no extension_id is on the bundle, so the static SIDEBAR_BUNDLES list is unaffected. * components/bundleItems.tsx -- wires the new actions in next to the chevron, plus a context-menu (right-click) capture that opens the same overflow trigger so keyboard / mouse / right-click all share one source of truth. * types/index.ts -- BundleItemProps.item gains an optional extension_id; took the opportunity to extract the SidebarBundle interface and tighten three pre-existing `any` types. * customization/feature-flags.ts -- ENABLE_EXTENSION_RELOAD gate, off by default until the bundle-list endpoint that populates extension_id per bundle ships. * controllers/API/helpers/constants.ts -- EXTENSIONS URL constant. * Tests: 7 component tests + 3 mutation-hook tests, all green. Total sidebar + extensions test count: 479 passing. Deferred: * Event-pipeline subscription is left as an inline TODO. The mutation response carries enough information to drive the toasts on its own today; once the events service lands the toast wiring will move to a `bundle_reloaded` / `bundle_reload_failed` listener so multi-tab and multi-worker swaps surface exactly once. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Build successful! ✅ |
|
Build successful! ✅ |
Cristhianzl
left a comment
There was a problem hiding this comment.
⛔ Blockers (resolve before merge)
B1 — Hard file-size-limit violations on the riskiest modules
Files:
src/lfx/src/lfx/extension/loader/_orchestrator.py— 801 linessrc/lfx/src/lfx/extension/reload.py— 721 linessrc/lfx/src/lfx/cli/_extension_commands.py— ~670 linessrc/lfx/src/lfx/extension/discovery.py— 558 linessrc/lfx/src/lfx/extension/init_template.py— 521 lines
Issue: The structural limit is 500 lines, with 600–700 tolerated only if SRP and responsibility-separation are impeccable. _orchestrator.py (801) and reload.py (721) exceed even the 700 ceiling, and both mix multiple responsibility prefixes in one module:
_orchestrator.py: path-safety (_resolve_bundle_path), directory walking/registration (_load_bundle_directory), manifest loading (load_extension), inline JSON parsing + discovery (_read_inline_bundle_json,discover_inline_bundles), and two startup-discovery flows (load_installed_extensions,load_seed_extensions).reload.py: pipeline orchestration, source resolution,sys.modulessurgery (_swap_sys_modules,_drop_staging_modules,_retag_component), diffing, result building, and a post-swap hook registry.
Why it matters: These are the two highest-risk files in the change (dynamic import + runtime sys.modules mutation). Per the project rules this is an explicit rejection criterion, and the sys.modules surgery primitives are a cohesive, separately-testable unit currently buried in an 800-line file.
Suggested fix:
_orchestrator.py→ split into core loader,_inline.py(inline bundle discovery/parsing),_startup.py(load_installed_extensions/load_seed_extensions).reload.py→ extractreload_swap.py(thesys.modulesprimitives) and fold the post-swap hook registry into the events stub orreload_hooks.py._extension_commands.py→ one module per command (or at least splitdevlaunch helpers).discovery.py→ splitinstalledvsseedhalves.init_template.py→ move the static content builders intoinit_template_content.py.
B2 — --execute-imports strips only LANGFLOW_*/LFX_*, leaking developer cloud credentials into untrusted bundle import
File: src/lfx/src/lfx/extension/validate.py:675
Issue: The probe docstring states the subprocess runs isolated so the bundle "can't read server state," but the env filter is a denylist of two prefixes:
Code reference
env = {k: v for k, v in os.environ.items() if not k.startswith(("LANGFLOW_", "LFX_"))}Every other environment variable — AWS_SECRET_ACCESS_KEY, OPENAI_API_KEY, GITHUB_TOKEN, CI secrets — is inherited by the subprocess, which then exec_module()s every .py in the (untrusted) bundle with no sandbox.
Why it matters: A malicious bundle's top-level import can exfiltrate the developer's cloud credentials during the step the author was explicitly told is the safe way to inspect an untrusted extension. The implementation contradicts its own documented guarantee.
Suggested fix: Invert to an allowlist — keep only PATH, LANG, LC_*, SYSTEMROOT, TMPDIR, and the temp HOME you set. Separately, downgrade the validator's security wording (see I2) and document in the --execute-imports CLI help that it executes untrusted code with no sandbox.
B3 — Discovered installed/seed manifests reach the dynamic loader without validate_extension / path-safety
Files: src/lfx/src/lfx/extension/discovery.py:224-281 (_build_installed_record), src/lfx/src/lfx/extension/discovery.py:443-468 (_build_seed_record), src/lfx/src/lfx/extension/discovery.py:397-418 (_iter_seed_subdirectories)
Issue: manifest.py's own docstring states that real path-safety (symlink escape, traversal) "is enforced by validate_extension" — BundleRef._validate_path_shape only rejects literal ../absolute syntactically. But the production-install discovery path (discover_installed_extensions / discover_seed_extensions) parses the manifest and emits a DiscoveredExtension that flows into the loader (which imports code) without ever calling validate_extension or an equivalent path-escape check. Compounding this, _iter_seed_subdirectories follows symlinks via child.is_dir() with no containment check, while its docstring claims the opposite:
Code reference — discovery.py:402-418
"""... Symlinks are followed, but only if their target stays on
disk -- a dangling link is silently dropped."""
...
for child in children:
if child.name.startswith(".") or child.name in skip_names:
continue
try:
if not child.is_dir(): # follows symlink to ANYWHERE; no resolve()/containment
continue
except OSError:
continue
yield childThe sibling loader (loader/_discovery.py:69-74) does perform resolved.relative_to(bundle_resolved) — discovery does not, so the two walkers disagree on the trust boundary.
Why it matters: The seed dir is operator-staged (lower-trust than request input), but the documented trust model — "path safety is enforced by validate_extension" — is simply not honored on the new production path, and the docstring actively misleads a reviewer into thinking it is. A symlinked seed subdir or a manifest bundles[0].path symlink reaches exec_module() unchecked.
Suggested fix: Run the path-safety pass (or full validate_extension) on every DiscoveredExtension before it is registered/loaded; add the resolve()-and-relative_to containment check to _iter_seed_subdirectories (reuse the helper from B5); and correct the docstring to match real behavior. Add a test pinning that a symlink-escaping seed dir is rejected with a typed error.
B4 — Three divergent implementations of bundle shadow/precedence that already disagree on collision behavior
Files: src/lfx/src/lfx/extension/discovery.py:515-558 (discover_all_extensions), src/lfx/src/lfx/interface/components.py (_resolve_bundle_shadowing), src/lfx/src/lfx/extension/registry.py:248-269 (ExtensionRegistry._register)
Issue: There are three independent resolvers for the same conceptual conflict, with different keys and different outcomes:
discover_all_extensionsdedupes onextension_id, emitsseed-bundle-shadowed._resolve_bundle_shadowingdedupes on bundle name across 4 sources with a different precedence tuple and mints 2 different codes; the registry-population loop theninstall_bundles by bundle name — last-wins, silent.ExtensionRegistry._registerkeys onextension_idand raisesDuplicateExtensionError.
So whether a same-named collision surfaces a typed error or silently overwrites depends entirely on which resolver/registry the caller reaches and on whether the collision is on id vs name. The production palette path (import_extension_components → BundleRegistry.install_bundle) is the silent last-wins one.
Why it matters: This is a CRITICAL DRY violation and a correctness bug — the divergence (id-keyed vs name-keyed) is not theoretical; the implementations already produce different shadow outcomes for the same input. The duplicated _distribution_manifest_path / _pyproject_declares_extension between loader/_plugins.py and discovery.py already disagree on editable installs (the _plugins copy has an _editable_manifest_path fallback the discovery copy lacks).
Suggested fix: Converge on one identity (id or name) and one resolver. The production palette path should consume discover_all_extensions/build_registry_from_discovery rather than reimplement precedence in the interface-cache layer; make install_bundle reject (not silently overwrite) a same-key record. Collapse the duplicated manifest-resolution helpers onto a single implementation.
B5 — Three independent copies of the symlink-escape trust boundary
Files: src/lfx/src/lfx/extension/loader/_orchestrator.py:87-130 (_resolve_bundle_path), src/lfx/src/lfx/extension/loader/_discovery.py:57-74 (per-file re-resolve), plus the missing-but-needed check in _iter_seed_subdirectories (B3) and _plugins/discovery.
Issue: The "resolve and relative_to(root)" containment check is reimplemented at least three times with subtly different strictness, and the byte-identical skip-dir set ({"__pycache__", ".git", ".venv", "venv", "node_modules", ".pytest_cache"}) is duplicated between discovery.py:404 and loader/_discovery.py:36 (SKIP_DIR_NAMES).
Why it matters: This is a security boundary. Three copies that can drift independently means a future tightening applied to one path silently leaves the others permissive — which is exactly the state B3 documents (one path has the check, another does not).
Suggested fix: Extract a single audited is_within(child: Path, root: Path) -> bool and a single SKIP_DIR_NAMES, imported by every walker (_resolve_bundle_path, iter_bundle_python_files, _iter_seed_subdirectories, the inline path, _plugins).
⚠️ Important (preferably this PR)
I1 — Synchronous reload_bundle runs on the async event loop
File: src/backend/base/langflow/api/v1/extensions.py:116
Issue: async def reload_extension_bundle calls result = reload_bundle(registry, bundle_name) directly — no await, no threadpool offload. reload_bundle does blocking disk I/O, importlib execution of arbitrary extension code, file hashing, and takes a threading.RLock.
Why it matters: A slow or large bundle import freezes the entire event loop for that worker — every other HTTP request stalls until reload completes. The reload.py docstring asserts "concurrent readers" work, but the async server is single-threaded and this entry point is not loop-friendly. (Blast radius limited: Mode A only, opt-in flag, authenticated.)
Suggested fix: await run_in_threadpool(reload_bundle, registry, bundle_name) (or asyncio.to_thread). Document that the entry point must be called off-loop.
I2 — AST "no top-level I/O" / "no import *" checks are trivially bypassable but presented as security
File: src/lfx/src/lfx/extension/validate.py:56-75
Issue: _find_top_level_io / _find_top_level_import_star only match literal open/socket/subprocess names on bare ast.Name/Name.Attribute callees. __import__("os").system(...), getattr(os, "sys"+"tem"), exec(base64.b64decode(...)), eval, importlib.import_module, aliased imports, or I/O via any third-party lib are all undetected. The module docstring frames false negatives as "how malicious bundles slip through."
Why it matters: This is import-time hygiene, not a security boundary, and is being marketed as one — a false sense of security for operators.
Suggested fix: Add exec, eval, __import__, compile to the name set, and downgrade the module/CLI wording to "best-effort hygiene lint, not a sandbox."
I3 — Destructive, non-rollback half-swap window in _swap_sys_modules
File: src/lfx/src/lfx/extension/reload.py:578-595
Issue: Old modules are popped from sys.modules (578-580) before zip(staging_list, new_list, strict=True) builds the rename map (583-585). If strict=True ever raises (length mismatch), the old modules are already gone, the new ones were never installed, and the finally only purges staging — it does not restore the dropped prod modules.
Why it matters: The invariant means this "should never" fire, but if it does, the failure mode is the worst possible: every in-flight resolution of the old classes breaks with no rollback — the exact broken state the module exists to prevent.
Suggested fix: Build the rename map before dropping old modules, or snapshot the popped modules and restore them in an except before re-raising. Add a Why: comment marking the strict=True as an invariant tripwire and make it non-destructive.
I4 — The load-bearing cls.__module__ retag is silently suppressed
File: src/lfx/src/lfx/extension/reload.py:592-594, 606-608
Issue: Both module.__name__ and cls.__module__ rewrites use contextlib.suppress(AttributeError, TypeError). The block's own docstring states that if cls.__module__ is not retagged, inspect.getmodule(cls) returns None and the post-swap cache rebuild silently breaks (the "empty palette after reload" bug this feature exists to fix).
Why it matters: Swallowing the exception that signals the exact failure the module is designed to prevent lets that bug silently regress with zero diagnostics.
Suggested fix: Log at warning inside the suppress (or use try/except that logs), and surface a ReloadResult.warnings entry for the cls.__module__ case.
I5 — dev_registry._read_state swallows unreadable/corrupt state as an empty registry
File: src/lfx/src/lfx/extension/dev_registry.py:142-143
Issue: except (OSError, ValueError): return [] — a PermissionError, a partially-written file, or a hand-edit typo makes every registered dev extension silently vanish from the next langflow run with no diagnostic. A PermissionError does not "self-heal."
Why it matters: Classic silent-failure in a config-load path; the author burns a debug cycle wondering why their extension disappeared. Also: the dev state file feeds arbitrary path strings straight into load_extension (code load) — any process able to write dev_extensions.json gets code loaded at startup. Consider 0600 perms on write and documenting the trust boundary.
Suggested fix: Distinguish "file absent" (legitimately empty) from "present but unreadable/corrupt" — surface a typed warning to the startup log in the latter case instead of return [].
I6 — _pyproject_declares_extension collapses real OSError into "not a manifest"
File: src/lfx/src/lfx/extension/discovery.py:200-221
Issue: except OSError: return False treats a permission/I/O error on pyproject.toml identically to "no [tool.langflow.extension] section." A package that does ship an extension but whose pyproject is briefly unreadable is silently dropped — the manifest-unreadable typed error in _build_installed_record is never reached.
Why it matters: Failed-check-passes pattern; a legitimately broken install disappears instead of surfacing an actionable typed error.
Suggested fix: Distinguish "section absent" (skip) from "could not read file" (surface or propagate so the typed-error path is reached).
I7 — _entry_point_loads_to_component executes third-party code at filter time, inconsistent with its sibling
File: src/lfx/src/lfx/extension/loader/_plugins.py:381-397
Issue: The default predicate calls ep.load() (imports the target module) purely to classify an entry point, with except BaseException swallowing SystemExit/KeyboardInterrupt into a debug log. The sibling _manifest_via_entry_point (lines 130-164) deliberately uses find_spec "so a manifest-discovery pass at startup does not trigger arbitrary side-effects." The riskier behavior is the default and the asymmetry is unexplained.
Why it matters: Module-level code in a manifest-shipping distribution runs as a side effect of a filter, potentially at startup, contradicting the stated design of the sibling helper.
Suggested fix: Make the predicate lazier (avoid ep.load()), or document why eager load is acceptable here when _manifest_via_entry_point explicitly avoids it. Narrow the except BaseException so it does not swallow KeyboardInterrupt/SystemExit unconditionally.
I8 — Cross-source @official bundle-name collision is undetected
File: src/lfx/src/lfx/extension/loader/_orchestrator.py (load_installed_extensions / load_seed_extensions)
Issue: Inline discovery has first-wins dedupe (seen_names), but two distributions with different canonical names but the same bundle.name both load at @official into _lfx_ext.official.<name>.*, silently clobbering each other in sys.modules. The only cross-source check (duplicate-distribution) keys on distribution name, not bundle name.
Why it matters: The second bundle's modules overwrite the first's; LoadedComponent.klass references from the first become orphaned; namespaced_id collides at the registry with no loader diagnostic. (Closely related to B4 — same root cause, different surface.)
Suggested fix: Add a bundle-name-collision check across the aggregated installed+seed results (analogous to the inline seen_names), or document and enforce uniqueness at a single named layer.
I9 — Frontend: reload success warnings are computed then silently dropped
File: src/frontend/src/pages/FlowPage/components/flowSidebarComponent/components/bundleHeaderActions.tsx:114-118
Issue: renderTypedErrorList(data.warnings) is computed and passed as list into setSuccessData, but alertStore.setSuccessData is typed (newState: { title: string }) => void and only forwards title. The spread (...(list ? { list: list.list } : {})) also evades type-checking.
Why it matters: A reload that succeeds with warnings (deprecated/partial component) shows a green toast and the user never sees the warnings — exactly the case where feedback matters.
Suggested fix: Extend setSuccessData to accept/forward list and render it, or route reload-with-warnings through setNoticeData/setErrorData which support list.
💡 Recommended (can ship as a follow-up)
R1 — Migration logging uses {}+kwargs while the rest of the PR uses %s
File: src/lfx/src/lfx/graph/graph/base.py:1199-1206
Issue: logger.warning("extension migration: {code} flow_id={flow_id} ...", code=..., flow_id=...). The logger here is structlog, so this does not crash (kwargs are captured as structured fields, which is the documented intent) — but the human-readable message string renders with literal {code} placeholders, and the style is inconsistent with every other logging call in the PR (logger.error("...: %s", x)).
Why it matters: Log-quality / consistency only — verified non-fatal. Worth aligning so console output is readable.
Suggested fix: Either use structured-only logging consistently, or match the %s style used elsewhere. Add a test that loads a flow with an unmapped reference and asserts no exception (cheap regression guard).
R2 — Unbounded difflib suggestion cost on the flow-load request path
File: src/lfx/src/lfx/extension/migration/rewrite.py:132-142, 378-396
Issue: migrate_flow_payload runs on every flow load (including API-supplied payloads). _closest_matches calls difflib.get_close_matches per unmapped node with no cap on node count. The table is tiny today, but it is append-only and grows every release.
Why it matters: An adversarial/large flow with thousands of unmapped nodes triggers O(nodes × table) ratio computations on the request path. Low impact now, latent later.
Suggested fix: Only compute suggestions when node count is below a threshold, or memoize per legacy_value.
R3 — _components_path_extension_paths compares paths by raw string equality
File: src/lfx/src/lfx/interface/components.py (_components_path_extension_paths)
Issue: if raw == BASE_COMPONENTS_PATH: continue then Path(raw). A trailing slash, ./ prefix, symlink, or case difference (Windows/macOS) means the base components dir is not excluded and gets walked as an inline-bundle root, producing duplicate/garbage palette entries.
Suggested fix: Compare resolved paths: Path(raw).resolve() == Path(BASE_COMPONENTS_PATH).resolve().
R4 — components_index.json write swallows all OSError silently
File: src/lfx/src/lfx/extension/bundle_registry.py:274-281
Issue: Bare except OSError: pass. A persistent permission/disk-full error yields a permanently stale index with zero log output, indistinguishable from "never written."
Suggested fix: Keep non-fatal behavior but add logger.warning("failed to write components_index.json: %s", exc).
R5 — Add adversarial tests for the reload entry point and de-flake the concurrency test
Files: src/lfx/tests/unit/extension/test_reload.py
Issue: Test coverage is strong and adversarial overall (symlink-escape, concurrent readers, ambiguous migration markers, etc.), but: (a) no traversal-style input test feeds ..//absolute bundle_name/extension_id to reload_bundle/the route — the highest-impact failure mode for a coordinate that lands in a filesystem path; (b) test_concurrent_readers_see_pre_or_post_state relies on time.sleep(0.05) to open the race window, which can silently degrade under CI load (pass without exercising the race).
Suggested fix: Add a traversal/absolute-path test for the reload entry point; replace the sleep with a threading.Barrier/started-event so the race window is deterministic.
R6 — No coverage metric stated in the PR
Issue: The PR description is detailed but states no coverage percentage and shows no coverage report. The rule set asks for coverage to be run and shown (≥75%, target 80%) for created tests.
Suggested fix: Run coverage on the new lfx/extension package and paste the summary into the PR description.
🟢 Nice to have
- N1 —
src/lfx/src/lfx/extension/loader/_orchestrator.py:375: errorcode="multi-bundle-deferred-in-this-milestone"bakes a transient milestone phrase into a stable, machine-matched typed code (others are stable nouns:path-escape,bundle-empty). Usemulti-bundle-unsupported; convey the milestone nuance inmessage/hint. - N2 —
BUNDLE_NAME_RE/ extension-id regex duplicated betweenmanifest.pyandinit_template.py:65,70(the docstring even admits "Drift here is caught by the AC test"). Import the single source frommanifest.pyinstead of recompiling a security-relevant identifier pattern. - N3 —
src/lfx/src/lfx/extension/loader/_orchestrator.py:412-416: comment says "Onlyversionis read in v0," butidis read at lines 514/637 and becomesextension_id(registry attribution) withoutBUNDLE_NAME_REvalidation, unlike the directory name. Update the comment and validate thebundle.jsonid. - N4 —
src/lfx/src/lfx/cli/_extension_reload_client.py:93,114: connection-refused/DNS/non-JSON failures surface ascode="reload-source-missing", whose message text is about a missing path — misleading for a transport failure. Use a transport-specific code or a plain connectivity message.
✅ Action checklist for the author
Blockers (resolve before merge):
- B1 — Split the 5 over-limit files;
_orchestrator.py(801) andreload.py(721) exceed even the 700 ceiling and mix responsibilities. - B2 —
--execute-imports: switch the env filter to an allowlist; stop leaking cloud credentials into untrusted bundle import. - B3 — Run path-safety/
validate_extensionon discovered installed+seed manifests; add containment check to the seed walk; fix the misleading docstring. - B4 — Converge the three shadow/precedence resolvers onto one identity + one resolver; make
install_bundlereject same-key records. - B5 — Extract a single audited
is_within()and oneSKIP_DIR_NAMES, used by every walker.
Important (preferably this PR):
- I1 — Offload
reload_bundleoff the event loop (run_in_threadpool/asyncio.to_thread). - I2 — Add
exec/eval/__import__/compileto the AST checks; downgrade the security wording to "hygiene lint." - I3 — Build the rename map before dropping old modules (non-destructive tripwire).
- I4 — Log (don't suppress) the load-bearing
cls.__module__retag failure. - I5 —
dev_registry._read_state: surface unreadable/corrupt state instead of returning[]; consider 0600 perms. - I6 —
_pyproject_declares_extension: distinguish "absent" from "unreadable." - I7 — Make the entry-point predicate lazy or justify the
ep.load()side-effect asymmetry; narrow theexcept BaseException. - I8 — Detect cross-source
@officialbundle-name collisions. - I9 — Frontend: surface reload success-path warnings instead of dropping them.
Recommended (can ship as a follow-up PR):
- R1 — Align migration logging style; add a "flow with unmapped ref does not raise" test.
- R2 — Cap/memoize
difflibsuggestion cost on the flow-load path. - R3 — Compare resolved paths in
_components_path_extension_paths. - R4 — Log the silent
components_index.jsonwrite failure. - R5 — Add reload traversal test; de-flake the concurrency test.
- R6 — Run and post coverage for the new
lfx/extensionpackage.
Test suggestions (post-B3/B5):
- Symlink-escaping seed subdirectory is rejected with a typed error (not loaded).
- Discovered installed/seed manifest with a symlink
bundles[0].pathis rejected before import. -
reload_bundlewith..//absolutebundle_namereturns a typed error, touches nothing outside the bundle root. - Two distributions shipping the same
bundle.nameat@officialproduce a typed collision error, not a silentsys.modulesclobber. -
--execute-importssubprocess env contains no credential-bearing variables (assert allowlist).
Notes for the author (comprehension)
Several blockers are not "bugs" — they are unverified assumptions baked into best-effort suppressions on the exact paths the feature exists to protect (I3, I4, B3's misleading docstring). For each, please capture the rationale durably (a Why: comment or a test name) rather than relying on "the invariant holds." The migration schema/loader layer and the frontend slice are the strongest parts of this PR — tight Pydantic validation, defensive 409/422 parsing, no palette-color or dangerouslySetInnerHTML issues, and genuinely adversarial tests. The concentration of risk is in discovery/precedence and the two oversized dynamic-import modules.
|
Thank you @Cristhianzl ! All ⛔ blockers, ⛔ Blockers — RESOLVEDB1 — Oversized modulesNew files created (extraction, not duplication):
Size after splits:
The two highest-risk dynamic-import modules ( B2 —
|
✅ Test Coverage AdvisorNo source changes detected without accompanying tests. Thanks for keeping coverage up! 🎉
|
|
Build successful! ✅ |
|
Build successful! ✅ |
1 similar comment
|
Build successful! ✅ |
Code Review Summary — Round 2 (verification of resolution)
This is a re-review of the author's resolution of the Round 1 findings, verified line-by-line against branch HEAD All 24 Round 1 findings (B1–B5, I1–I9, R1–R6, N1–N4) are genuinely and substantively resolved. The changes are real code, not prose: three new cohesive modules were extracted, the security-critical paths were corrected, and — notably — the non-obvious decisions are now captured in durable Verdict: Approve with comments ✅ Round 1 findings — verification resultVerified against source on Blockers — all resolved
Important — all resolved
Recommended & Nice-to-have — all resolved
|
|
Build successful! ✅ |
|
Build successful! ✅ |
1 similar comment
|
Build successful! ✅ |
Langflow Extension System (Phase 1)
Combined ticket-by-ticket work on the Bundle Separation iteration. Lands the v0 Extension System end to end: manifest contract, offline validator, loader, production install (pip + seed), atomic-swap reload, init/dev CLIs, flow migration, frontend palette integration, the LE-1023
lfx-duckduckgopilot extraction, and the user-facing extension docs.What this is for
Extensions are the distribution unit Langflow uses to ship Bundles (named groups of Components) outside the core repo. v0 supports a single Bundle per Extension, installed via
pip(Mode A), built into the image (Mode B), or seeded via on-disk subdirectories (Mode C). This PR is the foundation every later Extension ticket builds on.Scope by ticket
lfx extension validate, typed error formatterLANGFLOW_COMPONENTS_PATHdiscoverylfx extension initandlfx extension devauthoring CLIslfx-duckduckgopilot extraction + migration table + dogfood checklistArchitecture
What's in the box
Manifest contract (LE-1014)
ExtensionManifest/BundleRef/LfxCompatPydantic models (extra="forbid").services,routes,hooks,starter_projects,userConfig) reserved asNone-only so non-null values produce a dedicatedfield-deferred-in-this-milestoneerror instead of a generic schema wall.multi-bundle-deferred-in-this-milestone).BUNDLE_API_VERSION = 1+manifest.lfx.compat: list[StrictStr]for forward-compat declarations.extension.jsonor[tool.langflow.extension]inpyproject.toml.lfx extension validateruns four passes: manifest discovery → path-safety (no.., no absolutes, no symlink escape) → AST inspection (syntax, Component subclass,build()declared, noimport *, no top-level I/O) → opt-in--execute-importssubprocess probe with strippedLANGFLOW_*/LFX_*env.lfx extension schemaemits the publishable v1 JSON Schema.Loader (LE-1015)
load_extension(root)for installed Extensions (registers at@officialslot).discover_inline_bundles()forLANGFLOW_COMPONENTS_PATH(registers at@extraslot, first-wins across paths)._lfx_ext.<slot>.<bundle>.<dotted>so a Bundle namedjsondoesn't shadow the stdlib.module_namespaceparameter lets the reload pipeline stage parallel loads in__reload_staging__.<id>.langflow.pluginsentry points;installed_extension_roots()+manifest_owning_distributions()for downstream discovery.module-import-failed,duplicate-component-name,duplicate-distribution,duplicate-inline-bundle,inline-bundle-name-invalid,inline-path-missing,inline-path-unreadable,bundle-json-invalid.Authoring CLIs (LE-1016)
lfx extension init <name>scaffolds a working extension fromBASIC_TEMPLATE(manifest + bundle dir + minimal Component).lfx extension dev <path>adds a working tree to a per-user state file; loaded at startup at the@officialslot until unregistered.extension-target-exists,extension-target-invalid,local-extension-missing.Production install (LE-1022)
discover_installed_extensions()walks every distribution shipping a manifest entry point.discover_seed_extensions()walks$LANGFLOW_SEED_DIR(default/opt/langflow/bundles) for Mode B/C images.load_seed_extensions()is the loader-side counterpart toload_installed_extensions(); both are now invoked fromimport_extension_components()so seed bundles register at@officialat server startup, populate theBundleRegistry, and become reload-target eligible.seed-bundle-shadowederror surfaces through the diagnostics path so the operator sees the misconfiguration instead of a silent overwrite.ExtensionRegistryrecords each entry as immutable:autoUpdate=Falseis forced; mutation verbs (uninstall/disable/enable) raiseExtensionImmutableErrorwith a typedinstalled-extension-immutable/seed-directory-immutablepayload.lfx extension listreads the registry; text + JSON output; non-zero exit on discovery errors.installed-extension-immutable,seed-directory-immutable,seed-directory-not-found,seed-bundle-shadowed,duplicate-extension-id.Atomic-swap reload (LE-1018)
__reload_staging__.<id>) → validate → swap (under registry write lock) → cleanup (purge oldsys.modulesentries) → notify (logger emission today; the typedbundle_reloadedevents stream lands with LE-1017).reload-in-progress(HTTP 409).POST /api/v1/extensions/{extension_id}/bundles/{bundle_name}/reloadgated byget_current_active_userand a per-requestenable_extension_reloadsetting (Mode A only — the route returnsextension-reload-disabledon Mode B/C deployments).lfx extension reload <id> [--bundle <name>]HTTP client; honors$LANGFLOW_HOST/$LANGFLOW_API_KEY; exit codes 0/1/2 map to ok/failed/argument-error.reload-in-progress,reload-bundle-not-installed,reload-bundle-name-mismatch,reload-source-missing,reload-post-swap-hook-failed,extension-reload-disabled.Frontend (LE-1019, frontend slice)
useReloadBundlemutation unwraps the typed 409reload-in-progressdetail into a stable, parseableErrorso the UI can branch without reading status codes.components_added/components_removedcounts), in-progress collision, generic typed error.bundleHeaderActions.test.tsx,bundleItems.test.tsx,sidebarBundles.test.tsx, plus theuseReloadBundlehook tests.Migration system (LE-1020)
migration_table.json(versioned, append-only) maps three legacy reference forms (full import path, package import path, bare class name) to their post-Phase-A namespaced IDs.data.type/data.node.template.<x>.valuereferences, returns aMigrationReport(counts + per-nodeNodeRewriteRecord).MergeDataComponent,SplitTextComponent,SubFlowComponent) are recorded underambiguous_bare_namesinstead and surfacecomponent-name-ambiguousso the user gets a fix hint rather than a silent wrong-bundle bind. The append-only guard now covers this list too — markers cannot be removed once added.component-not-found-with-hint(single match miss) andcomponent-name-ambiguous(multiple matches).migration-table-missing,migration-table-unreadable,migration-table-invalid.User-facing docs (LE-1021)
Three new pages under Develop > Build extensions in the docs sidebar:
extensions-quickstart.mdx— three-minute scaffold-validate-run-reload-publish path.extensions-manifest.mdx— field-by-field reference derived from the LE-1014 schema, with the$schemaURL and thelfx extension schemaexport workflow.extensions-author-guide.mdx— long-form authoring guidance: bundles vs. extensions, slot model, authoring rules, migration table, reload semantics, porting recipe pointer, publishing checklist. The "First-delivery scope" callout names LE-1017 explicitly so authors see the events deferral up-front instead of discovering it from code.LE-1023 pilot bundle (
lfx-duckduckgo)DuckDuckGoSearchComponentfromlfx.components.duckduckgo/into a standalone distribution undersrc/bundles/duckduckgo/with its ownpyproject.tomland innerextension.json.pip install langflowbehaviour is unchanged.BUNDLE_API.mdat repo root enumerates the v0 surface and seeds the changelog.scripts/migrate/check_bare_names.pywalkssrc/lfx/src/lfx/components/**andsrc/bundles/**, builds a class-name → bundle-folder map, and asserts everybare_class_namemigration entry maps to a class found in exactly one folder.scripts/migrate/check_bundle_api_changelog.pyensures any PR that modifies an in-scope public-surface file (manifest, loader, errors, registry, reload, etc.) also adds a## Changelogentry toBUNDLE_API.md.scripts/migrate/check_router_trust.pystatically checks that no handler under/api/v1/extensions/**mounts an install/uninstall/registry-mutation verb. Resolves dotted attribute chains and__init__.pyrelative imports so a router imported viafrom .submodule import routerdoesn't slip past the guard.extension-migration-checks.ymlworkflow so they run on every PR.Error contract
Every error from validate / loader / reload / migration / discovery surfaces as a structured envelope:
format_extension_error()is the single renderer — no other code in the system formats error strings. CLI emits to stderr + non-zero exit; HTTP returns the same body in non-2xx responses. Adding a new code requires (1) adding it toERROR_CODES, (2) adding a_BRANCH_TEMPLATESentry, (3) adding a snapshot test — enforced bytest_every_known_code_has_snapshot.Threat model
--execute-importssubprocess pass stripsLANGFLOW_*/LFX_*env vars, sets temporary HOME / TMPDIR / LANGFLOW_CONFIG_DIR, and is opt-in only — never invoked by pack/publish/install/registry-ingest pipelines..., absolute paths, or symlinks escaping the extension root are rejected (path-escape)./api/v1/extensions/**. Production install ispip installin a Dockerfile (Mode B/C); there is no runtime install path.Test coverage
409 unit + integration tests under
src/lfx/tests/unit/extension/andsrc/lfx/tests/integration/extension/:x-deferred-fieldspopulated.--execute-imports.@official, pathsep splitting, malformed-manifest surfaces, non-extension subdir skip, deterministic ordering.validate.duplicate-extension-id.BundleRegistryfor inline + seed bundles; installed-vs-seed shadowing emits the typedseed-bundle-shadowedevent through the diagnostics path; post-swap reload hook refreshes the component cache.lfx-duckduckgodistribution is importable and ships its manifest whereimportlib.metadata.files()finds it; and the loader resolvesext:duckduckgo:DuckDuckGoSearchComponent@officialto the same class symbol the bundle exports, with the canonicalinput_valueinput anddataframeoutput preserved so existing flows' wiring stays valid.test_router_trust_script.py,test_append_only_script.py,test_bare_names_script.py,test_bundle_api_changelog_script.py).Frontend:
bundleHeaderActions.test.tsx,bundleItems.test.tsx,sidebarBundles.test.tsx, plus theuseReloadBundlehook tests.What's deferred
These are intentionally out of scope and tracked separately:
bundlesaccepts a list but length > 1 is rejected withmulti-bundle-deferred-in-this-milestone.services/routes/hooks/starter_projects/userConfig— reserved asNone-only; non-null values producefield-deferred-in-this-milestone.ExtensionEventsService,GET /api/v1/extensions/events, theuse-extension-eventsReact hook, and the typedbundle_reloaded/flow-migratedevent payloads land in a follow-up. Reload and migration emit through the standard logger today; the HTTP response carries the typed result body so the palette gets immediate feedback through the existing mutation hook. The author's-guide:::infocallout names this deferral up-front.lfx extension reload --all— gated on the LE-1019 list endpoint; current CLI requires--bundle <name>.lfx-arxiv— second pilot bundle, follow-up PR [#13047](feat: Port Arxiv to the Extension Framework #13047).M1 dogfood gate
The proof-of-delivery gate for LE-1023 has two halves:
[test_pilot_duckduckgo_upgrade.py](https://github.com/langflow-ai/langflow/blob/ee5693e8bc640dbe38e4e65e65b763f3fa736224/src/lfx/tests/integration/extension/test_pilot_duckduckgo_upgrade.py)covers every legacy reference form rewriting to the canonical target, the bundle being importable + manifest-discoverable, and the migration target resolving to the same class symbol the bundle exports with canonical inputs/outputs preserved.[src/bundles/duckduckgo/M1_DOGFOOD_CHECKLIST.md](https://github.com/langflow-ai/langflow/blob/ee5693e8bc640dbe38e4e65e65b763f3fa736224/src/bundles/duckduckgo/M1_DOGFOOD_CHECKLIST.md)— save a flow on pre-migration Langflow, upgrade, confirm it loads and runs identically. Must be performed by an engineer not on the Extension team. The checklist's status banner is currently:hourglass: OPEN; the filled-in checklist must be linked here under "M1 dogfood evidence" before merge.M1 dogfood evidence: (paste link to completed checklist before merge)
Test plan
cd src/lfx && uv sync --dev && uv run pytest tests/unit/extension/ tests/integration/extension/cd src/frontend && npm test -- bundleHeaderActions bundleItems sidebarBundles use-reload-bundlelfx extension init mytest && cd mytest && lfx extension validate— scaffolded extension passes its own validatorpip install -e fixture-extension && lfx extension list— installed package shows up at@officialLANGFLOW_SEED_DIR=/path/to/bundles langflow run— seed-directory bundles register at@officialand appear in the palettePOST /api/v1/extensions/<id>/bundles/<name>/reloadagainst an installed Bundle — returns the typed resultcomponent-not-found-with-hintcd docs && npm run build) — three new extension pages render without broken-link errorsSummary by CodeRabbit
New Features
Documentation
Bug Fixes & Improvements