feat: Add support for vLLMs and custom URL support#10336
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThis PR adds vLLM as a supported language model provider by introducing frontend icon components and extending the backend configuration to handle vLLM initialization via base URL and optional API key, with model name as a text input rather than a dropdown selection. Changes
Sequence DiagramsequenceDiagram
participant User
participant UI as Language Model UI
participant Backend
participant ChatOpenAI
User->>UI: Select vLLM provider
UI->>Backend: update_build_config(provider="vLLM")
Note over Backend: Convert model_name to text input<br/>Set base_url visibility<br/>Mark API key optional
Backend-->>UI: Updated configuration
User->>UI: Enter base_url & model_name
User->>UI: (Optional) Enter API key
User->>UI: Build model
UI->>Backend: build_model(provider="vLLM", base_url, model_name, api_key?)
alt base_url present
Backend->>ChatOpenAI: Initialize with base_url & api_key
ChatOpenAI-->>Backend: ChatOpenAI client
Backend-->>UI: Model ready
else base_url missing
Backend-->>UI: ValueError: base_url required
end
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested labels
Suggested reviewers
Pre-merge checks and finishing touches❌ Failed checks (1 error, 4 warnings)
✅ Passed checks (2 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
enhancement
jpramos123
force-pushed
the
support-vllm
branch
from
bceca8a to
b397b90
Compare
github-actions
Bot
added
enhancement
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/lfx/src/lfx/components/models/language_model.py (1)
107-113: The web search confirms the review comment's assertion. The langchain_openai ChatOpenAI usesmodelparameter, notmodel_name, and the API key parameter isapi_key(with alias toopenai_api_keyfrom older versions). Looking at the code at lines 107–113 ofsrc/lfx/src/lfx/components/models/language_model.pyand line 139-143 (vLLM branch), both use the incorrect kwargsmodel_nameandopenai_api_keyinstead of the correctmodelandapi_key.Fix ChatOpenAI kwargs for OpenAI and vLLM paths.
The langchain_openai package expects
modelandapi_keyparameters (notmodel_nameandopenai_api_key). Both the OpenAI path (lines 107–113) and vLLM path (lines 139–143) use incorrect kwargs:return ChatOpenAI( - model_name=model_name, + model=model_name, temperature=temperature, streaming=stream, - openai_api_key=self.api_key, + api_key=self.api_key, base_url=self.base_url, )Apply the same fix to both occurrences (lines ~107–113 and ~139–143).
src/frontend/src/icons/eagerIconImports.ts (1)
109-115: Add vLLM to lazy icon mapping.vLLM was added to eagerIconImports.ts but is missing from the lazyIconsMapping in lazyIconImports.ts. Add the entry between VertexAI and WatsonxAI to ensure consistent lazy-loading support, using the key
vLLMto match the eager import exactly.
🧹 Nitpick comments (2)
src/lfx/src/lfx/components/models/language_model.py (2)
49-55: Consider StrInput for base_url or validate URL format.MessageTextInput accepts Message/iterators; base_url is a plain string. Either switch to a simple string input or add a light URL validator/normalizer (e.g., ensure it includes /v1 and no trailing slash quirks).
If you prefer to keep MessageTextInput, confirm other components also use it for plain strings for consistency.
187-193: Null-safety for o1 detection.Defensive check avoids attribute errors if model_name is ever None or not str.
- elif field_name == "model_name" and field_value.startswith("o1") and self.provider == "OpenAI": + elif field_name == "model_name" and isinstance(field_value, str) and field_value.startswith("o1") and self.provider == "OpenAI": ... - elif field_name == "model_name" and not field_value.startswith("o1") and "system_message" in build_config: + elif field_name == "model_name" and isinstance(field_value, str) and not field_value.startswith("o1") and "system_message" in build_config:
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (1)
src/frontend/src/icons/vLLM/vLLM.svgis excluded by!**/*.svg
📒 Files selected for processing (4)
src/frontend/src/icons/eagerIconImports.ts(3 hunks)src/frontend/src/icons/vLLM/index.tsx(1 hunks)src/frontend/src/icons/vLLM/vLLM.jsx(1 hunks)src/lfx/src/lfx/components/models/language_model.py(5 hunks)
🧰 Additional context used
📓 Path-based instructions (4)
src/frontend/src/**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.cursor/rules/frontend_development.mdc)
src/frontend/src/**/*.{ts,tsx,js,jsx}: All frontend TypeScript and JavaScript code should be located under src/frontend/src/ and organized into components, pages, icons, stores, types, utils, hooks, services, and assets directories as per the specified directory layout.
Use React 18 with TypeScript for all UI components in the frontend.
Format all TypeScript and JavaScript code using the make format_frontend command.
Lint all TypeScript and JavaScript code using the make lint command.
Files:
src/frontend/src/icons/vLLM/vLLM.jsxsrc/frontend/src/icons/eagerIconImports.tssrc/frontend/src/icons/vLLM/index.tsx
src/frontend/src/icons/**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.cursor/rules/frontend_development.mdc)
Use Lucide React for icons in the frontend.
Files:
src/frontend/src/icons/vLLM/vLLM.jsxsrc/frontend/src/icons/eagerIconImports.tssrc/frontend/src/icons/vLLM/index.tsx
src/frontend/src/icons/*/*.@(js|jsx|ts|tsx)
📄 CodeRabbit inference engine (.cursor/rules/icons.mdc)
Create a new directory for your icon in
src/frontend/src/icons/YourIconName/and add your SVG as a React component (e.g.,YourIconName.jsx). The SVG component must use theisDarkprop to support both light and dark mode.
Files:
src/frontend/src/icons/vLLM/vLLM.jsxsrc/frontend/src/icons/vLLM/index.tsx
src/frontend/src/icons/*/index.tsx
📄 CodeRabbit inference engine (.cursor/rules/icons.mdc)
Create an
index.tsxin your icon directory that exports your icon usingforwardRefand passes theisDarkprop.
Files:
src/frontend/src/icons/vLLM/index.tsx
🧠 Learnings (5)
📚 Learning: 2025-07-28T15:56:47.865Z
Learnt from: CR
PR: langflow-ai/langflow#0
File: .cursor/rules/icons.mdc:0-0
Timestamp: 2025-07-28T15:56:47.865Z
Learning: Applies to src/frontend/src/icons/*/*.@(js|jsx|ts|tsx) : Create a new directory for your icon in `src/frontend/src/icons/YourIconName/` and add your SVG as a React component (e.g., `YourIconName.jsx`). The SVG component must use the `isDark` prop to support both light and dark mode.
Applied to files:
src/frontend/src/icons/vLLM/vLLM.jsx
📚 Learning: 2025-07-28T15:56:47.865Z
Learnt from: CR
PR: langflow-ai/langflow#0
File: .cursor/rules/icons.mdc:0-0
Timestamp: 2025-07-28T15:56:47.865Z
Learning: Applies to src/frontend/src/icons/lazyIconImports.ts : Add your icon to the `lazyIconsMapping` object in `src/frontend/src/icons/lazyIconImports.ts` with a key that matches the backend icon string exactly.
Applied to files:
src/frontend/src/icons/eagerIconImports.ts
📚 Learning: 2025-07-28T15:56:47.865Z
Learnt from: CR
PR: langflow-ai/langflow#0
File: .cursor/rules/icons.mdc:0-0
Timestamp: 2025-07-28T15:56:47.865Z
Learning: Applies to src/frontend/src/icons/*/index.tsx : Create an `index.tsx` in your icon directory that exports your icon using `forwardRef` and passes the `isDark` prop.
Applied to files:
src/frontend/src/icons/vLLM/index.tsx
📚 Learning: 2025-06-23T12:46:52.420Z
Learnt from: CR
PR: langflow-ai/langflow#0
File: .cursor/rules/icons.mdc:0-0
Timestamp: 2025-06-23T12:46:52.420Z
Learning: Export custom icon components in React using React.forwardRef to ensure proper ref forwarding and compatibility with parent components.
Applied to files:
src/frontend/src/icons/vLLM/index.tsx
📚 Learning: 2025-06-16T11:14:04.200Z
Learnt from: dolfim-ibm
PR: langflow-ai/langflow#8394
File: src/frontend/src/icons/Docling/index.tsx:4-6
Timestamp: 2025-06-16T11:14:04.200Z
Learning: The Langflow codebase consistently uses `React.PropsWithChildren<{}>` as the prop type for all icon components using forwardRef, rather than `React.SVGProps<SVGSVGElement>`. This is an established pattern across hundreds of icon files in src/frontend/src/icons/.
Applied to files:
src/frontend/src/icons/vLLM/index.tsx
🧬 Code graph analysis (3)
src/frontend/src/icons/eagerIconImports.ts (1)
src/frontend/src/icons/vLLM/index.tsx (1)
VLLMIcon(5-9)
src/frontend/src/icons/vLLM/index.tsx (1)
src/frontend/src/icons/vLLM/vLLM.jsx (1)
SvgVLLM(1-148)
src/lfx/src/lfx/components/models/language_model.py (1)
src/lfx/src/lfx/inputs/inputs.py (6)
DropdownInput(474-499)MessageInput(185-212)MessageTextInput(215-266)MultilineInput(269-279)SecretStrInput(295-350)SliderInput(651-652)
🔇 Additional comments (2)
src/frontend/src/icons/vLLM/index.tsx (1)
1-9: LGTM — forwardRef wrapper matches icon conventions.src/lfx/src/lfx/components/models/language_model.py (1)
173-187: Update vLLM model name and endpoint examples for clarity.The /v1 path is a standard part of vLLM's OpenAI-compatible API endpoint, and typical model names follow conventions like "Qwen/Qwen2.5-1.5B-Instruct". The example "gemini-2.5-pro" is a Google model and does not reflect common vLLM deployments. The suggested changes align with best practices:
- build_config["model_name"]["info"] = "Enter the model name (e.g., gemini-2.5-pro)" + build_config["model_name"]["info"] = "Enter the model name (e.g., meta-llama/Meta-Llama-3.1-8B-Instruct, mistralai/Mixtral-8x7B-Instruct)" - build_config["base_url"]["info"] = "vLLM server endpoint URL (e.g., http://localhost:8000/v1)" + build_config["base_url"]["info"] = "vLLM server endpoint URL (must include /v1, e.g., http://localhost:8000/v1)"
| return ChatOpenAI( | ||
| model_name=model_name, | ||
| temperature=temperature, | ||
| streaming=stream, | ||
| openai_api_key=self.api_key or "EMPTY", # vLLM may not require API key | ||
| base_url=self.base_url, | ||
| ) |
There was a problem hiding this comment.
Fix ChatOpenAI kwargs for vLLM path (model/api_key).
Same issue as OpenAI path; use model and api_key. Keep base_url.
- return ChatOpenAI(
- model_name=model_name,
- temperature=temperature,
- streaming=stream,
- openai_api_key=self.api_key or "EMPTY", # vLLM may not require API key
- base_url=self.base_url,
- )
+ return ChatOpenAI(
+ model=model_name,
+ temperature=temperature,
+ streaming=stream,
+ api_key=self.api_key or "EMPTY", # vLLM may not require API key
+ base_url=self.base_url,
+ )📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| return ChatOpenAI( | |
| model_name=model_name, | |
| temperature=temperature, | |
| streaming=stream, | |
| openai_api_key=self.api_key or "EMPTY", # vLLM may not require API key | |
| base_url=self.base_url, | |
| ) | |
| return ChatOpenAI( | |
| model=model_name, | |
| temperature=temperature, | |
| streaming=stream, | |
| api_key=self.api_key or "EMPTY", # vLLM may not require API key | |
| base_url=self.base_url, | |
| ) |
🤖 Prompt for AI Agents
In src/lfx/src/lfx/components/models/language_model.py around lines 139 to 145,
the ChatOpenAI instantiation for the vLLM path is using the OpenAI-style kwargs
(model_name and openai_api_key); change these to use the vLLM-style kwargs by
passing model=self.model or model=model (matching surrounding variable) and
api_key=self.api_key or "EMPTY" (or empty string as before), while keeping
base_url and the other existing args (temperature, streaming/stream) unchanged.
b397b90 to
1277b60
Compare
github-actions
Bot
added
enhancement
jpramos123
force-pushed
the
support-vllm
branch
from
5edeccf to
9de548b
Compare
github-actions
Bot
added
enhancement
|
Please run the following command locally and commit the changes: make build_component_indexOr alternatively: LFX_DEV=1 uv run python scripts/build_component_index.pyThen commit and push the updated |
github-actions
Bot
added
enhancement
|
jpramos123
force-pushed
the
support-vllm
branch
2 times, most recently
from
7cefccd to
0c2b19c
Compare
github-actions
Bot
added
enhancement
jpramos123
force-pushed
the
support-vllm
branch
from
5c25292 to
6c22e43
Compare
github-actions
Bot
added
enhancement
jpramos123
force-pushed
the
support-vllm
branch
from
4c98c6a to
343c14d
Compare
github-actions
Bot
added
enhancement
jpramos123
force-pushed
the
support-vllm
branch
from
df0f085 to
eaef3de
Compare
|
@erichare @rodrigosnader Can you help us understand why is the pipeline failing? I already ran the |
|
@jpramos123 the failing pipeline shouldnt block the merging of this - I think we just need an approving review. There are enough frontend changes that it would be good to have multiple eyes on this: @ogabrielluiz @Cristhianzl @lucaseduoli and @edwinjosechittilappilly might be good candidates! |
|
@ogabrielluiz @Cristhianzl @lucaseduoli and @edwinjosechittilappilly Hey guys, are you able to review this PR? Thank you! |
|
|
||
| if hasattr(self, "system_prompt"): | ||
| input_dict: dict[str, str | list[BaseMessage]] = {} | ||
| if hasattr(self, "system_prompt") and self.system_prompt and self.system_prompt.strip(): |
There was a problem hiding this comment.
Is this strictly required for this PR, or a more general fix?
There was a problem hiding this comment.
Sorry, removed my last message
I think this is needed for models like granite, which don't use their internal system_prompt template when anything is set (even "")
But this is crucial to get e.g. tool calling right.
So we need to send the perfectly formatted system prompt for this model or don't send any system_prompt - like really skipping the value…
| async def send_message_noop( | ||
| message: Message, | ||
| text: str | None = None, # noqa: ARG001 | ||
| background_color: str | None = None, # noqa: ARG001 |
There was a problem hiding this comment.
Similar question here - are these changes necessary for the PR?
There was a problem hiding this comment.
but this looks like a CI-fix artefact - @jpramos123 what do you think?
There was a problem hiding this comment.
@schuellerf Correct! This was needed to get the CI up and running. These values are not being used at all here
erichare
left a comment
erichare
left a comment
There was a problem hiding this comment.
This is a fantastic PR, but i do have two questions that i'd like addressed first if possible - a couple changes that may be necessary for the vLLMs, but just want to be careful about backwards compat
| ] | ||
| messages = [] | ||
|
|
||
| # Only include system message if system_prompt is provided and not empty |
There was a problem hiding this comment.
@erichare like here - this is really needed for models like granite which behave strangely when you want tool calling and don't provide the recommended system prompt (or skip the parameter to force loading the deployed one)
|
@jpramos123 I think the CI wants you to update the component index with |
Hey @schuellerf, yes always use this make command to refresh the Looks like the conflict happens because it always changes after a new commit is merged into master |
Thanks! That's a good explanation, so we don't need to update this always, I guess! |
|
/retest |
3 participants
Extend the Language Model component to support language models using vLLM OpenAI compatible server. [1]
The component uses the langchain ChatOpenAI to access the model.
The component has a custom model name, URL and API key.
The user should be responsible to feed abovinformationns correctly in order for the models to work properly
Summary by CodeRabbit