-
Notifications
You must be signed in to change notification settings - Fork 2.8k
feat(flux2): add FLUX.2 klein model support #8768
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
feat(flux2): add FLUX.2 klein model support #8768
Conversation
- Add new invocation nodes for FLUX 2: - flux2_denoise: Denoising invocation for FLUX 2 - flux2_klein_model_loader: Model loader for Klein architecture - flux2_klein_text_encoder: Text encoder for Qwen3-based encoding - flux2_vae_decode: VAE decoder for FLUX 2 - Add backend support: - New flux2 module with denoise and sampling utilities - Extended model manager configs for FLUX 2 models - Updated model loaders for Klein architecture - Update frontend: - Extended graph builder for FLUX 2 support - Added FLUX 2 model types and configurations - Updated readiness checks and UI components
FLUX.2 VAE uses Batch Normalization in the patchified latent space (128 channels). The decode must: 1. Patchify latents from (B, 32, H, W) to (B, 128, H/2, W/2) 2. Apply BN denormalization using running_mean/running_var 3. Unpatchify back to (B, 32, H, W) for VAE decode Also fixed image normalization from [-1, 1] to [0, 255]. This fixes washed-out colors in generated FLUX.2 Klein images.
…nkuchensack/InvokeAI into feature/flux2-klein-support
…ompatibility - Add FLUX.2 transformer loader with BFL-to-diffusers weight conversion - Fix AdaLayerNorm scale-shift swap for final_layer.adaLN_modulation weights - Add VAE batch normalization handling for FLUX.2 latent normalization - Add Qwen3 text encoder loader with ComfyUI FP8 quantization support - Add frontend components for FLUX.2 Klein model selection - Update configs and schema for FLUX.2 model types
- Add isFlux2Klein9BMainModelConfig and isNonCommercialMainModelConfig functions - Update MainModelPicker and InitialStateMainModelPicker to show license icon - Update license tooltip text to include FLUX.2 Klein 9B
|
I tested with two models: the diffusers model at Initial observations:
|
|
Here's a trivial fix for the error |
Backend: - Add klein_4b/klein_9b variants for FLUX.2 Klein models - Add qwen3_4b/qwen3_8b variants for Qwen3 encoder models - Validate encoder variant matches Klein model (4B↔4B, 9B↔8B) - Auto-detect Qwen3 variant from hidden_size during probing Frontend: - Show variant field for all model types in ModelView - Filter Qwen3 encoder dropdown to only show compatible variants - Update variant type definitions (zFlux2VariantType, zQwen3VariantType) - Remove unused exports (isFluxDevMainModelConfig, isFlux2Klein9BMainModelConfig)
Distinguish between FLUX.2 Klein 9B (distilled) and Klein 9B Base (undistilled) models by checking guidance_embeds in diffusers config or guidance_in keys in safetensors. Klein 9B Base requires more steps but offers higher quality.
Backend changes: - Update text encoder layers from [9,18,27] to (10,20,30) matching diffusers - Use apply_chat_template with system message instead of manual formatting - Change position IDs from ones to zeros to match diffusers implementation - Add get_schedule_flux2() with empirical mu computation for proper schedule shifting - Add txt_embed_scale parameter for Qwen3 embedding magnitude control - Add shift_schedule toggle for base (28+ steps) vs distilled (4 steps) models - Zero out guidance_embedder weights for Klein models without guidance_embeds UI changes: - Clear Klein VAE and Qwen3 encoder when switching away from flux2 base - Clear Qwen3 encoder when switching between different Klein model variants - Add toast notification informing user to select compatible encoder
|
The recent commit is causing the previously-installed ZImage Qwen3 encoders to fail validation with warnings like this one: |
|
Another issue I just noticed. When I try to generate with Flux.1 (schnell or krea), I am getting "Node with id pos_cond_collect is not found." [Edit]: I think this was a one-time mismatch between the front and back end; I'll report if it happens again. |
- Configure scheduler with FLUX.2 Klein parameters from scheduler_config.json (use_dynamic_shifting=True, shift=3.0, time_shift_type="exponential") - Pass mu parameter to scheduler.set_timesteps() for resolution-aware shifting - Remove manual shift_schedule parameter (scheduler handles this automatically) - Simplify get_schedule_flux2() to return linear sigmas only - Remove txt_embed_scale parameter (no longer needed) This matches the diffusers Flux2KleinPipeline behavior where the FlowMatchEulerDiscreteScheduler applies dynamic timestep shifting based on image resolution via the mu parameter. Fixes 4-step distilled Klein 9B model quality issues.
The posCondCollect node was created with getPrefixedId() which generates a random suffix (e.g., 'pos_cond_collect:abc123'), but g.getNode() was called with the plain string 'pos_cond_collect', causing a node lookup failure. Fix by declaring posCondCollect as a module-scoped variable and referencing it directly instead of using g.getNode().
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Pfannkuchensack Generations are looking good now. If you are satisfied with the image quality, then I think we can make a point release as soon as the following remaining issues are addressed:
- The standalone model (e.g. flux-2-klein-4b.safetensors) cannot automatically find its Qwen3 encoder in the main model
- Define FLUX.2 Klein starter models (including a compatible standalone encoder)
- Define default generation parameters for the klein models (steps, CFG, etc)
- Fix pydantic validation errors for the ZiT qwen3 encoder models
- Add FLUX.2 model requirements (minimum hardware needed) to the docs
Please also give me an idea of when these missing features may be added, so that I can address user expectations in the release notes:
- img2img
- inpainting
- outpainting
- support for quants / GGUF
Is the image editing feature accessed through img2img from the raster layer, or is there an additional head for the reference image?
- Add flux2_vae_encode invocation for encoding images to FLUX.2 latents - Integrate inpaint_extension into FLUX.2 denoise loop for proper mask handling - Apply BN normalization to init_latents and noise for consistency in inpainting - Use manual Euler stepping for img2img/inpaint to preserve exact timestep schedule - Add flux2_img2img, flux2_inpaint, flux2_outpaint generation modes - Expand starter models with FP8 variants, standalone transformers, and separate VAE/encoders - Fix outpainting to always use full denoising (0-1) since strength doesn't apply - Improve error messages in model loader with clear guidance for standalone models
I'll take a look at it later.
I'll do it later, or someone else can take care of it. I did some testing with inpaiting, outpaiting and both together. i2i works fine. |
…Klein - Add Main_GGUF_Flux2_Config for GGUF-quantized FLUX.2 transformer models - Add VAE_Diffusers_Flux2_Config for FLUX.2 VAE in diffusers format - Add Flux2GGUFCheckpointModel loader with BFL-to-diffusers conversion - Add Flux2VAEDiffusersLoader for AutoencoderKLFlux2 - Add FLUX.2 Klein 4B/9B hardware requirements to documentation - Update starter model descriptions to clarify dependencies install together - Update frontend schema for new model configs
…nkuchensack/InvokeAI into feature/flux2-klein-support
- Improve FLUX.2 variant detection for GGUF/checkpoint models (BFL format keys) - Fix guidance_embeds logic: distilled=False, undistilled=True - Add FP8 weight dequantization for ComfyUI-style quantized models - Prevent FLUX.2 models from being misidentified as FLUX.1 - Preserve user-editable fields (name, description, etc.) on model reidentify - Improve Qwen3Encoder detection by variant in starter models - Add defensive checks for tensor operations
Previously num_attention_heads was hardcoded to 24, which is correct for Klein 4B but causes size mismatches when loading Klein 9B checkpoints. Now dynamically calculates num_attention_heads from the hidden_size dimension of context_embedder weights: - Klein 4B: hidden_size=3072 → num_attention_heads=24 - Klein 9B: hidden_size=4096 → num_attention_heads=32 Fixes both Checkpoint and GGUF loaders for FLUX.2 models.
Previously the encoder was cleared whenever switching between any Klein models, even if they had the same variant. Now compares the variant of the old and new model and only clears the encoder when switching between different variants (e.g., klein_4b to klein_9b). This allows users to switch between different Klein 9B models without having to re-select the Qwen3 encoder each time.
|
@Pfannkuchensack Fantastic work! I am doing testing now. Hope to be able to approve and merge Friday. |
|
@Pfannkuchensack With this PR I'm getting config validation errors on previously-installed Z-Image Turbo models. I can reinstall the models and they work. Is this expected? |
I dont have a z-image installed in my test instance, so i cant check that. Do you have message from the console ? |







Summary
Add initial support for FLUX 2 klein models architecture.
New Invocation Nodes
Backend Support
flux2module with denoise and sampling utilitiesFrontend Updates
Related Issues / Discussions
QA Instructions
Merge Plan
Standard merge after review.
Checklist
What's Newcopy (if doing a release after this PR)