Update start.sh #1
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fixed issue with npm install and npm run build which was preventing the frontend from building properly when running start.sh
dotsimulate
pushed a commit
that referenced
this pull request
Sep 22, 2025
Fix diffusers version and update diffusers.models.autoencoder_tiny submodule name
dotsimulate
pushed a commit
that referenced
this pull request
Sep 22, 2025
…rolnet Preprocessors, Config System (#1) * initial webcam test * spike * initial test with controlnet * lineart script, controlnet script test * general purpose single controlnet pipe, addnl configs * restore demo script, readme * fix controlnet preprocess, update depth example * add some models to readme for referece * update default lineart controlnet * add tile * multi demo * move examples * remove old script * preprocess optim * gpu prepro * sdturbo and sdxl controlnet # Conflicts: # examples/controlnet/controlnet_webcam_demo.py * sdturbo streamdiffusion controlnet * SDXLTurbo controlnet streamdiffusion * monolithic pipeline refactor * high performance controlnet adaptation demo * consolidate sd1.5 + sdturbo * update examples * replace strength w t_index_list * remove unneeded branching * ignore onnx * sdxl tensor fix * refactor - sdxl broken this is to refactor to follow the existing codebase architecture more closely. with a tight timeline, SDXL support is not a priority. TODO todo: look at state before this commit and fix SDXL * spike * model detection update * reference documents * remove duplicate * trt compiling * yaml update * dynamic controlnet + dynamic conditioning streen * working example multicontrolnet * remove extranneous scrtipt use the regular webcam demo * varying spatial dimensions * remove print statements * remove profiling * image conversions * cached preprocessor * fix overwriting config * phase 2 plan * tensorrt controlnet * remove debug scripts * restore test script * remove async compialtion * batch dimension * batch size detection for controlnets * partial cnet trt - black frames * working controlnet trt * delete test * webcam gui * gui improvement * frame buffer size, delta support * frame buffer delta demo * restrict frame buffer size * notes * demo frame buffer size * clarity * config use denoising batch * remvoe unneeded tensor cloning in controlnet engin * optimizations * optimization * optimize postprocess pipeline * tensorrt buffer allocation * get_controlnet_conditioning cleanup * remove extranneous scripts * update readme * update multi controlnet example * stand alone simple pipeline * remove controlnet config abstraction removes nonessential dataclass abstraction. Streamdiffusion uses dicts * remove demo function * remove legacy demo * update docstrings * update docstring * update readme * stack trace in controlnet fallback * add yolonas pose tensorrt preprocessor * attribution * demo exit behavior * selectable config * video test * mediapipe preprocessor spike * touch * add engine only mode * todo * mediapipe segementation preprocessor * theoretical multi-stage processing plan * tensorrt buffer allocation fix * controlnet support in web demo, partial tunuous support for default pipeline * standard lineart preprocessor spike * standar lineart optimization * simple preprocessing parallelism * live prompt update * realtime tindex list update * add tindex real time update * update interface, tindex list * respect acceleration * remove amorphous yamls * clean up examples and readme * consistent naming convention * config system controlnet agnostic * add additional realtime params * remove notes * respect engine dir in wrapper * respect engine dir in demo * remove antiquated control methods * demo header * parameter update documentation * TODO * remove antiquated method from test script * add optional config to standalone pipeline demo * remove gui demo * use stream resolution * use utility method to load config * respect engine dir * fix variable * duplicate cfg logic, comment reduction * remove duplicative update_controlnet_scale this is in wrapper * remove duplicative get_last_processed_image this is in wrapper.py * remove two dead methods * docstring & comment reduction * remove get input output names dead methods * docstring fluff and comment reduction * remove dead methods * docstring and comment reduction * docstring & comment reduction * remove deprecated method * remove deprecated save_contronet_config * remove unused method * Revert "remove dead methods" This reverts commit 9913223. * docstring & comment reduction * restore update controlnet scale * cleanp docstring and comments * docstring & comment reduction * docstring comment reduction * comment cleanup * readme update * readme file name * update stand alone example * wrapper: Merge with ai-runner and export from lib (cumulo-autumn#3) * Copy changes from ai-runner wrapper * Remove copied header We're original now * Export StreamDiffusionWrapper for ai-runner So we can avoid copying the whole thing over there. * Move wrapper to main streamdiffusion package * Fix rebase typo * obey config resolution * unet multires trt * controlnet engine naming convention * preprocessor refactor This commit reduces code duplication without changing the public interface or performance. Next steps: more work on consistent GPU first approach BEFORE REFACTOR -------------------------------------------------------------------------------- Preprocessor Avg Time (s) Min (s) Max (s) Std (s) Output Size -------------------------------------------------------------------------------- Passthrough_PIL 0.000 0.000 0.000 0.000 (512, 512) Passthrough_Tensor 0.001 0.000 0.001 0.000 (512, 512) Canny_PIL 0.001 0.001 0.002 0.000 (512, 512) Canny_Tensor 0.001 0.001 0.002 0.000 (512, 512) StandardLineart_PIL 0.003 0.002 0.004 0.001 (512, 512) StandardLineart_Tensor 0.004 0.003 0.004 0.001 (512, 512) Depth_PIL 0.024 0.021 0.028 0.002 (512, 512) Depth_Tensor 0.024 0.022 0.025 0.001 (512, 512) Lineart_PIL 0.157 0.155 0.159 0.002 (512, 512) Lineart_Tensor 0.164 0.158 0.169 0.004 (512, 512) OpenPose_PIL 0.362 0.351 0.368 0.007 (512, 512) OpenPose_Tensor 0.361 0.348 0.371 0.009 (512, 512) MediaPipePose_PIL 0.024 0.024 0.026 0.001 (512, 512) MediaPipePose_Tensor 0.027 0.026 0.029 0.001 (512, 512) MediaPipeSegmentation_PIL 0.004 0.003 0.004 0.001 (512, 512) MediaPipeSegmentation_Tensor 0.006 0.005 0.006 0.000 (512, 512) PoseTensorRT_PIL 0.021 0.020 0.022 0.001 (512, 512) PoseTensorRT_Tensor 0.015 0.015 0.016 0.000 (512, 512) DepthTensorRT_PIL 0.005 0.004 0.006 0.001 (512, 512) DepthTensorRT_Tensor 0.002 0.002 0.002 0.000 (512, 512) AFTER CHANGES Preprocessor Avg Time (s) Min (s) Max (s) Std (s) Output Size -------------------------------------------------------------------------------- Passthrough_PIL 0.000 0.000 0.000 0.000 (512, 512) Passthrough_Tensor 0.000 0.000 0.000 0.000 (512, 512) Canny_PIL 0.001 0.000 0.001 0.000 (512, 512) Canny_Tensor 0.001 0.001 0.002 0.000 (512, 512) StandardLineart_PIL 0.003 0.002 0.004 0.001 (512, 512) StandardLineart_Tensor 0.004 0.004 0.005 0.001 (512, 512) Depth_PIL 0.024 0.022 0.026 0.002 (512, 512) Depth_Tensor 0.024 0.023 0.025 0.001 (512, 512) Lineart_PIL 0.160 0.158 0.164 0.003 (512, 512) Lineart_Tensor 0.160 0.155 0.164 0.003 (512, 512) OpenPose_PIL 0.355 0.352 0.358 0.002 (512, 512) OpenPose_Tensor 0.367 0.346 0.383 0.014 (512, 512) MediaPipePose_PIL 0.024 0.023 0.025 0.001 (512, 512) MediaPipePose_Tensor 0.027 0.026 0.027 0.000 (512, 512) MediaPipeSegmentation_PIL 0.003 0.003 0.004 0.000 (512, 512) MediaPipeSegmentation_Tensor 0.005 0.005 0.006 0.000 (512, 512) PoseTensorRT_PIL 0.021 0.020 0.023 0.001 (512, 512) PoseTensorRT_Tensor 0.016 0.014 0.018 0.002 (512, 512) DepthTensorRT_PIL 0.004 0.004 0.005 0.001 (512, 512) DepthTensorRT_Tensor 0.002 0.002 0.004 0.001 (512, 512) * external preprocessor * move config * update img2img demo import * seed update * provide dummy inputs --------- Co-authored-by: RyanOnTheInside <7623207+ryanontheinside@users.noreply.github.com> * remove pipeline type and clean up model loading * remove infer_model_type * remove infer_controlnet_type * remove pipeline_type from example configs * remove ppipeline_type from demo * documentation * remove pipeline from config * remove pipline_type from demo * remove resolution * remove .lower * happy path * Copy changes from ai-runner wrapper * Remove copied header We're original now * Move wrapper to main streamdiffusion package * obey config resolution * move config * use direct instead of refelction * prompt blending working * seed blending * prompt and seed blending * interface update * support text to image configs * fix dupes from rebase * add optional weighting for prompt and seed * soft edge preprocessor * HED preprocessor * remove custom network * update public interface and use it * remove overly defensive checks * complete parameter updater forwarding methods * update dynamic controls * cleanup prints * vectorize middle joints * precompute edges and colors * reduce cuda available system calls * prediction list and tuple creation * reduce detection resolution * vectorize * video mode, gpu delegate * pipelined processing this results in 20%-30% fps improvement when using four controlnets * orchestration module * add normalize consistency * generic cache stats and normalization * unified blending control * rename interpolation method to prompt interpolation method * forward legacy single prompt * Set "normalize weight" params w/ update params func (cumulo-autumn#24) * Multi-Resolution (Pipeline Reload) + WebUI Upgrades (cumulo-autumn#15) * Added api-only mode to server, update svelte vite and frontend packages, built multiresolution support with tensorrt, built resolution picker. * Added realtime resolution updates. * Small fix. * Controlnet work. * ControlNet testing. * Rebuilt resolution swapping by restarting pipeline, re-added prompt input to UI for testing. * Fix missing file. * Fix to config updating the resolution. * fix input profiles * restore sdxl functionality * remove legacy prompt box * fix prompt blending * unified blending control and fixes * Set dynamic size from 384-1024 with opt at 704. * updater: Remove "dynamic" resolution update logic * Remove debug logs * Remove more debug logs... --------- Co-authored-by: RyanOnTheInside <7623207+ryanontheinside@users.noreply.github.com> Co-authored-by: Victor Elias <victorgelias@gmail.com> Co-authored-by: Victor Elias <victor@livepeer.org> * configurable pytorch fallback * remove legacy prompt box # Conflicts: # demo/realtime-img2img/frontend/src/lib/components/PromptBlendingControl.svelte # demo/realtime-img2img/frontend/src/routes/+page.svelte * fix prompt blending # Conflicts: # demo/realtime-img2img/frontend/src/lib/components/SeedBlendingControl.svelte # demo/realtime-img2img/main.py * unified blending control and fixes # Conflicts: # demo/realtime-img2img/frontend/src/lib/components/BlendingControl.svelte * rename interpolation method * remove legacy files * IPAdapters + Controlnet and Tensorrt for SDXL (cumulo-autumn#40) * testing * text to image working * img2img working * example update * update examples * successful engine build * ipadapter tensorrt acceleration * remove test scripts * remove test script * safe import * update test script * working * button up * bake with strength | name engine dir * cleanup wrapper * cache embeddings * comment some prints * backtrack multiple ipadapter support this will be done in subsequent phase * remove test scripts * dynamic ipadapter token * plus support * rm erroneous commit * move config * erroneous comment * demo spike * prompts working * composable conditioning wrapper * bugfix wrapper * wrapper fix * remove print statement * fix prompt embeds * add ipadapter support to demo * remove extranneous exports * remove extranneous backwards compat * remove extrannous controlnet exports * comment cleanup * rm ipadapter readme * cleanup prints * remove debug script * cleanup config logic * remove defensive import handling * remove example bloat * remove extranneous demo * rm print statements * remove prints, update todo * fix config upload * fix style image update * remove copywritten image * leverage preprocessor system * remove print statements * prompt update perf fix * remove heuristic ipadapter detection # Conflicts: # src/streamdiffusion/ipadapter/base_ipadapter_pipeline.py * Ryan/feat/02 ipadapter on sdxl (cumulo-autumn#32) * merge sdxl Merged changes from ryan/feat/ipadapter-on-sdxl branch with marco/feat/sdxl branch, resolving conflicts in: src/streamdiffusion/pipeline.py: Integrated SDXL UNet calling conventions with comprehensive conditioning support, retained SDXL added_cond_kwargs handling and TensorRT vs PyTorch UNet detection demo/realtime-img2img/main.py: Combined IPAdapter info response with resolution calculation, merged pipeline creation logic for both IPAdapter and SDXL support src/streamdiffusion/acceleration/tensorrt/__init__.py: Unified embedding dimension detection for both SDXL (2048) and SD1.5 (768), integrated IPAdapter and ControlNet TensorRT support src/streamdiffusion/controlnet/base_controlnet_pipeline.py: Merged ControlNet conditioning logic with SDXL compatibility src/streamdiffusion/stream_parameter_updater.py: Integrated IPAdapter embedding enhancers with parameter update system src/streamdiffusion/wrapper.py: Resolved TensorRT engine compilation conflicts, unified IPAdapter and SDXL model loading paths * handle sdxl in conditioning wrapper * image encoder download fix * remove extranneous ipadapter pipeline abstraction * remove heuristic ipadapter detection * consolidate model detection * Ryan/feat/03 controlnet on ipadapter on sdxl (cumulo-autumn#33) * controlnet enabled unet engine * remove prints * cnet engine support * consolidate utilities logging * uncomment fallback * consolidate sdxl unet wrapper * Fixed model detection. * Rebuilt model detection mechanism, cleanup. * Removal of fluff. * More reliance on detect_model function, fix of SDXL detection for onnx export. * Bit of cleanup and additional logging. * Reverted controlnet/ipadapter model detection to original functions. * add controlnet model detection * add controlnet model detection to export --------- Co-authored-by: BuffMcBigHuge <marco@bymar.co> --------- Co-authored-by: BuffMcBigHuge <marco@bymar.co> --------- Co-authored-by: RyanOnTheInside <7623207+ryanontheinside@users.noreply.github.com> Co-authored-by: BuffMcBigHuge <marco@bymar.co> * rename to export wrapper convention * move sdxl controlnet export * rename conditioning wrapper * rename controlnet_wrapper.py * rename ipadapter_wrapper.py * rename sdxl_support.pt, move SDXLExportWrapper * move export wrappers to directory * move models * rename engine.py * move engines to directory * fix straggle imports * move preprocessors from src/streamdiffusion/controlnet/ --> src/streamdiffusion * move engine pool from runtime folder * install and use diffusers_ipadapter package * remove legacy docs * set use cuda graph true in wrapper * Ryan/fix/01 fix perf regression (cumulo-autumn#46) * remove logging, input/output validation * remove logging, inputoutput from infer() * remove logging and input output validation from unet * remove overhead from hotpath remove logging, input/output validation, make vram montioring in debug mode only * remove logging from controlnet engine * remove logger.debug * remove HybridControlnet abstraction and pytorch fallback * remove dead code * add feedback preprocessor * add feedback preprocessor support * add blend to preprocessor * add preprocessor metadata * dynamic preprocessor controls * remove erroneous variable * unify tensorrt engine management * remove unicode chars * import fix * change default acceleration * fullscreen * add route to get default image and makes ipadapter use it * rgb conversion fix * static video input control * minimalist resolution picker * increase max delta value * Update resolution button --------- Co-authored-by: Victor Elias <victor@livepeer.org> Co-authored-by: Victor Elias <victorgelias@gmail.com> Co-authored-by: Buff <marcotundo@gmail.com> Co-authored-by: BuffMcBigHuge <marco@bymar.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixed issue with npm install and npm run build which was preventing the frontend from building properly when running start.sh