-
Notifications
You must be signed in to change notification settings - Fork 6.7k
[WIP][Docs] Use DiffusionPipeline Instead of Child Classes when Loading Pipeline #2809
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[WIP][Docs] Use DiffusionPipeline Instead of Child Classes when Loading Pipeline #2809
Conversation
…g a checkpoint using from_pretrained() instead of a child class (e.g. StableDiffusionPipeline) where possible.
|
The documentation is not available anymore as the PR was closed or merged. |
|
Sorry, I just realized that the docs in |
|
Looked at the docs for
Left unchanged because I think the specific pipelines (e.g.
Since this page discusses the Similarly to the other training docs, since these pages discuss specific pipelines, I think it makes sense to use them rather than the generic
For the first example, using
The examples discuss specific tasks (e.g. inpainting) so I think it makes sense to use the specific pipeline for that task (e.g. |
|
I realize that instead of documenting specific cases where a change is or is not made, it might make more sense to use a bright line rule about when using (I would also greatly appreciate it if people could give feedback on the verbosity of this PR, so that I can improve in the future.) |
|
Thanks for the PR @dg845 - I agree with your reasoning. I think we should use I'm in favor of merging this PR as it though, so LGTM. @yiyixuxu could you take a final look? |
yiyixuxu
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
…ng Pipeline (huggingface#2809) * Change the docs to use the parent DiffusionPipeline class when loading a checkpoint using from_pretrained() instead of a child class (e.g. StableDiffusionPipeline) where possible. * Run make style to fix style issues. * Change more docs to use DiffusionPipeline rather than a subclass. --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
…ng Pipeline (huggingface#2809) * Change the docs to use the parent DiffusionPipeline class when loading a checkpoint using from_pretrained() instead of a child class (e.g. StableDiffusionPipeline) where possible. * Run make style to fix style issues. * Change more docs to use DiffusionPipeline rather than a subclass. --------- Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
This PR edits the docs to use
DiffusionPipelineinstead of its child classes (such asStableDiffusionPipeline) where possible when loading a pipeline. The rationale behind this change is thatDiffusionPipelineshould always create a pipeline of the right class when loading a checkpoint and thus should be more user friendly. See #2135 for more discussion; in particular, this PR corresponds to item 4 in this comment.Below I've documented pages with examples that use a child class of
DiffusionPipelinewhen loading a checkpoint that I've opted not to change, along with a short explanation of why I think it should stay as is./docs/source/en/using-diffusers/loading.mdx(link)The examples here specifically discuss loading directly from the appropriate pipeline class (e.g.
StableDiffusionPipeline)./docs/source/en/using-diffusers/schedulers.mdx(link)The JAX/Flax example uses
FlaxStableDiffusionPipeline; since it deals specifically with Flax, I think it makes sense to use the more specific class./docs/source/en/using-diffusers/img2img.mdx(link)/docs/source/en/using-diffusers/inpaint.mdx(link)/docs/source/en/using-diffusers/depth2img.mdx(link)Since these pages talk about specific Stable Diffusion pipelines, I think it's appropriate to reference the specific pipeline class when loading the checkpoints.
/docs/source/en/using-diffusers/reproducibility.mdx(link)The running example on this page uses
DDIMPipeline, which I think is relevant to the discussion on reproducibility. I haven't tested the numbers produced if we substituteDiffusionPipelineforDDIMPipeline(presumably, they are close if not the same).Edit: in
0.15.0.dev0the numbers are less important to the exposition, so it might make more sense to replaceDDIMPipelinewithDiffusionPipeline./docs/source/en/using-diffusers/contribute_pipeline.mdx(link)I think it makes sense that there is an example that discusses loading a checkpoint explicitly from a custom pipeline (in this case,
UnetSchedulerOneForwardPipeline)./docs/source/en/using-diffusers/using_safetensors.mdx(link)I imagine substituting
DiffusionPipelineforStableDiffusionPipelinein the loading speed example could make a difference to the final loading numbers (I don't have the required hardware to test this myself)./docs/source/en/optimization/onnx.mdx(link)I think discussing the difference between
StableDiffusionPipelineandStableDiffusionOnnxPipelineis valuable, although perhaps there could be a note about usingDiffusionPipelineto load the checkpoint as well.Edit: in
0.15.0.dev0ORTStableDiffusionPipelinereplacesStableDiffusionPipeline. I think the comment above remains valid, although I'm not sure if 'DiffusionPipelineis an acceptable substitute forORTStableDiffusionPipeline`./docs/source/en/optimization/habana.mdx(link)Using
GaudiStableDiffusionPipelinehere instead of the genericDiffusionPipelineprobably makes more sense. (I'm not familiar with Habana Gaudi.)/docs/source/en/training/text_inversion.mdx(link)/docs/source/en/training/text2image.mdx(link)/docs/source/en/training/lora.mdx(link)These pages specifically discuss fine-tuning a Stable Diffusion model, so it feels weird to replace
StableDiffusionPipelinewithDiffusionPipeline.(Note that for
/docs/source/en/training/dreambooth.mdx(link), some of the code snippets useDiffusionPipelineand some useStableDiffusionPipeline, so I've changed all checkpoint loading examples to useDiffusionPipeline.)/docs/source/en/api/outputs.mdx(link)In the pipeline loading example, we follow up a call to
from_pretrained(...)with a call to__call__(...), so I think it makes sense to use a specific pipeline such asDDIMPipelinerather thanDiffusionPipeline(which doesn't implement a__call__method).(Sorry for making the PR so long!)
Edit: changed the links to point to the PR docs endpoint.