Add timestep_spacing to schedulers #3929

pcuenca · 2023-07-03T13:56:31Z

No description provided.

pcuenca · 2023-07-03T13:56:51Z

src/diffusers/schedulers/scheduling_ddim.py

        self.num_inference_steps = num_inference_steps

-        # "leading" and "trailing" corresponds to annotation of Table 1. of https://arxiv.org/abs/2305.08891
+        # "leading" and "trailing" corresponds to annotation of Table 2. of https://arxiv.org/abs/2305.08891


missing linspace from ddim, will add later

pcuenca · 2023-07-03T13:57:19Z

src/diffusers/schedulers/scheduling_ddpm.py

        dynamic_thresholding_ratio: float = 0.995,
        clip_sample_range: float = 1.0,
        sample_max_value: float = 1.0,
+        timestep_spacing: str = "leading",


default is leading here

pcuenca · 2023-07-03T13:57:58Z

src/diffusers/schedulers/scheduling_ddpm.py

+            if self.config.timestep_spacing == "linspace":
+                timesteps = np.linspace(0, self.config.num_train_timesteps - 1, num_inference_steps).round()[
+                    ::-1
+                ].copy().astype(np.int64)


Need to be careful here as some schedulers use float and others use ints.

How about performing the type-casting after all the conditions have been traversed?

Good idea, I'll check it out.

pcuenca · 2023-07-03T13:58:30Z

src/diffusers/schedulers/scheduling_lms_discrete.py

        trained_betas: Optional[Union[np.ndarray, List[float]]] = None,
        use_karras_sigmas: Optional[bool] = False,
        prediction_type: str = "epsilon",
+        timestep_spacing: str = "linspace",


This one defaults to linspace

HuggingFaceDocBuilderDev · 2023-07-03T14:02:36Z

The documentation is not available anymore as the PR was closed or merged.

pcuenca · 2023-07-03T14:13:45Z

src/diffusers/schedulers/scheduling_lms_discrete.py

+    @property
+    def init_noise_sigma(self):
+        # standard deviation of the initial noise distribution
+        if self.config.timestep_spacing == "linspace":


I think this should be the case for trailing too.

pcuenca · 2023-07-03T14:19:13Z

src/diffusers/schedulers/scheduling_pndm.py

+        timestep_spacing (`str`, default `"leading"`):
+            The way the timesteps should be scaled. Refer to Table 2. of [Common Diffusion Noise Schedules and Sample
+            Steps are Flawed](https://arxiv.org/abs/2305.08891) for more information.


Note: steps_offset already existed here, I put timestep_spacing before it for consistency.

pcuenca · 2023-07-03T14:21:53Z

src/diffusers/schedulers/scheduling_pndm.py

-        self._timesteps += self.config.steps_offset
+        # "linspace", "leading", "trailing" corresponds to annotation of Table 2. of https://arxiv.org/abs/2305.08891
+        if self.config.timestep_spacing == "linspace":
+            self._timesteps = np.linspace(0, self.config.num_train_timesteps - 1, num_inference_steps).round().astype(np.int64)


Note they are not reversed in this case

pcuenca · 2023-07-03T15:53:51Z

One problem is that PNDMScheduler defaults to leading, but others default to linspace. So for example when changing an scheduler after loading the pipeline, this would behave differently as before:

import torch
from diffusers import DiffusionPipeline
from diffusers import HeunDiscreteScheduler

pipe = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
pipe = pipe.to("cuda")

pipe.scheduler = HeunDiscreteScheduler.from_config(pipe.scheduler.config)
# Here we are using heun with `leading` instead of the previous default `linspace`

@patrickvonplaten how do you think we should deal with this?

Not sure the range is the way it was intended.

patrickvonplaten · 2023-07-03T17:28:23Z

One problem is that PNDMScheduler defaults to leading, but others default to linspace. So for example when changing an scheduler after loading the pipeline, this would behave differently as before:
import torch
from diffusers import DiffusionPipeline
from diffusers import HeunDiscreteScheduler

pipe = DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)
pipe = pipe.to("cuda")

pipe.scheduler = HeunDiscreteScheduler.from_config(pipe.scheduler.config)
# Here we are using heun with `leading` instead of the previous default `linspace`
@patrickvonplaten how do you think we should deal with this?

I thought about this too previously. Can we do the following here: We add a mechanism for values that are not loaded from the config: https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/scheduler/scheduler_config.json and then default to the value in the __init__ function that stores these as a list in the hidden_config_dict here, under "_use_default_values: List[Any]" and then pop those values from the config when loading it again?

For params not present in the original configuration. This makes it possible to switch pipeline schedulers even if they use different timestep_spacing (or any other param).

src/diffusers/configuration_utils.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

tests/schedulers/test_schedulers.py

This test currently fails in main. When switching from DEIS to UniPC, solver_type is "logrho" (the default value from DEIS), which gets translated to "bh1" by UniPC. This is different to the default value for UniPC: "bh2". This is where the translation happens: https://github.com/huggingface/diffusers/blob/36d22d0709dc19776e3016fb3392d0f5578b0ab2/src/diffusers/schedulers/scheduling_unipc_multistep.py#L171

pcuenca · 2023-07-04T13:01:03Z

This is ready for review now. I'm not fully sure that the leading and trailing implementation for DEIS, UniPC, DPMSolverSDE, DPMSolverMultiStep, DPMSolverSingleStep are as intended.

Fixes a bug when switching from UniPC from another scheduler (i.e., DEIS) that uses a different solver type. The solver is now the same as if we had instantiated the scheduler directly.

patrickvonplaten · 2023-07-04T17:45:27Z

PR looks great to me - however could we merge this into "main" and not the sd_xl branch so that in can go into this weeks' release?

pcuenca · 2023-07-04T18:20:00Z

Rebased in #3947.

Add timestep_spacing to DDPM, LMSDiscrete, PNDM.

f905958

pcuenca commented Jul 3, 2023

View reviewed changes

pcuenca changed the title ~~Add timestep_spacing to DDPM, LMSDiscrete, PNDM.~~ Add timestep_spacing to schedulers Jul 3, 2023

pcuenca marked this pull request as draft July 3, 2023 13:59

Remove spurious line.

d9b68b9

pcuenca commented Jul 3, 2023

View reviewed changes

pcuenca added 3 commits July 3, 2023 16:27

More easy schedulers.

eedf428

Add linspace to DDIM

9f0db7b

Noise sigma for trailing.

2bde078

Add timestep_spacing to DEISMultistepScheduler.

2ad9bb1

Not sure the range is the way it was intended.

pcuenca added 4 commits July 3, 2023 20:06

Fix: remove line used to debug.

fcc5531

Support timestep_spacing in DPMSolverMultistep, DPMSolverSDE, UniPC

f6e4782

Fix: convert to numpy.

5cede2b

Use sched. defaults when instantiating from_config

04b9a76

For params not present in the original configuration. This makes it possible to switch pipeline schedulers even if they use different timestep_spacing (or any other param).

patrickvonplaten reviewed Jul 4, 2023

View reviewed changes

src/diffusers/configuration_utils.py Outdated Show resolved Hide resolved

patrickvonplaten reviewed Jul 4, 2023

View reviewed changes

src/diffusers/configuration_utils.py Outdated Show resolved Hide resolved

pcuenca and others added 4 commits July 4, 2023 12:02

Apply suggestions from code review

e9dc9fd

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

Missing args in DPMSolverMultistep

a30f92a

Test: default args not in config

a9ccade

Style

1ac79fe

pcuenca commented Jul 4, 2023

View reviewed changes

tests/schedulers/test_schedulers.py Outdated Show resolved Hide resolved

Fix scheduler name in test

27c134d

pcuenca added 2 commits July 4, 2023 13:25

Remove duplicated entries

c6fe8b0

pcuenca marked this pull request as ready for review July 4, 2023 12:59

UniPC: use same default for solver_type

1774aad

Fixes a bug when switching from UniPC from another scheduler (i.e., DEIS) that uses a different solver type. The solver is now the same as if we had instantiated the scheduler directly.

pcuenca mentioned this pull request Jul 4, 2023

Add timestep_spacing and steps_offset to schedulers #3947

Merged

pcuenca closed this Jul 4, 2023

yiyixuxu mentioned this pull request Jun 27, 2024

Confusion about the use of from_config and its relation to _use_default_values #7626

Open

Add timestep_spacing to schedulers #3929

Add timestep_spacing to schedulers #3929

Uh oh!

Conversation

pcuenca commented Jul 3, 2023

Uh oh!

pcuenca Jul 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pcuenca Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

pcuenca Jul 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

pcuenca Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

pcuenca Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Jul 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pcuenca Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

pcuenca Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

pcuenca Jul 3, 2023

Choose a reason for hiding this comment

Uh oh!

pcuenca commented Jul 3, 2023

Uh oh!

patrickvonplaten commented Jul 3, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pcuenca commented Jul 4, 2023

Uh oh!

patrickvonplaten commented Jul 4, 2023

Uh oh!

pcuenca commented Jul 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

pcuenca Jul 3, 2023 •

edited

Loading

pcuenca Jul 3, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Jul 3, 2023 •

edited

Loading