Qwen edit 2511 training speed

[preethamp0197]

When training with zero2offloading, I have 95% of GPU memory utilization but but only 20% of GPU utilization. And with no offloading it gives me OOM error. I am training it on 8 H100s.
HOw can I increase my GPU utilization
Also would like to know if I have to train it on 5 nodes (each with 8 h100s) Whats the best configuration? Can I use deepspeed 3? Or something like Deepspeed zero++?

[preethamp0197]

I tried data process and then sft. but it takes around 15-25 sec/iteration. And in this 10-15 secs taken by backward pass. Any tips on how to make it faster?

[lyz20020125]

@preethamp0197 I encountered the same issue while training Qwen Edit 2509. Have you managed to resolve it?

[preethamp0197]

With Qwen edit 2511 I am using deepspeed 3 and no offloading. Just make sure to load the models in cpu first and then do accelerate prepare.

[lyz20020125]

@preethamp0197 Thank you so much for your prompt reply! I have another question: regarding what you mentioned—loading the models on the CPU first and then performing accelerate prepare—is this still implemented within the DiffSynth framework?

[lyz20020125]

Could you please provide a more detailed explanation of how to achieve this effect? Truly appreciate your help!

With Qwen edit 2511 I am using deepspeed 3 and no offloading. Just make sure to load the models in cpu first and then do accelerate prepare.

[preethamp0197]

In train.py
I made device='cpu', when calling QwenImageTrainingModule and then after that I created my accelerator
accelerator = accelerate.Accelerator(
gradient_accumulation_steps=args.gradient_accumulation_steps,
kwargs_handlers=[accelerate.DistributedDataParallelKwargs(find_unused_parameters=args.find_unused_parameters)],
)

[lyz20020125]

In train.py I made device='cpu', when calling QwenImageTrainingModule and then after that I created my accelerator accelerator = accelerate.Accelerator( gradient_accumulation_steps=args.gradient_accumulation_steps, kwargs_handlers=[accelerate.DistributedDataParallelKwargs(find_unused_parameters=args.find_unused_parameters)], )

Yes! Exactly as you mentioned—I also modified this part following your previous guidance:

1. Set device='cpu' in QwenImageTrainingModule,
2. Moved the initialization of accelerator to after the QwenImageTrainingModule instantiation.

My YAML configuration includes the following section:
deepspeed_config: gradient_accumulation_steps: 1 offload_optimizer_device: 'none' offload_param_device: 'none' zero3_init_flag: true zero_stage: 3

I'm using four H20 GPUs (each with 96GB of VRAM), yet I'm still encountering an OOM error. Is this expected behavior?
Thank you so much—once again, you've come to my rescue in a time of need!

[preethamp0197]

I am using this config
compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
deepspeed_multinode_launcher: standard
gradient_accumulation_steps: 1
offload_optimizer_device: none
offload_param_device: none
zero3_init_flag: true
zero_stage: 3
zero3_save_16bit_model: true
distributed_type: DEEPSPEED
downcast_bf16: 'no'
enable_cpu_affinity: false
main_training_function: main
mixed_precision: bf16
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
num_machines: 4
num_processes: 32
machine_rank: 0
main_process_ip: master_ip
main_process_port: 18888

I am able to train Qwenedit 2511 with this on 1 node with h100 gpus

[preethamp0197]

Can you check when is it giving OOM error? Is it while loading or is it during the forward process?

[lyz20020125]

Can you check when is it giving OOM error? Is it while loading or is it during the forward process?

The OOM error occurs not during model loading:
0%| | 0/20 [00:13<?, ?it/s]
[rank0]: Traceback (most recent call last):
[rank0]: File "/DiffSynth-Studio/examples/qwen_image/model_training/train17.py", line 176, in
[rank0]: launcher_map[args.task](accelerator, dataset, model, model_logger, args=args)
[rank0]: File "/DiffSynth-Studio/diffsynth/diffusion/runner.py", line 41, in launch_training_task
[rank0]: accelerator.backward(loss)
[rank0]: File "/env/qwen/lib/python3.10/site-packages/accelerate/accelerator.py", line 2844, in backward
[rank0]: self.deepspeed_engine_wrapped.backward(loss, sync_gradients=self.sync_gradients, **kwargs)

compute_environment: LOCAL_MACHINE
debug: false
deepspeed_config:
gradient_accumulation_steps: 1
offload_optimizer_device: 'none'
offload_param_device: 'none'
zero3_init_flag: true
zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: 'no'
enable_cpu_affinity: false
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

Moreover, when I set gradient_accumulation_steps to 1, the above OOM error occurs immediately; when set to 4, it runs for 3 steps and then triggers an OOM error.
15%|███████████ | 3/20 [00:33<03:08, 11.06s/it]

Perhaps I should try setting num_processes: 8 (when more GPUs are available)？
Thank you for your patient guidance!