Update trainer for easier handling of accumulate, compile fixes, and proper reporting by muellerzr · Pull Request #34511 · huggingface/transformers

muellerzr · 2024-10-30T14:35:37Z

What does this PR do?

Alternative to #34442

TL;DR we just need to remove lru_cache and everything will work fine. (and adds a test)

This PR also takes the full lessons from my article and adds it to the Trainer for a simpler solution to the grad accum calculation (we shouldn't rely on accelerator from now on bc it can't handle the nuances with the grad accum fix at the highest level API, so we use a lower level version instead)

Fixes #34402

Would recommend a patch after this

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @Rocketknight1

HuggingFaceDocBuilderDev · 2024-10-30T15:06:27Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Rocketknight1

Couple comments about the test!

Rocketknight1

Tests look clean to me now, and I'm trusting you on the accelerate side of things! 😅

cc @LysandreJik / @ArthurZucker for core maintainer review

muellerzr · 2024-10-30T18:11:38Z

+                    context = (
+                        functools.partial(self.accelerator.no_sync, model=model)
+                        if i == len(batch_samples) - 1
+                        else contextlib.nullcontext
+                    )


For an explanation on what we have going on here @Rocketknight1 , during DDP we use model.no_sync() to only communicate across all GPUs during the next step outside it (so we speed up training when not needed when doing gradient accumulation). accelerator.no_sync() is the lower-level accumulate() API which makes that op backed-independent (so on a single GPU it just does nullcontext)

ArthurZucker · 2024-10-31T13:35:05Z

@Milad335t just warning you to stop spamming or we'll have to block you 😢

ArthurZucker

Thanks, let's hope this gets stabilized!

ArthurZucker · 2024-10-31T13:41:25Z

-                num_items_in_batch = sum(
-                    [data_batch["labels"][..., 1:].ne(-100).sum().item() for data_batch in batch_samples]
-                )
+                num_items_in_batch = sum([(batch["labels"].ne(-100)).sum() for batch in batch_samples])


weird to me that we have to use -100 here, instead of a general parameter but whit was already the case

IIRC we use -100 for padding by default in the Trainer. I can align it to self.processor if it exists else -100 if that's better?:)

Actually our padding index is -100 everywhere.

okay sounds good then sorry

No worries, it's weird for me too :)

Why do we no longer need to shift labels ["labels"][...,1:] when getting num_items_in_batch?

sure, I also think we need to shift labels before computing the num_items_in_batch. Otherwise, the value is incorrect as the first element in labels may not be -100

ArthurZucker

Thanks patching today!

ArthurZucker · 2024-11-04T11:44:16Z

-                num_items_in_batch = sum(
-                    [data_batch["labels"][..., 1:].ne(-100).sum().item() for data_batch in batch_samples]
-                )
+                num_items_in_batch = sum([(batch["labels"].ne(-100)).sum() for batch in batch_samples])


okay sounds good then sorry

…proper reporting (#34511) * Update trainer for easier handling of accumulate + proper reporting * test * Fixup tests * Full fix * Fix style * rm comment * Fix tests * Minimize test + remove py 311 check * Unused import * Forward contrib credits from discussions * Fix reported metrics * Refactor, good as it's going to get * rm pad tok id check * object detection and audio are being annoying * Fin * Fin x2 --------- Co-authored-by: Gyanateet Dutta <Ryukijano@users.noreply.github.com>

…proper reporting (huggingface#34511) * Update trainer for easier handling of accumulate + proper reporting * test * Fixup tests * Full fix * Fix style * rm comment * Fix tests * Minimize test + remove py 311 check * Unused import * Forward contrib credits from discussions * Fix reported metrics * Refactor, good as it's going to get * rm pad tok id check * object detection and audio are being annoying * Fin * Fin x2 --------- Co-authored-by: Gyanateet Dutta <Ryukijano@users.noreply.github.com>

hiyouga · 2024-12-11T13:35:57Z

        else:
-            if num_items_in_batch is not None:
-                if self.compute_loss_func or self.model_accepts_loss_kwargs:
-                    loss *= self.args.gradient_accumulation_steps


I'm confused that the loss is no longer multiplied by the gradient accumulation steps here, because the loss has been multiplied by the data parallel size in https://github.com/huggingface/transformers/pull/34511/files#diff-ed55888e6665791fe92cc8fc0c499da54f4ace6738551cd9a2591881cda076deR3702-R3703

I see, it should be solved in #35207

muellerzr added 4 commits October 30, 2024 09:41

Update trainer for easier handling of accumulate + proper reporting

ec49756

test

4cdee53

Fixup tests

fb8070f

Full fix

9c6ed74

muellerzr requested review from ArthurZucker and Rocketknight1 October 30, 2024 14:35

muellerzr mentioned this pull request Oct 30, 2024

Fix loss function compatibility with torch dynamo #34442

Closed

muellerzr added 2 commits October 30, 2024 10:37

Fix style

aab5467

rm comment

c56ffe6

muellerzr added 2 commits October 30, 2024 11:15

Fix tests

4a8a2a3

Merge branch 'main' into muellerzr-final-gradaccum-check

18abcb5

Rocketknight1 reviewed Oct 30, 2024

View reviewed changes

Comment thread tests/trainer/test_trainer.py

Comment thread src/transformers/testing_utils.py Outdated

Minimize test + remove py 311 check

2f41eb7

muellerzr requested a review from Rocketknight1 October 30, 2024 17:25

Unused import

27eaadc

Rocketknight1 approved these changes Oct 30, 2024

View reviewed changes

muellerzr commented Oct 30, 2024

View reviewed changes

Ryukijano and others added 4 commits October 30, 2024 14:23

Forward contrib credits from discussions

6fcb0b5

Fix reported metrics

43f6a2f

Refactor, good as it's going to get

238c985

rm pad tok id check

93f36e8

object detection and audio are being annoying

c633219

ArthurZucker reviewed Oct 31, 2024

View reviewed changes

muellerzr added 2 commits October 31, 2024 09:44

Fin

c452194

Fin x2

7739461

ArthurZucker approved these changes Nov 4, 2024

View reviewed changes

muellerzr merged commit ef976a7 into main Nov 4, 2024

muellerzr deleted the muellerzr-final-gradaccum-check branch November 4, 2024 12:47

Jintao-Huang mentioned this pull request Nov 8, 2024

Compat transformers 4.46.2 loss modelscope/ms-swift#2413

Merged

fzyzcjy mentioned this pull request Nov 25, 2024

Loosen dependency version requirements OpenRLHF/OpenRLHF#534

Closed

ArthurZucker mentioned this pull request Nov 25, 2024

Torch.compile Graph break introduced due to new loss function api #34615

Closed

4 tasks

hiyouga reviewed Dec 11, 2024

View reviewed changes

hiyouga mentioned this pull request Dec 27, 2024

Pass correct num_items_in_batch value into the training_step function #35438

Merged

5 tasks

hiyouga mentioned this pull request Mar 24, 2025

Paligemma: fix generation with Gemma2 #36044

Merged

w32zhong mentioned this pull request Aug 9, 2025

Previous PRs introduced a bug on Accumulated Gradients Losses #40052

Closed

4 tasks

Conversation

muellerzr commented Oct 30, 2024

What does this PR do?

Before submitting

Who can review?

Uh oh!

HuggingFaceDocBuilderDev commented Oct 30, 2024

Uh oh!

Rocketknight1 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Rocketknight1 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurZucker commented Oct 31, 2024

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hiyouga Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

hiyouga Dec 11, 2024 •

edited

Loading