Fix: NotebookProgressCallback crash when evaluating with the Trainer by Charly21r · Pull Request #44949 · huggingface/transformers

Charly21r · 2026-03-23T16:07:50Z

What does this PR do?

This PR fixes an issue with NotebookProgressCallback in the Trainer where calling evaluate() before or after training would crash due to the training tracker being None. The callback now properly handles evaluation even if training has not yet started or if it has already finished, ensuring metrics can be computed and displayed.

Previously, the on_evaluate method assumed that self.training_tracker was always initialized, but:

Before training: self.training_tracker has not being initialised by on_train_begin yet.
After training: on_train_end sets self.training_tracker to None, so calling on_evaluate afterwards would fail.

Fix: on_evaluate now checks whether self.training_tracker exists before using it, and safely handles cases where it is None. This prevents crashes and ensures evaluation can run regardless of training state.

Additionally, new unit tests were added to ensure that evaluation works in this scenario, and existing notebook callback tests were updated to cover this case. This improves robustness of notebook-based workflows, especially in Jupyter or Colab environments.

Code Agent Policy

I confirm that this is not a pure code agent PR.

Before submitting

Did you read the contributor guideline, Pull Request section?
Did you write any new necessary tests?

Who can review?

@SunMarc

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

SunMarc · 2026-03-24T15:10:21Z

does this fix your issue @HenrikEilers ?

Charly21r · 2026-03-26T07:25:54Z

@SunMarc Just checking in, happy to make any changes if needed. If the original reporter isn’t available, I can also provide more details or tests to help validate the fix.

HenrikEilers · 2026-03-27T17:05:54Z

does this fix your issue @HenrikEilers ?

I cant really test it right now but if it now allows for trainer.evaluate() to be called after training then yes

HenrikEilers · 2026-04-03T20:05:44Z

Is there anything left to do for us to get the pr aproved?

Charly21r · 2026-04-07T14:03:18Z

Just checking in again, this PR should resolve the original issue by allowing trainer.evaluate() after training, as discussed above. Tests are passing on my side, and I’m happy to add more coverage if needed.

@SunMarc I'd really appreciate a review when you have time :)

SunMarc

Thanks ! Left a comment

SunMarc · 2026-04-07T14:37:33Z

+        if self.training_tracker is None:
+            return control


Ok this will work but we are not outputing anything in this case no like tt.write_line(values). Can you check what is the output that we get and if it makes sense, maybe we should add it.

Good point, I’ve updated on_evaluate to display the metrics as a standalone HTML table when there's no training tracker, so the user still sees the output. I also included the first_column computation into on_evaluate directly so it doesn't depend on `on_train_begin having run.

can you show me what you get ?

Calling evaluate() after train():

Calling evaluate() before train():

I noticed Model Preparation Time shows up as a column when calling it before train, should I filter it out along with the other runtime metrics?

Yeah, Model preparation time shouldn't show up. We shouldn't necessarily filter this one in particular but find out why it shows up when it doesn't when calling evaluate after train. Thanks for testing !

Found it. The trainer adds eval_model_preparation_time to the metrics dict when the model hasn't been prepared yet (self.accelerator._models is empty), which only happens when evaluate() is called before train(). After training, the model is already prepared so the metric is never added. The fix is just adding metrics.pop(f\"{metric_key_prefix}_model_preparation_time\", None) alongside the other filtered metrics.

SunMarc

Thanks, a few nits

SunMarc · 2026-04-09T15:15:36Z


    def on_evaluate(self, args, state, control, metrics=None, **kwargs):
-        tt = _require(self.training_tracker, "on_train_begin must be called before on_evaluate")
+        self.first_column = "Epoch" if args.eval_strategy == IntervalStrategy.EPOCH else "Step"


why we need to overwrite that ? we shouldn't have to

Because on_evaluate can also be called before on_train_begin (which is the bug this PR fixes), but self.first_column wouldn't exist yet since it's only initialized in on_train_begin. Another option would be to move the initialization to __init__ with a default of "Step" instead, so on_evaluate doesn't need to overwrite it. Defaulting to "Step" could make sense here since if training hasn't started, there are no epochs to reference. So I can do that if you prefer it.

understood. let's keep what you did, maybe just add a comment about that above

SunMarc

Thanks ! Just fix the issue about the logs that shows up incorrectly when doing evaluation first and we can merge this

HuggingFaceDocBuilderDev · 2026-04-10T14:03:46Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

…r training

…sCallback

SunMarc

Thanks for iterating !

…uggingface#44949) * Fix NotebookProgressCallback to allow evaluate() before and after train * Add unit test for NotebookProgressCallback evaluating before and after training * Skip NotebookProgressCallback tests when IPython is not installed * Display eval metrics when training tracker is None on NotebookProgressCallback * Add is_ipython_available and require_ipython test decorator * Filter model_preparation_time metric and add code comments in on_eval

Charly21r force-pushed the fix-evaluate-after-train branch from e61e591 to e25f2a6 Compare March 23, 2026 17:20

SunMarc reviewed Apr 7, 2026

View reviewed changes

Charly21r force-pushed the fix-evaluate-after-train branch 2 times, most recently from 08f50cb to 7fcca92 Compare April 8, 2026 06:50

SunMarc reviewed Apr 9, 2026

View reviewed changes

Charly21r force-pushed the fix-evaluate-after-train branch from 7bcd9b1 to 055eb9f Compare April 9, 2026 18:12

SunMarc approved these changes Apr 10, 2026

View reviewed changes

Charly21r added 6 commits April 10, 2026 17:30

Fix NotebookProgressCallback to allow evaluate() before and after train

199217c

Add unit test for NotebookProgressCallback evaluating before and afte…

2e20fce

…r training

Skip NotebookProgressCallback tests when IPython is not installed

cb1df8d

Display eval metrics when training tracker is None on NotebookProgres…

a2d67e2

…sCallback

Add is_ipython_available and require_ipython test decorator

7a01ca9

Filter model_preparation_time metric and add code comments in on_eval

2d98716

Charly21r force-pushed the fix-evaluate-after-train branch from 055eb9f to 2d98716 Compare April 10, 2026 15:30

SunMarc approved these changes Apr 13, 2026

View reviewed changes

SunMarc enabled auto-merge April 13, 2026 13:27

SunMarc added this pull request to the merge queue Apr 13, 2026

Merged via the queue into huggingface:main with commit 0b5dbfc Apr 13, 2026
28 checks passed

Conversation

Charly21r commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Before submitting

Who can review?

Uh oh!

SunMarc commented Mar 24, 2026

Uh oh!

Charly21r commented Mar 26, 2026

Uh oh!

HenrikEilers commented Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

HenrikEilers commented Apr 3, 2026

Uh oh!

Charly21r commented Apr 7, 2026

Uh oh!

SunMarc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SunMarc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SunMarc left a comment

Choose a reason for hiding this comment

Uh oh!

HuggingFaceDocBuilderDev commented Apr 10, 2026

Uh oh!

SunMarc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Charly21r commented Mar 23, 2026 •

edited

Loading

HenrikEilers commented Mar 27, 2026 •

edited

Loading