fix: qwen2_5_7b_squad ckpt robustness thresholds for transformers v5.5

[adil-a]

Summary

• Bump ci.checkpoint_robustness.hf_kl_threshold in examples/llm_finetune/qwen/qwen2_5_7b_squad.yaml from 9e-3 to 2.5e-2 and add resume_loss_threshold: 5e-2, to restore the qwen2_5_7b_squad / sft_ckpt_robustness CI job that started failing after the transformers v5.5 upgrade (ci: Update to transformers v5.5 #1734).
• Follows the same pattern as fix: relax KL thresholds and remove invalid kwargs in Qwen3Next linear attn #1867 (qwen3_moe, gpt_oss) and fix: gemma_3_270m_squad HF KL regression in ckpt robustness #1932 (gemma_3_270m_squad) for the Phase 4 HF-KL bump, and the baichuan_2_7b_squad.yaml precedent for resume_loss_threshold: 5e-2 on TP=2 SFT.

Evidence

Pre-fix, CI job 301287531:

[Phase 3] Automodel-from-consolidated max KL: 0.000000e+00 (threshold: 0.000000e+00)
[Phase 4] HF-loaded max KL: 5.140897e-03 (threshold: 9.000000e-03)
[Phase 5] Cross-TP (tp_size=2) max KL: 0.000000e+00 (threshold: 9.000000e-03)
...
AssertionError: SFT loss mismatch after resume at step 5:
  baseline=4.089181, resume=2.453070, diff=1.636111e+00

Phase 3 (automodel-from-consolidated) KL is exactly 0, so the save/reload path is bit-exact. Phases 4 and 5 also pass in the CI run. The failure is Phase 6 (training resumption) with the 5e-3 default resume_loss_threshold. Reproducing on cw-dfw 8xH100 with transformers 5.5.0 and the CI launcher overrides (--step_scheduler.max_steps 5 --step_scheduler.ckpt_every_steps 5 --step_scheduler.val_every_steps 5 --step_scheduler.global_batch_size 32 --step_scheduler.local_batch_size 2) shows the same failure mode — Phase 6 loss diffs drifting between runs and occasionally exceeding 5e-3; multiple runs also showed Phase 4 HF KL drifting up to ~1.1e-2 above the 9e-3 threshold. This is the same kind of forward-pass + optimizer-step numerical drift (TP=2 bf16 accumulation + v5.5's revised Qwen2 HF forward) that the sibling PRs already addressed via threshold bumps.

Post-fix verification on cw-dfw 8xH100 (transformers 5.5.0, same CI overrides):

[Phase 3] Automodel-from-consolidated max KL: 0.000000e+00 (threshold: 0.000000e+00)
[Phase 4] HF-loaded max KL: 4.951302e-03 (threshold: 2.500000e-02)
[Phase 5] Cross-TP (tp_size=2) max KL: 0.000000e+00 (threshold: 9.000000e-03)
[Phase 6] Step 5: baseline_loss=2.035934, resume_loss=2.037734, diff=1.799345e-03
[Phase 6] Step 6: baseline_loss=1.800883, resume_loss=1.803557, diff=2.673864e-03
[Phase 6] Step 7: baseline_loss=1.887681, resume_loss=1.891772, diff=4.091144e-03
[Phase 6] Training resumption verified (3 steps compared) ✓
================== 1 passed, 24 warnings in 222.65s (0:03:42) ==================

Test plan

• Reproduce the CI failure on cw-dfw (transformers 5.5.0, same launcher overrides) and confirm Phase 3 KL = 0 (save/reload is bit-exact).
• Apply the threshold bumps and rerun the same test end-to-end — Phases 1-6 all pass with the new thresholds.
• Verify Phase 3 KL still 0 and Phase 5 KL still 0 — no regression in save/reload correctness.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[adil-a]

Superseded by #1984. YAML changes absorbed. Per #1971 policy, hf_kl_threshold unified to 1e-1 in the batched PR (yours had 2.5e-2); resume_loss_threshold: 5e-2 preserved as you wrote. Relying on your separate-env re-verification that this recipe's resume dynamics work on current sqsh.