Update FLAIR results (relMASE 0.838, relCRPS 0.587)

[TakatoHonda]

Summary

Update FLAIR (v0.6.0) on GIFT-Eval. Follows up on PR #122 with the reproduction script you asked for.

• relMASE: 0.9201 → 0.8384
• relCRPS: 0.6903 → 0.5871
• replication_code_available: Yes
• code_link: examples/gift_eval_reproduction.py

Reproduction

pip install flaircast
git clone https://github.com/TakatoHonda/FLAIR.git
export GIFT_EVAL=/path/to/gift-eval-data
python FLAIR/examples/gift_eval_reproduction.py

What changed since PR #122

• Unified BIC with P=1 null model (rejects periodicity when a rank-1 fit doesn't justify the extra Shape parameters)
• Gavish-Donoho 2014 optimal Frobenius shrinkage of the rank-1 singular value
• Dynamic Ridge DoF guard (requires n_train >= 2p)
• Horizon-adaptive phase-noise deflation
• James-Stein per-phase bias shrinkage
• pandas 2.2+ frequency alias support

Package: pip install flaircast (v0.6.0)
GitHub: https://github.com/TakatoHonda/FLAIR

[cuthalionn]

Hey @TakatoHonda,

Thanks for the new script it is very easy to use!

One issue: I tried to replicate results for some datasets and I am getting some differences in the results. I wonder if there is a seed that I would need to set, or randomness is expected naturally? I tried to follow the exact steps that you shared above.

Results I got for the subset of datasets I tried (I also added submitted MASE for ease of comparison):

22500it [01:22, 271.54it/s]
  [  1/97] bitbrains_fast_storage/5T/short               MASE=0.9997 CRPS=0.4706 (83s) 
Submitted MASE=0.9904
2500it [00:25, 99.61it/s]
  [  2/97] bitbrains_fast_storage/5T/medium              MASE=1.1207 CRPS=0.6333 (25s)
Submitted MASE=1.1347
2500it [00:34, 73.16it/s]
  [  3/97] bitbrains_fast_storage/5T/long                MASE=1.0496 CRPS=0.6660 (34s)
Submitted MASE=1.0968
2500it [00:07, 355.89it/s]
  [  4/97] bitbrains_fast_storage/H/short                MASE=1.3213 CRPS=0.6425 (7s)
Submitted MASE=1.2639
9000it [00:33, 268.81it/s]
  [  5/97] bitbrains_rnd/5T/short                        MASE=1.9100 CRPS=0.4989 (33s)
Submitted MASE=1.8986
1000it [00:09, 101.45it/s]
  [  6/97] bitbrains_rnd/5T/medium                       MASE=4.5308 CRPS=0.6017 (10s)
Submitted MASE=4.5296
1000it [00:13, 74.80it/s]
  [  7/97] bitbrains_rnd/5T/long                         MASE=3.5089 CRPS=0.6267 (13s)
Submitted MASE=3.4974
1000it [00:02, 356.18it/s]
  [  8/97] bitbrains_rnd/H/short                         MASE=6.2281 CRPS=0.5913 (3s)
Submitted MASE=5.9843
15it [00:00, 246.89it/s]
  [  9/97] bizitobs_application/10S/short                MASE=2.2751 CRPS=0.0246 (0s)
Submitted MASE=3.7251
2it [00:00, 93.84it/s]
  [ 10/97] bizitobs_application/10S/medium               MASE=2.4852 CRPS=0.0425 (0s)
Submitted MASE=6.9703

Any suggestions or insight about this gap?
Thanks!

[TakatoHonda]

Hi @cuthalionn, thanks for running the reproduction. That gap flagged a real issue.

It isn't randomness. The script is deterministic (seed = md5 of item_id) and your reproduction is correct. What broke was the submission itself: the CSV I uploaded had the right aggregate relMASE / relCRPS, but about 48/97 of the per-config MASE rows and 57/97 of the CRPS rows had corrupted seasonal-error denominators from gluonts. The per-item MAE / MSE values were fine, so the predictions were right; only the MASE scaling was inconsistent. My best guess is environment contamination during the original evaluation run.

I re-ran the full 97 configs in an isolated git worktree (flaircast==0.6.0, gluonts==0.15.1, pandas==2.2.3, numpy<2, N_SAMPLES=200) and pushed the regenerated CSV as 3288fa4. Two independent re-runs produced bit-identical MASE/CRPS.

Every one of the 10 configs you posted matches the new CSV to 4 decimals:

bitbrains_fast_storage/5T/short    ours 0.9997  yours 0.9997
bitbrains_rnd/H/short              ours 6.2281  yours 6.2281
bizitobs_application/10S/short     ours 2.2751  yours 2.2751
bizitobs_application/10S/medium    ours 2.4852  yours 2.4852
...

The aggregate numbers (relMASE=0.8384, relCRPS=0.5871) stay the same; they were computed on a separate normalization path that wasn't affected. Sorry for the confusion and thanks for catching it.

[TakatoHonda]

Second bug, same review. Sorry about this.

Looking at the script itself I noticed the final print block labels gm(raw MASE) / gm(raw CRPS) as relMASE / relCRPS. They aren't. Running it from scratch prints relMASE ≈ 1.17, not the 0.838 this PR claims. Anyone following the README would hit that inconsistency on the first run. Which is embarrassing; the whole point of shipping a replication script is that it reproduces the headline number.

Fix pushed to TakatoHonda/FLAIR@d1d2cb4:

• Rename the raw geomeans to MASE_gm / CRPS_gm so they aren't confused with rel-*
• Pull the Seasonal Naive baseline from results/seasonal_naive/all_results.csv and divide each row
• Print actual relMASE / relCRPS as the geomean of per-config (model / SN)
• Update the docstring's expected numbers to 0.8384 / 0.5871

After the fix the script prints:

MASE_gm = 1.1719
CRPS_gm = 0.1480

Aggregate normalized by Seasonal Naive (matched 97/97 configs):
  relMASE = 0.8384
  relCRPS = 0.5871

That matches the PR body and the CSV at 3288fa4.

The code_link already points at main, so it serves the fixed script now without touching the PR body. Between this and the CSV regeneration the PR should finally be reproducible. Thanks again.