Feature: Calibration Error Metrics and Losses

[theo-barfoot]

Is your feature request related to a problem? Please describe.

Currently, MONAI does not provide built-in support for calibration metrics, such as Expected Calibration Error, or auxiliary calibration losses that can be used to improve the calibration of medical image segmentation networks. I have implemented these features, using the MONAI framework, which can be found at this repo, with unit testing, and corresponding publications here and here

---

Describe the solution you’d like

I propose adding my calibration metrics and handlers, as well as auxiliary calibration losses to MONAI.

1. Calibration Metrics

   • CalibrationErrorMetric (subclass of CumulativeIterationMetric) supporting Expected (ECE), Average (ACE), and Maximum (MCE) reductions, with batched, per-class, and background-exclusion settings.

2. Differentiable Calibration Losses

   • HardL1ACELoss and its compound variants (HardL1ACEandCELoss, HardL1ACEandDiceLoss, HardL1ACEandDiceCELoss)
   • SoftL1ACELoss and its compound variants (SoftL1ACEandCELoss, SoftL1ACEandDiceLoss, SoftL1ACEandDiceCELoss)
     These losses implement the L1 Average Calibration Error (ACE) in hard- and soft-binned form, with options for background exclusion, one-hot encoding, custom activation, and class weighting.

3. Ignite Handlers

   • CalibrationError inheriting from IgniteMetricHandler, to attach calibration metrics to training and evaluation engines, with automatic logging and CSV export of per-image details.

Optionally, I also have visualisation method for reliability diagrams and reliability dataset histograms, however these may be harder to integrate nicely into the current MONAI framework.

Visualisation Utilities (maybe)

• Plotting functions draw_case_reliability_diagrams and draw_dataset_reliability_diagrams for reliability diagrams and histograms, with customisable figure size, colormaps, and annotations. And their associated ReliabilityDiagramMetric and ReliabilityDiagramHandler

---

All components are already implemented and thoroughly unit-tested in Average-Calibration-Losses, and integrate into MONAI’s bundle-based pipelines out of the box.

---

Describe alternatives you’ve considered

• Third-party libraries: Calibration Error Metric do exist in torchmetrics and net:cal but are not implemented via the CumulativeIterationMetric used in MONAI. Differentiable calibration losses for semantic segmentation are not implemented elsewhere.

Integrating these features into MONAI directly will give users a standardised, well-tested interface for calibration, reduce duplication, and promote reproducible, robust model evaluation.

---

I’m happy to contribute these components as a PR to MONAI. Please let me know which of these features would be useful and how best to align with MONAI’s architecture and naming conventions, and any feedback on API design or testing guidelines!

[ericspod]

Hi @theo-barfoot thanks for proposing these additions. I don't see any particular challenges with integrating this into MONAI as it is. The code in your repo looks fine but we will review with a full PR anyway and adjust some minor things like formatting. One thing that we will need is a clear tutorial in the Tutorials repo to along with the new losses to demonstrate what they, what they mean, and where to use them. I don't expect we'd explain everything about calibration but some links to good sources in the tutorial would help, and links to relevant concepts in the doctrings of the definitions themselves. You should also cite your paper in the doctings.

So I think we have a few actions:

• Put together the losses, metrics, handlers, and associated tests into a PR and we review from there.
• Write up a tutorial illustrating the use of these components in the Tutorials repo.
• Add in the visualisation components as a separate PR if these are useful (I haven't looked at these in detail so couldn't say right now if they are).

We don't have anything on calibration so these would be good additions, it would further be good to have tutorials on the subject in general but that's for another day!