Quantitative assessment of the performance of pyCompressor + GANs

[Radonirinaunimi]

Compare the standard compressor with the Compressor + GANs using PDF-like estimators and/or metric distances:

• PDF-PDF correlation matrices
• Central Value-Standard Deviation relative uncertainties
• Frechet Inception Distance.
• KL distance.

@scarrazza: In addition to the above, are there any estimators/metrics you would like to consider?

[scarrazza]

Yes, we should include the PDF distance through the validphys action.

[Radonirinaunimi]

Below are some quantitative comparisons using FID. As a side note, the prior here contains 1000 MC replicas, which was enhanced to get 3000 MC replicas. Both the prior and enhanced sets are compressed down to 100. Each FID value (for all the flavours) measures the similarity between the prior and X, where X can just be a random replicas of size 100 (in blue), a compressed set from the prior (orange), or a compressed set from the enhanced (green). The FID is defined such that the smaller its value is, the closer X is to the prior.

Considering the difference between the blue and orange, the difference between the orange and green is somehow good. And, indeed, when looking at the correlations for instance, we see that the correlations from the enhanced look closer to the prior as compared to the standard.

EDIT: Add random to correlation plots.

[Radonirinaunimi]

The figures below show the comparison of the distributions along a given x-axis using the KL test. Both the standard and the enhanced distributions are tested against the prior. The smaller the value of v (or the larger the value of the probability p), the more the distribution is similar to the prior. As shown in the example below, in most of the cases, the enhanced yields a smaller value of v (or a larger value of p).

[scarrazza]

@Radonirinaunimi for these last plots, is the prior binning different from the compressions?

[Radonirinaunimi]

@Radonirinaunimi for these last plots, is the prior binning different from the compressions?

The binings are the same (even though the size of the datapoints are not), but the distributions are normalized such that the area under the histograms are all one.

[scarrazza]

Ok, thanks.

[Radonirinaunimi]

Concerning the PDF-PDF covariance matrices, are there ways to compare/assess two covariance matrices? Currently, I am computing the covariance matrix according to eq. 2 of the mc2hessian paper. Hence, for each set (prior, standard, enhanced), one gets a 4-dimensional matrix. What would be the test to compare them?

[scarrazza]

Why 4D? You take a grid of x points and compute for each flavour pair the correlation coefficient.

[Radonirinaunimi]

Why 4D? You take a grid of x points and compute for each flavour pair the correlation coefficient.

This I have done. And the same question above also applies here, especially since looking at the correlation matrices alone, one cannot assess what are the differences between the sets. For example, below are the correlation matrices for the 1000 prior and the 100 compressed (standard & enhanced) at 100 GeV:

I tried plotting the difference as in the compressor paper but the difference is not noticeable. I also had a look at some statistical tests (such as the Box's M test), but that was not conclusive. So, in the comparison of correlation matrices, we also need a way to assess the difference.

But coming back to the covariance matrix, I thought this was different and defined as below. Or what you actually meant was the above?

EDIT: Change plots to 100 GeV and add colorbar.

[scarrazza]

You should take these matrices and compute the difference (or ratio) between them, i.e.: compressed-prior, standard-prior.

[Radonirinaunimi]

When computing the difference (likewise for the ratio), we get similar figures (unnoticeable difference) for standard-prior & enhanced-prior. The figures are generated with 70 compressed at 100 GeV.

When computing the difference between the correlation matrices, the prior-enhanced seems a bit clearer than the prior-standard. But, looking at these figures might not be enough to assess the real difference.

EDIT: Rectify bugs in plotting figures that removes the total white.

[scarrazza]

Strange, please remove the vmax and vmin constrains from [-1,1], the differences should be above 1%.

[scarlehoff]

Am I right to understand from the plots above (not the heatmaps) that the enhanced is (at least!) as good as the standard in all cases?

The standard would be exactly the same as one would get using the old compressor right?

(I'm trying to make sense of this plus the thread with Stefano to have a full picture of the last two weeks developments before the meeting later!)