Implementation of ATLAS WPWM 8TeV - excluded from 4.0

[vschutze-alt]

No description provided.

[scarlehoff]

fwiw, for the error, rebase on top of master (you can use the button "update with rebase" here) and submit the bot with the "regenerate-data" label

[scarlehoff]

For this one the grids are already available: https://github.com/NNPDF/theories_slim/pull/64
so you can add the theory to the commondata file (and even create the fktable and test the implementation!)

[vschutze-alt]

Regarding the luminosity uncertainty in this dataset, I can see that the paper mentions that the luminosity uncertainty is 1.9%, but I cannot identify any of the uncertainties in the helpdata files as the luminosity uncertainty. Does this mean that I have to add it myself manually? Thanks!

[enocera]

Regarding the luminosity uncertainty in this dataset, I can see that the paper mentions that the luminosity uncertainty is 1.9%, but I cannot identify any of the uncertainties in the helpdata files as the luminosity uncertainty. Does this mean that I have to add it myself manually? Thanks!

Yes!

[ecole41]

@enocera @juanrojochacon I have re-run the data-theory plots using the grids in theories_slim and using this branch with the new fix to the shifts. We are seeing a good agreement:

However the $\chi^2$ values are very large:

Exp chi2: 132.5
Exp+th chi2: 99.62
Exp+th+pdf chi2: 52.56

Do you understand why this may be?

[juanrojochacon]

Thanks @ecole41 this is weird: from the shifted plots one can compute the chi2 almost by hand, and should be of order 1 per data point. At least once PDF errors are accounted for.

[juanrojochacon]

So it seems to me that the chi2 values are not consistent with what is being shown in the data vs theory comparison plot?

[enocera]

@ecole41 What's the blue uncertainty in your plots? The PDF uncertainty, the MHOU uncertainty or the sum in quadrature of the two? The shift does include only the effect of experimental correlations. But in principle there are also PDF and theory correlations that are not currently accounted in the shift. So it could be (and this happens, e.g. for the ATLAS ZpT 13 TeV data set) that the chi2 would be of order one once you incorporate only the diagonal term of the theory covariance matrix (which I guess is what is displayed in your plots).

[ecole41]

@ecole41 What's the blue uncertainty in your plots? The PDF uncertainty, the MHOU uncertainty or the sum in quadrature of the two? The shift does include only the effect of experimental correlations. But in principle there are also PDF and theory correlations that are not currently accounted in the shift. So it could be (and this happens, e.g. for the ATLAS ZpT 13 TeV data set) that the chi2 would be of order one once you incorporate only the diagonal term of the theory covariance matrix (which I guess is what is displayed in your plots).

I am not so sure which uncertainties are included in the blue points. This is plotted using the plot_fancy function in the API script to plot grids, I believe that it only included a standard error across replica predictions (PDF uncertainties). But it would be a good idea to clarify this with e.g @scarlehoff

[juanrojochacon]

Well we should be able to know exactly what goes on into the plots, else we are in trouble.

As far as I understand, the shifted data vs theory comparison plots include only PDF uncertainties.

The value Exp chi2: 132.5 is consistent with the plot, but not those of the chi2 once MHOU and PDF errors are accounted for. Considering only central PDF errors brings the chi2 down to 1, as @enocera indicates.

I would be very surprised if accounting for PDF corrections makes the chi2 increase so dramatically.

So there is something that we need to understand here.

[scarlehoff]

Indeed, plot_fancy includes only PDF errors.
For theory uncertainties you must use e.g.,

nnpdf/validphys2/examples/data_theory_comparison_w_sv.yaml

Line 44 in 3db217b

{@ plot_fancy_sv_dataspecs @}

[scarlehoff]

@ecole41 please repeat the plot with plot_fancy_sv_dataspecs (you might need to add the **config that is added to the chi2 for that)

norm_threshold might be useful as well

[enocera]

I have recomputed the chi2 by regularising the experimental covariance matrix. The plots are unchanged, as it should, and I get the following values of the chi2:
Exp chi2: 1.834
Exp+th chi2: 1.504
Exp+th+pdf chi2: 1.27
In my opinion, these values are in line with what we observe at the level of the shifted plots.
For comparison, I attach the unshifted plot, where you can clearly see that there is a normalisation shift between data and theory.


So, all in all, I think that this falls in the usual category of data sets for which we are overcorrelating systematic uncertainties. Some work may be needed to understand which ones we are actually overcorrelating. Interestingly, on Hepdata there is a another version of the data set in which some of the uncertainties are combined. It may be useful to implement this version as a variant and see if we can get a better chi2 without regularisation.

[enocera]

And just to complete the picture, here are the data-theory comparison plots including theory uncertainties (the blue bars are the sum in quadrature of the PDF uncertainty and of the diagonal elements of the MHOU covariance matrix)


Of course, the dominant MHOU component to the blue bar is strongly correlated across data points, hence the chi2 drops only by a moderate, and very reasonable, amount when including MHOU uncertainties (1.8->1.5)

[enocera]

If one makes the luminosity uncertainty uncorrelated, something that is of course unphysical, one gets
Exp chi2: 1.935
Exp+th chi2: 0.1147
Exp+th+pdf chi2: 0.1117
This is peculiar, and very similar to what was observed in the companion Z measurement, see #2270. If I had to make a bet, I would say that the experimentalists got the normalisation of the central value of their data sets wrong.

[enocera]

So, all in all, I don't see anything wrong in the implementation of this data set. The unshifted plots reveal a ~2.5% normalisation shift, which is very close to the luminosity uncertainty, so there's no surprise that when one decorrelates an uncertainty of the same order of the shift the chi2 drops to about one. My suggestion is to merge this PR. The data set will then be excluded at the level of the data set selection.
cc @scarlehoff