Estimator's error bars

[1ucian0]

A derived discussion of #51 about Estimator's representation of error bars:

2. We propose that the error portion of the data should represent error bars on the expectation values, somehow. That is, standard error instead of standard deviation or variance, or possibly, a two-sided confidence interval.

For reference, see:

• Updated Estimator interface #51 (comment)
• Updated Estimator interface #51 (comment)
• Updated Estimator interface #51 (comment)

With the aim to extracting that conversation from the RFC on the details on the how (having agreed on the spirit of the need for some for of error represetation in TaskResult), please continue the discussion in this issue.

This discussion might result in one of the following out comes:

• Amending or extending Updated Estimator interface #51
• An agreement on this issue
• It's own RFC

[1ucian0]

cc @ajavadia @blakejohnson @jakelishman

[jakelishman]

Personally my feeling is that if the "error bars" are going to be in the abstract interface, then returning some arbitrary confidence interval is the strongest trade-off between statistical approximations, not constraining implementations, and allowing users to do some very basic statistics on the output in the abstract. The absolute most basic Estimator use (estimate a single bit probability) isn't normally distributed, so having the abstract interface be estimations of the first and second statistical moments (without any sort of covariance) seems like an awkward initial choice, because the underlying distribution will be known to have significant skew for means away from 0.5.

Detailed statistics can be returned separately as an implementation detail, and given how specialised these would be per implementation, it's not clear to me that anything would be bought by trying to abstract that component (just as the RFC doesn't).

I don't feel strongly about this. My vote would go to returning the 16-84 confidence interval ($1\sigma$) such that the standard error is also trivially accessible for distributions that are normally distributed. I don't feel strongly about whether the confidence interval is represented absolutely or relative to the returned mean. I think matplotlib uses relative error bars, but it's one of the things I always have to look up every time I draw a graph with it.

[blakejohnson]

Reporting a confidence interval is about the closest thing we can get to allowing a user to ask for an "error bar" without making assumptions about the underlying statistics. So, I agree with Jake here that the interface should be explicit about returning an interval. The 1σ interval also makes sense as a default (a 68% confidence interval).