Geometry fit uncertainty

[rbooth200]

Brodie Norfolk asked about generating an uncertainty estimate on the geometry fit. One way to estimate the uncertainty would be to use bootstrap methods - it would be good to have a tutorial showing how this could be done using draw_bootstrap_sample.

On top this, I realised that our geometry fits are equivalent to a maximum likelihood fit where the priors on the inc, PA, and phase centre are all flat. It would be better if the prior was flat in cos(inc). I haven't thought this through fully, but it should be easy enough to make the change.

[jeffjennings]

Just looking back at this, I've found with mock data in the last couple days that the uncertainty on the inc, PA and phase center as returned by the least squares optimizer in FitGeometryGaussian and FitGeometryFourierBessel can grossly underestimate the error (the true values being >> 3 sigma from the fitted values). Maybe not surprising, but just to note.

[jeffjennings]

A prior flat in cos(inc) is just prior = 1 / inc unless I'm mistaken.

[rbooth200]

HI Jeff, thanks for this. Do you know whether the very large errors are associated with the fit getting stuck in a local minimum?

[jeffjennings]

I don't suspect so, only because the tendency to underestimate the errors is consistent across several initializations of a guess for the geometry (though all for the same mock dataset). But I could be wrong, and I haven't looked into this exhaustively.

I've also varied diff_step in least_squares (https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.least_squares.html) from 1e-3 to 1e2, which hasn't altered the results.

There might be some unique quirk to this dataset that makes the geometry fit difficult, though I don't see what that would be. I'm generating mock visibilities using galario and using estimate_weights though, so there could be potentially be contributions to the error upstream of the actual geometry fit. I added a couple plots to slack to show.

[rbooth200]

It might be, but I suspect that the errors are just bad. That's sort of expected though

[jeffjennings]

Hey @rbooth200 do you want to apply the prior in cos(inc) to the geometry fitting routines? I think it should just be prior = 1 / inc.

In #164 I added a note to the docs about how the geometry fit won't be accurate for a disc that's highly non-Gaussian, like one with a cavity. For an uncertainty estimate on ~Gaussian discs, I can check if draw_bootstrap_sample will give a realistic uncertainty estimate. Shouldn't we also be able to get an estimate straight from FitGeometryGaussian and FitGeometryFourierBesselthough, since they both use least_squares?

[rbooth200]

Can you check whether the bootstrap gives a reasonable uncertainty estimate?

However, I wonder if the best thing to do might be to note that these automated procedures are not perfect. We could edit the docs to discuss this better, suggest typical uncertainties and point to the appropriate papers that discuss it instead.