Lock files attempt 2

[qrilka]

Please include the following checklist in your PR:

• Any changes that could be relevant to users have been recorded in the ChangeLog.md
• The documentation has been updated, if necessary.

Tested manually and with local run of integration tests
Has some code and resolves #4550
Resolves #4288

[qrilka]

One note about the implementation: it persist all dependency locations and that means that stack.yaml.lock is quite large because it contains all snapshot packages so loading it takes noticeable time.

[snoyberg]

I haven't looked at the code yet, but based on your comment @qrilka: do you think it makes more sense to take a different approach here? Two things I can think of:

• Don't include all packages. This seems like it will make it quite difficult to come up with a SourceMapHash
• Use a more efficient format, either as the primary format or as an optimized format for lookup. In the latter, I mean keeping the .lock file as a YAML file, but keeping a .lock.bin file next to it (or elsewhere) in a more efficient binary format, or maybe even storing in the SQLite database

[qrilka]

@snoyberg I think I need to check the current timings to get a bit more clear understanding of the problem. But in any case it looks odd to add an optimization which makes Stack slower.

[qrilka]

@snoyberg on Monday I've tried to optimize sourcemap-based lockfiles and I had some improvements but still if we parse almost the same what we have in a snapshot it takes some time. So in 40bf6f0 I've simplified everything and in that commit lock file contains only non-trivial location completions (e.g. snapshot locations won't get stored there) - it simplifies logic quite a lot and also it gives some speed improvements (up to 20% compared to optimized sourcemap-based locks).
I think we should go this simpler and a bit faster way.
One visible difference with the previous version is that we don't check snapshots but I don't see how that could cause any troubles.
If we choose this simple variant then lock_files.md needs to be rewritten.

[qrilka]

BTW regarding timings I have this gist - https://gist.github.com/qrilka/d5e220f3955c8d8ea70ad40f720f5f12

[qrilka]

@snoyberg 1 of integration test failed, so I need to fix that before it will be ready for a proper review

[qrilka]

the problem with 1438-configure-options looks to be unrelated to this change so this PR is ready to be reviewed

[snoyberg]

Looks really solid, and I'm happy with the results of testing it so far! One request: I didn't notice a test for removing a package from extra-deps and checking that it was removed from the lock file as well. If I missed it: ignore me. If I didn't miss it: can you add such a test?

[snoyberg]

Very happy to see this

[qrilka]

Thanks to @psibi :)

[snoyberg]

Good helper function

[qrilka]

@snoyberg please take another look, especially into the doc

[snoyberg]

to form a new lock file

Can you clarify this a bit? I'm guessing the logic is "form a new lock file in memory and compare against the one on disk, writing if there are any differences." Is that the case?

[qrilka]

Yes, I'll add this

[qrilka]

done