Implicit snapshots (aka: Unify extra-deps and extensible snapshots)

[borsboom]

There's a lot of overlap between extra-deps and extensible snapshots. Both give you a to add some "extra stuff" to a snapshot. The main difference is that extra-deps are installed into <project>/.stack-work and treated as slightly weird local packages, while extensible snapshots will go into ~/.stack/snapshots and are immutable. There's a fair amount of confusion around extra-deps (and packages that have extra-dep: true) , so instead of having two ways to do similar things, they should be unified into a single coherent approach.

Particular issues currently:

• Most people don't want extra-deps to behave like local packages, especially as regards GHC options
• extra-deps cannot be shared between projects (or copies of the same project)
• The current stack.yaml configuration is confusing

Now that snapshots can share packages between them, there's no longer a good reason why extra-deps have to be local to the project. Instead, they could form an implicit extended snapshot.

So here, after discussing with @snoyberg, is my proposal for steps to take:

1. Ensure that extensible snapshots (Extensible snapshots #863) can support everything required to support extra-deps. That would include:

   • Setting custom ghc-options for packages
   • Getting package source from a remote Git repository or local directory [edit: local directory for extended snapshot would not be a good idea, since contents could change]

2. Modify stack so that an implicit custom snapshot is created for extra-deps, without changing the existing stack.yaml configuration. The implicit custom snapshot's name would be generated by taking a hash the package source location, name, version, flags, and ghc-options (at least) for each package

3. Rationalize the stack.yaml configuration. In particular, do away with the extra-deps section. All additions to the custom snapshot will be added using a flag in the packages section (similar to the existing extra-dep flag, but perhaps with a different name like extend-snapshot). The existing configuration would be deprecated but still supported for some time.

   Edit: alternatively, keep a separate section for the extended snapshot (akin to current extra-deps section), but have it accept exactly the same syntax as the packages section (but not accepting local directories). Pro: a bit more clear; Con: more work to switch packages between local and extended snapshot (you have to move stuff around in stack.yaml instead of just toggling a flag).

ping @mgsloan @zudov @bitemyapp

[3noch]

👍

[mgsloan]

For (3), I'm thinking doing the following makes sense:

• Add a new dependencies section, which uses the same syntax as packages.
• Deprecate use of extra-deps.
• Deprecate use of extra-dep: True in packages list, and disallow it in the dependencies list. Hopefully this will clear up some of the confusion in Stack should be more clear about packages vs. dependencies #1217
• Make it so that the packages and dependencies list can specify a hackage package version. Does it make sense to be able to specify a hackage package as a local package? Seems a little weird, but it may well be useful, and it seems consistent given that we allow using source control repos as local packages.

Major release 0.2 (or perhaps 1.0!) would remove the deprecated options. How hard would it be to automatically upgrade the stack.yaml? Comment preservation is tricky!

[bitemyapp]

@mgsloan

Does it make sense to be able to specify a hackage package as a local package? Seems a little weird, but it may well be useful, and it seems consistent given that we allow using source control repos as local packages.

Errr, vendoring? If I read you correctly, this is something I've done many times, although it wasn't "local", it was a git repository with modifications divergent from an available Hackage version. I had to do this for Snap 1.0 and it was only due to Stack that I was able to manage the mess.

Deprecate use of extra-dep: True in packages list, and disallow it in the dependencies list. Hopefully this will clear up some of the confusion in #1217

That's satisfactory for what I was concerned about, although we'll want a big-bold-headline explaining package vs. dependency since the former can be read either way.

Everything else seems fine.

[mgsloan]

If I read you correctly, this is something I've done many times, although it wasn't "local", it was a git repository with modifications divergent from an available Hackage version

packages:
- hackage: lens-4.13

vs

packages:
- location:
    git: https://github.com/ekmett/lens.git
    commit: v4.13

The main reason to add it is consistency. I've discussed this with @borsboom and he's in favor. Probably the way it will work is that packages will treat strings as dir paths, whereas dependencies will treat strings as hackage versions. Paths won't be allowed in dependencies. In all other ways they will work the same.