Custom snapshots

[UnkindPartition]

Custom snapshots are essential for me and generally important for user freedom.

By 'custom' I mean the ability to specify both custom constraints and a custom download repository.

I am happy to try to work on this myself, but wanted to find out your thoughts first.

[snoyberg]

Definitely in favor. For the download repository, see #29. This is already somewhat supported by the (undocumented) URL overriding feature, something like:

urls:
    package-index-http-url: "some-url"

For custom snapshots, there are two approaches we can take:

• Continue with the extra-deps approach, and possibly add a field extra-deps-file so that the custom snapshot info can be kept in a separate file from stack.yaml. Then you could combine this with something like resolver: ghc-7.10
• Allow a new resolver type for custom snapshots. The advantage of this is that the custom snapshot database could be shared among multiple projects (as opposed to extra-deps, which are project-specific). This is actually pretty easy to do, we'd just have to decide on the naming. In particular, we'd need some assurances that a custom snapshot doesn't change arbitrarily, otherwise your installed snapshot database could become corrupted. The two approaches I can think of are (1) trust the user, and (2) make the name based off of a hash of the snapshot contents.

[snoyberg]

@feuerbach Did you have any thoughts on which direction you think this should take?

[UnkindPartition]

I'd certainly like to add a new resolver type.

Although I understand the need for immutability, I'm a bit confused about "the naming". Do you mean naming the snapshot itself (which is presumably done by the user), or naming the internal package database, or something else?

I don't like relying on a dumb hash; e.g. adding packages shouldn't be a problem, only changing the existing versions is. Right? Although a dumb hash could serve as a fast check.

Do I understand correctly that a resolver in general doesn't assume one version per package (e.g. the ghc resolver doesn't have this property), but a "snapshot resolver" does?

Should we also try to check that a custom package store doesn't replace foo-1.0 with a different foo-1.0? I don't think this will lead to any breakage; in the worst case the change will simply be ignored (if foo-1.0 is already installed).

[snoyberg]

The naming situation I'm worried about is for the internal directory names. Imagine if you name your custom snapshot "roche1". Then stack installs packages in ~/.stack/snapshots/roche1/pkgdb (or something like that). If you then tweak the version of some package, stack would have to know to unregister that package and anything that depends on it, which it doesn't know about now (since it works under the assumption that snapshots are immutable). That's not a big deal of course, we can add that logic. The difficult thing is: what if a different project you're working on also uses roche1, but has a slightly different definition? Then you end up in the cabal-ish situation where each build will break your other project.

The answer to all of this may be: caveat emptor, custom snapshots are an advanced feature, and users need to think about these things when using them. I'm OK with that being the answer, I just want to make sure we make such a decision consciously instead of accidentally.

[UnkindPartition]

FYI, I'm studying stack's codebase to implement this feature, but it isn't going fast, so in case anyone wants to do this, go ahead and don't be blocked on me. But if not, I'm still determined to get this done eventually.

[snoyberg]

I'm deep inside the multiple package indices right now, which is a prereq for this. I'd be happy to help you get through this at the same time (it would be good to have more people familiar with the codebase). Let me know if there's any assistance I can give you. If you're still stuck on it when I get my other high priority tasks for this milestone complete, let's touch base again.

[UnkindPartition]

If we could get on a call (say, later today?) and walk around the codebase together, that'd be super helpful, actually.

[snoyberg]

Cool let's do it. I'm free most of the morning and early afternoon, just
ping me on hangout/jabber

On Mon, Jun 1, 2015, 9:05 AM Roman Cheplyaka notifications@github.com
wrote:

If we could get on a call (say, later today?) and walk around the codebase
together, that'd be super helpful, actually.

—
Reply to this email directly or view it on GitHub
#111 (comment).

[UnkindPartition]

Here's my design so far (there's also some code, but it's not finished yet).

The snapshot.yaml has the following format:

packages: >
  base-1.0
  foo-bar-2.1.3
flags:
  unification-fd:
    base4: true

(the flags section is optional). The rationale behind this format (as opposed to a more structured mapping from PackageName to Version) is to be able to copy-paste the output of ghc-pkg list.

In stack.yaml, a URL to snapshot.yaml is given. It can be file:, http:, or https:. We store this file locally under ~/.stack upon first download and never re-download it, which ensures immutability of the snapshot.

The name of the package db contains the host, the path, and a short hash of the full URL of the snapshot, so it is both human readable and is uniquely determined by the URL. The same name is used for the local copy of snapshot.yaml.

[snoyberg]

Sounds like a good approach to me.