Usage of alternative package indices

[snoyberg]

Stack—like cabal-install—has support for specifying alternative package indices, where the default package index is Hackage, or more specifically, FP Complete's S3-backed mirror. Providing alternative package indices is useful for two completely different use cases in my experience:

1. Providing for packages that are not on Hackage
2. Providing for a local mirror of Hackage, such as one behind a corporate firewall

There's no issue at all with the second use case. However, there are some major issues with the first use case:

• It breaks the meaning of snapshots. foo-1.2.3 means "foo version 1.2.3 as uploaded on Hackage." With an alternative package index, that could mean something quite different.
• Stack already provides a more direct way of providing non-Hackage packages: via tarball URLs or Git repos in either stack.yaml or custom snapshot files.

I would like to move in the direction of more reliably checking package hashes to ensure that build plan you're getting actually matches what a snapshot references. This may have an impact on how we deal with package indices. So I'm reaching out now to find out: is anyone using alternative package indices in ways they would be worried would be broken by changes here?

[centromere]

We use stack in production and would not be impacted by your changes.

[DanBurton]

I would like to move in the direction of more reliably checking package hashes to ensure that build plan you're getting actually matches what a snapshot references.

I was under the impression that it already does this. Specifically, I thought that since stackage snapshots specify hashes, that these were used in order to select the desired revision of a package. Is this behavior disabled by the use of alternative package indices?

[DanBurton]

I see a 3rd potential use case, which would be something similar to HEAD.hackage, facilitating the development and testing of packages against GHC HEAD (or alpha/rc ghc versions) and corresponding pre-release versions of libraries.

I am unaware of anyone currently using stack's alternative package index in this way, but I'd like for it to be something that people start doing.

[snoyberg]

That's actually an example of exactly the kind of use case I'd like to avoid: there's no way to ensure reproducibility of builds. I'd rather see a tool that creates custom snapshots that use a specific set of packages from an overlay.

[DanBurton]

Good point. I agree 100%

So in such a scenario, the custom snapshot based on an overlay would still need some way to reference things that aren't available on the usual hackage package index. As long as there is some sort of "download by hash", it doesn't really matter where it comes from.

[gridaphobe]

Sorry to be so late to the discussion here...

At Bloomberg we have an internal mirror of Hackage that we also publish internal packages to, i.e. we do both (1) and (2). This is very convenient because we have a single place to download packages and browse documentation, we can have internal haddocks cross-link to public packages, etc.

I understand the concern about breaking snapshots due to name clashes, and don't have a problem with checking hashes more aggressively. If, somehow, we got to a point where foo-1.2.3 pointed to a different package in our internal mirror, I would want stack to complain! But we've been careful about package names and haven't had any issues so far, so it would be unfortunate if stack made changes that broke the broader workflow of having an internal Hackage mirror that also contains private packages.

That being said, it's not clear to me whether stack has actually made any changes that would break our workflow. We configure stack with a single package index, our internal mirror, and use the newer 01-index. Package revisions have been an issue for us, the mirror tool doesn't seem to mirror them correctly, but I'd also like to fix that if possible.

[snoyberg]

As described, I think the changes to Stack should work without problem for the use case you describe.

[ozzzzz]

That being said, it's not clear to me whether stack has actually made any changes that would break our workflow. We configure stack with a single package index, our internal mirror, and use the newer 01-index. Package revisions have been an issue for us, the mirror tool doesn't seem to mirror them correctly, but I'd also like to fix that if possible.

Hi, @gridaphobe! Can you write few words more about how you use only one snapshot? Because in BIOCAD we have the same workflow as you, but we still need two package-indices: one public and one private.

[gridaphobe]

Hi @ozzzzz, there's not much to it to be honest. We run an internal Hackage server and use the bundled hackage-mirror tool to mirror the public Hackage to our internal server. Then we configure stack to only talk to our internal mirror, which hosts both public and internal packages.