Pantry does not work with git annex

[dbaynard]

General summary/comments (optional)

Pantry does not support a git repo which uses git-annex.

Steps to reproduce

For example, using master:

1. stack new no-annex --resolver https://gist.github.com/dbaynard/9693107a883a6282f52f22785d0072e2/raw/edd32d61464cf0e19236081e250aebbe8cea1112

The resolver includes a git repository which uses git-annex.

Expected

Stack to create the directory and populate with a stack.yaml file.

Actual

Stack creates the directory, then attempts to resolve the git repos in the snapshot, fails with https://github.com/dbaynard/cascade, and so doesn't create the stack.yaml file. The command succeeds on 1.9.3.

Downloading template "new-template" to create project "no-annex" in no-annex/ ...
Looking for .cabal or package.yaml files to use to init the project.
Using cabal packages:
- no-annex/

Selected resolver: 8.6.3-13.5-baynard-snapshot
Loading a snapshot from a SnapshotDef: 8.6.3-13.5-baynard-snapshot
Loading a snapshot from a SnapshotDef: lts-13.5
Loading a snapshot from a SnapshotDef: ghc-8.6.3
Cloning 9699b1a6f61c28ce154f271aef9bb784f65fd47e from https://github.com/dbaynard/cascade
Unsupported tarball from /tmp/get-repo18829/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader

Verbose log

Version 1.10.0 (6902 commits) x86_64 hpack-0.31.1
2019-02-10 17:30:20.358305: [debug] Opening local template: "new-template.hsfiles"
2019-02-10 17:30:20.358606: [info] Downloading template "new-template" to create project "no-annex" in no-annex/ ...
2019-02-10 17:30:20.358685: [debug] Downloading /commercialhaskell/stack-templates/master/new-template.hsfiles
2019-02-10 17:30:20.795431: [debug] Opening local template: "/home/db/.stack/templates/new-template.hsfiles"
2019-02-10 17:30:20.809348: [info] Looking for .cabal or package.yaml files to use to init the project.
2019-02-10 17:30:20.811577: [info] Using cabal packages:
2019-02-10 17:30:20.812241: [info] - no-annex/

2019-02-10 17:30:20.812622: [debug] Running hpack on /home/db/Projects/stack-triage/no-annex/package.yaml
2019-02-10 17:30:20.816604: [debug] hpack generated a modified version of /home/db/Projects/stack-triage/no-annex/no-annex.cabal
2019-02-10 17:30:20.967687: [info] Selected resolver: 8.6.3-13.5-baynard-snapshot
2019-02-10 17:30:20.970425: [debug] Trying to decode /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache
2019-02-10 17:30:20.970564: [debug] Exception ignored when attempting to load /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache: /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache: openBinaryFile: does not exist (No such file or directory)
2019-02-10 17:30:20.970653: [debug] Failure decoding /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/1545069179e0277c8097bb0fe0f8991eff5edc7a7b5ee97d99b9163581cb9335.cache
2019-02-10 17:30:20.970713: [info] Loading a snapshot from a SnapshotDef: 8.6.3-13.5-baynard-snapshot
2019-02-10 17:30:20.970814: [debug] Trying to decode /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/d7f31aacdd7580839a49a29174d1025edd557547229c8e84bdc74bf8aee94c56.cache
2019-02-10 17:30:21.103588: [debug] Success decoding /home/db/.stack/loaded-snapshot-cached/x86_64-linux/__snapshot_hints__/d7f31aacdd7580839a49a29174d1025edd557547229c8e84bdc74bf8aee94c56.cache
2019-02-10 17:30:21.103745: [debug] Parsing cabal file for Repo from https://github.com/jameysharp/corrode, commit 34053342c2f1ca04f23ad94d67057f14e74d9fb9
2019-02-10 17:30:21.106157: [debug] Parsing cabal file for Repo from https://github.com/mgattozzi/curryrs, commit 6d3ecbc3d1786ccc16e2ff20c6d6da53734f6b03
2019-02-10 17:30:21.108578: [debug] Parsing cabal file for Repo from https://github.com/smallhadroncollider/taskell, commit 3c2b5bb272df616670202bb3da42e1dc0606015a
2019-02-10 17:30:21.114289: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict
2019-02-10 17:30:21.118805: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-db
2019-02-10 17:30:21.120918: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-html
2019-02-10 17:30:21.122066: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-json
2019-02-10 17:30:21.123896: [debug] Parsing cabal file for Repo from https://github.com/jkarni/verdict, commit a36aaf04598d4cffba18abed44d2e58186e597cc in subdir verdict-quickcheck
2019-02-10 17:30:21.125179: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir forestay
2019-02-10 17:30:21.127342: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir forestay-data
2019-02-10 17:30:21.129108: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir forestay-serial
2019-02-10 17:30:21.130675: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/haskell, commit ccbf50014bcb8e233b9a99604d7c2e3610611f58 in subdir readp
2019-02-10 17:30:21.132285: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/hserv, commit 736555ca4869da6d8b3d818df5d32a0831a58488
2019-02-10 17:30:21.133937: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/htidy, commit a4139a34f128a858169218211565cae7e1ba5752
2019-02-10 17:30:21.135793: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/pandoc-include, commit ad1f72c485c4c00908326a90efe94ead9bb73c77
2019-02-10 17:30:21.137710: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/lucid-from-html, commit 901bb8edf5e77f227cde847763ee429c620ef62b in subdir lucid-from-html
2019-02-10 17:30:21.140073: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/cascade, commit 9699b1a6f61c28ce154f271aef9bb784f65fd47e in subdir cascade
2019-02-10 17:30:21.140958: [info] Cloning 9699b1a6f61c28ce154f271aef9bb784f65fd47e from https://github.com/dbaynard/cascade
2019-02-10 17:30:21.141851: [debug] Run process within /tmp/get-repo19260: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-10 17:30:22.705593: [debug] Process finished in 1563ms: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-10 17:30:22.705788: [debug] Run process within /tmp/get-repo19260/cloned: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-10 17:30:22.714857: [debug] Process finished in 9ms: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-10 17:30:22.715085: [debug] Run process within /tmp/get-repo19260/cloned: /usr/bin/git submodule update --init --recursive
2019-02-10 17:30:22.785615: [debug] Process finished in 70ms: /usr/bin/git submodule update --init --recursive
2019-02-10 17:30:22.785991: [debug] Run process within /tmp/get-repo19260/cloned: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo19260/foo.tar HEAD
2019-02-10 17:30:22.798792: [debug] Process finished in 12ms: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo19260/foo.tar HEAD
2019-02-10 17:30:22.802218: [debug] parseArchive of GZIP-ed tar file: ZlibException (-3)
Unsupported tarball from /tmp/get-repo19260/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader

Stack version

$ stack --version
Version 1.10.0 (6902 commits) x86_64 hpack-0.31.1

I've no idea why it doesn't show the git ref.

Method of installation

Built from git repo.

[dbaynard]

Note that if the directory already exists, stack build fails for the same reason.

[dbaynard]

I'd like stack to support git-annex, but I'm happy to consider it out of scope. This error ought to be handled, though. Would you mind taking a look, @qrilka?

[dbaynard]

Actually, in this case I don't need any git-annex files to build the package so it ought to work.

[qrilka]

Sure, I will try to find time to investigate this

[qrilka]

@dbaynard this doesn't appear to be related to tar problems, e.g. I see the following verbose output:

2019-02-13 00:02:05.712578: [debug] Parsing cabal file for Repo from https://github.com/dbaynard/cascade, commit 9699b1a6f61c28ce154f271aef9bb784f65fd47e in subdir cascade
2019-02-13 00:02:05.712919: [info] Cloning 9699b1a6f61c28ce154f271aef9bb784f65fd47e from https://github.com/dbaynard/cascade
2019-02-13 00:02:05.713000: [debug] Run process within /tmp/get-repo10938: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-13 00:02:07.521022: [debug] Process finished in 1808ms: /usr/bin/git clone https://github.com/dbaynard/cascade cloned
2019-02-13 00:02:07.521206: [debug] Run process within /tmp/get-repo10938/cloned: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-13 00:02:07.526820: [debug] Process finished in 6ms: /usr/bin/git reset --hard 9699b1a6f61c28ce154f271aef9bb784f65fd47e
2019-02-13 00:02:07.526981: [debug] Run process within /tmp/get-repo10938/cloned: /usr/bin/git submodule update --init --recursive
2019-02-13 00:02:07.554099: [debug] Process finished in 27ms: /usr/bin/git submodule update --init --recursive
2019-02-13 00:02:07.554269: [debug] Run process within /tmp/get-repo10938/cloned: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo10938/foo.tar HEAD
2019-02-13 00:02:07.557223: [debug] Process finished in 3ms: /usr/bin/git -c core.autocrlf=false archive -o /tmp/get-repo10938/foo.tar HEAD
2019-02-13 00:02:07.558883: [debug] parseArchive of GZIP-ed tar file: ZlibException (-3)
Unsupported tarball from /tmp/get-repo10938/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader

See git archive being used - that's the way Pantry currently deals with Git repos - it just gets an archive and uses archive handling code and as your archive contains annex files and Pantry doesn't do anything about them you get broken links in tar file and Pantry fails.
@snoyberg could you give more details why we go through archives for Git repos?

[snoyberg]

Yes:

1. It can be more efficient than asking git to populate the filesystem
2. On filesystems without proper symbolic link support or case insensitivity, it's the only way to ensure we get reproducible repo contents, otherwise the tree hashes would be different on different OSes

We could theoretically make Pantry simply ignore broken symlinks, but the error we're getting here is due to more rigorous error checking.

[qrilka]

@dbaynard the only solution I could propose in this case is to add fonts export-ignore to .gitattributes of your git-annex repo. If that sounds as a good workaround then it makes sense to document it.

[dbaynard]

Yes, fixing the archive seems like the right way to do this. I'll try your suggestion. I wonder if there are other cases where archiving fails, too?

---

Meanwhile:

parseArchive of GZIP-ed tar file: ZlibException (-3)
Unsupported tarball from /tmp/get-repo10938/foo.tar: Symbolic link dest not found from fonts/Cardo-Bold.otf to see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader, looking for fonts/see f7fad052a98cc66eec3bee49a62ac84139337aa1.paxheader

This snippet shows the code which is throwing this error (the upper case is the end of the definition of toSimple).

stack/subs/pantry/src/Pantry/Archive.hs

Lines 384 to 393 in 7a076ab

	case Map.lookup dest files of
	Nothing -> Left $ "Symbolic link dest not found from " ++ mePath me ++ " to " ++ relDest ++ ", looking for " ++ dest
	Just me' ->
	case meType me' of
	METNormal -> Right $ SimpleEntry dest FTNormal
	METExecutable -> Right $ SimpleEntry dest FTExecutable
	METLink _ -> Left $ "Symbolic link dest cannot be a symbolic link, from " ++ mePath me ++ " to " ++ relDest
	case traverse toSimple files of
	Left e -> throwIO $ UnsupportedTarball loc $ T.pack e

Would line 385 be a good place to add that stack doesn't support git-annex, and provide a link to the workaround?

[qrilka]

@dbaynard that code deals with archives in general so I'm not sure we could talk explicitly about git-annex here, probably we could give some hint that it could be the reason behind such a failure but also there are other options - e.g. archive could come not from git at all and bad symlinks could get put into it my some accident.

[qrilka]

@dbaynard what do you think could be a proper resolution for this issue?