Add support for Git repositories with submodules

[hsyl20]

Add support for Git submodules (recursive or not) in Pantry. Useful when a git repository containing submodules is used as a Stack "extra-deps".

A test has been added: it tries to stack setup a package that has an "extra-deps" on a package contained in a sub-submodule of a Git repo.

Also export the function withRepo that I use to implement #4567.

• Any changes that could be relevant to users have been recorded in the ChangeLog.md
• The documentation has been updated, if necessary: not necessary

[dbaynard]

Hi @hsyl20,

Thanks for the PR. I misread 'submodules' as 'subdirectories' the first time.

Am I correct to say: currently pantry doesn't support git submodules; this change introduces that support? If not, please correct me.

Also, would you please give a brief description of how your change handles recursive submodules.

There are a few issues at the moment concerning pantry and git (#4579, #4580) so I suggest @qrilka take a look at this PR at the same time as them.

Thanks!

[hsyl20]

Hi @dbaynard,

Am I correct to say: currently pantry doesn't support git submodules; this change introduces that support?

Yes exactly. Pantry uses git archive to create a tarball containing the repository files. But git archive doesn't include files from submodules.

Also, would you please give a brief description of how your change handles recursive submodules.

I use git submodule foreach --recursive to execute some commands in each submodule of the top-level repository recursively:

• git archive --prefix=path/from/toplevel/ bar.tar to create an archive containing the files of the submodule prefixed with the relative path to the top-level repository's root directory
• tar -Af bar.tar path/to/toplevel/archive.tar to append the submodule archive to the top-level one

[qrilka]

Overall it looks good but we need a passing integration test

[qrilka]

@hsyl20 it still fails for me, now with a different error:

       Running: echo 'extra-deps:' >> stack.yaml
       Running: echo "- git: $PWD/../tmpRepo" >> stack.yaml
       Running: echo "  commit: $(cd ../tmpRepo && git rev-parse HEAD)" >> stack.yaml
       Running: echo '  subdir: sub/sub/pkg' >> stack.yaml
       Running: /home/qrilka/ws/h/stack/.stack-work/install/x86_64-linux-tinfo6/custom-snapshot-for-building-stack-with-ghc-8.2.2-M5bKQSIGuB0R/8.2.2/bin/stack setup
       Cloning 6eb1e7d2f7f8c950a8238faeb69ee32057c6f87c from /tmp/stack-integration-git-submodules13863/tmpPackage/../tmpRepo
       No cabal file found for Repo from /tmp/stack-integration-git-submodules13863/tmpPackage/../tmpRepo, commit 6eb1e7d2f7f8c950a8238faeb69ee32057c6f87c in subdir sub/sub/pkg
       Main.hs: Exited with exit code: ExitFailure 1
       CallStack (from HasCallStack):
         error, called at /home/qrilka/ws/h/stack/test/integration/lib/StackTest.hs:52:34 in main:StackTest
         stack, called at /home/qrilka/ws/h/stack/test/integration/tests/git-submodules/Main.hs:41:7 in main:Main
       )

[hsyl20]

@qrilka I have pushed a new version of the test that doesn't use a relative path ("..") in the git repo path. I am not sure it fixes the issue you get though.

Could it be that you don't have the "tar" program in your PATH? Can you show the verbose output of stack?

[qrilka]

@hsyl20 it looks like I just forgot to rebuild local Stack executable before running the test. Probably we need to have some way to do that automatically but that's outside of this PR for sure :)