Add support for git submodules

[g00ds0n]

In git.yml there's no support for cloning git submodules. This can be fixed by setting recursive=yes in the git ansible module, but I'm not sure if there's any side effects to just adding that in.

Also, git checkout-index doesn't seem to be submodule aware so using it to copy over the git repository doesn't work with submodules.

[tgrzemski]

From 1.6 this is default value for recursive.

I also think that git checkout-index is not so good approach. It is not taking all files from repository. I used rsync to do that.

[ricardclau]

I agree git checkout-index might not the best approach but the whole idea behind it was to keep the git content only in one place, outside of your app directory, and only move the actual application files for your chosen version / branch to the current release directory

But of course, this does not work with submodules so an option to support them could be creating a new gitsubmodules strategy not relying on git checkout-index. What are your thoughts on that?

[g00ds0n]

I'm assuming you're just trying to get rid of all the .git directories? This isn't nearly as clean and efficient but it seems to accomplish the same task. If you do something like this then it would still cover both cases without the need for a new strategy.

- name: ANSISTRANO | GIT | Export a copy of the repo
  command: cp -rp repo {{ ansistrano_release_path.stdout }}
  args:
    chdir: "{{ ansistrano_deploy_to }}"

- name: ANSISTRANO | GIT | find .git directories
  command: find -name .git
  args:
    chdir: "{{ ansistrano_release_path.stdout }}"
  register: git_dirs

- name: ANSISTRANO | GIT | delete .git directories
  command: rm -rf {{ item }}
  args:
    chdir: "{{ ansistrano_release_path.stdout }}"
  with_items: git_dirs.stdout_lines

[ricardclau]

I know this is a long standing issue but I was thinking... with the changes introduced in 1.10.0 using git archive. Are submodules now working? Do you have any open source repo to test (have not used submodules for many years :()?

[EvanK]

@ricardclau Submodules are checked out to the repo, but git archive appears to exclude any submodule content by design. There are bash and python based workarounds, but I can't speak to their stability

[EvanK]

@ricardclau Unless there's a reason for using git archive that I haven't picked up on, you could bypass it (and the tarring -> untarring process altogether) by listing the git versioned files and piping that list into rsync (with some filtering in between to handle path prefixes):

- name: ANSISTRANO | GIT | Sync repo files to release path
  shell: >-
           {
             git ls-files --with-tree="{{ ansistrano_git_branch }}";
             git submodule foreach --recursive --quiet 'git ls-files --with-tree="$sha1" | sed "s#^#$path/#"';
           }
           | grep "^$prefix"
           | sed "s#^$prefix/##"
           | rsync -a --files-from=- "./$prefix/" {{ ansistrano_release_path.stdout }}/
  args:
    chdir: "{{ ansistrano_deploy_to }}/repo/"
  environment:
    prefix: "{{ ansistrano_git_real_repo_tree }}"

It's a non trivial mess of commands and pipes, so it's broken across several lines in the yaml for readability, but the >- block scalar indicator will collapse it all into a one liner for the shell module to run.

It lists all file entries in the superproject:

git ls-files;
git ls-files --with-tree="{{ ansistrano_git_branch }}";

Then subsequently lists all file entries in each submodule, prefixing them by the submodule's relative path:

git submodule foreach --recursive --quiet 'git ls-files --with-tree="$sha1" | sed "s#^#$path/#"';

We then filter all file entries to match the prefix (effectively a no-op if prefix is empty):

| grep "^$prefix"

And remove the path prefix from filtered file entries (again, a no-op if prefix is empty):

| sed "s#^$prefix/##"

Finally, the resulting file entries are piped into rsync, with a source relative to the current directory plus the prefix (just the current directory if prefix is blank):

| rsync -a --files-from=- "./$prefix/" {{ ansistrano_release_path.stdout }}/

[ricardclau]

Hi

Sorry about the delay here, I am a bit confused with

'git ls-files --with-tree="$sha1

Where is this $sha1 variable coming from?

[EvanK]

As with $path, I believe it is coming from the git submodule foreach. Effectively, it is the current committish of the checked out submodule.

From git help submodule:

       foreach
           Evaluates an arbitrary shell command in each checked out submodule. The command has access to the variables $name, $path, $sha1 and $toplevel: $name
           is the name of the relevant submodule section in .gitmodules, $path is the name of the submodule directory relative to the superproject, $sha1 is
           the commit as recorded in the superproject, and $toplevel is the absolute path to the top-level of the superproject. Any submodules defined in the
           superproject but not checked out are ignored by this command. Unless given --quiet, foreach prints the name of each submodule before evaluating the
           command. If --recursive is given, submodules are traversed recursively (i.e. the given shell command is evaluated in nested submodules as well). A
           non-zero return from the command in any submodule causes the processing to terminate. This can be overridden by adding || : to the end of the
           command.

           As an example, git submodule foreach 'echo $path `git rev-parse HEAD`' will show the path and currently checked out commit for each submodule.