trying to push to a different remote, that has any objects, not present in GVFS remote fails

[turbonaitis]

Here's the scenario

$ git remote add upstream c:personal/ignas/testing3
$ git show
commit 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 (HEAD -> master, origin/master)
$ git push --no-verify -- upstream  4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85:refs/tags/T1
Initialized empty Git repository in /var/lib/gitolite3/repositories/personal/ignas/testing3.git/
Enumerating objects: 347, done.
Counting objects: 100% (347/347), done.
Delta compression using up to 8 threads
Compressing objects: 100% (234/234), done.
Writing objects: 100% (347/347), 11.02 MiB | 3.46 MiB/s, done.
Total 347 (delta 93), reused 318 (delta 66)
remote: Resolving deltas: 100% (93/93), done.
To c:personal/ignas/testing3
 * [new tag]         4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 -> T1

In another checkout:
$ git remote add upstream c:personal/ignas/testing3
$ git checkout -b new_branch
Switched to a new branch 'new_branch'
$ vim banana.txt
$ git commit -am "Simple change"
[new_branch 0d39d0d] Simple change
 1 file changed, 1 insertion(+)
$ git push upstream new_branch
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 8 threads
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 307 bytes | 307.00 KiB/s, done.
Total 3 (delta 1), reused 0 (delta 0)
To c:personal/ignas/testing3
 * [new branch]      new_branch -> new_branch

Back in previous checkout
$ git push --no-verify -- upstream  4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85:refs/tags/T2
CRASH! (because 

So when we push to a new remote, that has no objects, everything works fine, but when we push to a remote, that has objects, which we don't have, read-object hook is invoked, tries to download the missing object from GVFS, fails and the whole process dies.

Below are traces for the failing push

$ GIT_TRACE_PACKET=1 GIT_TRACE=1 git push --no-verify -- upstream  4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85:refs/tags/T2
18:35:34.977320 run-command.c:664       trace: run_command: /testing3/src/.git/hooks/pre-command push --no-verify -- upstream 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85:refs/tags/T2 --git-pid=53585
18:35:35.101879 git.c:498               trace: built-in: git push --no-verify -- upstream 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85:refs/tags/T2
18:35:35.115549 run-command.c:664       trace: run_command: unset GIT_PREFIX; ssh c 'git-receive-pack '\''personal/ignas/testing3'\'''
18:35:41.005561 pkt-line.c:80           packet:         push< 0d39d0dab690d7389ac06812af74f486db563f38 refs/heads/new_branch\0report-status delete-refs side-band-64k quiet atomic ofs-delta agent=git/2.20.1
18:35:41.005614 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/T1
18:35:41.005627 pkt-line.c:80           packet:         push< 0000
18:35:41.008441 pkt-line.c:80           packet:         push> 0000000000000000000000000000000000000000 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/T2\0 report-status side-band-64k agent=git/2.20.1.vfs.1.1.47.g534dbe1ad1
18:35:41.008800 pkt-line.c:80           packet:         push> 0000
18:35:41.009246 run-command.c:664       trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
18:35:41.016168 git.c:498               trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
18:35:41.022843 run-command.c:664       trace: run_command: /Users/ignas/Uber/testing3/src/.git/hooks/read-object
18:35:41.024362 pkt-line.c:80           packet:          git> git-read-object-client
18:35:41.024387 pkt-line.c:80           packet:          git> version=1
18:35:41.024396 pkt-line.c:80           packet:          git> 0000
18:35:41.027295 pkt-line.c:80           packet:          git< git-read-object-server
18:35:41.027312 pkt-line.c:80           packet:          git< version=1
18:35:41.027330 pkt-line.c:80           packet:          git< 0000
18:35:41.027335 pkt-line.c:80           packet:          git> capability=get
18:35:41.027340 pkt-line.c:80           packet:          git> 0000
18:35:41.027379 pkt-line.c:80           packet:          git< capability=get
18:35:41.027391 pkt-line.c:80           packet:          git< 0000
18:35:41.027400 pkt-line.c:80           packet:          git> command=get
18:35:41.027420 pkt-line.c:80           packet:          git> sha1=0d39d0dab690d7389ac06812af74f486db563f38
18:35:41.027426 pkt-line.c:80           packet:          git> 0000

While without GVFS, same push looks like this:

$ GIT_TRACE_PACKET=1 GIT_TRACE=1 git push --no-verify -- upstream  4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85:refs/tags/T2
18:38:09.719088 git.c:498               trace: built-in: git push --no-verify -- upstream 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85:refs/tags/T2
18:38:09.730635 run-command.c:664       trace: run_command: unset GIT_PREFIX; ssh c 'git-receive-pack '\''personal/ignas/testing3'\'''
18:38:15.614498 pkt-line.c:80           packet:         push< 0d39d0dab690d7389ac06812af74f486db563f38 refs/heads/new_branch\0report-status delete-refs side-band-64k quiet atomic ofs-delta agent=git/2.20.1
18:38:15.614551 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/T1
18:38:15.614564 pkt-line.c:80           packet:         push< 0000
18:38:15.617145 pkt-line.c:80           packet:         push> 0000000000000000000000000000000000000000 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/T2\0 report-status side-band-64k agent=git/2.20.1.vfs.1.1.47.g534dbe1ad1
18:38:15.617558 pkt-line.c:80           packet:         push> 0000
18:38:15.618265 run-command.c:664       trace: run_command: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
18:38:15.624875 git.c:498               trace: built-in: git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
Total 0 (delta 0), reused 0 (delta 0)
18:38:15.981209 pkt-line.c:80           packet:     sideband< \1000eunpack ok0014ok refs/tags/T20000
18:38:15.981288 pkt-line.c:80           packet:         push< unpack ok
18:38:15.981298 pkt-line.c:80           packet:         push< ok refs/tags/T2
18:38:15.981304 pkt-line.c:80           packet:         push< 0000
18:38:16.329504 pkt-line.c:80           packet:     sideband< 0000
To c:personal/ignas/testing3
 * [new tag]         4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 -> T2

It seems, that git version with GVFS is unneccessarily trying to get ALL missing objects, even though getting the new branch is unneccessary in this particular case

[turbonaitis]

Actually, the crashing part was due to our server returning a 502, instead of a 404, when the object is not found, with that fixed, the problem is still there, but different:

git push --no-verify  -- gitolite@code.uber.internal:staging/ignas/testing3 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85:refs/tags/I69                                                                                    !10186
# Using gitolite-code primary gitolite07-sjc1
Failed to connect socket, pipeName: /testing3/.gvfs/GVFS_NetCorePipe, error: 61
error: external process '/testing3/src/.git/hooks/read-object' failed
Failed to connect socket, pipeName: /testing3/.gvfs/GVFS_NetCorePipe, error: 61
error: external process '/testing3/src/.git/hooks/read-object' failed
fatal: bad object 0d39d0dab690d7389ac06812af74f486db563f38
error: remote unpack failed: eof before pack header was fully read
error: failed to push some refs to 'gitolite@code.uber.internal:staging/ignas/testing3'

It seems either read-object hook, or it's interaction with git doesn't work well, when object is not found. The original git code would treat it as a normal case though:
https://github.com/git/git/blob/d62dad7a7dca3f6a65162bf0e52cdf6927958e78/send-pack.c#L41

[Ignas]

Adding a git trace that shows what's happening behind the scenes:

$ GIT_TRACE_PACKET=1 git push --no-verify  -- gitolite@code.uber.internal:staging/ignas/testing3 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85:refs/tags/I70
18:34:18.779441 pkt-line.c:80           packet:         push< 0d39d0dab690d7389ac06812af74f486db563f38 refs/heads/new_branch\0report-status delete-refs side-band-64k quiet atomic ofs-delta agent=git/2.20.1
18:34:18.780546 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/I1
18:34:18.780557 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/I2
18:34:18.780565 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/I4
18:34:18.780573 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/I5
18:34:18.780581 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/I6
18:34:18.780590 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/I8
18:34:18.780598 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/I9
18:34:18.780606 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/T1
18:34:18.780614 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/T2
18:34:18.780622 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/T3
18:34:18.780630 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/T4
18:34:18.780638 pkt-line.c:80           packet:         push< 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/T5
18:34:18.780645 pkt-line.c:80           packet:         push< 0000
18:34:18.780881 pkt-line.c:80           packet:         push> 0000000000000000000000000000000000000000 4b6a28890fc09aa2505e6e6a3bb34f0e65fb0e85 refs/tags/I70\0 report-status side-band-64k agent=git/2.20.1.vfs.1.1.47.g534dbe1ad1
18:34:18.780897 pkt-line.c:80           packet:         push> 0000
18:34:18.792838 pkt-line.c:80           packet:          git> git-read-object-client
18:34:18.793681 pkt-line.c:80           packet:          git> version=1
18:34:18.793686 pkt-line.c:80           packet:          git> 0000
18:34:18.795357 pkt-line.c:80           packet:          git< git-read-object-server
18:34:18.795382 pkt-line.c:80           packet:          git< version=1
18:34:18.795394 pkt-line.c:80           packet:          git< 0000
18:34:18.795399 pkt-line.c:80           packet:          git> capability=get
18:34:18.795403 pkt-line.c:80           packet:          git> 0000
18:34:18.795471 pkt-line.c:80           packet:          git< capability=get
18:34:18.795514 pkt-line.c:80           packet:          git< 0000
18:34:18.795523 pkt-line.c:80           packet:          git> command=get
18:34:18.795538 pkt-line.c:80           packet:          git> sha1=0d39d0dab690d7389ac06812af74f486db563f38
18:34:18.795543 pkt-line.c:80           packet:          git> 0000
18:34:18.798301 pkt-line.c:80           packet:          git< status=error
18:34:18.798345 pkt-line.c:80           packet:          git< 0000
18:34:18.798829 pkt-line.c:80           packet:          git> command=get
18:34:18.798845 pkt-line.c:80           packet:          git> sha1=0d39d0dab690d7389ac06812af74f486db563f38
18:34:18.798853 pkt-line.c:80           packet:          git> 0000
18:34:18.800846 pkt-line.c:80           packet:          git< status=error
18:34:18.800862 pkt-line.c:80           packet:          git< 0000
fatal: bad object 0d39d0dab690d7389ac06812af74f486db563f38
18:34:19.050208 pkt-line.c:80           packet:     sideband< \10031unpack eof before pack header was fully read0024ng refs/tags/I70 unpacker error0000
18:34:19.050247 pkt-line.c:80           packet:     sideband< 0000
18:34:19.050321 pkt-line.c:80           packet:         push< unpack eof before pack header was fully read
error: remote unpack failed: eof before pack header was fully read

Seems like interaction between read-object and git send-pack goes wrong, though I am not sure how exactly and whether issue is on read-object side or on git side.