git status returns different results when core.fscache enabled

[latkin]

• I was not able to find an open
  or closed issue
  matching what I'm seeing

Setup

Win10, x64
git version 2.6.3.windows.1
Default install as far as I recall
Problem repros in all shells

Details

• Clone https://github.com/latkin/filetest
  • foo:bar file has invalid Windows filename. It is copied as size-0 file foo.
  • Add foo to gitignore
• With core.fscache false, working directory is shown as clean by git status
• With core.fscache true, file foo:bar is flagged with deleted status.

Expected: Working directory status is reported the same with/without core.fscache, in all cases.

Motivation: I work on windows but most of my team is on OSX. They put files named like foo:bar in the repo. I can happily ignore them with a private gitignore when core.fscache is turned off, but they pollute my working directory when I try to use core.fscache true (which I'd like to do for perf reasons).

[kgybels]

@latkin gitignore is only for untracked files. Use git update-index --assume-unchanged foo:bar instead, see git-update-index for more information.

The behavior with core.fscache on seems to be correct, there is no file foo:bar, so it is reported as deleted. That being said, having git deal with paths in your tree, that it can't represent in the working copy seems futile.

[latkin]

git update-index --assume-unchanged solves my problem perfectly, thank you for the pointer.

I will be on my merry way, but it does seem like something to investigate that fscache true/false changes results here. FWIW I agree the fscache=true result seems more accurate in this case.

[latkin]

Poked around a bit and see where the root cause is (but not how to fix it).

For whatever reason, _wfopen creates files on Windows even if there are bogus chars in the path (at least :, haven't tested others), and you can write content into such file.

If you look up such a file by full path with GetFileAttributesEx (which is what mingw_lstat does), Windows tells you the file is there, has nonzero length, etc. Thus in fscache = false case, Windows says the file is there so git does not report it as deleted.

If you instead enumerate files in a directory via FindFirstFile/FindNextFile (which is what fscache_lstsat ultimately does), the file is not reported. Thus with fscache = true, the file is reported as deleted.

There is also a non-hidden file that's created, matching the filename before the : char. e.g. create file foo:bar and a standard file foo is created with 0 content, and also a phantom file foo:bar that's invisible to FindNextFile

[latkin]

Alright, finally have real clarity. Git for windows doesn't currently handle Alternate File Streams correctly, this issue is simply a compilcation due to that.

File paths in Windows can't truly contain a : character, but : can be used in a path-like way when calling various system APIs, in order to access alternate streams. e.g. fopen("C:\\src\\foo:bar") opens the bar stream of the file C:\src\foo. The file c:\src\foo will be created if it doesn't already exist.

As mentioned above, if you call APIs like fopen, GetFileAttributesEx, etc using such a stream-qualified path, they will not report an invalid file name, they will simply start working with the custom stream. An API like FindFirstFile will only return proper file names, i.e. stream-unqualified. Thus there is inconsistency between behavior of fscache true/false.

Without being too expert on mingw or git internals, it seems to me like it would make sense for the various mingw_XYZ and fscache_XYZ functions to check for : in filenames that could be misinterpreted as a stream qualifier, and report the same error as if it was an invalid file path. I don't see any scenario where direct access to alternate file streams is the right behavior in git.

I'm happy to provide such a patch if this sounds right.

[dscho]

We already disallow tracking such illegal file names in Git for Windows, so I do not really see why we need to care all that much: you are talking about untracked files which are completely uninteresting to Git until the time when they are staged.

Please read this as my vote to close this ticket as WONTFIX.

[PhilipOakley]

I think the original problem is that the 'file with colon' is already in the repo he is trying to use - see the original

Motivation:I work on windows but most of my team is on OSX. They put files named likefoo:barin the repo. I can happily ignore them with a privategitignorewhencore.fscacheis turned off, but they pollute my working directory when I try to usecore.fscache` true (which I'd like to do for perf reasons).'

However, he has an existing workaround, which isn't great but works, so it may be useful if it's possible to give @latkin a pointer as to if his line of reasoning as to a suggested 'fix' (to the particular problem of ignoring those colon-files under core.fscache true) is in the right direction (it's his itch to scratch after that ..)

[PhilipOakley]

@latkin Seeing as you have a possible solution, a draft pull request may be a suitable method of exploring the issue (of ignoring colon-files when core.fscache is true). At least then @dscho can see if it is a small tweak, or a major code churn, and how far it reaches.

[latkin]

@PhilipOakley yes, that's correct - the issue is mysterious/confusing behavior for unsuspecting Windows users when working with repos of unix origin. I'm not concerned with malicious/determined Windows users who are deliberately trying to do bogus stuff.

Consider this toy repo: https://github.com/latkin/filetest It contains 3 files: foo:bar1, foo:bar2, and foo:bar3.

If you clone that repo today on Windows, the clone succeeds and no errors are reported. However there are problems:

• A mysterious untracked, size-0 file foo has been created
• Although none of the actual files has been created, for some reason nothing is reported as deleted
  • Until you change to core.fscache = true and now suddenly all 3 files are reported as deleted

This is very confusing and unintuitive. Expected experience is to simply report all 3 files as having invalid paths.

I've sent a proposal PR which is very tiny and fixes this for the 99% case - it simply blocks checkout of files with : in the name. Malicious windows users can still screw with things by deliberately creating alternate streams, but that's probably not worth worrying about.

[PhilipOakley]

----- Original Message -----

@PhilipOakley yes, that's correct - the issue is mysterious/confusing behavior for unsuspecting Windows users when working with repos of unix origin. I'm not concerned with malicious/determined Windows users who are deliberately trying to do bogus stuff.

Consider this toy repo: https://github.com/latkin/filetest It contains 3 files: foo:bar1, foo:bar2, and foo:bar3.

If you clone that repo today on Windows, the clone succeeds and no errors are reported. However there are problems:

a.. A mysterious untracked, size-0 file foo has been created 
b.. Although none of the actual files has been created, for some reason nothing is reported as deleted 
  a.. Until you change to core.fscache = true and now suddenly all 3 files are reported as deleted 

This is very confusing and unintuitive. Expected experience is to simply report all 3 files as having invalid paths.

I've sent a proposal PR which is very tiny and fixes this for the 99% case - it simply blocks checkout of files with : in the name. Malicious windows users can still screw with things by deliberately creating alternate streams, but that's probably not worth worrying about.

I think the commit message is missing the original "why" explanation that in the wild, in mixed communities, G4W should be able to cope with repositories where files-with-colons are already present.

The commit message should also cover the "how to use it" after the fix - does it still require a local gitignore to be set?, and confirm the patch fixes the core.fscache=true issue (and it still works when false).

Finally, as an aside, have you considered the sparse checkout options to mark those files as --assume-unchanged, so they can be in the index, but not in the file system (most descriptions on the web get that option wrong - it's actually about personal FS 'memory' management).

If you amend the commit message you can force push the commit to github and re-use the existing PR - this keeps the thread together.

If some of the confirmatory points don't need to be in the main part of the commit, then stick them after a three dashes line.

Philip

[latkin]

Thanks @PhilipOakley, I've amended the commit message and PR comment.

The change doesn't fix the underlying inconsistency between standard lstat and fscache lstat, it simply blocks problematic files from being checked out in the first place. Not perfect, but it makes the fix considerably less invasive, while still covering the primary scenario.

A full fix would require changes to a bunch of mingw APIs, which seemed too much for an admittedly minor bug.

[LunNova]

I am experiencing a similar problem when using sparse checkout. Is it the same or should I open a new issue?

core.sparseCheckout is set to true.
core.fscache can be either true or false
core.preloadindex can be either true or false
git version shows git version 2.9.0.windows.1
A sparse-checkout file is used to exclude a folder with many small files in it.

I checkout a branch. git status shows no changes. I cherry-pick a commit from a different branch. The cherry-pick succeeds without needing to correct any conflicts. Now running git status shows a deleted file for every single file which was excluded by the sparse-checkout file. None of these files were changed by the cherry-picked commit.

Running git reset --hard HEAD at this point (since there are no actual changes and it's fine to reset) works as a workaround to fix the broken git status results.

Does this sound like the same problem?

edit: Sorry, after reading through the comments it seems obvious that this doesn't have the same root cause. Not sure why I posted this here. Opened a new issue #811

[amalic]

Same here when checking out cBioPortal on Windows.
--> https://github.com/cBioPortal