WCOW: Build context with nested directories fails with "unclean path"

[snorrk]

Using nested directories in build context fails with error: failed to solve: unclean path test2/hello.txt: invalid argument.

Repro

Set-Content Dockerfile @"
FROM mcr.microsoft.com/windows/nanoserver:ltsc2022
USER ContainerAdministrator
COPY . C:/
"@

md test1\test2
"hello" >> test1\test2\hello.txt

buildctl build `
--frontend=dockerfile.v0 `
--local context=test1 `
--local dockerfile=. `
--output=type=image,name=hello-buildkit

[+] Building 0.3s (5/6)
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 128B                                                                                                                                                                   0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022                                                                                                                         0.1s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => ERROR [internal] load build context                                                                                                                                                                0.1s
 => => transferring context: 59B                                                                                                                                                                       0.0s
 => CACHED [1/2] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7                                                            0.1s
 => => resolve mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7                                                                   0.1s
------
 > [internal] load build context:
------
error: failed to solve: unclean path test2/hello.txt: invalid argument

Version

buildctl --version
buildctl github.com/moby/buildkit v0.13.0-rc2 596ef8f

Windows 11 23H2

[profnandaa]

Thanks for opening the issue! I suspect this must be related to #4696 -- can you try "hello" >> test1/test2/hello.txt instead and lmk?

[profnandaa]

Cross posting here from our side-channel discussions. Now, sorry I had mistaken your initial post, the commands there:

md test1\test2
"hello" >> test1\test2\hello.txt

are not inside the dockerfile.

I can't repro the issue, my other suspicion is the containerd version.

[profnandaa]

I can repro the issue now on WS2022:

PS C:\Users\Administrator\repro-4741> buildctl build `
>> --frontend=dockerfile.v0 `
>> --local context=test1 `
>> --local dockerfile=. `
>> --output=type=image,name=hello-buildkit
[+] Building 0.5s (5/6)
 => [internal] load build definition from Dockerfile                                                                                    0.1s
 => => transferring dockerfile: 128B                                                                                                    0.0s
 => [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022                                                          0.2s
 => [internal] load .dockerignore                                                                                                       0.0s
 => => transferring context: 2B                                                                                                         0.0s
 => ERROR [internal] load build context                                                                                                 0.0s
 => => transferring context: 59B                                                                                                        0.0s
 => CANCELED [1/2] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c7192  0.0s
 => => resolve mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:64b22e42a69ebcdb86e49bf50780b64156431a508f7f06ac3050c71920fe57b7    0.0s
------
 > [internal] load build context:
------
error: failed to solve: unclean path test2/hello.txt: invalid argument

Will take a look.

[doctorpangloss]

Is there a situation where this doesn't occur but nested directories work?

I'm here because dockerd building is also full of bugs

[profnandaa]

I'm still investigating this but I suspect this is around path.Clean() nuances on Windows - https://github.com/tonistiigi/fsutil/blob/7525a1af2bb545e89dc9bced785bff7a3b7f08c2/validator.go#L31

[crazy-max]

I'm still investigating this but I suspect this is around path.Clean() nuances on Windows - https://github.com/tonistiigi/fsutil/blob/7525a1af2bb545e89dc9bced785bff7a3b7f08c2/validator.go#L31

Looks similar to #4723 (comment)

[profnandaa]

@crazy-max -- that's true. So, turns out filepath.Clean() will turn / to \\ for Windows, that's causing the first failure.
When I fix that, I end up with another path issue downstream:

error: failed to solve: invalid file request test2\hello.txt

ref: vendor/github.com/tonistiigi/fsutil/receive.go:240:

I suspect this is because of the path not having /.

Yes, turns out, the path is being stored with / in the map, hence can't be found.

(dlv) p p
"test2\\hello.txt"
(dlv) p r.files
map[string]uint32 [
        "test2/hello.txt": 1,
]
(dlv) bt
0  0x00000000011b1676 in github.com/tonistiigi/fsutil.(*receiver).asyncDataFunc
   at C:/dev/container-core/buildkit/vendor/github.com/tonistiigi/fsutil/receive.go:240
1  0x00000000011bcd4c in github.com/tonistiigi/fsutil.(*receiver).asyncDataFunc-fm
   at <autogenerated>:1
2  0x00000000011a277d in github.com/tonistiigi/fsutil.(*DiskWriter).processChange
   at C:/dev/container-core/buildkit/vendor/github.com/tonistiigi/fsutil/diskwriter.go:255
3  0x00000000011a2485 in github.com/tonistiigi/fsutil.(*DiskWriter).requestAsyncFileData.func1
   at C:/dev/container-core/buildkit/vendor/github.com/tonistiigi/fsutil/diskwriter.go:231
4  0x0000000000b84269 in golang.org/x/sync/errgroup.(*Group).Go.func1
   at C:/dev/container-core/buildkit/vendor/golang.org/x/sync/errgroup/errgroup.go:75
5  0x00000000003525a1 in runtime.goexit
   at C:/Program Files/Go/src/runtime/asm_amd64.s:1695

I'm still on it.

/cc. @gabriel-samfira

[profnandaa]

Now fixed it in a hacky way, cleaning it up and will do the PRs first to fsutil (and later vendor it); and then here.

 buildctl build `
>>  --output type=image,name=docker.nandaa.dev/test,push=false `
>>  --progress plain "-frontend=dockerfile.v0" `
>>  --local context=C:/dev/play/dockerfiles/repro-4741/test1 `
>>  --local dockerfile=C:\dev\play\dockerfiles\repro-4741
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 29B
#1 transferring dockerfile: 124B 19.9s
#1 transferring dockerfile: 128B 19.9s done
#1 DONE 19.9s

#2 [internal] load metadata for mcr.microsoft.com/windows/nanoserver:ltsc2022
#2 DONE 2.6s

#3 [internal] load .dockerignore
#3 transferring context:
#3 transferring context: 2B done
#3 DONE 0.4s

#4 [1/2] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:6e6053f0358f9522d2d14693f9bc152f47fe04c82c53dc8c6d127a5a823c8720
#4 resolve mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:6e6053f0358f9522d2d14693f9bc152f47fe04c82c53dc8c6d127a5a823c8720 0.1s done
#4 DONE 0.2s

#5 [internal] load build context
#5 ...

#4 [1/2] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:6e6053f0358f9522d2d14693f9bc152f47fe04c82c53dc8c6d127a5a823c8720
#4 ...

#5 [internal] load build context
#5 transferring context: 87B 18.0s done
#5 DONE 18.3s

#4 [1/2] FROM mcr.microsoft.com/windows/nanoserver:ltsc2022@sha256:6e6053f0358f9522d2d14693f9bc152f47fe04c82c53dc8c6d127a5a823c8720
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 0B / 116.94MB 0.2s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 6.29MB / 116.94MB 3.9s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 12.58MB / 116.94MB 7.8s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 18.87MB / 116.94MB 11.3s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 25.17MB / 116.94MB 12.3s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 31.46MB / 116.94MB 16.5s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 37.75MB / 116.94MB 20.3s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 44.04MB / 116.94MB 24.0s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 50.33MB / 116.94MB 25.2s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 56.62MB / 116.94MB 29.0s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 62.91MB / 116.94MB 32.9s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 69.21MB / 116.94MB 36.5s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 75.50MB / 116.94MB 37.5s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 81.79MB / 116.94MB 41.3s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 88.08MB / 116.94MB 44.9s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 94.37MB / 116.94MB 48.8s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 100.66MB / 116.94MB 49.8s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 106.95MB / 116.94MB 53.6s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 113.25MB / 116.94MB 57.0s
#4 sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 116.94MB / 116.94MB 57.5s done
#4 extracting sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee
#4 extracting sha256:e47e27b77a3ea500a5f75962d5b3feb20e0b4c7949a9b47c8c091b708a697bee 24.1s done
#4 DONE 81.8s

#6 [2/2] COPY . C:/
#6 DONE 0.8s

#7 exporting to image
#7 exporting layers
#7 exporting layers 0.9s done
#7 exporting manifest sha256:d1b63c3def742a9f1adaceea08cc5ccf3fb25106d82462f3f1ee08b2b7640b93 0.1s done
#7 exporting config sha256:3c1f55a4d88478ad39dd9239a2f4cb7e513587b11d59acb2731057420e01f768 0.1s done
#7 naming to docker.nandaa.dev/test
#7 naming to docker.nandaa.dev/test 0.0s done
#7 DONE 1.1s

Will also add some more path tests cases in our integration tests suite.

[profnandaa]

Fix in tonistiigi/fsutil#196 ; will be vendored here once merged.