Recent Change to Dockerignore Parsing

[BusterNeece]

One of my GitHub Actions involves a build which uses a .dockerignore file that basically looks like this:

**
!web
!util/docker/web
!composer.json

The intent is to automatically ignore anything that isn't expressly specified later in the file.

Up until today, on my GitHub Actions, this worked without any issue. Today, however, I started getting failed builds:

#7 [2/8] COPY ./util/docker/web /bd_build/
#7 DONE 0.4s

#8 [3/8] RUN chmod a+x /bd_build/*.sh     && /bd_build/prepare.sh     && /bd_build/add_user.sh     && /bd_build/setup.sh     && /bd_build/cleanup.sh     && rm -rf /bd_build
#8 0.070 chmod: cannot access '/bd_build/*.sh': No such file or directory

Removing the .dockerignore file caused the problem to be resolved and the build to continue as normal. Something has changed in the last 24 hours with some part of the stack that enables building Docker images via Buildkit on GitHub Actions that has affected how the .dockerignore file is parsed.

My GitHub Actions report that my current version of buildkit is moby/buildkit:buildx-stable-1 => buildkitd github.com/moby/buildkit v0.9.3 8d2625494a6a3d413e3d875a2ff7dd9b1ed1b1a9

[slavovojacek]

Had the same issue, everything good locally but GH actions were failing for us. Figured it was .dockerignore related, we did the below to fix.

  **/*
  !*.json
- !prisma
- !src
+ !prisma/**
+ !src/**

[tonistiigi]

Could either of you post a full example?

[crazy-max]

@SlvrEagle23 If you have a link to your repo that would be great!

Ok I was able to repro with buildx 0.7.0. In the meantime you can use the previous buildx release in your workflow with:

      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v1
        with:
          version: v0.6.3

[tonistiigi]

@aaronlehmann This looks to be related to your * updates in patternmatcher. The vendor of buildkit&moby seems to be regression point #775

[aaronlehmann]

This is related to moby/moby#42676

@tonistiigi: I want to confirm the expected behavior here. Given the .dockerignore file:

**
!util/docker/web

It sounds like we expect a file util/docker/web/foo not to be ignored because of the exclusion.

What if .dockerignore was this instead?

**
!util/docker/web
util/docker/web/foo

In that case, would we expect the file to be ignored because there is a more specific exclude pattern?

The logic currently getting applied is to use the parent dir's match result as a default if none of the patterns directly match a file. In the case here, the ** matches all files, so the parent dir result never comes into play.

If we want the behavior I described above, it sounds like we need to keep track of these parent results at a per-pattern level. For example, with the .dockerignore:

**
!util/docker/web
util/docker/web/foo

...when checking util/docker/web/foo:

• ** matched parent dir util/docker/web => we don't care if ** matches util/docker/web/foo; default to whatever we decided for util/docker/web
• !util/docker/web matched parent dir util/docker/web => we don't care if !util/docker/web matches util/docker/web/foo; continue with the default
• util/docker/web/foo did not match parent dir util/docker/web => therefore we check this pattern, and since it matches, we must exclude util/docker/web/foo

[tonistiigi]

In that case, would we expect the file to be ignored because there is a more specific exclude pattern?

I think logically this would be correct. But priority would be to fix up the regressions from the old behavior. If these cases didn't work before as well then they are not that time-critical.

[thaJeztah]

I think this matches .gitignore behavior; https://git-scm.com/docs/gitignore

It is not possible to re-include a file if a parent directory of that file is excluded. Git doesn’t list excluded directories for performance reasons, so any patterns on contained files have no effect, no matter where they are defined.

mkdir test && cd test
mkdir -p util/docker/web
touch util/docker/web/foo
touch util/docker/web/bar
git init

Without .gitignore

git status

On branch main

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	util/

nothing added to commit but untracked files present (use "git add" to track)

Adding a .gitignore:

cat > ./.gitignore <<'EOF'
**
!util/docker/web
util/docker/web/foo
EOF

git status

On branch main

No commits yet

nothing to commit (create/copy files and use "git add" to track)

Checking with git check-ignore:

git check-ignore -v util/docker/web
.gitignore:1:**	util/docker/web

git check-ignore -v util/docker/web/foo
.gitignore:1:**	util/docker/web/foo

git check-ignore -v util/docker/web/bar
.gitignore:1:**	foo/bar/baz

So to re-include a file, the parent directory that contains it must be included first, which can become complex quickly. The gitignore docs has an example to "exclude everything, except for some files, but things get complex quickly;

cat > ./.gitignore <<'EOF'
/*
!/util
/util/*
!util/docker
/util/docker/web/*
!/util/docker/web/foo
EOF

git check-ignore -v util/docker/web/foo
.gitignore:6:!/util/docker/web/foo	util/docker/web/foo

git check-ignore -v util/docker/web/bar
.gitignore:5:/util/docker/web/*	util/docker/web/bar

git status -uall
On branch main

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	util/docker/web/foo

nothing added to commit but untracked files present (use "git add" to track)

Changing that .gitignore to use ** shows that everything is ignored again;

cat > ./.gitignore <<'EOF'
/**
!/util
/util/*
!util/docker
/util/docker/web/*
!/util/docker/web/foo
EOF

git status -uall
On branch main

No commits yet

nothing to commit (create/copy files and use "git add" to track)

[thaJeztah]

That said, with BuildKit only sending files that are included in the COPY, perhaps a better pattern would be to

COPY util/docker/web

Unlike the classic (non-buildkit) builder, that should not send anything except for that path

[tonistiigi]

That said, with BuildKit only sending files that are included in the COPY, p

These paths still go through dockerignore filtering.

[thaJeztah]

Yes, so I meant that the .dockerignore file then wouldn't be needed in that case

…

On 19 Nov 2021, at 21:24, Tõnis Tiigi ***@***.***> wrote:  That said, with BuildKit only sending files that are included in the COPY, p These paths still go through dockerignore filtering. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[aaronlehmann]

Opened moby/moby#43037 and tonistiigi/fsutil#114 to restore the old behavior.