`failed to solve: duplicate output 0` in `FileOpSolver`

[jedevc]

Encountered while running the binaries CI job for linux/riscv64

https://github.com/moby/buildkit/actions/runs/4533691754/jobs/7986851892?pr=3747#step:7:220

...

#25 [linux/amd64 runc-src 1/2] WORKDIR /usr/src
#25 ERROR: duplicate output 0

#17 [linux/amd64 golatest 1/1] FROM docker.io/library/golang:1.20-alpine3.17@sha256:1db127655b32aa559e32ed3754ed2ea735204d967a433e4b605aed1dd44c5084
#17 extracting sha256:935e6c44a52cfcbd8b70f453843a8ac311407c9fe45e21f251024fe1a4e18570 1.9s done
#17 DONE 3.7s

#26 [linux/amd64 gobuild-base 1/2] RUN apk add --no-cache file bash clang lld pkgconfig git make
#26 CANCELED

#27 [linux/amd64 releaser 3/4] WORKDIR /work
#27 DONE 0.1s
------
 > [linux/amd64 runc-src 1/2] WORKDIR /usr/src:
------
Dockerfile:54
--------------------
  52 |     FROM git AS runc-src
  53 |     ARG RUNC_VERSION
  54 | >>> WORKDIR /usr/src
  55 |     RUN git clone https://github.com/opencontainers/runc.git runc \
  56 |       && cd runc && git checkout -q "$RUNC_VERSION"
--------------------
ERROR: failed to solve: duplicate output 0
make: *** [Makefile:27: release] Error 1
Error: Process completed with exit code 2.

The stack trace from buildkit:

  time="2023-03-27T15:24:55Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = duplicate output 0"
  duplicate output 0
  1 v0.11.5 buildkitd --debug
  github.com/moby/buildkit/solver/llbsolver/ops.(*FileOpSolver).Solve
  	/src/solver/llbsolver/ops/file.go:331
  github.com/moby/buildkit/solver/llbsolver/ops.(*fileOp).Exec
  	/src/solver/llbsolver/ops/file.go:171
  github.com/moby/buildkit/solver.(*sharedOp).Exec.func2
  	/src/solver/jobs.go:889
  github.com/moby/buildkit/util/flightcontrol.(*call).run
  	/src/util/flightcontrol/flightcontrol.go:121
  sync.(*Once).doSlow
  	/usr/local/go/src/sync/once.go:74
  sync.(*Once).Do
  	/usr/local/go/src/sync/once.go:65
  runtime.goexit
  	/usr/local/go/src/runtime/asm_amd64.s:1594

[tonistiigi]

@sipsma @cpuguy83 Any way this could be related to #3732 ?

Looks like incorrectly marshaled LLB(or bug in handler).

[sipsma]

@tonistiigi I can't really see how. The change on line 213 just means that if you have already handled all the Inputs for a vertex then you don't need to recurse to its inputs a second time. If you have handled all the inputs (and recursive descendents) then I don't see how it could make any difference to skip checking for mutations again as I don't see how there could be any further mutations possible.

And the change on line 278 just handles a case that otherwise would have been an infinite loop, so I don't see how it could cause a new bug.

That's obviously all theoretical though, only way to know for sure is with a repro.

[tonistiigi]

Looking at that error, my best guess would be that https://github.com/moby/buildkit/blob/master/solver/llbsolver/ops/file.go#L171 gets called multiple times for some reason. Don't know exactly why or if any recent change would make it more likely. If you look at that implementation, then it looks like the second call to FileOpSolver.Solve() will add to previous outs. The other cases don't make much sense as it appears on WORKDIR that creates a FileOp that only has one action (mkdir).

Calling ops.Exec() multiple times is not something that is strictly forbidden. The only practical case where I know this can happen is if the previous Exec() was canceled, but then another part of the build still requested the result before the Op was cleared. I don't know what would have triggered cancellation in this case.

[jedevc]

Just saw this in a recent PR run based off of master: https://github.com/moby/buildkit/actions/runs/4595412950/jobs/8115654416

#27 [linux/amd64 gobuild-base 1/2] RUN apk add --no-cache file bash clang lld pkgconfig git make
#27 CANCELED
------
 > [linux/amd64 runc-src 1/2] WORKDIR /usr/src:
------
Dockerfile:54
--------------------
  52 |     FROM git AS runc-src
  53 |     ARG RUNC_VERSION
  54 | >>> WORKDIR /usr/src
  55 |     RUN git clone https://github.com/opencontainers/runc.git runc \
  56 |       && cd runc && git checkout -q "$RUNC_VERSION"
--------------------
ERROR: failed to solve: duplicate output 0
make: *** [Makefile:27: release] Error 1

Doesn't seem relevant to the changes in the PR (although it was also full of errors).

[tonistiigi]

CI does not use master buildkit version https://github.com/moby/buildkit/blob/master/.github/workflows/buildkit.yml#L26

[jedevc]

🤦