vendor: github.com/containerd/containerd v1.7.7

[thaJeztah]

vendor: github.com/containerd/continuity v0.4.2

full diff: containerd/continuity@v0.4.1...v0.4.2

vendor: github.com/stretchr/testify v1.8.4

full diff: stretchr/testify@v1.8.3...v1.8.4

vendor: golang.org/x/sys v0.10.0

full diff: golang/sys@v0.8.0...v0.10.0

vendor: golang.org/x/crypto v0.11.0, golang.org/x/text v0.11.0

full diff: golang/crypto@v0.2.0...v0.11.0
full diff: golang/text@v0.9.0...v0.11.0

vendor: golang.org/x/net v0.13.0

full diff: golang/net@v0.10.0...v0.13.0

vendor: github.com/containerd/containerd v1.7.7

full diff:

• microsoft/hcsshim@v0.10.0-rc.8...v0.11.1
• containerd/containerd@v1.7.2...v1.7.7

[tonistiigi]

Looks like the windows error is legit

[tonistiigi]

Same as #4301 (comment)

=== FAIL: util/purl TestPURLToRef/pkg:docker/busybox@latest?platform=windows%2Famd64 (0.00s)
    image_test.go:155: 
        	Error Trace:	D:/a/buildkit/buildkit/util/purl/image_test.go:155
        	Error:      	Not equal: 
        	            	expected: v1.Platform{Architecture:"amd64", OS:"windows", OSVersion:"", OSFeatures:[]string(nil), Variant:""}
        	            	actual  : v1.Platform{Architecture:"amd64", OS:"windows", OSVersion:"10.0.20348", OSFeatures:[]string(nil), Variant:""}
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -3,3 +3,3 @@
        	            	  OS: (string) (len=7) "windows",
        	            	- OSVersion: (string) "",
        	            	+ OSVersion: (string) (len=10) "10.0.20348",
        	            	  OSFeatures: ([]string) <nil>,
        	Test:       	TestPURLToRef/pkg:docker/busybox@latest?platform=windows%2Famd64
    --- FAIL: TestPURLToRef/pkg:docker/busybox@latest?platform=windows%2Famd64 (0.00s)

=== FAIL: util/purl TestPURLToRef (0.00s)

[crazy-max]

This one is expected, see #4316

[thaJeztah]

I removed this "reset", per #4301 (comment)

But I'm somewhat wondering if that's "correct"; are package-URLs meant to be a canonical URL for a specific platform, and if so, should (could) they be os-version specific? (and include the os-version?)

Looking at containerd/containerd#8778 again, I'm starting to wonder if that change was the right thing to do (although, admitted, "Parse" was already filling in missing information). The main issue is that it does not provide for a os-version element, but now does "dream up" one, and it somewhat feels like the paints shut the option of adding an element for os-version in the string-representation.

I wonder if Parse and <insert defaults> should be separate things, but maybe that ship already sailed when it was decided to "only architecture OR platform are required, everything else being optional".

[tonistiigi]

Yeah, I don't see at all how containerd/containerd#8778 could be correct. The Parse() converts from input string to struct, but with that PR it doesn't use the input string, but the properties of the machine that does the parsing, which is completely unrelated.

Same for this test. If the input for the test is empty OSVersion then one should not just magically appear from purl conversion.

[thaJeztah]

I was considering doing the reverse, and update the Purl function to explicitly reset the os-version field; but that's assuming the string format would never support an os-version field. It would perhaps be slightly more correct though; WDYT?

[thaJeztah]

@tonistiigi @jedevc I updated the PR, and updated PURLToRef to explicitly exclude the OSVersion (and OSFeatures) from the result.

[jedevc]

cc @ktock (from #4301)

[ktock]

Dockerfile should also be updated (can be another PR)

buildkit/Dockerfile

Line 4 in 4c93208

ARG CONTAINERD_VERSION=v1.7.2

[thaJeztah]

Rebased to get a fresh run of CI, and I noticed that the require github.com/containerd/log v0.1.0 was not grouped with other dependencies in go.mod

[jedevc]

Dockerfile should also be updated (can be another PR)

buildkit/Dockerfile

Line 4 in 4c93208

ARG CONTAINERD_VERSION=v1.7.2

Is there a strong reason to not do this? If there isn't, we should always keep these in sync.

[thaJeztah]

Is there a strong reason to not do this?

No, don't think there's a strong reason (honestly, forgot doing so)

I do think it's good to do so in either a separate commit, or a separate PR though to verify that updating just the binary / just the vendor code / vice-versa all works; updating them individually should work, and if updating one of them breaks CI, there's a serious issue in containerd.

I'll open a PR to update the binaries as well.

[thaJeztah]

• opened Dockerfile: update containerd binary to v1.7.7 #4339

[tonistiigi]

The problem in here looks pretty clearly a bug in the new version of platforms.Parse() where it does not do the job of parsing the string to a struct https://github.com/containerd/containerd/blame/main/platforms/platforms.go#L240-L253 . We shouldn't try to patch this by covering up these side-effects in the caller to make the test pass. There are 10+ calls of platforms.Parse() in buildkit. The only reason this showed up in purl test is likely that the integration tests are not running on windows atm. (might be missing some coverage as well).

I think the containerd patches should be reverted and if a string can be parsed to a struct with OSVersion/OSFeatures then the first step is to define how OSVersion is represented in string form.

cc @dmcgowan

[dmcgowan]

Which PR specifically? Seems this should be easy to unit test but for some reason none of these have unit tests. Microsoft folks last changed this code, we can request better test coverage around it. I don't know enough about what the behavior should even be for these fields.

[tonistiigi]

Changes to parser.Parse in containerd/containerd#8778 . They read the configuration of the current system instead of parsing the input, so the return value will always be different depending on the caller's operating system or windows versions.

[thaJeztah]

Yes, I agree that the latest changes to Parse() are far from ideal. It's digging in deeper into an already taken direction, where Parse() doesn't just parse but also "fills in the missing pieces" by applying defaults that are based on the host;

• amd64 (arch only) gets "parsed" into linux/amd64, darwin/amd64, windows/amd64 (etc) depending on the host
• linux (os only) gets "parsed" into linux/amd64, linux/arm64 (etc), depending on the host
• and now windows gets "parsed" into windows/<host-arch>/<host OS version>

The first two are already somewhat problematic (depending on "what" the result is used for), but the os-version is (imo) even more problematic because there is no string-representation of that field.

I guess in an ideal world, Parse would do just that; parse the string representation into a struct (possibly with some amount of normalizing), but leave it to the consumer to fill in defaults (if any). Some options could be to;

• make Parse accept functional argument(s) to apply defaults (and/or normalizing)
• make Parse accept a struct with defaults

And ideally also have a string representation of the additional fields (os-version, os-features ?), so that information can be preserved when serializing to a string.

[tonistiigi]

The first two are already somewhat problematic (depending on "what" the result is used for), but the os-version is (imo) even more problematic because there is no string-representation of that field.

Yes, the first two could maybe be better with a more explicit API, but these are not blocking for us. These are the "incomplete" values that make it more obvious that the behavior might be undefined and if we wanted to we could just not allow these (we never document/promote them anyway and users should never use them imho). But for the new windows one, these are the full representations and there is no way to provide input where the parser would just parse the input in a stable form.

[gabriel-samfira]

CC @kiashok @profnandaa for visibility on this from the MSFT side.

To summarise, Parse() should just parse and return a struct with fields accurately populated based on the string we've just parsed. We absolutely need OSVersion on Windows to satisfy the matcher and fetch an image that can actually run on the host we're running on, but that value should be explicitly set (if needed) before sending it to the matcher. We should not add defaults that can vary based on the environment we're running on, when parsing.

[thaJeztah]

To summarise, Parse() should just parse and return a struct with fields accurately populated based on the string we've just parsed.

Correct; parsing should just parse what's in the string. The current "fill in host OS or host Architecture if missing" is something we should look at deprecating (but not currently the highest priority as it's not a commonly used or advertised feature).

We absolutely need OSVersion on Windows to satisfy the matcher and fetch an image that can actually run on the host we're running on, but that value should be explicitly set (if needed) before sending it to the matcher.

Yes, matching should be a separate concern from Parsing; in an ideal world, the platform's string representation would also allow for os_version to be included, in which case matching should probably skip heuristics (and just do what's requested) but that's not the case yet.

But I guess a specific MatchComparer could be used to either fill in the missing os_version based on the host/environment, or to set the os_version before matching.

[TBBle]

It definitely needs to be up to the MatchComparer that decides how the host's details are intermingled, because the same Platform object can be a match or a non-match depending on the type of container to be run: Process-Isolated, HyperV-isolated, and Host-Process containers all have different rules for OSVersion values are selectable (They're approximately super-sets in that order, as it happens; empty OSVersion is semantically meaningful: "there's no OS present in this image" which only Host-Process mode can execute) and MatchComparer is the thing we want to vary in containerd to achieve this; I don't recall if the actual mechanism for this variation has been implemented yet though, I recall that it turns out to be a frontend (containerd/CRI, buildkit) concern and there was some issues found in a containerd/cri prototype, but I haven't looked for a while.

So the platforms.Parse (and presumably the reverse?) is intermingling concerns that should be in the Windows-specific MatchComparer implementations. (Which is already a thing: you can only currently get one via the Windows-build DefaultComparer call, as I recall; that could be nicer... And it implements the Process-Isolation logic)

It's a fair way down the track, but at some point we may also want, e.g., --platform windows/amd64//10.0.0.19242 to be able to be used with a HyperV runtime to build Windows-ABI-incompatible images from the same Dockerfile on a single host, rather than needing to maintain a build-farm of down-level hosts. Presumably that'll rely on being able to parse an OSVersion out of the string, either in buildkit or internally to containerd.

Outside MatchComparers, purl's use of the same function has different purpose, highlighting the problem. Here, we need to specify the string form of a specific image rather than an execution platform, and cannot assume the host's details; so for this code to be correct, we need OSVersion in the canonical string format, or any purl formed for for a Windows image index reference will only be able to round-trip back to a HostProcess image, breaking the current use-case of attestations/provenance on Windows. (That said, GitHub Search says the only caller of PurlToRef is the BuildKit test suite, so incorrect parsing is probably a less-pressing issue than ambiguous stringification)

I assume from those current use-cases that a partial Platform value (filled in by host-provided-defaults) isn't desirable here.

Poking around at the purl docs (I haven't looked at them before AFAIK) it seems like platform is not currently specified for the docker type, but I assume we're treating it as "containerd's platforms.Parse implementation-defined string serialisation". I see arch is specified for the oci type, but buildkit doesn't support parsing the oci type, and the other Platform fields are not present there.

[tonistiigi]

It's a fair way down the track, but at some point we may also want, e.g., --platform windows/amd64//10.0.0.19242 to

Yes, I don't see any other way around it. I would just make it windows/amd64/10.0.0.19242, windows/arm64/10.0.0.19242, windows/arm64/v9/10.0.0.19242 as we know it is only used for windows. I'd also suggest that if platform is just windows/amd64 then the OSVersion is not taken from host (even for matcher or build target platform) but there is a defined value that is normalized to empty string. Same as v8 for arm64 and v7 for arm.

but I assume we're treating it as "containerd's platforms.Parse implementation-defined string serialisation

Yes, we don't define any other serialization in buildkit.