Support for passing through raw buildkit policies

[cpuguy83]

While we consider coming up with a nice DX for policies, allow passing through the raw buildkit format.

[cpuguy83]

Notes from @tonistiigi from slack: https://dockercommunity.slack.com/archives/C7S7A40MP/p1679946536583609?thread_ts=1679946076.766389&cid=C7S7A40MP

Recording them here in case the Slack history disappears:

• probably fine as an env. maybe we should add UNSTABLE or EXPERIMENTAL to the env name
• would be nice if bake --print showed that policy was applied to target
• there is a wip work atm going on with introducing controller interface. we need to be careful about reading env in these cases as if the build goes through controller it has different env. basically we need to make sure that all params configured by env are declared in controller types and we don’t expect controller side to call GetEnv()

last point is similar to #1675

[tonistiigi]

@AkihiroSuda Afaics, the points from the last comment have not been updated yet.

[jedevc]

@cpuguy83 are you still interested in looking at this? We're coming up to a buildx release soon, I imagine you'd probably want this in?

If you're busy, I can try and carry this with @tonistiigi's suggested points 🎉

[cpuguy83]

I think I've mostly got it. Sent you a message in slack but I'll post here as well:

Trying to decide how to deal with bake.
Specifically for --print.
If we want to show the policy in there then I need to expose it to the Target type, which opens it up to being set directly from the bake file.
Is this desirable?
Alternatively I could wrap the Target type for printing and include the source policy there.
(note I really mean the path to the source policy, not the policy itself)

[cpuguy83]

Updated this with the requested changes.

Note that while policies will apply to bakes it does not currently print that information with docker buildx bake --print.

[tonistiigi]

If we decide to remove the experimental or expose this more visibly, then it should show up in bake --print as well. Atm. I don't think it matters much.

[jedevc]

I think we still need a couple changes - we need to avoid calling ReadSourcePolicy completely inside the build package. I wrote a fixup that I've tested and works correctly with build (and both local and remote controllers) and bake. 972f40a

I'm also not convinced on the env name - I'd personally prefer to use a combination of our already existing BUILDX_EXPERIMENTAL=1 and BUILDKIT_SOURCE_POLICY? Though, then we have the issue that it's not possible to just enable source policy, without also enabling the controller API, so I'm happy to keep it separate for now.

[cpuguy83]

Updated with the changes from your fixup commit.

[jedevc]

cc @dvdksn we might want some docs changes to note this somewhere?

[tianon]

If I wanted to update this to also allow passing the policy itself via environment variable,

1. would such a change be considered?

2. should that be implemented with this same variable (some kind of fallback either after reading the file or after trying to parse the JSON string?) or via a different variable?

😇

[tonistiigi]

would such a change be considered?

I think this could be a bit messy to pass a big json/proto value with env. Any other option that would avoid creating temp file if that is the issue?

@cpuguy83

[tianon]

I guess something like EXPERIMENTAL_BUILDKIT_SOURCE_POLICY=<(... generate policy here ...) docker buildx build ... would probably work, given we read the entire file all at once and don't make any assumptions about it being an actual file (like trying to seek or something).