allow setting compression level on export

[tonistiigi]

-o compression-level=n to configure the compressor.

Second commit tries to fix the multi-blob inline cache that I think was broken already (even before zstd PR). I think it is still broken with this PR as well and to fully fix it I need to change LoadRemote signature to LoadRemotes([]compression) []Remotes. @ktock

[tonistiigi]

This isn't very nice but didn't have any other ideas for the cache export to see the compressor. For other exports compressor should be set directly but for inline, it needs to use the same as what is used for the image.

[tonistiigi]

Should we change force to compression-force in follow-up and deprecate old field?

[ktock]

@tonistiigi During thinking about this, I've noticed LoadRemtoe and Load have a very similar logic, except Load calls Worker.FromRemote but LoadRemote doesn't.
The current logic of FromRemote does lazy-pulling so Load doesn't perform pull until the blob is really needed. Now that what is the difference between Load and LoadRemote? Why does solver.*exporter.ExportTo() treat LoadRemote as a special case?

cc @sipsma

[tonistiigi]

@ktock LoadRemote is allowed to return nil

buildkit/worker/cacheresult.go

Line 82 in 78e8856

return nil, nil // ignore error. loadRemote is best effort

(it calls createBlobsIfNeeded=false). So it can be used to get fast access to remote but ignore it if fast access can't be achieved. In that case, some records will be skipped over and some will perform the slow Load+Diff to get the remote, depending on the cache mode.

[ktock]

Thank you for the clarification. Opened #2405 to try to move this forward.

[tonistiigi]

replaced by #2591