[remotecache/s3] fix: "updated_at" in http header can't pass some http gateway, "_" -> "-"

[LomotHo]

the word updated_at in s3 Metadata will be converted to x-amz-meta-updated_at in the http header, but the http headers which contain _ can't pass some http gateway, such as tencent cloud object storage gateway.

[LomotHo]

[+] Building 0.0s (0/0)                                                                                                                                                                                            
[+] Building 58.2s (19/20)                                                                                                                                                                                         
[+] Building 58.3s (20/20) FINISHED                                                                                                                                                                                
 => [internal] load build definition from Dockerfile.operator                                                                                                                                                 0.2s
 => => transferring dockerfile: 1.04kB                                                                                                                                                                        0.2s
 => [internal] load .dockerignore                                                                                                                                                                             0.2s
 => => transferring context: 171B                                                                                                                                                                             0.2s
 => [internal] load metadata for txharbor.xxxxx.com/xxxxx-pub/distroless/static:nonroot                                                                                                                   1.6s
 => [internal] load metadata for docker.io/library/golang:1.19                                                                                                                                                1.2s
 => [auth] xxxxx-pub/distroless/static:pull token for txharbor.xxxxx.com                                                                                                                                  0.0s
 => importing cache manifest from s3:2848418040738151787                                                                                                                                                      0.2s
 => [stage-1 1/3] FROM txharbor.xxxxx.com/xxxxx-pub/distroless/static:nonroot@sha256:c6fb839d535f59c6dcd4009e04212c8e06778e027b8d6982c3af7b62cf1c078c                                                     0.5s
 => => resolve txharbor.xxxxx.com/xxxxx-pub/distroless/static:nonroot@sha256:c6fb839d535f59c6dcd4009e04212c8e06778e027b8d6982c3af7b62cf1c078c                                                             0.0s
 => => sha256:fbad7aa519f7da86dde82ab83e3130d8024e9ffebc315cead1fecb230e208340 799.93kB / 799.93kB                                                                                                            0.2s
 => => extracting sha256:fbad7aa519f7da86dde82ab83e3130d8024e9ffebc315cead1fecb230e208340                                                                                                                     0.2s
 => [builder 1/9] FROM docker.io/library/golang:1.19@sha256:660f138b4477001d65324a51fa158c1b868651b44e43f0953bf062e9f38b72f3                                                                                  0.1s
 => => resolve docker.io/library/golang:1.19@sha256:660f138b4477001d65324a51fa158c1b868651b44e43f0953bf062e9f38b72f3                                                                                          0.0s
 => [internal] load build context                                                                                                                                                                             0.8s
 => => transferring context: 118.99kB                                                                                                                                                                         0.8s
 => [builder 2/9] WORKDIR /workspace                                                                                                                                                                          5.0s
 => [builder 3/9] COPY go.mod go.mod                                                                                                                                                                          0.1s
 => [builder 4/9] COPY go.sum go.sum                                                                                                                                                                          0.1s
 => [builder 5/9] RUN go mod download                                                                                                                                                                         5.0s
 => [builder 6/9] COPY api/ api/                                                                                                                                                                              0.3s
 => [builder 7/9] COPY cmd/gpucheck-operator cmd/gpucheck-operator                                                                                                                                            0.1s
 => [builder 8/9] COPY pkg/ pkg/                                                                                                                                                                              0.1s
 => [builder 9/9] RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -o main cmd/gpucheck-operator/main.go                                                                                                20.0s
 => [stage-1 2/3] COPY --from=builder /workspace/main .                                                                                                                                                       3.8s
 => exporting to image                                                                                                                                                                                        3.8s
 => => exporting layers                                                                                                                                                                                       2.1s
 => => exporting manifest sha256:2d411d76190708c95ab187262c134443ebecc65b90528c1bfb187fb398b0d022                                                                                                             0.0s
 => => exporting config sha256:07618da383a22ae6082e5af09c456158335d7c9386077f0aa3ddfecb98da1ea9                                                                                                               0.0s
 => => pushing layers                                                                                                                                                                                         1.0s
 => => pushing manifest for txharbor.xxxxx.com/koyomi-test/image-builder:latest@sha256:2d411d76190708c95ab187262c134443ebecc65b90528c1bfb187fb398b0d022                                                     0.8s
 => ERROR exporting cache to s3                                                                                                                                                                              16.1s
 => => preparing build cache for export                                                                                                                                                                      15.7s
 => => writing layer sha256:0f17e994bfa4017f5219e147d3c7f90bf0fbe36c2605a5053b53ddee5e5af93c                                                                                                                  0.1s
------
 > exporting cache to s3:
------
error: failed to solve: failed to touch file: operation error S3: CopyObject, https response error StatusCode: 403, RequestID: NjNhZWIxNzZfN2I1MGIwYl84ZTBhXzZkNjJlNzA=, HostID: , api error SignatureDoesNotMatch: The Signature you specified is invalid.

this problem cost me about 4 hours, i upload the layer cache to the s3 compatible object storage privoded by tencent cloud, it's fine at the begining, untill yesterday, the request failed with SignatureDoesNotMatch , buildkit will refresh date of the file when it existing for 24 hours, then it calls the "touch" function, which send request with header x-amz-meta-updated_at.

finally i found the problem was the http header x-amz-meta-updated_at missing when the request passing the tencent cloud object storage gateway.

[thaJeztah]

I'm not very familiar with this code and how this information is used, but saw the PR; wondering; would there be code that expects the previous (updated_at) name? And if so, should this set both values (old and new) for compatibility?

i.e, something like;

updatedAt := time.Now.String()

// ....

Metadata:          map[string]string{"updated_at": updatedAt, "updated-at": updatedAt},

[LomotHo]

the head x-amz-meta-updated_at will be dropd when it pass some gateway, it result a change to the request, which will cause a SignatureDoesNotMatch error.

[crazy-max]

I think we should just remove this metadata. Don't think it has any purpose for BuildKit. cc @bpaquet.

[bpaquet]

No, to do a touch, we need to change a meta data. No issue to change the name :)