Improve support for registries with self-signed certs (x509: certificate signed by unknown authority)

[balchua]

Expected behavior

Skaffold should allow the use of private registries using self signed certificates.

Actual behavior

Skaffold complains about the certificate being signed by unknown authority.

Information

• Skaffold version: tested with bleeding edge (the version with skipTLS option)
• Operating system: RedHat 7.5.
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta15
kind: Config
profiles:
 - name: sub
    build:
      artifacts:
      - image: my-private-registry.com/balchu/gonuts-sub
        context: sub
        kaniko:
          dockerfile: Dockerfile
          skipTLS: true
          buildContext:
            localDir: {}
          cache:
            repo: my-private-registry.com/balchu/gonuts-sub          
      cluster:
        dockerConfig: 
          secretName: regcred
        namespace: gonuts
      insecureRegistries: #Use this for local registry.  such as microk8s registry.
      - my-private-registry.com
    deploy:
      helm:
        releases:
          - name: gonuts-sub
            chartPath: k8s-manifest/sub
            namespace: gonuts
            wait: true
            values:
              image.repository: my-private-registry.com/balchu/gonuts-sub

Steps to reproduce the behavior

1. Find a private registry with a self signed certificate
2. Make sure your Dockerfile's base images are also using the private registry.

For example:

# Use base golang image from Docker Hub
FROM my-private-registry.com/golang:1.12.10 as build

WORKDIR /src/github.com/balchua/gonuts

# Copy go.mod and go.sum 
ADD ./go.mod /src/github.com/balchua/gonuts/
ADD ./go.sum /src/github.com/balchua/gonuts/
# Install dependencies in go.mod and go.sum
RUN go mod download

# Copy application source code
COPY ./main.go /src/github.com/balchua/gonuts

# Compile the application to /app.
RUN go build -o /app -v .

# Now create separate deployment image
FROM my-private-registry.com/distroless/base
COPY --from=build /app /app
# Cause full tracebacks; also serves to identify this image as a Go image for `skaffold debug`
ENV GOTRACEBACK=all
ENTRYPOINT ["/app"]

Skaffold complains about the private registry's certificates and then it fails. It does not even start the Kaniko pod.

Thanks!

[raul1991]

I am getting the same error as well. Is there any workaround yet ?

[balopat]

Hi, thanks for opening, can you share logs please?

[balchua]

@balopat its a bit tricky to provide the logs. Company policy restricts it. Maybe @raul1991 can help provide logs.

[raul1991]

Here are the logs

out.log

[balopat]

@raul1991 I see that these are minikube docker daemon logs - can you try following https://minikube.sigs.k8s.io/docs/tasks/registry/insecure/ to setup the insecure registry for the daemon? Unfortunately this is not something Skaffold can do currently to reconfigure the daemon inside minikube.

@balchua would you be able to send the exact error message at least? That can help identify which part of the process we are failing. Thanks!

[balchua]

Hi @balopat im taking some screenshot and upload it here.

But i got it working somehow. Strangely enough if i have an environment variable DOCKER_CERT_PATH set to the same location where i keep certs needed to talk to docker daemon i get past the point where kaniko pods starts and builds successfully.
I digged through the code and it actually goes thru the code path where the docker api is used.
It arrives to this line.

skaffold/pkg/skaffold/docker/client.go

Line 57 in 057c22a

env, apiClient, err := newAPIClient(runCtx.KubeContext)

The docker api actually gets all those environment variables it needs such as DOCKER_HOST and DOCKER_CERT_PATH.

Just right after kaniko completes the build it fails again somewhere here.

skaffold/pkg/skaffold/docker/remote.go

Line 94 in b758988

func remoteImage(identifier string, insecureRegistries map[string]bool) (v1.Image, error) {

And then i end up setting this environment variable SSL_CERT_FILE pointing to the registry's ca.crt the same one you normally see in /etc/docker/certs.d/

In short, kaniko build succeeds but processes before and after fails.

[balchua]

- This screenshot is the error i get before adding the DOCKER_CERT_PATH environment variable. The kaniko pod was not started.

- This screenshot is the error i get if I do not set SSL_CERT_FILE environment variable.

[balchua]

Although this issue #3116 is talking about jib, i think it could be related.

[balopat]

@balchua thanks for sharing the screenshots!!!
As a workaround - can you try if --cache-artifacts=false helps with the post-failure - I think that is code path that is trying to get the remote config from the registry and fails...

[balchua]

@balopat will give this a go. Thanks

[balchua]

@balopat, same error x509: certificate signed by unknown authority