extended test cannot talk to integrated docker registry

[miminar]

Extended tests trying to curl integrated registry directly fail with this error:

failed to get image "sha256:a4948c96a4784dc2a5b6e9c0dbaca7e14942c3f1185b32081219e20cdea74512" manifest: Get http://docker-registry.default.svc:5000/v2/: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

The Get request made to the registry doesn't generate any entry in registry's log. Pull/push made by Docker daemon works like a charm.

Examples of failed tests:

Details

    Failure [11.080 seconds]
    [Feature:ImagePrune] Image prune
    /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/images/prune.go:127
      with default --all flag
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/images/prune.go:107
        should prune both internally managed and external images [It]
        /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/images/prune.go:106

        Expected error:
            <*url.Error | 0xc4215c1560>: {
                Op: "Get",
                URL: "http://172.30.228.91:5000/v2/extended-test-prune-images-hxlqx-hj7c0/origin-release/blobs/sha256:ae60289ae57f6a7da4a23756109b6078c209a2e280be031a70bf00e198b6b1a0",
                Err: {
                    what: "malformed HTTP response",
                    str: "\x15\x03\x01\x00\x02\x02",
                },
            }
            Get http://172.30.228.91:5000/v2/extended-test-prune-images-hxlqx-hj7c0/origin-release/blobs/sha256:ae60289ae57f6a7da4a23756109b6078c209a2e280be031a70bf00e198b6b1a0: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
        not to have occurred

        /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/images/prune.go:267

Details

    • Failure [119.714 seconds]
    [imageapis][registry] image signature workflow
    /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/registry/signature.go:113
      can push a signed image to openshift registry and verify it [It]
      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/registry/signature.go:112

      Expected
          <string>: error verifying signature sha256:a4948c96a4784dc2a5b6e9c0dbaca7e14942c3f1185b32081219e20cdea74512@3009b28d3715141e65f04d51ff25a066 for image sha256:a4948c96a4784dc2a5b6e9c0dbaca7e14942c3f1185b32081219e20cdea74512 (verification status will be removed): failed to get image "sha256:a4948c96a4784dc2a5b6e9c0dbaca7e14942c3f1185b32081219e20cdea74512" manifest: Get http://docker-registry.default.svc:5000/v2/: malformed HTTP response "\x15\x03\x01\x00\x02\x02"
      to contain substring
          <string>: identity is now confirmed

      /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/registry/signature.go:106

See jobs
https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_image_registry/3/consoleFull
and https://ci.openshift.redhat.com/jenkins/job/test_pull_request_origin_extended_image_registry/2/consoleFull

[miminar]

@stevekuznetsov Do you have a clue why http requests made from the test binary to the registry fail while docker can talk to the registry just fine?

[stevekuznetsov]

Not obvious to me. This is a single-node install. Maybe @sdodson knows that these things are configured differently?

[miminar]

By the way, there are some expected environment variables unset (which were previously set):

• Failure [6.378 seconds]
[Feature:ImageQuota] Image limit range
/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/imageapis/limitrange_admission.go:225
  should deny an import of a repository exceeding limit on openshift.io/image-tags resource [It]
  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/imageapis/limitrange_admission.go:224

  Expected error:
      <*errors.errorString | 0xc421424e40>: {
          s: "MAX_IMAGES_BULK_IMAGES_IMPORTED_PER_REPOSITORY needs to be set",
      }
      MAX_IMAGES_BULK_IMAGES_IMPORTED_PER_REPOSITORY needs to be set
  not to have occurred

  /go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/imageapis/limitrange_admission.go:190

Can be found in the same job as the errors above. It may be related.

This particular variable should be exported by os::util::environment::setup_images_vars().

[stevekuznetsov]

Hm. We launch the jobs with this line:

OPENSHIFT_SKIP_BUILD='true' \
KUBECONFIG=/etc/origin/master/admin.kubeconfig \
TEST_ONLY=true \
JUNIT_REPORT='true' \
make test-extended SUITE=core FOCUS="\[Feature:Image|\[registry\]"

Maybe because we are skipping the build they are not called? @miminar do you have the cycles to chase that?

[miminar]

@stevekuznetsov I can check this out locally in a VM. I'll let your know.

[miminar]

@stevekuznetsov the problem is with TEST_ONLY=true. The environment variable
MAX_IMAGES_BULK_IMAGES_IMPORTED_PER_REPOSITORY get set here. But it's never called when TEST_ONLY is true.

However it's wrong to move the line before the TEST_ONLY check because we would expect that the server was configured with our default settings - which may be wrong. The right thing is to set this variable in a parent job of test_pull_request_origin_extended_image_registry.yml so that the same setting is used to generate the master config and in the test.

[stevekuznetsov]

SGTM

[miminar]

The registry is secured now for the extended tests!

time="2017-08-15T18:09:52.910859221Z" level=info msg="listening on :5000, tls" go.version=go1.8.3 instance.id=45b65968-1af4-46cd-a5ad-657a43c87348 openshift.logger=registry

@stevekuznetsov please correct me if I'm wrong. It looks like the extended_image_registry job now depends on openshift-ansible to install the registry, which secures it by default.

I'll make sure the tests can cope with that.