HTTP 400 on Push and Analyze

[goffinf]

Similar problem as #18 (Detailed Guidance for setup of CLAIRCTL), however since that issue is now discussing other topics I wanted to return to the basic problem of getting the http error: 400 client quit unexpectedly error for push or analyze to see what it is that I am missing ..

Clairctl health-check ...

`
$ clairctl health --log-level debug
2017-06-04 16:08:01.755098 D | config: Using config file: /home/docker/clairctl.yml
2017-06-04 16:08:01.756459 D | clair: requesting health on: http://172.18.0.4:6061/health

Clair: ✔
`

But PUSH ...

`
$ clairctl push hello-world:latest --log-level debug

2017-06-04 15:59:48.660635 D | config: Using config file: /home/docker/clairctl.yml
2017-06-04 15:59:48.662555 D | dockerdist: Downloading manifest for hello-world:latest
2017-06-04 15:59:48.664492 D | dockerdist: Retrieving repository client
2017-06-04 15:59:48.666448 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true
2017-06-04 15:59:52.316571 D | dockerdist: manifest type: *schema2.DeserializedManifest
2017-06-04 15:59:52.316594 D | dockerdist: retrieved schema2 manifest, no verification
2017-06-04 15:59:52.317774 I | config: retrieving interface for local IP
2017-06-04 15:59:52.317787 D | config: no interface provided, looking for docker0
2017-06-04 15:59:52.319476 I | clair: Pushing Layer 1/1 [sha256:78445]
2017-06-04 15:59:52.320515 D | clair: Saving sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3[https://registry-1.docker.io/v2]
2017-06-04 15:59:53.022542 I | clair: adding layer 1/1 [sha256:78445]: receiving http error: 400
client quit unexpectedly
2017-06-04 15:59:53.024542 C | cmd: pushing image "hello-world:latest": receiving http error: 400
`
Setup is as follows ...

Clairctl binary installed (not running inside a container).
Clairctl version: 1.2.6

Docker version: 17.05.0-ce

clairctl.yml:
clair: port: 6060 healthPort: 6061 uri: http://clair report: path: ./reports format: html docker: insecure-registries: - "localhost:5000"

docker-compose.yml:

`
version: '2'
services:
postgres:
container_name: clair_postgres
image: postgres:latest
restart: unless-stopped
environment:
POSTGRES_PASSWORD: password

clair:
container_name: clair_clair
#image: quay.io/coreos/clair:latest
image: quay.io/coreos/clair-git:latest
restart: unless-stopped
depends_on:
- postgres
ports:
- "6060-6061:6060-6061"
links:
- postgres
volumes:
- /tmp:/tmp
- ./clair_config:/config
command: [-config, /config/config.yaml]

registry:
container_name: registry
image: registry:2
restart: unless-stopped
ports:
- "5000:5000"
`

Pull appears to work ok ..

`
docker@clairctl:~$ clairctl pull hello-world:latest --log-level debug
2017-06-04 16:07:19.617159 D | config: Using config file: /home/docker/clairctl.yml
2017-06-04 16:07:19.625274 D | dockerdist: Downloading manifest for hello-world:latest
2017-06-04 16:07:19.625920 D | dockerdist: Retrieving repository client
2017-06-04 16:07:19.626598 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true
2017-06-04 16:07:25.464485 D | dockerdist: manifest type: *schema2.DeserializedManifest
2017-06-04 16:07:25.464544 D | dockerdist: retrieved schema2 manifest, no verification

Image: docker.io/library/hello-world:latest
1 layers found
➜ sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3
`

I have tried using localhost and the clair container IP as the uri: value in clairctl.yml but neither work.

nsloookup clair fails

Really not sure what else to try. I assume clair cannot communicate with clairctl. I note mention of the need for a reverse proxy but thought that came OOTB.

Help appreciated.

Regards

Fraser.

[jgsqware]

Hello, thanks for reporting, have you logged into docker with the docker login command before? To download layer of official images you need to be logged Le dim. 4 juin 2017 18:14, goffinf <notifications@github.com> a écrit :

…

Similar problem as #18 <#18> (Detailed Guidance for setup of CLAIRCTL), however since that issue is now discussing other topics I wanted to return to the basic problem of getting the *http error: 400 client quit unexpectedly* error for push or analyze to see what it is that I am missing .. Clairctl health-check ... ` ***@***.***:$ ***@***.***:$ clairctl health --log-level debug 2017-06-04 16:08:01.755098 D | config: Using config file: /home/docker/clairctl.yml 2017-06-04 16:08:01.756459 D | clair: requesting health on: http://172.18.0.4:6061/health Clair: ✔ ` But PUSH ... ` $ clairctl push hello-world:latest --log-level debug 2017-06-04 15:59:48.660635 D | config: Using config file: /home/docker/clairctl.yml 2017-06-04 15:59:48.662555 D | dockerdist: Downloading manifest for hello-world:latest 2017-06-04 15:59:48.664492 D | dockerdist: Retrieving repository client 2017-06-04 15:59:48.666448 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true 2017-06-04 15:59:52.316571 D | dockerdist: manifest type: *schema2.DeserializedManifest 2017-06-04 15:59:52.316594 D | dockerdist: retrieved schema2 manifest, no verification 2017-06-04 15:59:52.317774 I | config: retrieving interface for local IP 2017-06-04 15:59:52.317787 D | config: no interface provided, looking for docker0 2017-06-04 15:59:52.319476 I | clair: Pushing Layer 1/1 [sha256:78445] 2017-06-04 15:59:52.320515 D | clair: Saving sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3[ https://registry-1.docker.io/v2] 2017-06-04 15:59:53.022542 I | clair: adding layer 1/1 [sha256:78445]: receiving http error: 400 client quit unexpectedly 2017-06-04 15:59:53.024542 C | cmd: pushing image "hello-world:latest": receiving http error: 400 ` Setup is as follows ... Clairctl binary installed (not running inside a container). Clairctl version: *1.2.6* Docker version: *17.05.0-ce* *clairctl.yml:* clair: port: 6060 healthPort: 6061 uri: http://clair report: path: ./reports format: html docker: insecure-registries: - "localhost:5000" *docker-compose.yml:* ` version: '2' services: postgres: container_name: clair_postgres image: postgres:latest restart: unless-stopped environment: POSTGRES_PASSWORD: password clair: container_name: clair_clair #image: quay.io/coreos/clair:latest image: quay.io/coreos/clair-git:latest restart: unless-stopped depends_on: - postgres ports: - "6060-6061:6060-6061" links: - postgres volumes: - /tmp:/tmp - ./clair_config:/config command: [-config, /config/config.yaml] registry: container_name: registry image: registry:2 restart: unless-stopped ports: - "5000:5000" ` Pull appears to work ok .. ` ***@***.***:~$ clairctl pull hello-world:latest --log-level debug 2017-06-04 16:07:19.617159 D | config: Using config file: /home/docker/clairctl.yml 2017-06-04 16:07:19.625274 D | dockerdist: Downloading manifest for hello-world:latest 2017-06-04 16:07:19.625920 D | dockerdist: Retrieving repository client 2017-06-04 16:07:19.626598 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true 2017-06-04 16:07:25.464485 D | dockerdist: manifest type: *schema2.DeserializedManifest 2017-06-04 16:07:25.464544 D | dockerdist: retrieved schema2 manifest, no verification Image: docker.io/library/hello-world:latest 1 layers found ➜ sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3 ` I have tried using localhost and the clair container IP as the uri: value in clairctl.yml but neither work. nsloookup clair fails Really not sure what else to try. I assume clair cannot communicate with clairctl. I note mention of the need for a reverse proxy but thought that came OOTB. Help appreciated. Regards Fraser. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#49>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADJrq7v6wuYQVmOUsua8tCGDKRE0QX2Dks5sAtf4gaJpZM4Nvaqx> .

[goffinf]

Yes,

docker@clairctl:~$ docker login
Username: goffinf
Password:
Login Succeeded

docker@clairctl:~$ clairctl analyze hello-world:latest --log-level debug
2017-06-04 22:28:34.957602 D | config: Using config file: /home/docker/clairctl.yml
2017-06-04 22:28:34.958019 D | dockerdist: Downloading manifest for hello-world:latest
2017-06-04 22:28:34.958238 D | dockerdist: Retrieving repository client
2017-06-04 22:28:36.358459 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true
2017-06-04 22:28:38.323521 D | dockerdist: manifest type: *schema2.DeserializedManifest
2017-06-04 22:28:38.323543 D | dockerdist: retrieved schema2 manifest, no verification
2017-06-04 22:28:38.323943 I | config: retrieving interface for local IP
2017-06-04 22:28:38.323967 D | config: no interface provided, looking for docker0
2017-06-04 22:28:38.324653 I | clair: Pushing Layer 1/1 [sha256:78445]
2017-06-04 22:28:38.325434 D | clair: Saving sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3[https://registry-1.docker.io/v2]
2017-06-04 22:28:38.911448 I | clair: adding layer 1/1 [sha256:78445]: receiving http error: 400
client quit unexpectedly
2017-06-04 22:28:38.911486 C | cmd: pushing image "hello-world:latest": receiving http error: 400

I also thought that with clairctl I could use a private registry. As you see from the docker-compose above I started a docker registry on localhost:5000, but if I push to that and then run analyze it fails in exactly the same way ..

docker@clairctl:$ docker push localhost:5000/hello-world
The push refers to a repository [localhost:5000/hello-world]
98c944e98de8: Pushed
latest: digest: sha256:2075ac87b043415d35bb6351b4a59df19b8ad154e578f7048335feeb02d0f759 size: 524
docker@clairctl:~$ clairctl analyze localhost:5000/hello-world:latest --log-level debug
2017-06-04 22:24:25.528790 D | config: Using config file: /home/docker/clairctl.yml
2017-06-04 22:24:25.529508 D | dockerdist: Downloading manifest for localhost:5000/hello-world:latest
2017-06-04 22:24:25.530102 D | dockerdist: Retrieving repository client
2017-06-04 22:24:25.531270 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true
2017-06-04 22:24:25.553704 D | dockerdist: manifest type: *schema2.DeserializedManifest
2017-06-04 22:24:25.553725 D | dockerdist: retrieved schema2 manifest, no verification
2017-06-04 22:24:25.553971 I | config: retrieving interface for local IP
2017-06-04 22:24:25.553981 D | config: no interface provided, looking for docker0
2017-06-04 22:24:25.555101 I | clair: Pushing Layer 1/1 [sha256:983bf]
2017-06-04 22:24:25.555248 D | clair: Saving sha256:983bfa07a342e316f08afd066894505088de985d46a9af743920aa9cafd17e7a[http://localhost:5000/v2]
2017-06-04 22:24:25.711773 I | clair: adding layer 1/1 [sha256:983bf]: receiving http error: 400
client quit unexpectedly
2017-06-04 22:24:25.711803 C | cmd: pushing image "localhost:5000/hello-world:latest": receiving http error: 400

Note: I changed the clairctl.yml uri value to point to localhost since 'http://clair' did not resolve

`
docker@clairctl:~$ cat clairctl.yml

clair:
port: 6060
healthPort: 6061
# uri: http://clair
uri: http://localhost
report:
path: ./reports
format: html
docker:
insecure-registries:
- "localhost:5000"
`

Any other thoughts ?

Kind Regards

Fraser

[goffinf]

From the clair container logs ...

When trying .. clairctl analyze hello-world:latest

{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:135","Time":"2017-06-04 23:08:38.866291","status code":401}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-06-04 23:08:38.866450","error":"could not find layer","layer":"sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3","path":"https://registry-1.docker.io/v2/library/hello-world/blobs/sha256:78445dd45222097f5f8d5a16e48dc19c4ca162dcdb80010ab6f1ccfc7e2c0fa3"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-06-04 23:08:38.866736","elapsed time":370576353,"method":"POST","remote addr":"172.18.0.1:52322","request uri":"/v1/layers","status":"400"}

... and .... if trying ... clairctl analyze localhost:5000/hello-world:latest

{"Event":"could not download layer","Level":"warning","Location":"driver.go:129","Time":"2017-06-04 23:12:25.108460","error":"Get http://localhost:5000/v2/hello-world/blobs/sha256:983bfa07a342e316f08afd066894505088de985d46a9af743920aa9cafd17e7a: dial tcp 127.0.0.1:5000: getsockopt: connection refused"}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-06-04 23:12:25.109273","error":"could not find layer","layer":"sha256:983bfa07a342e316f08afd066894505088de985d46a9af743920aa9cafd17e7a","path":"http://localhost:5000/v2/hello-world/blobs/sha256:983bfa07a342e316f08afd066894505088de985d46a9af743920aa9cafd17e7a"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-06-04 23:12:25.115614","elapsed time":86736193,"method":"POST","remote addr":"172.18.0.1:52338","request uri":"/v1/layers","status":"400"}

[gauravjos]

Facing same issue +1

[jdel]

I cannot reproduce this issue:

$ clairctll analyze hello-world

Image: docker.io/hello-world:latest
 1 layers found

  ➜ Analysis [sha256:78445] found 0 vulnerabilities.


$ clairctll version

Clairctl version b2094f1

edit: clairctll is an alias i use to run clairctl from a docker container

[goffinf]

Thx @jdel . I'm not surprised you can't replicate because I'm sure the mis-configuration is on my (our) side. Could you possibly share how you are running Clair and clairctl and the config you are using for both, where these are located and whether you did any other setup steps which we might have missed.

If we could follow your steps precisely I'm sure we would get to the bottom of it.

Much appreciated

Fraser.

[gauravjos]

@jdel If its possible please share the clairctl config.yml

[jdel]

@goffinf, @gauravjos: you can find my setup in #47

The easiest would be to clone my fork https://github.com/jdel/clairctl.git

Then cd in the directory and run IMAGE_NAME=your-hub-account/clairctl hooks/build in a terminal. This will create a clairctl container.

Mind that you will have to amend the docker-compose.yml and change the image jgsqware/clairctl image with your-hub-account/clairctl.

Then, you can run docker-compose up -d to run everything. Clair will need some time to pull all vulnerabilities list before you can use it.

Finally, you can run docker-compose exec clairctl clairctl health, docker-compose exec clairctl clairctl version, docker-compose exec clairctl clairctl push XXX... and so on.

For reference, the clair.yml config file I use is here https://github.com/jdel/clairctl/blob/master/docker-compose-data/clair-config/config.yml

[goffinf]

Thx Julien,

after a few minor adjustments I was able to get things working using your fork.

Regards

Fraser.

[jdel]

@goffinf: my fork has been merged here. you can get the latest docker image built from master at: jgsqware/clairctl:master

Other tags (latest, 1.6.2, ...) will be coming shortly.

[gauravjos]

I am still getting error when try to scan image

>docker-compose exec clairctl clairctl push gauravsj/demo-php-app:blue
client quit unexpectedly
2017-06-13 09:38:06.811711 C | cmd: pushing image "gauravsj/demo-php-app:blue": receiving http error: 400

Psst .. I have already logged in using docker login cmd.

Following is clair container log
{"Event":"processing layer","Level":"debug","Location":"worker.go:73","Time":"2017-06-13 09:38:05.758614","engine version":3,"format":"Docker","layer":"sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93","parent layer":"","path":"https://registry-1.docker.io/v2/gauravsj/demo-php-app/blobs/sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93"}
{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:135","Time":"2017-06-13 09:38:06.811360","status code":401}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2017-06-13 09:38:06.811407","error":"could not find layer","layer":"sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93","path":"https://registry-1.docker.io/v2/gauravsj/demo-php-app/blobs/sha256:7095154754192bfc2306f3b2b841ef82771b7ad39526537234adb1e74ae81a93"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-06-13 09:38:06.811556","elapsed time":1053002203,"method":"POST","remote addr":"172.21.0.3:33576","request uri":"/v1/layers","status":"400"}

[pireslaert]

+1 , having the exact same issue using the clairctl locally outside the container

[lcgkm]

@gauravjos @pireslaert
Patch #32 or #52

[pireslaert]

Thanks @lcgkm will hv a go with the latest patch

[jgsqware]

@gauravjos
can you run in debug like this:

docker-compose exec clairctl clairctl --log-level debug push gauravsj/demo-php-app:blue

and paste the logs?

can you pull the image too?

docker-compose exec clairctl clairctl --log-level debug pull gauravsj/demo-php-app:blue