tests.integration.service_test test_scale_with_custom_container_name_outputs_warning

[dnephin]

(Aliased as "Test that calling scale on a service that has a custom container name" from the docstring)

Seems to be failing fairly frequently against docker 1.8.1

• https://jenkins.dockerproject.org/job/Compose-PRs/1118/console
• https://jenkins.dockerproject.org/job/Compose-PRs/1122/console
• https://jenkins.dockerproject.org/job/Compose-PRs/1117/console
• https://jenkins.dockerproject.org/job/Compose-PRs/1115/console
• https://jenkins.dockerproject.org/job/Compose-PRs/1112/console

(a sample from today)

I suspect there is an underlying docker bug here. We should confirm and make sure it's known upstream, and/or find a way to make the test suite less flaky.

[aanand]

Yes, I've seen this a lot. Been unable to reproduce it so far.

Could it possibly be happening when two test suites run concurrently?

[dnephin]

I don't believe so. From what I've observed the runs queue, they don't run over each other (but we'd have to look at the config to confirm that).

The only other build is master which only ran twice yesterday during that time period

[dnephin]

Another more:

• https://jenkins.dockerproject.org/job/Compose-PRs/1151/console
• https://jenkins.dockerproject.org/job/Compose-PRs/1149/console

[dnephin]

"Test that when scaling if the API returns an error, that is not of type" seems to fail occasionally as well, but not nearly as often as this other one.

[aanand]

OK. I've reproduced it locally (not reliably, sadly), so that rules out the "concurrent test suites" theory for good.

[dnephin]

For some reason, when I run this locally (against docker 1.8.1), I actually see a different error, and 0 containers. The error is this:

ERROR: for 1  b'Cannot start container eafa544eaa691e812581a88aa70ef514f16ab25ae453a86637c96dcd4a8e5a54: a container has already joined the endpoint' 
ERROR: for 3  b'Cannot start container c2ee56f269dcd6965871c0fb3c24b20d3ff722bf8d182a351a2ce5bc229e1308: a container has already joined the endpoint' 

This seems to be a race condition where one container is created, but never started

$ docker inspect custom-container | jq '.[].State'
{
  "FinishedAt": "0001-01-01T00:00:00Z",
  "StartedAt": "0001-01-01T00:00:00Z",
  "Running": false,
  "Paused": false,
  "Restarting": false,
  "OOMKilled": false,
  "Dead": false,
  "Pid": 0,
  "ExitCode": 128,
  "Error": "a container has already joined the endpoint"
}

Not sure why we don't see this on jenkins. I hit it over 50% of the time if I run this test in a loop in my desktop (docker 1.8.1).

[aanand]

That looks suspiciously like a libnetwork error. @mavenugo / @dave-tucker?

[aanand]

This failure may well have been introduced by #1911 - this could be a regression somewhere between Docker 1.8.0-rc3 and 1.8.1.

[dnephin]

I agree. I started looking into it, and the error definitely comes from libnetwork/endpoint.go, but I wanted to make sure I could reproduce it before opening an issue.