Skip to content

[19.03 backport] Backport master latest commits to 19.03 #433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
tiborvass merged 16 commits into from
May 1, 2020
Merged

[19.03 backport] Backport master latest commits to 19.03 #433

tiborvass merged 16 commits into from
May 1, 2020

Conversation

@glours
Copy link
Contributor

@glours glours commented Feb 24, 2020 •
edited by thaJeztah
Loading

Backporting commits from master to be close as possible with the nightly docker-ce packaging in the 19.03 branch

backport of:

@glours glours requested review from StefanScherer, silvin-lubecki, thaJeztah and tiborvass and removed request for tiborvass February 24, 2020 16:34
thaJeztah
thaJeztah reviewed Feb 24, 2020
View reviewed changes
deb/Makefile
$(RM) -r debbuild
[ ! -d sources ] || $(CHOWN) -R $(shell id -u):$(shell id -g) sources
$(RM) -r sources
$(RM) engine-image
Copy link
Member

@thaJeztah thaJeztah Feb 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to discuss if we can already backport these changes

Copy link
Contributor Author

@glours glours Feb 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, that the main objective of this PR, what should we pick up to be close as possible of the nightly build without breaking the existing release process for previous version.

@thaJeztah
Copy link
Member

thaJeztah commented Feb 24, 2020

Can you sign-off the cherry-pick commits? Some checks expect the cherry-pick commits to have a sign-off (which should be the last line in the commit); when cherry-picking, the cherry-picked from line is added, causing those checks to fail;

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 2350856)

You can amend the existing commits, or use both -x and -s when cherry-picking

@glours glours force-pushed the backport-master-to-19.03 branch 2 times, most recently from 4c44962 to 5f3ca6b Compare February 25, 2020 15:46
@thaJeztah
Copy link
Member

thaJeztah commented Feb 27, 2020

Given that there's some bigger changes in this (removal of publishing the "engine" images); I suggest holding off merging this until after 19.03.7 has been done.

(I'm ok with a separate PR with a more limited number of changes if there's fixes in this one that are trivial and/or important to have)

@glours glours force-pushed the backport-master-to-19.03 branch from 5f3ca6b to 088c10d Compare March 5, 2020 14:54
@thaJeztah
Copy link
Member

thaJeztah commented Apr 14, 2020

hmm; why is static failing on this branch, but not on master?

+ pwd
+ make ENGINE_DIR=/home/ubuntu/workspace/docker-ce-packaging_PR-433/engine clean-engine
rm -rf /home/ubuntu/workspace/docker-ce-packaging_PR-433/engine
rm: cannot remove '/home/ubuntu/workspace/docker-ce-packaging_PR-433/engine/bundles/binary-daemon/vpnkit': Permission denied
rm: cannot remove '/home/ubuntu/workspace/docker-ce-packaging_PR-433/engine/bundles/binary-daemon/rootlesskit-docker-proxy': Permission denied
rm: cannot remove '/home/ubuntu/workspace/docker-ce-packaging_PR-433/engine/bundles/binary-daemon/ctr.sha256': Permission denied

@tiborvass
Copy link
Contributor

tiborvass commented Apr 30, 2020

If only we had these things called containers so we could reproduce CI issues locally... #NotAFanOfJenkins

@tiborvass
Copy link
Contributor

tiborvass commented Apr 30, 2020

I think what's happening is we chown the workspace to jenkins user, but then the generated binaries are not chowned as jenkins user (probably root) so it can't remove it.

@thaJeztah
Copy link
Member

thaJeztah commented Apr 30, 2020

Yes, there's some "chown-ing" happening in some parts; mostly curious why it fails in this branch but not on master

@thaJeztah
Copy link
Member

thaJeztah commented Apr 30, 2020

diff with master; https://github.com/glours/docker-ce-packaging/compare/backport-master-to-19.03..docker:master

@tiborvass
Copy link
Contributor

tiborvass commented Apr 30, 2020 •
edited
Loading

@thaJeztah is it normal that CI is using docker/engine ? I lost track of where we are with that.

@thaJeztah
Copy link
Member

thaJeztah commented Apr 30, 2020

is it normal that CI is using docker/engine ? I lost track of where we are with that.

Currently, yes, but I have a PR to change that; #457 (let me extract the first commit to a separate PR for easier review/merge)

kolyshkin and others added 12 commits April 30, 2020 22:47
@kolyshkin
deb/Makefile: simplify and fix
92b35fb 
1. A symbol % is not working as expected in PHONY targets, so
e.g. ubuntu-cosmic was not being rebuild each time, because
ubuntu-cosmic directory exists. The fix is to explicitly mark
every ubuntu-whatever target as phony.

2. Remove code duplication for making packages for different distros.

3. Add missing ubuntu (cosmic, disco) and debian (buster) to the
appropriate targets.

4. As a side effect, bash completion now lists all the distros
to be build.

5. Exclude raspbian from deb target as it can only be built on ARM.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit c048fc8)
Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@kolyshkin
rpm/Makefile: simplify and fix
6049c97 
1. A symbol % is not working as expected in PHONY targets, so
e.g. fedora-30 was not being rebuild each time, because
fedora-30 directory exists. The fix is to explicitly mark
every fedora-NN target as phony.

2. Remove code duplication for making packages for different distros.

3. As a side effect, bash completion now lists all the distros
to be build.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 62a8258)
Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@kolyshkin
Makefile: rely on targets in deb/rpm
fab64a8 
Instead of dynamically getting list of distros to build for,
rely on the corresponding targets in sub-Makefiles. This also
ensures that deb/Makefile and rpm/Makefile will have up-to-date
list of distros included.

This also fixes the following bug:

> $ make deb
> for p in raspbian-stretch ubuntu-bionic ubuntu-disco ubuntu-xenial debbuild/ubuntu-disco ubuntu-cosmic debian-buster debian-stretch; do \
> ...

As you can see, `debbuild/ubuntu-disco` should not be included but it
is. Could be prevented by using `-maxdepth 1` argument to `find`.

While at it, amend the sub-Makefiles to print out the distro
that we build for.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 8ac858c)
Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@thaJeztah
Dockerfiles: remove unused GO_VERSION
6c93862 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 2350856)


Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@thaJeztah
plugins: remove (now) redundant .gitkeep
27e2692 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 689992d)


Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@tao12345666333
Update packages list
429b236 
Signed-off-by: Jintao Zhang <zhangjintao9020@gmail.com>
(cherry picked from commit fedbde9)


Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@zelahi
ADDED gomodules=off to docker-ce-packaging files
5201da0 
Signed-off-by: Zuhayr Elahi <elahi.zuhayr@gmail.com>
(cherry picked from commit 1b52d15)


Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@zelahi
Re-order common env variables
91e273a 
Signed-off-by: Zuhayr Elahi <elahi.zuhayr@gmail.com>
(cherry picked from commit c1a42f9)


Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@seemethere
deb: Only recommend aufs-tools on amd64
e806a09 
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
(cherry picked from commit 3ed5f77)


Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@thaJeztah
Dockerfiles: re-use DISTRO, SUITE build-args where possible
201d868 
Re-use these build-args, so that there's only one place to set them per Dockerfile.

To verify this change:

Check if the env-vars are set on the image:

```bash
make UBUNTU_VERSIONS=ubuntu-bionic DEBIAN_VERSIONS="" RASPBIAN_VERSIONS="" CLI_DIR=$GOPATH/src/github.com/docker/cli ENGINE_DIR=$GOPATH/src/github.com/docker/docker deb

docker image inspect debbuild-ubuntu-bionic/x86_64 --format '{{json .Config.Env }}' | jq .
```

Which should output something like;

```json
[
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/go/bin",
  "GOPROXY=direct",
  "GO111MODULE=off",
  "GOPATH=/go",
  "DOCKER_BUILDTAGS=apparmor seccomp selinux",
  "RUNC_BUILDTAGS=apparmor seccomp selinux",
  "DISTRO=ubuntu",
  "SUITE=bionic"
]
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit a996681)


Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@thaJeztah
Remove image-based build (for docker engine activate)
1c83b4a 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 5b8545f)


Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@thaJeztah
Remove some remnants related to image-based builds
64c2d9a 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ca70795)


Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@thaJeztah
Makefile: enable buildkit when building images
6d5aec9 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 863ec11)


Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@thaJeztah
Remove more remnants from image-based engine
882e614 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f5975a7)


Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@thaJeztah
update list of reviewers in unir config
399b434 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f2d6d9c)

Signed-off-by: Guillaume Lours <guillaume.lours@docker.com>
@tiborvass tiborvass force-pushed the backport-master-to-19.03 branch from 088c10d to 399b434 Compare April 30, 2020 22:47
@tiborvass
Copy link
Contributor

tiborvass commented Apr 30, 2020

I simply rebased

@thaJeztah thaJeztah mentioned this pull request May 1, 2020
Merged
@thaJeztah
Copy link
Member

thaJeztah commented May 1, 2020

opened #462 and cherry-picked it here to see if that resolves the problem

@thaJeztah
Makefile: chown engine dir during cleanup
593dcd2 
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit d2395a3)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah thaJeztah force-pushed the backport-master-to-19.03 branch from a1f9018 to 593dcd2 Compare May 1, 2020 12:41
@thaJeztah
Copy link
Member

thaJeztah commented May 1, 2020

All green now, with #462 cherry-picked (we should merge that one first though)

@tiborvass tiborvass merged commit 4814b6e into docker:19.03 May 1, 2020
@thaJeztah thaJeztah changed the title Backport master latest commits to 19.03 [19.03 backport] Backport master latest commits to 19.03 May 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@thaJeztah thaJeztah thaJeztah left review comments

@silvin-lubecki silvin-lubecki Awaiting requested review from silvin-lubecki

@StefanScherer StefanScherer Awaiting requested review from StefanScherer

+1 more reviewer

@tiborvass tiborvass tiborvass approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@glours @thaJeztah @tiborvass @kolyshkin @tao12345666333 @zelahi @seemethere