This repository was archived by the owner on Oct 13, 2023. It is now read-only.
[18.09 backport] Jenkinsfile, CI, and Makefile updates#338
Merged
andrewhsu merged 27 commits intodocker-archive:18.09from Sep 23, 2019
Merged
[18.09 backport] Jenkinsfile, CI, and Makefile updates#338andrewhsu merged 27 commits intodocker-archive:18.09from
andrewhsu merged 27 commits intodocker-archive:18.09from
Conversation
thaJeztah
force-pushed
the
18.09_backport_jenkinsfile
branch
from
51ed975 to
3dcd669
Compare
In case of an exception, it makes great sense to print out some information telling where exactly it happened. _.InvocationInfo.PositionMessage gives script name, line number, character position and (depending on the PS version) highlights the part where error has happened. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> (cherry picked from commit d2788cb) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I noticed this in the build output; ``` 16:05:07 [100%] Built target tini-static 16:05:07 + mkdir -p /build/ 16:05:07 + cp tini-static /build//docker-init ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 5d5adcd) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Update to the latest patch release of 17.06.2. This keeps the same API requirements. This also enables pre-built binaries for armhf instead of compiling from source. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 468eb93) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Rong Gao <gaoronggood@163.com> (cherry picked from commit 5e77399) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Pretty much cross-compile doesn't work because of this: > profiles/seccomp/seccomp.go:13:2: build constraints exclude all Go files in /go/src/github.com/docker/docker/vendor/github.com/seccomp/libseccomp-golang This changes adds a new Dockerfile target for cross compilation with the neccesary arch specific libseccomp packages and CC toolchains. Signed-off-by: Brian Goff <cpuguy83@gmail.com> Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 61a3285) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Not exactly sure why, but this line; build: DOCKER_CROSS ?= false Always overwrote `DOCKER_CROSS` when running `make cross`. Perhaps because it is set in `cross: DOCKER_CROSS := true`, and in a different scope? May also be dependent on the version of `make` in use. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 934b3a3) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
thaJeztah
force-pushed
the
18.09_backport_jenkinsfile
branch
2 times, most recently
from
672970a to
7504eea
Compare
This adds to the existing format of `<GOOS>/<GOARCH>` to allow for `<GOOS>/arm/v<GOARM>` Signed-off-by: Brian Goff <cpuguy83@gmail.com> (cherry picked from commit fbb001d) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This cross-compiling other binaries simpler. It would be nice if the cross Makefile target built all the required bins, but at least this is a first step. Signed-off-by: Brian Goff <cpuguy83@gmail.com> (cherry picked from commit f067a0a) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Adds `DOCKER_BINDDIR_MOUNT_OPTS` to easily tweak the BINDDIR mount options... primarily adding so I can control the caching mode for osxfs because compiling takes > 1min for me with the default and < 30s with both `cached` and `delegated`. Signed-off-by: Brian Goff <cpuguy83@gmail.com> (cherry picked from commit b1e6536) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Before:
Running `ls -la bundles/` before, and after removing:
ls -la bundles/
total 16
drwxr-xr-x 7 root root 224 Jul 12 12:25 .
drwxr-xr-x 1 root root 4096 Jul 12 12:30 ..
drwxr-xr-x 2 root root 64 Jul 12 10:00 dynbinary
drwxr-xr-x 6 root root 192 Jul 12 12:25 dynbinary-daemon
lrwxrwxrwx 1 root root 1 Jul 12 12:25 latest -> .
drwxr-xr-x 92 root root 2944 Jul 12 12:29 test-integration
Removing bundles/
ls -la bundles/
total 16
drwxr-xr-x 7 root root 224 Jul 12 12:25 .
drwxr-xr-x 1 root root 4096 Jul 12 12:30 ..
drwxr-xr-x 2 root root 64 Jul 12 10:00 dynbinary
drwxr-xr-x 6 root root 192 Jul 12 12:25 dynbinary-daemon
lrwxrwxrwx 1 root root 1 Jul 12 12:25 latest -> .
drwxr-xr-x 92 root root 2944 Jul 12 12:29 test-integration
After:
Running `ls -la bundles/` before, and after removing:
ls -la bundles/
total 16
drwxr-xr-x 7 root root 224 Jul 12 12:25 .
drwxr-xr-x 1 root root 4096 Jul 12 12:30 ..
drwxr-xr-x 2 root root 64 Jul 12 10:00 dynbinary
drwxr-xr-x 6 root root 192 Jul 12 12:25 dynbinary-daemon
lrwxrwxrwx 1 root root 1 Jul 12 12:25 latest -> .
drwxr-xr-x 92 root root 2944 Jul 12 12:29 test-integration
Removing bundles/
ls -la bundles/
total 4
drwxr-xr-x 2 root root 64 Jul 12 12:25 .
drwxr-xr-x 1 root root 4096 Jul 12 12:30 ..
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit f75f342)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com> (cherry picked from commit a557538) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Andrew Hsu <andrewhsu@docker.com> (cherry picked from commit aac6e62) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This is basically taking some stuff that make a custom shell function for. This takes a test filter, builds the appropriate TESTFLAGS, and sets the integration API test dirs that match the given filter to avoid building all test dirs. Signed-off-by: Brian Goff <cpuguy83@gmail.com> (cherry picked from commit 13064b1) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The "new test" detector in test-integration-flaky was a bit flaky since it would detect function signatures that are not new tests. In addition, the test calls `return` outside of a function which is not allowed. Signed-off-by: Brian Goff <cpuguy83@gmail.com> (cherry picked from commit e2b2449) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This function changed to the correct working directory before starting the tests (which is the same as on Linux), however the `ProcessStartInfo` process does not inherit this working directory, which caused Windows tests to be running with a different working directory as Linux (causing files used in tests to not be found). From the documentation; https://docs.microsoft.com/en-us/dotnet/api/system.diagnostics.processstartinfo.workingdirectory?view=netframework-4.8 > When `UseShellExecute` is `true`, the fully qualified name of the directory that contains > the process to be started. When the `UseShellExecute` property is `false`, the working > directory for the process to be started. The default is an empty string (`""`). This patch sets the `ProcessStartInfo.WorkingDirectory` to the correct working directory before starting the process. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 6ae46ae) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit b0835dd) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Using a build-arg so that we don't have to specify it for each `apt-get install`, and to preserve that the `DEBIAN_FRONTEND` is preserved in the image itself (which changes the default behavior, and can be surprising if the image is run interactively).` With this patch, some (harmless, but possibly confusing) errors are no longer printed during build, for example: ```patch Unpacking libgcc1:armhf (1:6.3.0-18+deb9u1) ... Selecting previously unselected package libc6:armhf. Preparing to unpack .../04-libc6_2.24-11+deb9u4_armhf.deb ... -debconf: unable to initialize frontend: Dialog -debconf: (TERM is not set, so the dialog frontend is not usable.) -debconf: falling back to frontend: Readline Unpacking libc6:armhf (2.24-11+deb9u4) ... Selecting previously unselected package libgcc1:arm64. Preparing to unpack .../05-libgcc1_1%3a6.3.0-18+deb9u1_arm64.deb ... Unpacking libgcc1:arm64 (1:6.3.0-18+deb9u1) ... Selecting previously unselected package libc6:arm64. Preparing to unpack .../06-libc6_2.24-11+deb9u4_arm64.deb ... -debconf: unable to initialize frontend: Dialog -debconf: (TERM is not set, so the dialog frontend is not usable.) -debconf: falling back to frontend: Readline ``` Looks like some output is now also printed on stdout instead of stderr Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 2ff9ac4) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This will run the DCO check in a lightweight alpine container, before running other stages, and before building the development image/container (which can take a long time). A Jenkins parameter was added to optionally skip the DCO check (skip_dco) Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit d6f7909) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Although the Dockerfile builds without it, adding wheel back should save some time ``` 00:45:28 moby#14 10.70 Building wheels for collected packages: pathspec, pyyaml 00:45:28 moby#14 10.70 Running setup.py bdist_wheel for pathspec: started 00:45:28 moby#14 10.88 Running setup.py bdist_wheel for pathspec: finished with status 'error' 00:45:28 moby#14 10.88 Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mbotnxes/pathspec/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpg9pl4u6kpip-wheel- --python-tag cp35: 00:45:28 moby#14 10.88 usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] 00:45:28 moby#14 10.88 or: -c --help [cmd1 cmd2 ...] 00:45:28 moby#14 10.88 or: -c --help-commands 00:45:28 moby#14 10.88 or: -c cmd --help 00:45:28 moby#14 10.88 00:45:28 moby#14 10.88 error: invalid command 'bdist_wheel' 00:45:28 moby#14 10.88 00:45:28 moby#14 10.88 ---------------------------------------- 00:45:28 moby#14 10.88 Failed building wheel for pathspec 00:45:28 moby#14 10.88 Running setup.py clean for pathspec 00:45:28 moby#14 11.05 Running setup.py bdist_wheel for pyyaml: started 00:45:28 moby#14 11.25 Running setup.py bdist_wheel for pyyaml: finished with status 'error' 00:45:28 moby#14 11.25 Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mbotnxes/pyyaml/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpyci_xi0bpip-wheel- --python-tag cp35: 00:45:28 moby#14 11.25 usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] 00:45:28 moby#14 11.25 or: -c --help [cmd1 cmd2 ...] 00:45:28 moby#14 11.25 or: -c --help-commands 00:45:28 moby#14 11.25 or: -c cmd --help 00:45:28 moby#14 11.25 00:45:28 moby#14 11.25 error: invalid command 'bdist_wheel' 00:45:28 moby#14 11.25 00:45:28 moby#14 11.25 ---------------------------------------- 00:45:28 moby#14 11.25 Failed building wheel for pyyaml 00:45:28 moby#14 11.25 Running setup.py clean for pyyaml 00:45:28 moby#14 11.44 Failed to build pathspec pyyaml 00:45:28 moby#14 11.45 Installing collected packages: pathspec, pyyaml, yamllint 00:45:28 moby#14 11.45 Running setup.py install for pathspec: started 00:45:29 moby#14 11.73 Running setup.py install for pathspec: finished with status 'done' 00:45:29 moby#14 11.73 Running setup.py install for pyyaml: started 00:45:29 moby#14 12.05 Running setup.py install for pyyaml: finished with status 'done' 00:45:29 moby#14 12.12 Successfully installed pathspec-0.5.9 pyyaml-5.1.2 yamllint-1.16.0 ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit ad70bf6) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Stefan Scherer <stefan.scherer@docker.com> (cherry picked from commit ca3e230) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This symlink was added in d427534, to allow finding the path to the latest built binary, because at the time, those paths were prefixed with the version or commit (e.g. `bundles/1.5.0-dev`). Commit bac2447 removed the version-prefix in paths, but kept the old symlink for backward compatiblity. However, many things were moved since then (e.g. paths were renamed to `binary-daemon`, and various other changes). With the symlink pointing to the symlink's parent directory, following the symlink may result into an infinite recursion, which can happen if scripts using wildcards / globbing to find files. With this symlink no longer serving a real purpose, we can probably safely remove this symlink now. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit dde1fd7) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The `libprotobuf-c0-dev` virtual package is no longer available in Debian Buster, but is provided by `libprotobuf-c-dev`, which is available. https://packages.debian.org/stretch/libprotobuf-c0-dev > Virtual Package: libprotobuf-c0-dev > > This is a virtual package. See the Debian policy for a definition of virtual packages. > > Packages providing libprotobuf-c0-dev > libprotobuf-c-dev > Protocol Buffers C static library and headers (protobuf-c) Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit d185ca7) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 917b0dc) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 8b65e05) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Failing to archive the bundles should not mark the build as failed. This can happen if a build is terminated early, or if (to be implemented) an optional build-stage is skipped / failed; ``` 2019-08-24T10:53:09.354Z] + bundleName=janky [2019-08-24T10:53:09.354Z] + echo Creating janky-bundles.tar.gz [2019-08-24T10:53:09.354Z] Creating janky-bundles.tar.gz [2019-08-24T10:53:09.354Z] + xargs tar -czf janky-bundles.tar.gz [2019-08-24T10:53:09.354Z] + find bundles -path */root/*overlay2 -prune -o -type f ( -name *-report.json -o -name *.log -o -name *.prof -o -name *-report.xml ) -print [2019-08-24T10:53:09.354Z] find: bundles: No such file or directory [2019-08-24T10:53:09.354Z] tar: Cowardly refusing to create an empty archive [2019-08-24T10:53:09.354Z] Try 'tar --help' or 'tar --usage' for more information. Error when executing always post condition: hudson.AbortException: script returned exit code 123 at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:569) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:515) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:461) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit a76ff63) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This was introduced in a76ff63: + find bundles -path */root/*overlay2 -prune -o -type f ( -o -name *.log -o -name *.prof ) -print find: invalid expression; you have used a binary operator '-o' with nothing before it. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit ca1e7a3) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Windows RS1 has problems with leaking NdisAdapters during the integration tests; the windows.ps1 script has a cleanup stesp to remove those leaked adapters. For internal testing at Microsoft on internal builds, this cleanup step was skipped, and only ran on the CI machines in our Jenkins. Due to the move to our new Jenkins, the names of Windows machines changed, and because of that, the cleanup step was never executed, resulting in the leaked adapters not being cleaned up: ``` 20:32:23 WARNING: There are 608 NdisAdapters leaked under Psched\Parameters 20:32:23 WARNING: Not cleaning as not a production RS1 server 20:32:24 WARNING: There are 608 NdisAdapters leaked under WFPLWFS\Parameters 20:32:24 WARNING: Not cleaning as not a production RS1 server ``` ``` 22:01:31 WARNING: There are 1209 NdisAdapters leaked under Psched\Parameters 22:01:31 WARNING: Not cleaning as not a production RS1 server 22:01:31 WARNING: There are 1209 NdisAdapters leaked under WFPLWFS\Parameters 22:01:31 WARNING: Not cleaning as not a production RS1 server ``` This patch removes the check for non-production builds, and unconditionally cleans up leaked adapters if they are found. Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 156ad54) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
thaJeztah
force-pushed
the
18.09_backport_jenkinsfile
branch
from
5a7a4db to
6a15a81
Compare
Member
Author
|
only one failure now on Janky, which is a flaky test (tracked through moby#34988); https://ci.docker.com/public/job/engine/job/PR-338/8/execution/node/163/log/?consoleFull Let me see if we had fixes/changes for that test |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backports of:
Conflicts in the Dockerfile at various points, due to rootless kit not being in this branch, but they were not too difficult to resolve; pay extra attention though if you see anything out of order.