[Dockerfile] Enable retries for apt-get when building Pulsar docker image #14513

lhotari · 2022-03-01T12:56:25Z

Motivation

prevents issues where apt repository doesn't respond
example failure: https://github.com/apache/pulsar/runs/5374589351?check_suite_focus=true#step:10:1458

[INFO] E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/b/base-files/base-files_11ubuntu5.5_amd64.deb  Could not connect to azure.archive.ubuntu.com:80 (52.250.76.244), connection timed out
[INFO] E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
[INFO] 
Error:  The command '/bin/sh -c sed -i "s|http://archive\.ubuntu\.com/ubuntu/|${UBUNTU_MIRROR:-mirror://mirrors.ubuntu.com/mirrors.txt}|g" /etc/apt/sources.list      && apt-get update      && apt-get -y dist-upgrade      && apt-get -y install --no-install-recommends openjdk-11-jdk-headless netcat dnsutils less procps iputils-ping                  python3 python3-yaml python3-kazoo python3-pip                  curl ca-certificates      && apt-get -y --purge autoremove      && apt-get autoclean      && apt-get clean      && rm -rf /var/lib/apt/lists/*' returned a non-zero code: 100

Modifications

set retries to 3
also reduce default timeout to 30 seconds

…mage - also reduce default timeout to 30 seconds - prevents issues where apt repository doesn't respond

eolivelli

I have seen this problem locally today.

+1

michaeljmarshall

LGTM

docker/pulsar/Dockerfile

…mage (apache#14513) - also reduce default timeout to 30 seconds - prevents issues where apt repository doesn't respond

tisonkun · 2022-09-20T09:10:26Z

docker/pulsar/Dockerfile


 # Install some utilities
 RUN sed -i "s|http://archive\.ubuntu\.com/ubuntu/|${UBUNTU_MIRROR:-mirror://mirrors.ubuntu.com/mirrors.txt}|g" /etc/apt/sources.list \
+     && echo 'Acquire::http::Timeout "30";\nAcquire::ftp::Timeout "30";\nAcquire::Retries "3";' > /etc/apt/apt.conf.d/99timeout_and_retries \


Interesting. Now when I build test images locally, it works fine; when I build the pulsar-all image, it fails with returned a non-zero code: 100.

~~But yes, it works well either on CI environment. See tisonkun#2.~~

Hmmm...It seems we build only test images on CI, not the pulsar-all image.

Related to other changes. Sorry for bothering.

…mage (apache#14513) - also reduce default timeout to 30 seconds - prevents issues where apt repository doesn't respond (cherry picked from commit d3f6fe3)

* [Dockerfile] Enable retries for apt-get when building Pulsar docker image (apache#14513) - also reduce default timeout to 30 seconds - prevents issues where apt repository doesn't respond (cherry picked from commit d3f6fe3) * PIP-155: Removed Python 2 support (apache#15376) * Remove Pulsar Client Build for Python 2.7 * Remove outdated homebrew files (source of truth is upstream homebrew) * Remove Python 2.7 build references; print error in some cases * Update python client tests to run with python client for python 3.5m * PIP-155: Removed Python 2 support * Fixed invocation in pulsar-build image * Fixed clang-format-10 indent differences * Fixed script invocation with wrong python * We don't need to rebuild the manylinux image each time * Fixed image name * Reverted back to use newer protobuf * Fixed image name * Fixed missing python3 in centos:7 image * Use python3 for gtest-parallel * Show bash commands in docker-tests.sh * Fixed gh action issue with git directory permissions * Fixed python to 3 * Fixed custom_logger_test.py * Fixed path in run_python_instance_tests.sh * Function runtime should use python3 * Fixed function runtime test python expectation * Fixed presto worker launcher * Fixed notes on how to format C++ code Co-authored-by: Michael Marshall <mmarshall@apache.org> (cherry picked from commit 2b2e0c5) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [improve][docker] Switch to Temurin JDK (apache#17129) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 4378856) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [refactor][ci] Build the docker image with docker-maven-plugin (apache#17148) (cherry picked from commit a68b58d) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [feat][build] Support ARM64-based docker images (apache#17733) (cherry picked from commit 9a2aeb2) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * PIP-209: Removed C++/Python clients from main repo (apache#17881) * PIP-209: Removed C++/Python clients from main repo * Removed python directory from Docekrfile * Fixed python client version argument scoping * Fixed handling of pulsar.functions.serde (cherry picked from commit f3c547b) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [improve][build] Avoid building image multiple times (apache#17208) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 79a97a9) * [improve] Allow to build and push multi-arch Docker images (apache#19432) Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com> Co-authored-by: Yong Zhang <zhangyong1025.zy@gmail.com> Co-authored-by: Zixuan Liu <nodeces@gmail.com> Co-authored-by: tison <wander4096@gmail.com> (cherry picked from commit 4190e40) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix publish image script (apache#20305) Signed-off-by: Zixuan Liu <nodeces@gmail.com> (cherry picked from commit 94c7bf3) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix the pulsar-all image may use the wrong upstream image (apache#20435) Signed-off-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@apache.org> (cherry picked from commit d7f3558) Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [feat][build] Adapt to Python client to be compatible with ARM arch Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Configure git-commit-id-plugin to skip git describe (apache#20550) (cherry picked from commit 05f7e62) * [improve][misc] Include native epoll library for Netty for arm64 From apache#22319 Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][misc] Rename all shaded Netty native libraries From apache#22415 Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [cleanup][build] Cleanup -Ddocker.nocache=true Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix ubuntu mirror Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Fix license Signed-off-by: Zixuan Liu <nodeces@gmail.com> * [fix][build] Downgrade docker-maven to 0.43.3 Signed-off-by: Zixuan Liu <nodeces@gmail.com> --------- Signed-off-by: Zixuan Liu <nodeces@gmail.com> Signed-off-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com> Co-authored-by: Matteo Merli <mmerli@apache.org> Co-authored-by: Michael Marshall <mmarshall@apache.org> Co-authored-by: tison <wander4096@gmail.com> Co-authored-by: Yong Zhang <zhangyong1025.zy@gmail.com> Co-authored-by: Zike Yang <zike@apache.org> Co-authored-by: Lari Hotari <lhotari@apache.org>

[Dockerfile] Enable retries for apt-get when building Pulsar docker i…

d8e7032

…mage - also reduce default timeout to 30 seconds - prevents issues where apt repository doesn't respond

lhotari added area/build doc-not-needed Your PR changes do not impact docs labels Mar 1, 2022

lhotari requested review from eolivelli and michaeljmarshall March 1, 2022 12:56

lhotari self-assigned this Mar 1, 2022

eolivelli approved these changes Mar 1, 2022

View reviewed changes

michaeljmarshall approved these changes Mar 1, 2022

View reviewed changes

nicoloboschi approved these changes Mar 1, 2022

View reviewed changes

michaeljmarshall reviewed Mar 1, 2022

View reviewed changes

docker/pulsar/Dockerfile Show resolved Hide resolved

lhotari closed this Mar 2, 2022

lhotari reopened this Mar 2, 2022

lhotari merged commit d3f6fe3 into apache:master Mar 2, 2022

codelipenghui added this to the 2.11.0 milestone Mar 3, 2022

tisonkun reviewed Sep 20, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dockerfile] Enable retries for apt-get when building Pulsar docker image #14513

[Dockerfile] Enable retries for apt-get when building Pulsar docker image #14513

Uh oh!

lhotari commented Mar 1, 2022 •

edited

Loading

Uh oh!

eolivelli left a comment

Uh oh!

michaeljmarshall left a comment

Uh oh!

Uh oh!

tisonkun Sep 20, 2022

Uh oh!

tisonkun Sep 20, 2022 •

edited

Loading

Uh oh!

tisonkun Sep 20, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[Dockerfile] Enable retries for apt-get when building Pulsar docker image #14513

[Dockerfile] Enable retries for apt-get when building Pulsar docker image #14513

Uh oh!

Conversation

lhotari commented Mar 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Uh oh!

eolivelli left a comment

Choose a reason for hiding this comment

Uh oh!

michaeljmarshall left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tisonkun Sep 20, 2022

Choose a reason for hiding this comment

Uh oh!

tisonkun Sep 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tisonkun Sep 20, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

lhotari commented Mar 1, 2022 •

edited

Loading

tisonkun Sep 20, 2022 •

edited

Loading