Kubernetes builders have issues with networking

[nuclearcat]

PASS
Found job on pod build-j12978-arm64-clang-14-cros---chromeos-5-15-arm64-chrwvc7f
Container Log:
++ nproc
+ echo nproc=8
+ df
nproc=8
Filesystem     1K-blocks     Used Available Use% Mounted on
overlay        259966896 26609932 233340580  11% /
tmpfs              65536        0     65536   0% /dev
tmpfs           16441384        0  16441384   0% /sys/fs/cgroup
tmpfs           16441384        0  16441384   0% /scratch
/dev/sda1      259966896 26609932 233340580  11% /etc/hosts
shm                65536        0     65536   0% /dev/shm
tmpfs           16441384       12  16441372   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs           16441384        0  16441384   0% /proc/acpi
tmpfs           16441384        0  16441384   0% /proc/scsi
tmpfs           16441384        0  16441384   0% /sys/firmware
+ free
               total        used        free      shared  buff/cache   available
Mem:        32882772      985860    26809652        1664     5087260    31439008
Swap:              0           0           0
+ whoami
root
+ export KDIR=/tmp/kci/linux
+ KDIR=/tmp/kci/linux
+ export CCACHE_DISABLE=true
+ CCACHE_DISABLE=true
+ cd /scratch
+ set +x
+ kci_build pull_tarball --url http://storage.chromeos.kernelci.org/kernelci/chromeos-stable/chromeos-stable-20220923.1/linux-src_kernelci_chromeos-stable.tar.gz --retries 3 --delete
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.9/socket.py", line 953, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/lib/python3.9/http/client.py", line 950, in send
    self.connect()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f8d848f91c0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='storage.chromeos.kernelci.org', port=80): Max retries exceeded with url: /kernelci/chromeos-stable/chromeos-stable-20220923.1/linux-src_kernelci_chromeos-stable.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8d848f91c0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/kci_build", line 4, in <module>
    __import__('pkg_resources').run_script('kernelci==1.1', 'kci_build')
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 651, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 1448, in run_script
    exec(code, namespace, namespace)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/EGG-INFO/scripts/kci_build", line 451, in <module>
    status = opts.command(configs, opts)
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/EGG-INFO/scripts/kci_build", line 444, in __call__
    return kernelci.build.pull_tarball(
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/kernelci/build.py", line 399, in pull_tarball
    if _download_file(url, dest_filename):
  File "/usr/local/lib/python3.9/dist-packages/kernelci-1.1-py3.9.egg/kernelci/build.py", line 384, in _download_file
    resp = requests.get(url, stream=True, headers=headers)
  File "/usr/local/lib/python3.9/dist-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='storage.chromeos.kernelci.org', port=80): Max retries exceeded with url: /kernelci/chromeos-stable/chromeos-stable-20220923.1/linux-src_kernelci_chromeos-stable.tar.gz (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8d848f91c0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

Which causes whole build to fail.
I suspect that container start to execute payload too fast and network stack is not ready yet.
Probably adding delay or script that check for network stack readiness might solve this issue.

[gctucker]

Or adding some retry feature to the download command.

[nuclearcat]

Great idea! Actually i think it will be good to have both features. Likely network stack coming very fast, so verification will not take more than few seconds, but download failures might happen due other reasons.

[nuclearcat]

As @gctucker suggested i am implementing retry mechanism in #1458
After thinking well i considered his suggestion as best and it might be enough without any other additional functions like checking network stack readiness.