Docker-in-docker - "sed: couldn't flush stdout: Device or resource" errors

[samruddhikhandale]

Tracks issues with docker startup due to sed: couldn't flush stdout: Device or resource errors. See https://github.com/devcontainers/features/actions/runs/5907721355/job/16026113620?pr=659#step:4:1968

Need to add a retry mechanism around this logic -->

features/src/docker-in-docker/install.sh

Lines 375 to 398 in d48a9d5

	set_cgroup_nesting()
	{
	# cgroup v2: enable nesting
	if [ -f /sys/fs/cgroup/cgroup.controllers ]; then
	# move the processes from the root group to the /init group,
	# otherwise writing subtree_control fails with EBUSY.
	# An error during moving non-existent process (i.e., "cat") is ignored.
	mkdir -p /sys/fs/cgroup/init
	xargs -rn1 < /sys/fs/cgroup/cgroup.procs > /sys/fs/cgroup/init/cgroup.procs || :
	# enable controllers
	sed -e 's/ / +/g' -e 's/^/+/' < /sys/fs/cgroup/cgroup.controllers \
	> /sys/fs/cgroup/cgroup.subtree_control
	fi
	}
	# Set cgroup nesting, retrying if necessary
	set +e
	set_cgroup_nesting
	if [ $? -ne 0 ]; then
	echo >&2 "cgroup v2: failed to enable nesting, retrying once..."
	set -e
	set_cgroup_nesting
	fi
	set -e

[samruddhikhandale]

docker-in-docker v2.4.0 includes the following changes, see #669

• Adds retries for docker daemon startup
• We have seen errors like sed: couldn't flush stdout: Device or resource errors which fails to start, this adds retries to fix such sed errors
• Adds a workflow which runs 100 jobs for validating docker startup --> 50 jobs validates dockerd after the container is started, and 50 jobs validates within the onCreateCommand. The stress test is 🟢

Important Note: /usr/local/share/docker-init.sh which starts/retries dockerd is added to the entrypoint command. This command runs in the background and is not a blocking script for the container startup. Since it's in the background, onCreateCommand/postCreateCommand/postStartCommand could all start executing before docker is fully running. If it takes docker too long, that could introduce flakiness in those lifecycle scripts.

Opened devcontainers/spec#299 which requests a new semantics to have "blocking" entrypoints that the CLI waits for. This way we can ensure that docker is already up and running for the mentioned ^ lifecycle scripts and is available in the container.

Closing in the favor of #671. Feel free to reopen if needed, or comment on #671 if you still run into docker not running issues. Thank you!