[docs]: missing build from source documentation

[daweedm]

Because of a bug in cloud-init v22.4.2-1 shipped with debian 12, i'm trying to get the version v23.1 compiled for debian but unfortunately intermediate release builds are not longer available in the testing repository. The latest version is not compatible with debian 12 because v25.x needs updating some dependencies (e.g. libc6) which may brake other packages in the distro.

My last resort is to build v23.1 from source but i can't find any documentation excepting this: https://github.com/canonical/cloud-init/tree/main/packages which seems incomplete. What needs to be installed & run to build the package and get a .deb file ?

[daweedm]

I tried to run the makefile with no luck :

git clone https://github.com/canonical/cloud-init.git
cd clout-init
git checkout tags/23.1
make deb

Got this error : dpkg-checkbuilddeps: error: Unmet build dependencies: dh-python python3-pytest-cov python3-pytest-mock python3-responses

Then I installed the missing dependencies :
apt-get install dh-python python3-pytest-cov python3-pytest-mock python3-responses

and again make deb but it ends up with an unclear error :

/usr/bin/debuild
python3 ./packages/bddeb
Creating a temp tarball using the 'make-tarball' helper
Extracting temporary tarball 'cloud-init_23.1-0-g242f6bc43.orig.tar.gz'
Creating a debian/ folder in '/run/cloud-init/tmp/tmpgtp1ie7e/cloud-init-23.1-0-g242f6bc43'
Running 'debuild -us -uc' in '/run/cloud-init/tmp/tmpgtp1ie7e/cloud-init-23.1-0-g242f6bc43'
Traceback (most recent call last):
  File "/var/lib/vz/template/iso/cloud-init/./packages/bddeb", line 319, in <module>
    sys.exit(main())
             ^^^^^^
  File "/var/lib/vz/template/iso/cloud-init/./packages/bddeb", line 293, in main
    subp.subp(cmd, capture=capture)
  File "/var/lib/vz/template/iso/cloud-init/cloudinit/subp.py", line 335, in subp
    raise ProcessExecutionError(
cloudinit.subp.ProcessExecutionError: Unexpected error while running command.
Command: ['debuild', '--preserve-envvar', 'INIT_SYSTEM', '-us', '-uc']
Exit code: 29
Reason: -
Stdout:  dpkg-buildpackage -us -uc -ui
        dpkg-buildpackage: info: source package cloud-init
        dpkg-buildpackage: info: source version 23.1-0-g242f6bc43-1~bddeb
        dpkg-buildpackage: info: source distribution UNRELEASED
        dpkg-buildpackage: info: source changed by Scott Moser <smoser@ubuntu.com>
         dpkg-source --before-build .
        dpkg-buildpackage: info: host architecture amd64
         debian/rules clean
        Can't exec "debian/rules": Permission denied at /usr/bin/dpkg-buildpackage line 834.
        dpkg-buildpackage: error: debian/rules clean subprocess failed with unknown status code -1
Stderr: debuild: fatal error at line 1182:
        dpkg-buildpackage -us -uc -ui failed
make: *** [Makefile:106: deb] Error 1

[aciba90]

Thanks for reporting this!

The recommended way to build cloud-init for testing/development is documented here too: https://cloudinit.readthedocs.io/en/latest/development/package_testing.html#package-testing.

I am not able to hit the issue you are facing. Would you, or other person hitting this, mind dropping a breakpoint in:

diff --git a/packages/bddeb b/packages/bddeb
index 4e06378c7..77f81d4c7 100755
--- a/packages/bddeb
+++ b/packages/bddeb
@@ -361,6 +361,7 @@ def main():
             "Running 'debuild %s' in %r" % (" ".join(args.debuild_args), xdir)
         )
         with util.chdir(xdir):
+            import pdb; pdb.set_trace()
             cmd = ["debuild", "--preserve-envvar", "INIT_SYSTEM"]
             if args.debuild_args:
                 cmd.extend(args.debuild_args)

and take a look on file ownership and permissions of xdir? In particular debian/rules under that directory as per your trace, it seems your user has no permissions to execute that file, presumably due to some sort of bug related to packages/bddeb. Thanks!

[DarkPhily]

A little debugging later, I found the issue 😁

The issue is, that you create the tmp dir in /run, or to be more precise: The issue is, that /run is mounted with noexec

root@cloud-init:/run/cloud-init/tmp/tmp9qt19dtq/cloud-init-25.2-62-gf57e6c26# mount | grep /run
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=793736k,mode=755,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=793732k,nr_inodes=198433,mode=700,inode64)

For the sake of completeness. Here are the permissions of the debian directory in the tmp directory:

root@cloud-init:~/cloud-init# ls -hal /run/cloud-init/tmp/tmp9qt19dtq/cloud-init-25.2-62-gf57e6c26/debian/
total 44K
drwxr-xr-x  3 root root  280 Sep  5 15:38 .
drwxrwxr-x 17 root root  840 Sep  5 15:38 ..
-rw-r--r--  1 root root  144 Sep  5 15:38 changelog
-rw-r--r--  1 root root  184 Aug 18 11:10 changelog.in
-rw-r--r--  1 root root  147 Aug 18 11:10 cloud-init.logrotate
-rw-r--r--  1 root root  153 Aug 18 11:10 cloud-init.postrm
-rw-r--r--  1 root root 3.3K Sep  5 15:38 control
-rw-r--r--  1 root root 2.9K Sep  2 16:46 control.in
-rw-r--r--  1 root root 1.6K Aug 18 11:10 copyright
-rw-r--r--  1 root root   69 Aug 18 11:10 dirs
-rw-r--r--  1 root root   65 Aug 18 11:10 manpages
-rwxr-xr-x  1 root root 1.2K Sep  2 16:46 rules
drwxr-xr-x  2 root root   60 Aug 18 11:10 source
-rw-r--r--  1 root root   91 Aug 18 11:10 watch

After some further digging I found the root cause.

cloud-init/packages/bddeb

Line 309 in 1955952

with temp_utils.tempdir() as tdir:

You are not setting the parameter needs_exe to True, but it's needed, if you are running bddeb as root:

cloud-init/cloudinit/temp_utils.py

Lines 27 to 40 in 1955952

	def _tempfile_dir_arg(odir=None, needs_exe: bool = False):
	"""Return the proper 'dir' argument for tempfile functions.
	When root, cloud-init will use /run/cloud-init/tmp to avoid
	any cleaning that a distro boot might do on /tmp (such as
	systemd-tmpfiles-clean).
	If the caller of this function (mkdtemp or mkstemp) was provided
	with a 'dir' argument, then that is respected.
	@param odir: original 'dir' arg to 'mkdtemp' or other.
	@param needs_exe: Boolean specifying whether or not exe permissions are
	needed for tempdir. This is needed because /run is mounted noexec.
	"""

[DarkPhily]

After setting this to True the command still fails, because some of the tests fail, all of the sudden :(
Running the tests via tox shows no failed tests.

       ====================================================================================================== short test summary info =======================================================================================================
FAILED tests/unittests/test_net.py::TestNetplanNetRendering::test_render[subnet_metric_in_dhcp] - AssertionError: assert False
FAILED tests/unittests/test_net.py::TestNetplanNetRendering::test_render[gateway_with_metric] - AssertionError: assert False
FAILED tests/unittests/test_net.py::TestNetplanNetRendering::test_render[static_routes_with_metrics] - AssertionError: assert False
FAILED tests/unittests/test_net.py::TestNetplanNetRendering::test_render[physical_gateway46] - AssertionError: assert False
FAILED tests/unittests/test_net.py::TestNetplanNetRendering::test_render[bond_gateway46] - AssertionError: assert False
FAILED tests/unittests/test_net.py::TestNetplanNetRendering::test_render[two_subnets_old_new_gateway46] - AssertionError: assert False
FAILED tests/unittests/test_net.py::TestNetplanNetRendering::test_render[one_subnet_old_new_gateway46] - AssertionError: assert False
FAILED tests/unittests/test_net.py::TestNetplanNetRendering::test_render[onlink_gateways] - AssertionError: assert False
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[bond_v1-yaml] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[bond_v2-yaml] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[small_v1-yaml] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[v4_and_v6-yaml_v1] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[v4_and_v6-yaml_v2] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[v1_ipv4_and_ipv6_static-yaml_v1] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[v2_ipv4_and_ipv6_static-yaml_v2] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[dhcpv6_only-yaml_v1] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[dhcpv6_only-yaml_v2] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[dhcpv6_accept_ra-yaml_v1] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[dhcpv6_reject_ra-yaml_v1] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[ipv6_slaac-yaml] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[dhcpv6_stateless-yaml] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[dhcpv6_stateful-yaml] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[wakeonlan_disabled-yaml_v2] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[wakeonlan_enabled-yaml_v2] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[large_v1-yaml] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_config[manual-yaml] - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_render_output_has_yaml_no_aliases - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/test_net.py::TestNetplanRoundTrip::test_render_output_supports_both_grat_arp_spelling - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/cmd/devel/test_net_convert.py::TestNetConvert::test_convert_output_kind_artifacts[netplan-outfile_content0-False] - py.error.ENOENT: [No such file or directory]: open('/tmp/pytest-of-root/pytest-9/test_convert_output_kind_artif0/etc/netplan/50-cloud-init.yaml', 'r')
FAILED tests/unittests/cmd/devel/test_net_convert.py::TestNetConvert::test_convert_output_kind_artifacts[netplan-outfile_content0-True] - py.error.ENOENT: [No such file or directory]: open('/tmp/pytest-of-root/pytest-9/test_convert_output_kind_artif1/etc/netplan/50-cloud-init.yaml', 'r')
FAILED tests/unittests/cmd/devel/test_net_convert.py::TestNetConvert::test_convert_netplan_passthrough[False] - py.error.ENOENT: [No such file or directory]: open('/tmp/pytest-of-root/pytest-9/test_convert_netplan_passthrou0/etc/netplan/50-cloud-init.yaml', 'r')
FAILED tests/unittests/cmd/devel/test_net_convert.py::TestNetConvert::test_convert_netplan_passthrough[True] - py.error.ENOENT: [No such file or directory]: open('/tmp/pytest-of-root/pytest-9/test_convert_netplan_passthrou1/etc/netplan/50-cloud-init.yaml', 'r')
FAILED tests/unittests/distros/test_netconfig.py::TestNetCfgDistroUbuntuNetplan::test_apply_network_config_v1_to_netplan_ub - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/distros/test_netconfig.py::TestNetCfgDistroUbuntuNetplan::test_apply_network_config_v1_ipv6_to_netplan_ub - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/distros/test_netconfig.py::TestNetCfgDistroUbuntuNetplan::test_apply_network_config_v2_passthrough_ub - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/distros/test_netconfig.py::TestNetCfgDistroUbuntuNetplan::test_apply_network_config_v2_passthrough_retain_orig_perms - AssertionError: assert '# This file ... version: 2\n' == 'a'
FAILED tests/unittests/distros/test_netconfig.py::TestNetCfgDistroUbuntuNetplan::test_apply_network_config_v2_passthrough_ub_old_behavior - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/distros/test_netconfig.py::TestNetCfgDistroUbuntuNetplan::test_apply_network_config_v2_full_passthrough_ub - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/distros/test_netconfig.py::TestNetCfgDistroArch::test_apply_network_config_v1_with_netplan - KeyError: '/etc/netplan/50-cloud-init.yaml'
FAILED tests/unittests/net/test_net_rendering.py::test_convert[no_matching_mac_v2-Renderer.Netplan|NetworkManager] - FileNotFoundError: [Errno 2] No such file or directory: '/tmp/pytest-of-root/pytest-9/test_convert_no_matching_mac_v0/netplan.yaml'
===================================================================== 
40 failed, 5243 passed, 12 skipped, 13 xfailed, 2 xpassed, 9 warnings in 106.12s (0:01:46) =====================================================================

[DarkPhily]

Something really weird is going on with the tests, which run during the build...
The tests fail, because some output files are not written to the /tmp/pytest-for-root/.

I can even observe that one of the tests is overwriting my /etc/netplan/50-cloud-init.yaml:

(cloud-init) root@ubuntu-2gb-hel1-1:~/cloud-init# cat /etc/netplan/50-cloud-init.yaml
network:
  version: 2
  ethernets:
    encc000: {}
    zz-all-en:
      match:
        name: "en*"
      dhcp4: true
    zz-all-eth:
      match:
        name: "eth*"
      dhcp4: true
  vlans:
    encc000.2653:
      addresses:
      - "10.245.236.14/24"
      nameservers:
        addresses:
        - 10.245.236.1
      gateway4: 10.245.236.1
      id: 2653
      link: "encc000"

I created a clean ubuntu 24.04 VM and only cloned cloud-init, installed the build dependencies and run dbbed

Edit: The tests fail, once you run bddeb as root