Use of/alignment with Anaconda

[Promaethius]

Add an option to the build command to output an initramfs.img and coreos.img to serve over PXE. This would support the https://github.com/coreos/matchbox project which would lead to tectonic integration. I'd be more than willing to contribute to this.

EDIT: I see a lot of parallels with https://github.com/weldr/lorax/blob/master/docs/livemedia-creator.rst

[cgwalters]

Yes. However, one important architectural point here is that currently lmc/lorax create loopback devices and require SELinux permissive or disabled. If we were to integrate them here, I would prefer we inject them into transient "utility" VMs and extract the output from that.

This way we don't affect the host system - we won't leak loopback devices on failure, we don't require the host to turn off SELinux. And more generally, we can "scale out" builds using e.g. Kubernetes as long as we have virt capabilities.

[cgwalters]

Looking quickly a bit more at LMC, I had forgotten it supports two modes, including one that does the heavy lifting in qemu.

[Promaethius]

Looking quickly a bit more at LMC, I had forgotten it supports two modes, including one that does the heavy lifting in qemu.

Yes that's the impression I got as well. In addition, it creates the squashfs and PXE scripts. It seems to be a nicely rounded tool. Doesn't it also see use in Pungi? As long as it still sees use in upstream fedora could it be useful here?

Reading through creator.py it includes some validation that might be useful:
Kickstart: L595-L638 https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L595

Disk Image: https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L642 -> https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L459 (Where @cgwalters was indicating the dual modes https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L478) into virt_install https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L484 -> https://github.com/weldr/lorax/blob/master/src/pylorax/installer.py#L486 into mkfsimage_from_disk https://github.com/weldr/lorax/blob/master/src/pylorax/installer.py#L550 -> https://github.com/weldr/lorax/blob/master/src/pylorax/imgutils.py#L508 (function completes and branches back up to creator.py) https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L698 into make_live_images https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L704 -> https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L496 (includes some disk logic between L518-L547) build initramfs rebuild_initrds_for_live https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L566 -> https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L210 (looks like the dracut build doesn't take params from the rpm-ostree definition, this makes sense given that lorax is ostree specific. I wonder how Pungi handles rpm-ostree to ostree to LMC dracut arguments? L219-L227 Dracut is run on L272. Looking at how dracut is built.)
Branch back up to virt_install into class QEMUInstall https://github.com/weldr/lorax/blob/master/src/pylorax/installer.py#L134 injected KS script at https://github.com/weldr/lorax/blob/master/src/pylorax/installer.py#L208 (If the ks script is anything like https://github.com/rvykydal/anaconda-kickstarts/blob/master/atomic/fedora-atomic-pxe-live.ks then adding ostreesetup commands to the kickstart script can easily rebase; however, this requires the ostree repo to be passed as a URL or installer.py needs to be modified to mount the ostree repo filesystem.)

TLDR;
LMC requires passing in the same dracut commands defined in a normal fedora-atomic-host-base.json rpm-ostree definition.
LMC uses kickstart commands to trigger an ostreesetup.
LMC can access a local ostree repo over http as is.

Should atomic-assembler consider using LMC to build artifacts rather than something else to maintain? Since LMC is used in upstream Fedora/Atomic releases, it might be worthwhile to investigate as artifact creation. As far as I can tell, it uses the same VM process atomic-assembler does.

If we continue using an in-house build system, there's a well documented python api which can generate a fsimage https://github.com/weldr/lorax/blob/master/src/pylorax/imgutils.py#L508, initramfs https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L210, and PXE scripts https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L572-L575 https://github.com/weldr/lorax/blob/master/src/pylorax/creator.py#L280

[cgwalters]

OK, first: I just put up #113

Now, long story time. I spent quite a while thinking about
this yesterday.

When Atomic/rpm-ostree was first created, I very intentionally
did not try to change (much) how the OS is configured (/etc, config
management, cloud-init) - and most notably, ostreepayload with
Anaconda kickstart works just fine, you can use the timezone
verb, etc.

(Further, ostree being agnostic about underlying block storage
means that you can install using Anaconda with dm-crypt or not,
whatever fs layout, etc.)

And to me, rpm-ostree excels at the "pet system" case - that
blends well with Anaconda which is often used to generate these
types of systems via interactive installs.

So I don't think we're going to delete the rpmostreepayload code
in Anaconda - if for no other reason than we probably still
need it for Silverblue (that's a whole other thing though).

However...the usage of Ignition for Fedora CoreOS
is a very opinionated model for how systems are configured. And
for many of our use cases, this is a good thing - people
who are seeking image-based systems like ostree tend to
want to go full "immutable infrastructure".

Using Ignition has a lot of consequences - it means that
for our users, we won't be exposing Kickstart or cloud-init.
And at a deeper technical level, it just works fundamentally
differently.

For example, see systemd/systemd#8268 (comment)
Because our ostree payload does not have /etc/machine-id,
if someone tried to use Anaconda to install it, the kickstart services
verb would not work.

Now potentially, we could teach anaconda about this - it
might work to simply systemd-machineid-setup in %post.
But the higher level point here stands - Ignition and Anaconda
overlap significantly.

And more importantly, I am convinced now for many reasons
that it's critical for Fedora CoreOS-derived systems
to offer the simple "dd to disk" install method:
coreos/fedora-coreos-tracker#18
We have had several requests for this over time.

Hence, I think the balance weighs towards not using Anaconda
at build time either.

And now here's the thing - all of the current image-building
tools in the Fedora ecosystem (ImageFactory, virt-install,
and lorax-composer/LMC) end up being wrappers around running
Anaconda in a VM.

Another major technical point here is that for ostree-based
systems, our "build" is really an ostree - VM images
are just "wrappers". This is an enormous difference from
people who are just doing "yum install foo bar baz" inside
kickstarts and using that to make AMIs or Vagrant boxes
or whatever.

And consequently, the usage of Anaconda is much much smaller.

As soon as you ship an OSTree-based system it has huge
implications for the delivery model.

The code today in coreos-assembler ties the ostree and
VM images together closely in a way that I'm quite
happy with so far - we never made something as nice as
this back in the Atomic days.

All this to say: I think we should share code with
lorax/LMC - if we end up using Python that will be easier.
And there are a lot of points where we do want to align;
it'd be good in particular to generate the same
PXE layout, as a lot of Fedora/Red Hat ecosystem tools
end up parsing that. But I think our best path here
is to have a custom high level tool rather than try
to wedge the things we do directly into e.g. lorax/LMC.

[cgwalters]

The current situation of using Anaconda both for per-system installs and as a "golden image" building tool creates a lot of tension and problems. For example, when generating a Fedora Cloud or Atomic Host image, we don't want the system go out and do geolocation to try to find the current timezone. We don't want it to persist /var/lib/systemd/random-seed. We don't want whatever network configuration the build-side VM has to be persisted.

Today, we often work around these sorts of things in kickstarts. But things feel a lot cleaner if Anaconda can primarily focus on "per-system" installs, and we have tooling (that again shares code where it makes sense) dedicated to making "unconfigured golden images".

Of course, since coreos-assembler obviously isn't going to be building "classic" images like Fedora Cloud, us not using Anaconda for that increases the likelihood of bugs there. But to me that's the past 😄 There's also the "LiveOS" style systems like current Fedora Workstation. That one falls in the middle, but it seems fine to me to keep using LMC for it. The real "middle" case is Silverblue...whether we use coreos-assembler to build that is a whole other question - whether we support Ignition for it...a messy topic.

[Promaethius]

@cgwalters cheers for the good read. There's a lot about the underlying utilities that I do not understand.

How does the current coreos-assembly differ from being a vm wrapper to install an ostree and ship as an image?
I see that coreos-assembler uses fedora_everything as the base image to install ostree into. Does this circumvent the need for anaconda since anaconda installs per-system? If I'm to understand the LMC correctly, it doesn't need an anaconda disk to use kickstart scripts.

One other difference between LMC and coreos-assembler's virt-install... the network is disabled and the ostree is mounted as a filesystem https://github.com/coreos/coreos-assembler/blob/master/src/virt-install#L112 so I'll mention that in the pull request.

[cgwalters]

I see that coreos-assembler uses fedora_everything as the base image to install ostree into. Does this circumvent the need for anaconda since anaconda installs per-system?

No, it currently just runs Anaconda in a VM (via virt-install) on the build system side, which has all of the issues listed above.

Today, this project (coreos-assembler) does not generate an installer ISO, or a dd-able disk image for bare metal. That's a lot of the discussion here.

[Promaethius]

So, where is this project headed in terms of Ignition vs Anaconda?

Of course, since coreos-assembler obviously isn't going to be building "classic" images like Fedora Cloud, us not using Anaconda for that increases the likelihood of bugs there. But to me that's the past 😄

I totally agree with this. Using an interpreted language as init system scripting feels wrong in terms of predictability. Ditching Anaconda feels like it should have its own issue; however, it ties in by branching the installation process away from LMC and the current coreos-assembly process.

In which case, how will base images be built? If we're not using fedora_everything, will the wrapper pass in build commands via qemu into the VM? Will we have to maintain our own base image?

Additionally, is there any issue with branding?

Removing Anaconda/kickstart might be feasible (oddly) through using Lorax instead of LMC. A quick read of the docs indicate that fedora_everything is built through Lorax. Merging a dnf squashfs with basic kernel to create a system image. Would it be possible to use this process? It would require some creativity with rebasing without initializing.

[cgwalters]

In which case, how will base images be built?

In my opinion, we should use supermin to inject content from the container into "utility VMs" - we'd create filesystems and install the bootloader etc. from there.

A major advantage of this is that we don't need to download a separate installer ISO - there is only the coreos-assembler container content.

And end users don't use it either - they will just have a dd-able disk image that is configured on firstboot via Ignition.

[Promaethius]

In my opinion, we should use supermin to inject content from the container into "utility VMs" - we'd create filesystems and install the bootloader etc. from there.

Looking at the example, supermin looks powerful and light!
The documentation indicates that packages and dependencies are installed from the host system. This creates a lot of flexibility when initiating the appliance given that the ostree package can be "injected" from the build container into the appliance without having to use any ks scripts. Additionally, there won't be any anaconda artifacts.

I guess this sets us back to square one though as far as extracting vmlinuz, initramfs, and squashfs from the appliance.

[cgwalters]

I guess this sets us back to square one though as far as extracting vmlinuz, initramfs, and squashfs from the appliance.

No, the appliance is a server-side only thing. As far as artifacts consumed by end users, I think we'd generate a build/$x/pxe subdirectory with kernel/initramfs, and a build/$x/live.iso or so?

It's interesting, it looks to me like Container Linux today puts the whole OS in the initramfs for iPXE, see https://coreos.com/os/docs/latest/booting-with-ipxe.html

[cgwalters]

In contrast, Anaconda today has only a slightly custom initramfs, and downloads the "stage2" - the much bigger squashfs with Anaconda inside - itself.

[Promaethius]

So supermin --prepare creates an appliance definition with packages, --build -f ext2 --copy-kernel builds vm artifacts (initrd, kernel, hda) with the host kernel, and then coreos-assembler uses guestfish to pass in the arguments to configure the ostree and generate a new initramfs in the qemu booted vm.

Are there any concerns with creating the kernel this way since ignition needs a kernel mod? Or is that up to the rpm-ostree generated dracut initramfs created inside the supermin generated vm? Will copying the device tree through --copy-kernel create any issues?