Bundle everything using "chroot" and "unshare"

[lvml]

Hi,

I wanted to give AppImage a try to create a "RawTherapee 5.1" package - which due to its dependencies is not available for even some recent distributions (like RHEL / CentOS 7.x).

I followed the "Documentation" link on https://appimage.org/, which led me to https://github.com/probonopd/AppImageKit/blob/master/README.md

I read there about LibcWrapGenerator, which I intended to try since it's easy to compile RawTherapee 5.1 on Arch, but it is a terrible pain to try this on CentOS.

Then I noticed that LibcWrapGenerator is not included (or used or mentioned) in the source code that is referred to by the "Source" link on the https://appimage.org/ page - which links to https://github.com/AppImage/AppImageKit instead of the other branch linked above, and both seem to contain pretty different content, with no clear explanation whether one or the other is outdated, and what happened to LibcWrapGenerator in the more recently changed of these two branches.

I also noticed that the "build.sh" script included with AppImageKit does create binaries under build/ - but there is no word on whether those should be installed somehow (if only by copying them to /usr/local/bin/ or something).

Can you clarify this confusion?

[TheAssassin]

You don't actually have to use CentOS, just use something sufficiently old. Debian (old)stable or Ubuntu 14.04 or openSUSE Leap 42.2 will work fine for that purpose, too. But Arch is a bad idea, please see below for details.

I am not really aware of LibcWrapGenerator and how it works, but it sounds hackish, so I'd avoid that.

To make your life easier and since you are obviously not necessarily interested in building AppImages for the latest (and probably greatest) development code, I'd recommend you to check out our latest innovation.

You probably haven't heard of that, but Open Build Service (including the free to use openSUSE build service) can build AppImages now. And guess what: The easiest way is to repackage existing RPMs.

openSUSE chairman Richard Brown and AppImage creator @probonopd explain why this is a good thing and why it greatly simplifies the creation of AppImages in these videos:

• https://media.ccc.de/v/1320-opensuse-jurrassic-park
• https://www.youtube.com/watch?v=8r1OwX-dNFo

Our wiki contains a really good guide to that: https://github.com/AppImage/AppImageKit/wiki/Using-Open-Build-Service

OBS contains a pretty up to date RPM in the graphics repository.

Now all you have to do is set up a project on OBS, add the graphics repository to your project on OBS, add the YML file describing the build and let OBS do the rest!

Features:

• automatic rebuilds of the AppImage whenever a bundled library or RawTherapee are updated
• automatic support for AppImageUpdate
• write the YML once, never touch it again

If you are interested in doing it that way, it would be a great help for our project, since we really need testers for the instructions and OBS, and I'd highly appreciate if you went that way. Only if that is what you want to do, of course.

If you need help with the details (you most likely will, as the OBS documentation isn't that complete, and we've probably faced the problems already), feel free to join us in #AppImage on Freenode at any time. We'll help you to make the AppImage recipe.

If you insist on building from source, OBS can do that too. If you don't wish to use OBS, that's fine too.

A few more comments on your initial problems:

• Please, do NOT build AppImages on Arch Linux. Even if you could technically make them with Arch and that wrapper thingy, it'd most likely be really hard for people to rebuild your AppImage. Arch, due to its rolling release character and the mostly questionable "package management" isn't at all suitable as a build system for anything.

• The README might be outdated. Please open another issue that describes what doesn't work, and we'll try to fix that.

• The build.sh script is intended to build AppImageTool of which an AppImage is created. Of course, we ship our build tools as AppImages, too. You can just go to the releases page, and download the latest copy of the AppImage. You do not have to install these AppImages. Following our AppImage spirit, just download them, make them executable and run them.

• Please note that we also provide linuxdeployqt, which aims to simplify the creation of AppImages. Don't interpret that name wrong, it's just written in Qt, but works well for any software.

I hope that helps you and gives you a good overview of the purpose of this repository.

[probonopd]

Hi @lvml, welcome to AppImage.

I also noticed that the "build.sh" script included with AppImageKit does create binaries under build/ - but there is no word on whether those should be installed somehow (if only by copying them to /usr/local/bin/ or something).

We generally recommend using our prebuilt binaries, https://github.com/AppImage/AppImageKit#building

If you want to build yourself, it should be sufficient to add the build/ directory to your $PATH.

[probonopd]

Please, do NOT build AppImages on Arch Linux. Even if you could technically make them with Arch and that wrapper thingy, it'd most likely be really hard for people to rebuild your AppImage. Arch, due to its rolling release character and the mostly questionable "package management" isn't at all suitable as a build system for anything.

And even more importantly, building on a Rolling Release distribution would mean that your AppImage would only run on the very latest target systems, as is explained in https://github.com/AppImage/AppImageKit/wiki/Creating-AppImages#binaries-compiled-on-old-enough-base-system

[lvml]

Hmm... thanks for your explanations, but they seem to address a different use case than the one I am currently contemplating about:

I own several computers, one of which runs CentOS 7, and I would like to run the same up-to-date version of RawTherapee on that one as on the other, Arch-Linux-running computers.
Compiling RawTherapee from its sources on Arch is a matter of minutes, but on CentOS 7, it's a bottomless pit of unresolved dependencies, if only because of the gtk3 absence.

My idea was to create an AppImage on Arch, for the sole purpose of running it on the CentOS 7 machine. Of course I am prepared to copy vast amounts of binaries, libraries and data files into that AppImage to make this work, I don't mind its size and would even be fine with putting ld-linux.so.* and libc with all its additional shared objects and data files in there, if required.

But compiling RawTherapee 5.1 on some older operating system like CentOS in order to create an AppImage from this completely defeats the purpose of the whole exercise: If I could make this work with reasonable effort, I could compile RawTherapee 5.1 on the target system in the first place and be done (no need for an AppImage, then).

Regarding:

building on a Rolling Release distribution would mean that your AppImage would only run on the very latest target systems

Maybe my understanding of AppImage was wrong, but I thought the very purpose of AppImage is to put everything required by an application (south of the kernel ABI) into a binary such that it can be run on any Linux distribution?
(By the way: That is what we do at work: Compile software on a very recent Linux system, and linking statically so everything is included and can run even on older systems.)

If "building on a recent system" means "cannot run on an older system", then AppImage may just not be meant for the purpose I'm after...

[probonopd]

@lvml you can decide what you put inside an AppImage, so you could do what you suggest. Please see AppImageCommunity/pkg2appimage#84

[probonopd]

We don't generally recommend to bundle everything though, since this could be percieved as bloated. Instead, we do not bundle libraries that can reasonably be expected to be part of every target system (e.g., glibc and friends). Which means that we need to compile on a system no newer than the oldest target system.

[lvml]

Thanks for the link to AppImageCommunity/pkg2appimage#84, will have a look at the options discussed there.

(BTW: Did anyone ever try to use https://github.com/rpodgorny/unionfs-fuse to user-mount the content of an AppImage as an "overlay" to the original root file-system? This would seem like an obvious idea to me for a "bundles everything" kind of AppImage, solving the issue that one can use absolute paths in the application just normally, even for the dynamic linker.)

Regarding "bloat": I agree that "bundling everything" might be undesirable for those who intend to publish an application of theirs to a large audience. However, for the use case of "just make application X currently running fine on computer Y run also on computer Z", a "bloated" AppImage is the least of my concerns. Should such an image of RawTherapee allocate 1GB of disk space - I woudn't mind, that is still below 1/1000 of the available storage space on the target, and I do not intend to populate the target with thousands of AppImages.

[probonopd]

Yes, https://github.com/rpodgorny/unionfs-fuse is an option. Whoever puts together an AppImage can put it inside, with a script called "AppRun" that sets it all up.

[lvml]

Success! (Albeight not using AppImage, but I wanted to let you know anyway since the information you provided in this thread was very helpful for my inspiration - thank you.)

I managed to run the binaries from an Arch Linux installation under CentOS:

I created a script that uses "strace ... rawtherapee" under Arch to record all the files that rawtherapee requires - including all distribution-provided shared libraries, data files and links.

Then I tar'ed those files, copied them to the CentOS installation, untar'ed them there (into /home/test7/rt_root), and used (as a normal user named "test7"):

unionfs -o dev,relaxed_permissions /home/test7/rt_root=RW:/=RW /home/test7/union_root

and then I can start (also as a normal user) rawtherapee using:

LD_LIBRARY_PATH=/lib64:/usr/lib64 unshare -r chroot union_root /usr/bin/rawtherapee

This very conveniently solves my use case (without requiring any re-compilation of RawTherapee under some older OS).

If I find the time, I'll try to wrap up my scripts into a piece of software that then could be used for just any binary, provided the kernel is "new enough" on the target system.

And of course it may also be a good idea to use something like https://github.com/solidsnack/arx to make the archive self-extracting / runnable.

[probonopd]

Well, you can throw this into an directory, write a bash script called AppRun that does the unionfs magic and launches the application, run appimagetool on it and have a working AppImage...

In fact, we'd be very interested in this.

[lvml]

@probonopd: Yes, that could make is unnecessary to unpack the archive on the target, just mounting the squashfs image instead. Will try that.

(Of course this will also require me to create statically linked versions of "unionfs" and "unshare" - but those should not be hard to create.)

BTW: The rt_root directory I assembled isn't that big, despite containing even libc, libstdc++ and their related data files - 210MB unpacked, 50MB tar.xz'ed - that space is really worth not having the hassle of trying to compile that software on the old distro ;-)

[probonopd]

unionfs can go into the AppImage as well, so that it doesn't have to be there on the target system...