Add Travis CI

[fscheiner]

This adds automated builds on Travis CI (on ppc64le arch for now) and supports the following tasks (see travis-ci/run_task_inside_docker.sh for details) for now:

• build (as there are no tests defined for UberFTP I renamed the tests task to build) - (test-)builds UberFTP
• tarballs(should maybe renamed to tarball) - creates the source tarball (make dist) for UberFTP

...and since today also:

• srpms
• rpms

Check possible results on https://app.travis-ci.com/github/fscheiner/UberFTP/builds

---

Based on @matyasselmeci's Travis-CI scripts from the GCT

[fscheiner]

@msalle, @matyasselmeci, @ellert:
Before implementing the srpms task (needed for uploads of source tarballs to repo.gridcf.org) and GitHub Actions (which will actually perform the uploads) we need to decide on a few things:

• 1. Should we adapt the /packaging/[...] directory structure of the GCT also for UberFTP? If yes, then we need to move the UberFTP RPM spec file there (i.e. /packaging/fedora/uberftp.spec)

Ok, I went for this "solution" now, see https://github.com/fscheiner/UberFTP/tree/automated-builds-continued, but I stumbled upon an issue I'll describe in a separate comment below.

UPDATE (2021-12-02): The following will be implemented in another PR.

- [ ] 2. Where should we put the UberFTP source tarball(s), into https://repo.gridcf.org/uberftp/sources? As long as it's separate from the GCT, https://repo.gridcf.org/gct6/sources makes no sense IMO.

[msalle]

I think I'm probably not the best to review this one. In any case, presuming it's mostly copy&paste from the rest of GCT, this seems all fine.
One thing: travis-ci/run_task_inside_docker.sh has an almost completely outdated list of platforms: CentOS6, F26 and F27 are all long out of support. OTOH we probably want RockyLinux, Almalinux or CentOS Stream 8 and currently supported Fedoras.

[fscheiner]

@msalle

One thing: travis-ci/run_task_inside_docker.sh has an almost completely outdated list of platforms: CentOS6, F26 and F27 are all long out of support. OTOH we probably want RockyLinux, Almalinux or CentOS Stream 8 and currently supported Fedoras.

I ignored thatt because I didn't want to change too much at once. Indeed, RockyLinux and Almalinux are long due for testing. but I'm unsure if they provide any Docker images for ppc64le (or even arm64/aarch64). If someone has experience with these, please let us know.

AFAIK CentOS Stream 8 is not "identical" to - and maybe not even binary compatible with - RHEL8, so I don't see it as a target for our build tests (i.e. something that will replace our CentOS7 target in the future). I'd rather go for one or both of the earlier mentioned OSes as target as they want to be what CentOS once was.

[msalle]

@msalle

One thing: travis-ci/run_task_inside_docker.sh has an almost completely outdated list of platforms: CentOS6, F26 and F27 are all long out of support. OTOH we probably want RockyLinux, Almalinux or CentOS Stream 8 and currently supported Fedoras.

I ignored thatt because I didn't want to change too much at once. Indeed, RockyLinux and Almalinux are long due for testing. but I'm unsure if they provide any Docker images for ppc64le (or even arm64/aarch64). If someone has experience with these, please let us know.

Both RockyLinux and AlmaLinux seem to have arm64, not ppc64le, in their repos and as docker images on docker hub. OTOH our reference platform is in any case amd64/x86_64.

AFAIK CentOS Stream 8 is not "identical" to - and maybe not even binary compatible with - RHEL8, so I don't see it as a target for our build tests (i.e. something that will replace our CentOS7 target in the future). I'd rather go for one or both of the earlier mentioned OSes as target as they want to be what CentOS once was.

The point is that different sites will be using different distros. CERN is probably going for CentOS Stream, Desy AFAIK for AlmaLinux and several others big sites for RockyLinux.
Building the GCT should be reasonably straightforward for all I guess.

[fscheiner]

Both RockyLinux and AlmaLinux seem to have arm64, not ppc64le, in their repos and as docker images on docker hub. OTOH our reference platform is in any case amd64/x86_64.

Yes, but for Travis-CI we can't use them as long as we don't have build credits, ppc64le and arm64 are not charged by Travis-CI as the machinery is provided by manufacturers.

The point is that different sites will be using different distros. CERN is probably going for CentOS Stream, Desy AFAIK for AlmaLinux and several others big sites for RockyLinux.

As long as it's compatible with each other and RHEL(8). Why CERN might want to use something different is out of my understanding.

@matyasselmeci
What will OSG use in the future as replacement for CentOS7 and why?

[fscheiner]

Both RockyLinux and AlmaLinux seem to have arm64, not ppc64le, in their repos and as docker images on docker hub.

Maybe IBM can convince the maintainers of RockyLinux and AlmaLinux to also support ppc64le? @gerrith3, @ezeeyahoo

[fscheiner]

@msalle
About the currently supported Fedoras: I don't know for sure, if our existing RPM spec files work unchanged on Fedora - here I mean the ones from the GCT, not the one from UberFTP whose spec file is not overly complex, so should probably work anywhere. I believe they differ from the ones @ellert is using over at Fedora, but I didn't compare them recently.

I mean, if we want to have test builds for UberFTP on multiple OSes, we should also implement that for the GCT.

...it was at this moment that he knew he had opened pandora's box... ;-)

[fscheiner]

Please ignore, as outdated with the latest changes.

---

Back to the original topic:
As per #16 (comment) I have a problem when running ./make_rpms.sh:

[johndoe@gridftp-5 travis-ci]$ ./make_rpms.sh
Wrote: /home/johndoe/git-projects/UberFTP/packaging/rpmbuild/SRPMS/uberftp-2.9-1.gct.src.rpm
*** Building uberftp ***
Error! Logs follow:
error: Failed build dependencies:
	globus-gssapi-gsi-devel is needed by uberftp-2.9-1.gct.x86_64
Installing /home/johndoe/git-projects/UberFTP/packaging/rpmbuild/SRPMS/uberftp-2.9-1.gct.src.rpm
Building target platforms: x86_64-linux
Building for target x86_64-linux

I don't want to make a PR for fscheiner@beeb89e until it works, so want to solve that issue first.

@matyasselmeci
In make_rpms.sh on line 107 I see that it removes all GCT packages (I extended that to also remove an installed UberFTP package) prior to the build:

[...]
remove_installed_gct_rpms()
{
    rpm -qa | egrep '^globus-|^myproxy|^uberftp' \
        | xargs --no-run-if-empty \
            sudo rpm -e $pkgs_to_rm
}
[...]

But how can this work if a build depends on a globus-*-devel package?

I know that run_task_inside_docker.sh should actually pre-install some build dependencies - I deactivated that and install the UberFTP deps via the standard packages list (see line 29) - but for example in the GCT this never installs any build requirements on globus* packages, though some spec files over at the GCT do require other globus* packages (e.g. globus-gridftp-server.spec). So I don't understand, why this could have worked for the GCT then. It doesn't work for UberFTP though.

UPDATE: Well, I just recognize that the deploy step only builds SRPMs, so building RPMs would not be needed for the upload of the tarball to repo.gridcf.org, but still it should work, as this was also meant to allow users to builds RPMs locally.

UPDATE2: I think I have a possible solution to make make_rpms.sh work as intended, see these changes in fscheiner@beeb89e for details.

[gerrith3]

Both RockyLinux and AlmaLinux seem to have arm64, not ppc64le, in their repos and as docker images on docker hub.

Maybe IBM can convince the maintainers of RockyLinux and AlmaLinux to also support ppc64le? @gerrith3, @ezeeyahoo

I will take a look but getting an entire distro supported can be difficult aka expensive. It may depend on where they are pulling their sources from - if they are using fedora or equivalent build environments that are copied into RockyLinux or AlmaLinux there may be a chance.

[fscheiner]

@gerrith3

I will take a look but getting an entire distro supported can be difficult aka expensive. It may depend on where they are pulling their sources from - if they are using fedora or equivalent build environments that are copied into RockyLinux or AlmaLinux there may be a chance.

Great to hear that, I assume both RockyLinux and AlmaLinux are using the official sources of RHEL8 (which supports ppc64le), i.e. like CentOS8 did until Red Hat changed that to CentOS Stream 8.

[msalle]

@msalle About the currently supported Fedoras: I don't know for sure, if our existing RPM spec files work unchanged on Fedora - here I mean the ones from the GCT, not the one from UberFTP whose spec file is not overly complex, so should probably work anywhere. I believe they differ from the ones @ellert is using over at Fedora, but I didn't compare them recently.

I mean, if we want to have test builds for UberFTP on multiple OSes, we should also implement that for the GCT.

...it was at this moment that he knew he had opened pandora's box... ;-)

I think that for simplicity it would probably then be better to just remove those completely. Building (or at least appearing to build?) something on those long outdated platforms doesn't make sense to me. And then you can keep the box closed for now (-;

[fscheiner]

I think that for simplicity it would probably then be better to just remove those completely. Building (or at least appearing to build?) something on those long outdated platforms doesn't make sense to me. And then you can keep the box closed for now (-;

I've done that with fscheiner@beeb89e and will incorporate that here soon.

[fscheiner]

@msalle:
Ok, all additional changes are now included, A successful build is available on https://app.travis-ci.com/github/fscheiner/UberFTP/builds/242419144. I'll now re-request your review if you don't mind. The review requests for @matyasselmeci and @ellert are still open, so if they like they can also review the complete PR.

[msalle]

I've left a view comments on things that I find a bit unclear. If it all works it is fine for me for the time being, but probably good to look at, at some point.

[matyasselmeci]

remove_all_gct_rpms() shouldn't be in there for the UberFTP build. It was in the GCT build to make sure we were only using the GCT binaries that we compiled, but that doesn't apply to UberFTP.

[gerrith3]

@gerrith3

I will take a look but getting an entire distro supported can be difficult aka expensive. It may depend on where they are pulling their sources from - if they are using fedora or equivalent build environments that are copied into RockyLinux or AlmaLinux there may be a chance.

Great to hear that, I assume both RockyLinux and AlmaLinux are using the official sources of RHEL8 (which supports ppc64le), i.e. like CentOS8 did until Red Hat changed that to CentOS Stream 8.

Hey @fscheiner I forgot to update you here but both RockyLinux and AlmaLinux are using our Power servers at OSU's OSL for testing, so both should have Power LE support (ppc64le).

[fscheiner]

@matyasselmeci

remove_all_gct_rpms() shouldn't be in there for the UberFTP build. It was in the GCT build to make sure we were only using the GCT binaries that we compiled, but that doesn't apply to UberFTP.

Thank you for the clarification. I had some thoughts about that and believe I understand it better now but correct me if I'm wrong: It works for the GCT, because - after all Globus related packages are removed - the build starts from the beginning (i.e. with globus-common which does not depend on any globus-* packages) and works through the build-dependency ordered list of packages (packaging/fedora/ORDERING) and builds and installs all RPMs in a single call to build_all_for_target() (from make_rpms.sh.

[fscheiner]

@gerrith3

Hey @fscheiner I forgot to update you here but both RockyLinux and AlmaLinux are using our Power servers at OSU's OSL for testing, so both should have Power LE support (ppc64le).

Thanks for the update, that's good news. Unfortunately both don't yet provide Docker images for ppc64le over at https://hub.docker.com/r/rockylinux/rockylinux/tags and https://hub.docker.com/_/almalinux?tab=tags but only for amd64 and arm64. Well, they currently also only offer installation disc images for amd64 and arm64 (https://rockylinux.org/download, https://mirrors.almalinux.org/isos.html) as I just found out, so I assume Docker images for ppc64le will eventually come.

[fscheiner]

@msalle:
My latest changes should address the comments from your last review.

Successful build here: https://app.travis-ci.com/github/fscheiner/UberFTP/builds/242806697

The addition of a GitHub Action/Worklfow for build tests and uploading source tarballs will be implemented in another PR.

So if there are no objections I'll merge these changes as is tomorrow.