Add support for copying yum repos from separate git repo

[saqibali-2k]

Add a new option --yumrepos option to cosa init to allow import of
yum repo files and content sets located outside of the source git.

Motivation: RHCOS needs yum repo definitions hidden behind a firewall.
Let's add a clean way to import them.

Co-authored-by: Jonathan Lebon jonathan@jlebon.com

[cgwalters]

Hmm. I'm OK with this, but IMO it also cuts against the "elegance" of the current architecture. For example, suddenly pull requests don't quite work because there are two git repositories involved. We're also today injecting metadata about the config git repository into the base container image. But that loses some meaning with two git repositories.

Related to that topic, this PR isn't injecting any metadata from (e.g. exact git rev) the "extconfig" git repo into the cosa metadata.

[jlebon]

Hmm. I'm OK with this, but IMO it also cuts against the "elegance" of the current architecture. For example, suddenly pull requests don't quite work because there are two git repositories involved. We're also today injecting metadata about the config git repository into the base container image. But that loses some meaning with two git repositories.

The use case for this is RHCOS, which sadly already suffers from this issue. The idea with this PR is essentially upstreaming and formalizing the cp hack that currently exists in the RHCOS pipeline into cosa.

Related to that topic, this PR isn't injecting any metadata from (e.g. exact git rev) the "extconfig" git repo into the cosa metadata.

👍 Yeah we should add that (and is actually another benefit of formalizing it in cosa directly).

[cgwalters]

Note: if we have lockfiles, then this extconfig repo degrades basically to a mirrorlist.

[jlebon]

Updated this now but still testing it locally!

[cgwalters]

CI failure looks legit

[jlebon]

Rebased and now tested locally!

[travier]

I think we should coordinate the changes in this one with those from #2934

[travier]

This is great. That will make things easier for SCOS!

[travier]

Thanks. This looks good to me. I would prefer another name for the option (repo, add(itional)-repo, private-repo, extra-repo?) but this is close to a bikeshed discussion so not blocking on that.

[travier]

The only concern I could find was about the additional repo becoming stale on developer systems. This will not be an issue in CI as we clone from scratch every time.

[travier]

One option to work around that is to include the minor RHEL version in the repo names thus the build would fail when the repos are missing.

[jlebon]

In a follow-up, I'll add support for symlinking. This will allow devs to point to their existing checkouts just like for the config repo, which should help counter that a bit.

Done in #3045.

[travier]

This likely broke RHCOS CI: openshift/os#959. Investigating:

+ cp src/config/repos/c9s.repo src/config/c9s.repo
+ curl -L http://base-4-12-rhel90.ocp.svc.cluster.local -o src/config/tmp.repo
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100   804  100   804    0     0   100k      0 --:--:-- --:--:-- --:--:--  112k
+ awk '/rhel-9-server-ose/,/^$/' src/config/tmp.repo
+ echo includepkgs=cri-o,cri-tools,openshift-clients,openshift-hyperkube
+ rm src/config/tmp.repo
+ cosa fetch
Config commit: v0.0.2-785-gf2b445d4302a85fc157526e766fe9c209158dbba
Using manifest: /tmp/cosa/src/config/manifest.yaml
ls: cannot access '/tmp/cosa/tmp/override/*.repo': No such file or directory
ERROR: no yum repo files were found
error: failed to execute cmd-fetch: exit status 1

[travier]

And I've realized late that this will also not work when we have two minor RHEL minor versions at the same time in the repo (9.0 & 9.2) as the repo names will be the same but not the URLs.

We thus have to make repo names minor version specific (this should also work for content sets) or rework this one.

[travier]

We also need to be able to specify a branch for the yumrepo git repo, unless we make all repo names explicitly include their version.

[travier]

Related docs update in openshift/os#958

[cgwalters]

We thus have to make repo names minor version specific (this should also work for content sets)

I'm in favor of this.

[travier]

I think the issue is here as this assumes that there is an overridesdir when there may not be

[jlebon]

Where this code is, we always create an overrides dir. ISTM like the issue OpenShift CI is hitting is the same that FCOS was hitting and should be fixed by 24ffe87 (#3045). Let's see what the CI rerun in openshift/os#959 says. :)

[jlebon]

And I've realized late that this will also not work when we have two minor RHEL minor versions at the same time in the repo (9.0 & 9.2) as the repo names will be the same but not the URLs.

We thus have to make repo names minor version specific (this should also work for content sets) or rework this one.

Yes, if the repo contents are different, we shouldn't be using the same repo ID. Otherwise we'll be claiming we're using the wrong content set for one of them. This is what this comment is about.

And this jogs my memory now. I said higher up:

We can't change the repo names unfortunately because they're used in the content set mapping.

This is not true. I was thinking instead I think of the fact we have to match the repo names of the mirrored versions in OpenShift CI, which I think we have some control over.

[travier]

Yes, we have control over the names in Prow CI. I'll make the PRs to change everything.