pj-rehearse did not create CM for new template

[petr-muller]

openshift/release#4938 added a new job together with a new template used by this new job. The rehearsal created for this job got stuck on failing to mount the template CM:

MountVolume.SetUp failed for volume "job-definition" : configmaps "rehearse-template-cluster-launch-installer-ipi-e2e-dd409aae" not found

The pj-rehearse output suggests that the CM was not even created:

time="2019-09-12T16:19:57Z" level=info msg="Rehearsing Prow jobs for a configuration PR" pr=4938
time="2019-09-12T16:20:00Z" level=info msg="templates changed" pr=4938 templates="[{ci-operator/templates/openshift/installer/cluster-launch-installer-ipi-e2e.yaml dd409aae952175ffec1a2c6f38822f919a85e1ab}]"
time="2019-09-12T16:20:00Z" level=info msg="Job has been chosen for rehearsal" diffs=" .Agent:   a: ''   b: 'kubernetes'" job-name=pull-ci-openshift-installer-master-e2e-ipi pr=4938 repo=openshift/installer
time="2019-09-12T16:20:00Z" level=info msg="Created a rehearsal job to be submitted" pr=4938 rehearsal-job=rehearse-4938-pull-ci-openshift-installer-master-e2e-ipi target-job=pull-ci-openshift-installer-master-e2e-ipi target-repo=openshift/installer
time="2019-09-12T16:20:00Z" level=info msg="Submitting a new prowjob." job=rehearse-4938-pull-ci-openshift-installer-master-e2e-ipi name=2b7aacd1-d579-11e9-9688-0a58ac100856 org=openshift pr=4938 repo=release type=presubmit
time="2019-09-12T16:20:00Z" level=info msg="Submitted rehearsal prowjob" job=rehearse-4938-pull-ci-openshift-installer-master-e2e-ipi name=2b7aacd1-d579-11e9-9688-0a58ac100856 org=openshift pr=4938 repo=release type=presubmit 

I have copied the commit from the problematic PR here for reference:
petr-muller/release@d69407a

[bbguimaraes]

The template needs to be known by updateconfig. The manner in which rehearse fails is not the clearest, but it at least signals an invalid configuration.

[petr-muller]

@bbguimaraes hm, so we do not need updateconfig for actually detecting the change (because pj-rehearse detected the template change, see the second line), but we do for creating the CM? We should at least fail better...

[bbguimaraes]

Correct, we use updateconfig to translate the template files to ConfigMaps. The side effect of a template not being present in the configuration is it not being created.

Technically, we reproduce production perfectly, but I agree this is not ideal. It's not even the first time a user finds that "bug".

[bbguimaraes]

For reference, this is the relevant code:

ci-tools/pkg/config/template.go

Lines 149 to 162 in 3be938f

	func (c *TemplateCMManager) createCMs(sources []ConfigMapSource, mapping map[string]string) error {
	changes, err := genChanges(c.releaseRepoPath, sources)
	if err != nil {
	return err
	}
	var errs []error
	for cm, data := range updateconfig.FilterChanges(replaceSpecNames(c.namespace, c.configUpdaterCfg, mapping), changes, c.logger) {
	c.logger.WithFields(logrus.Fields{"cm-name": cm.Name}).Info("creating rehearsal configMap")
	if err := c.createCM(cm.Name, data); err != nil {
	errs = append(errs, err)
	}
	}
	return kutilerrors.NewAggregate(errs)
	}

ci-tools/pkg/config/template.go

Lines 89 to 106 in 3be938f

	func (c *TemplateCMManager) createCM(name string, data []updateconfig.ConfigMapUpdate) error {
	cm := &v1.ConfigMap{
	ObjectMeta: metav1.ObjectMeta{
	Name: name,
	Labels: map[string]string{
	createByRehearse: "true",
	rehearseLabelPull: strconv.Itoa(c.prNumber),
	},
	},
	Data: map[string]string{},
	}
	if _, err := c.cmclient.Create(cm); err != nil && !kerrors.IsAlreadyExists(err) {
	return err
	} else if err := updateconfig.Update(osFileGetter{root: c.releaseRepoPath}, c.cmclient, cm.Name, "", data, nil, c.logger); err != nil {
	return err
	}
	return nil
	}

As I said, this "issue" is just us simulating production, not due to negligence. If we want to add more validation, that would be a whole new layer.