OTA-740: ci-operator/step-registry/ipi/conf/telemetry: Disable Telemetry (#32153)

* ci-operator/step-registry/ipi/conf/telemetry: Disable Telemetry

The OCM team is concerned about the volume of data submitted by
ephemeral CI clusters.  This commit disables Telemetry by default (for
the bulk of CI, which flows through this step), while still allowing
Telemetry to be enabled for jobs that have important data to report,
or jobs that want to excersise the Telemetry or Insights reporting
logic.

From [1]:

> You can modify your existing global cluster pull secret to disable
> remote health reporting. This disables both Telemetry and the
> Insights Operator.

In 4.12, the Insights operator is growing a new configuration
structure that allows disabling Insights [1,2] without affecting
Telemetry.  And it turns out that the monitoring operator has a
similar option to disable Telemetry [3], although the only documented
use for telemeterClient is for setting nodeSelector [4].

There are a number of existing steps poking at the
cluster-monitoring-config ConfigMap, including
ipi-conf-inframachineset setting telemeterClient's nodeSelector, so I
am following ipi-conf-user-workload-monitoring's use of yq patching
and manifest naming to fit in with the other steps.

I'm following 8a4696c (using yq installed from upi-installer image,
2022-08-25, #31256) to get yq from the upi-installer image instead of
curling it down dynamically.  With the 4.12 pin to get `yq`, we're
safe using `upi-installer`, even for CI jobs that are otherwise
looking at different 4.y.

[1]: https://github.com/openshift/enhancements/blob/ef85659d01738b9f89958d5f0da31cff05bb1182/enhancements/insights/insights-config-api.md
[2]: https://docs.openshift.com/container-platform/4.11/support/remote_health_monitoring/opting-out-of-remote-health-reporting.html
[3]: https://github.com/openshift/cluster-monitoring-operator/blob/8d331d78b22948d36c20da0552763ddd8a4e2093/pkg/manifests/config.go#L337
[4]: https://docs.openshift.com/container-platform/4.11/monitoring/configuring-the-monitoring-stack.html#moving-monitoring-components-to-different-nodes_configuring-the-monitoring-stack

* ci-operator/step-registry: Inject ipi-conf-telemetry after ipi-conf

Generated with:

  $ sed -i 's/^\( *- ref: ipi-conf\)$/\1\n\1-telemetry/' $(git grep -l '^ *- ref: ipi-conf$' ci-operator/step-registry)

to slot the new step in after the generic ipi-conf, so we can
configure Telemetry in all of the existing chains and workflows that
were flowing through ipi-conf.

* ci-operator/config/openshift/telemeter: TELEMETER_ENABLED=true

When making changes to the Telemeter client, we want to ensure that
we're still excercising uploads.

Author: wking

ci-operator/config/openshift/telemeter/openshift-telemeter-master.yaml

@@ -55,10 +55,14 @@ tests:
 - as: e2e-aws-ovn
   steps:
     cluster_profile: aws
+    env:
+      TELEMETRY_ENABLED: "true"
     workflow: openshift-e2e-aws
 - as: e2e-aws-upgrade
   steps:
     cluster_profile: aws
+    env:
+      TELEMETRY_ENABLED: "true"
     workflow: openshift-upgrade-aws
 - as: benchmark
   steps:

ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/byo-iam-role/provision/cucushift-installer-rehearse-aws-ipi-byo-iam-role-provision-chain.yaml

@@ -3,6 +3,7 @@ chain:
   steps:
     - ref: aws-provision-iam-role
     - ref: ipi-conf
+    - ref: ipi-conf-telemetry
     - ref: ipi-conf-aws-byo-iam-role
     - ref: ipi-conf-aws
     - ref: ipi-install-monitoringpvc

ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/byo-route53/provision/cucushift-installer-rehearse-aws-ipi-byo-route53-provision-chain.yaml

@@ -4,6 +4,7 @@ chain:
     - ref: aws-provision-vpc-shared
     - ref: aws-provision-route53-private-hosted-zone
     - ref: ipi-conf
+    - ref: ipi-conf-telemetry
     - ref: ipi-conf-aws-custom-vpc
     - ref: ipi-conf-aws-byo-route53
     - ref: ipi-conf-aws

ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/disconnected/private/provision/cucushift-installer-rehearse-aws-ipi-disconnected-private-provision-chain.yaml

@@ -6,6 +6,7 @@ chain:
     - ref: aws-provision-cco-manual-users-static
     - chain: mirror-images-by-oc-adm
     - ref: ipi-conf
+    - ref: ipi-conf-telemetry
     - ref: ipi-conf-mirror
     - ref: ipi-conf-aws-custom-vpc
     - ref: ipi-conf-manual-creds

ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/disconnected/private/sdn/provision/cucushift-installer-rehearse-aws-ipi-disconnected-private-sdn-provision-chain.yaml

@@ -6,6 +6,7 @@ chain:
   - ref: aws-provision-cco-manual-users-static
   - chain: mirror-images-by-oc-adm
   - ref: ipi-conf
+  - ref: ipi-conf-telemetry
   - ref: ipi-conf-mirror
   - ref: ipi-conf-aws-custom-vpc
   - ref: ipi-conf-manual-creds

ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/disconnected/private/sno/provision/cucushift-installer-rehearse-aws-ipi-disconnected-private-sno-provision-chain.yaml

@@ -6,6 +6,7 @@ chain:
     - ref: aws-provision-cco-manual-users-static
     - chain: mirror-images-by-oc-adm
     - ref: ipi-conf
+    - ref: ipi-conf-telemetry
     - ref: ipi-conf-mirror
     - ref: ipi-conf-aws-custom-vpc
     - ref: ipi-conf-manual-creds

ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/disconnected/private/techpreview/provision/cucushift-installer-rehearse-aws-ipi-disconnected-private-techpreview-provision-chain.yaml

@@ -6,6 +6,7 @@ chain:
     - ref: aws-provision-cco-manual-users-static
     - chain: mirror-images-by-oc-adm
     - ref: ipi-conf
+    - ref: ipi-conf-telemetry
     - ref: ipi-conf-mirror
     - ref: ipi-conf-aws-custom-vpc
     - ref: ipi-conf-manual-creds

ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/private/cco-manual-security-token-service/provision/cucushift-installer-rehearse-aws-ipi-private-cco-manual-security-token-service-provision-chain.yaml

@@ -2,6 +2,7 @@ chain:
   as: cucushift-installer-rehearse-aws-ipi-private-cco-manual-security-token-service-provision
   steps:
   - ref: ipi-conf
+  - ref: ipi-conf-telemetry
   - ref: aws-provision-vpc-shared
   - chain: aws-provision-bastionhost
   - ref: ipi-conf-aws-custom-vpc

ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/private/ovn/ipsec/provision/cucushift-installer-rehearse-aws-ipi-private-ovn-ipsec-provision-chain.yaml

@@ -2,6 +2,7 @@ chain:
   as: cucushift-installer-rehearse-aws-ipi-private-ovn-ipsec-provision
   steps:
     - ref: ipi-conf
+    - ref: ipi-conf-telemetry
     - ref: aws-provision-vpc-shared
     - chain: aws-provision-bastionhost
     - ref: ipi-conf-aws-custom-vpc

ci-operator/step-registry/cucushift/installer/rehearse/aws/ipi/private/provision/cucushift-installer-rehearse-aws-ipi-private-provision-chain.yaml

@@ -2,6 +2,7 @@ chain:
   as: cucushift-installer-rehearse-aws-ipi-private-provision
   steps:
     - ref: ipi-conf
+    - ref: ipi-conf-telemetry
     - ref: aws-provision-vpc-shared
     - chain: aws-provision-bastionhost
     - ref: ipi-conf-aws-custom-vpc