OSDOCS#5126: Using an Azure managed identity as an alternative to a s… by mjpytlak · Pull Request #62875 · openshift/openshift-docs

mjpytlak · 2023-07-27T18:42:49Z

Version(s):
4.14+

Issue:

This issues addresses osdocs-5126.

Link to docs preview:

IPI

Configuring an Azure account > Recording the subscription and tenant IDs
Configuring an Azure account > Supported identities to access Azure resources
Configuring an Azure account > Supported identities to access Azure resources > Required Azure permissions for installer-provisioned infrastructure
Configuring an Azure account > Supported identities to access Azure resources > Using Azure managed identities
Configuring an Azure account > Supported identities to access Azure resources > Creating a service principal
Installing a cluster quickly on Azure > Deploying the cluster [1]
Installing a cluster on Azure with customizations > Creating the installation configuration file [2]
Installing a private cluster on Azure > Deploying the cluster [3]

UPI

Installing a cluster on Azure using ARM templates > Recording the subscription and tenant IDs
Installing a cluster on Azure using ARM templates > Supported identities to access Azure resources
Installing a cluster on Azure using ARM templates > Required Azure permissions for user-provisioned infrastructure
Installing a cluster on Azure using ARM templates > Using Azure managed identities
Installing a cluster on Azure using ARM templates > Creating a service principal
Installing a cluster on Azure using ARM templates > Creating the installation configuration file

QE review:

QE has approved this change.

Additional information:

This is is 1 of 2 doc PRs covering how to deploy an Azure cluster using a managed identity. This PR addresses the concepts and procedures associated with meeting the minimum required Azure permissions to install the cluster, the ability to use either a service principal or managed identity to complete the installation, and how to interact with the installation program survey when using a system-assigned managed identity or a user-assigned managed identity.

The concepts and procedures associated with using the CCO to install a cluster using a managed identity will be addressed by the second PR. @jeana-redhat is covering the latter work.

[1] This section of the documentation represents a default installation. In this use case, an admin can use the installation program to deploy a cluster using the default values -- without explicitly creating an install-config.yaml file. It is at this point of the installation that an admin must account for a service principal or managed identity.

[2] This section of the documentation appears in 3 other installation use cases (assemblies). In this use case, an admin can use the installation program to create the install-config.yaml file. It is at this point in the installation that an admin must account for a service principal or managed identity.

[3] This section of the documentation appears in 1 other installation use case (assembly). In this use case, an admin must manually create the install-config.yaml file and reference it when running the installation program. Because the installation configuration file was manually created, it is at this point during the installation process that an admin must account for a service principal or managed identity.

ocpdocs-previewbot · 2023-07-27T19:05:29Z

🤖 Updated build preview is available at:
https://62875--docspreview.netlify.app

Build log: https://circleci.com/gh/ocpdocs-previewbot/openshift-docs/22675

mjpytlak

@r4f4 We have two installation use cases where the guidance is to create the install-config.yaml file manually [1] . A private cluster and a cluster being installed into a government region. in these cases, I am wondering if we always had a doc gap from the perspective of how an admin would define a service principal, and with the introduction of managed identities, the client ID of a user-assigned managed identity. In these cases, would be explicitly have to tell users to make sure that an osServicePrincipal.json file was present on the machine where the installation program is being run? Or in these cases, when a user runs ./openshift-install create cluster -- dir <location of manually configured install config file> will the installer prompt for the missing service principal information?

Or by the very nature of these use cases, is a service principal not required?

[1] https://docs.openshift.com/container-platform/4.13/installing/installing_azure/installing-azure-government-region.html#installation-initializing-manual_installing-azure-government-region

mjpytlak · 2023-07-31T21:16:09Z

+}
+----
+
+. Record the `tenantId` and `id` parameter values from the output. You need these values during the {product-title} installation.


As part of our demo this week @r4f4 , let's use part of the time to revisit this procedure. I am presently reusing existing content from the published docs [1], but even without access to an account and the Azure CLI, something does not feel correct about it. For example, steps 2a and 2b in the published output seem identical.
If possible, it would be good to tighten this writing up while we are here.

[1] https://docs.openshift.com/container-platform/4.13/installing/installing_azure/installing-azure-account.html#installation-azure-service-principal_installing-azure-account

I guess they are the same in this case because there is only one account/subscription involved. Here is an example output where 2a would return more than one result:

$ az account list [ { "cloudName": "WWT", "id": "<Redacted Id1>", "isDefault": true, "name": "Redhat-2", "state": "Enabled", "tenantId": "<Redacted T1>", "user": { "name": "XXX@wwtatc.com", "type": "user" } }, { "cloudName": "WWT", "id": "<Redacted Id2>", "isDefault": false, "name": "Redhat", "state": "Enabled", "tenantId": "<Redacted T1>", "user": { "name": "XXX@wwtatc.com", "type": "user" } } ]

But 2b shows:

$ az account show { "environmentName": "WWT", "id": "<Redacted Id1>", "isDefault": true, "name": "Redhat-2", "state": "Enabled", "tenantId": "<Redacted T1>", "user": { "name": "XXX@wwtatc.com", "type": "user" } }

mjpytlak · 2023-07-31T21:17:30Z

+... Depending on the Azure identity you are using to deploy the cluster, do one of the following when prompted for the *azure service principal client secret*:
+**** If you are using a service principal, enter its password.
+**** If you are using a system-assigned managed identity, leave this value blank.
+**** If you are using a user-assigned managed identity,leave this value blank.


Upon a closer look, I am not as concerned about the user experience as I was when I first reached out to you @r4f4 , but let's take a look at this to make sure I have it correct.

r4f4 · 2023-07-31T22:18:23Z

@r4f4 We have two installation use cases where the guidance is to create the install-config.yaml file manually [1] . A private cluster and a cluster being installed into a government region. in these cases, I am wondering if we always had a doc gap from the perspective of how an admin would define a service principal, and with the introduction of managed identities, the client ID of a user-assigned managed identity. In these cases, would be explicitly have to tell users to make sure that an osServicePrincipal.json file was present on the machine where the installation program is being run?

Or by the very nature of these use cases, is a service principal not required?

[1] https://docs.openshift.com/container-platform/4.13/installing/installing_azure/installing-azure-government-region.html#installation-initializing-manual_installing-azure-government-region

Those cases are not a problem since the Installer will always ask for authentication info if it cannot find $HOME/.azure/osServicePrincipal.json. Here is an example of a run with an existing install-config.yaml:

$ openshift-install create manifests --log-level debug --dir /tmp/azure/ocp4-workdir-pub/
DEBUG OpenShift Installer unreleased-master-8464-ga9971cff1d02bd365376b87eb786b1f513b94e5c
DEBUG Built from commit a9971cff1d02bd365376b87eb786b1f513b94e5c
DEBUG Fetching Master Machines...
DEBUG Loading Master Machines...
DEBUG   Loading Cluster ID...
DEBUG     Loading Install Config...
DEBUG       Loading SSH Key...
DEBUG       Loading Base Domain...
DEBUG         Loading Platform...
DEBUG       Loading Cluster Name...
DEBUG         Loading Base Domain...
DEBUG         Loading Platform...
DEBUG       Loading Networking...
DEBUG         Loading Platform...
DEBUG       Loading Pull Secret...
DEBUG       Loading Platform...
INFO Asking user to provide authentication info
? azure subscription id REDACTED
? azure tenant id REDACTED
? azure service principal client id REDACTED
? azure service principal client secret [? for help]
INFO Saving user credentials to "$HOME/.azure/osServicePrincipal.json"
INFO Credentials loaded from file "$HOME/.azure/osServicePrincipal.json"
[...]

mjpytlak · 2023-08-03T15:46:51Z

 .. Select the AWS region to deploy the cluster to.
 .. Select the base domain for the Route 53 service that you configured for your cluster.
 endif::aws[]
-ifdef::azure,ash[]


Worth noting that I removed the ash attribute from this content as I incorrectly introduced it when documenting the ASH installation procedures [1]. ASH does not have an install use case where the installation program is being run without an install-config.yaml file. This specific set of content is used for use cases (typically default installation) where an admin runs the installation program without an install-config.yaml file.
[1] e12ad15#diff-86f98220fbab62878a702dcf4ce45ed0c5cd235192d0a79bd889cff480811d2eR244

mjpytlak · 2023-08-03T16:28:38Z

+* If you are installing the cluster using a user-assigned managed identity, you have met these prerequisites:
+** You have its client ID.
+** You have assigned it to the virtual machine that you will run the installation program from.
+endif::azure[]


Worth noting that #37275 previously added the service principal req specifically for Azure, but did not add the azure attribute. As a result, the req appeared in all install use cases using this module. I removed the nutanix attribute, as I incorrectly added it (instead of azure) when I wrote the Nutanix doc [1] Correcting this oversight now.

[1] https://github.com/openshift/openshift-docs/pull/44537/files#diff-aa2bf98d200d7bbc6fbc805b89a39df30e54112c533f035e025e1160cfeeecd9R202

mjpytlak · 2023-08-04T17:58:44Z

@r4f4 Appreciate the help earlier this week. Docs are ready for SME review.

CC @jeana-redhat

r4f4

/lgtm
Thoroughly documented. I've just left a couple of nitpick comments.

r4f4 · 2023-08-10T15:25:52Z

+  {
+    "cloudName": "AzureCloud",
+    "id": "9xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
+    "isDefault": true,


Nit: I think only one can be the default.

r4f4 · 2023-08-10T15:31:23Z

+... Depending on the Azure identity you are using to deploy the cluster, do one of the following when prompted for the *azure service principal client secret*:
+**** If you are using a service principal, enter its password.
+**** If you are using a system-assigned managed identity, leave this value blank.
+**** If you are using a user-assigned managed identity,leave this value blank.


Suggested change

**** If you are using a user-assigned managed identity,leave this value blank.

**** If you are using a user-assigned managed identity, leave this value blank.

mjpytlak · 2023-08-10T16:25:34Z

@jinyunma Docs are ready for QE review. PTAL. Please keep in mind:

This is is 1 of 2 doc PRs covering how to deploy an Azure cluster using a managed identity. This PR addresses the concepts and procedures associated with meeting the minimum required Azure permissions to install the cluster, the ability to use either a service principal or managed identity to complete the installation, and how to interact with the installation program survey when using a system-assigned managed identity or a user-assigned managed identity.

The concepts and procedures associated with using the CCO to install a cluster using a managed identity will be addressed by the second PR. @jeana-redhat is covering the latter work.

r4f4

/lgtm

jinyunma · 2023-08-11T03:12:23Z

Installing a cluster quickly on Azure > Deploying the cluster [1]

I don't think that installer with managed identity can be added in doc Installing a cluster quickly on Azure, since it requires to customize install-config file to set credentialsMode as Manual after generating install-config.yaml file.

And this setting seems also not mentioned in other docs?

mjpytlak · 2023-08-11T11:44:12Z

Installing a cluster quickly on Azure > Deploying the cluster [1]

I don't think that installer with managed identity can be added in doc Installing a cluster quickly on Azure, since it requires to customize install-config file to set credentialsMode as Manual after generating install-config.yaml file.

You are absolutely right @jinyunma. Looks like I got carried away. Great catch. I am off today, but will update the docs on Monday. I will reach out when the update is ready to review.

And this setting seems also not mentioned in other docs?

Setting credentialsMode to Manual will be covered by the PR that address the concepts and procedures associated with using the CCO to install a cluster using a managed identity. That work is being tracked here. https://issues.redhat.com/browse/OSDOCS-5128

openshift-ci · 2023-08-14T21:19:36Z

New changes are detected. LGTM label has been removed.

mjpytlak · 2023-08-15T00:00:41Z

@jinyunma I updated Installing a cluster quickly on Azure > Deploying the cluster to reflect that only a service principal can be used when deploying a cluster with the default configuration setting. PTAL.

bscott-rh

Looks great. Left a few small comments for your consideration.

bscott-rh · 2023-08-15T14:13:38Z

+The installation program requires an Azure identity to complete the installation. You can use a service principal.
+
+Alternatively, you can use a managed identity.


Suggested change

The installation program requires an Azure identity to complete the installation. You can use a service principal.

Alternatively, you can use a managed identity.

The installation program requires an Azure identity to complete the installation. You can use a service principal or a managed identity.

minimalism suggestion

@bscott-rh I struggled with this abstract a bit. Given these steps are for creating the service principal, I was going to go with your suggestion originally, but thought it odd to "directly" mention a managed identity. Maybe something along the lines of the following?

If you are unable to use a service principal, you can use a managed identity.

The goal here is to simply alert users to the other option if they land in this part of the docs, without reading Supported identities to access Azure resources.

I think that's a great middle ground if we're "preferring" the service principal route.

bscott-rh · 2023-08-15T14:52:36Z

+The installation program requires an Azure identity to complete the installation. You can use either a system-assigned or user-assigned managed identity.
+
+Alternatively, you can use a service principal.


Suggested change

The installation program requires an Azure identity to complete the installation. You can use either a system-assigned or user-assigned managed identity.

Alternatively, you can use a service principal.

The installation program requires an Azure identity to complete the installation. You can use a system-assigned managed identity, a user-assigned managed identity, or a service principal.

minimalism suggestion

Same comment as #62875 (comment)

bscott-rh · 2023-08-15T14:55:48Z

 = Required Azure permissions for installer-provisioned infrastructure

-When you assign `Contributor` and `User Access Administrator` roles to the service principal, you automatically grant all the required permissions.
+The installation program requires access to an Azure service principal or managed identity with the necessary permissions to deploy the the cluster and to maintain the daily operation of it. These permissions must be granted on the Azure subscription that is associated with the identity.


Suggested change

The installation program requires access to an Azure service principal or managed identity with the necessary permissions to deploy the the cluster and to maintain the daily operation of it. These permissions must be granted on the Azure subscription that is associated with the identity.

The installation program requires access to an Azure service principal or managed identity with the necessary permissions to deploy the the cluster and to maintain its daily operation. These permissions must be granted to the Azure subscription that is associated with the identity.

Rewording suggestion

Thanks and while I was here, I removed the repeated instance of the that shockingly your eagle-eye missed. 😄

bscott-rh · 2023-08-15T14:56:54Z

 = Required Azure permissions for user-provisioned infrastructure

-When you assign `Contributor` and `User Access Administrator` roles to the service principal, you automatically grant all the required permissions.
+The installation program requires access to an Azure service principal or managed identity with the necessary permissions to deploy the the cluster and to maintain the daily operation of it. These permissions must be granted on the Azure subscription that is associated with the identity.


Suggested change

The installation program requires access to an Azure service principal or managed identity with the necessary permissions to deploy the the cluster and to maintain the daily operation of it. These permissions must be granted on the Azure subscription that is associated with the identity.

The installation program requires access to an Azure service principal or managed identity with the necessary permissions to deploy the the cluster and to maintain its daily operation. These permissions must be granted to the Azure subscription that is associated with the identity.

Rewording suggestion

Thanks. Took care of this one.

jinyunma · 2023-08-16T07:39:45Z

Setting credentialsMode to Manual will be covered by the PR that address the concepts and procedures associated with using the CCO to install a cluster using a managed identity. That work is being tracked here. https://issues.redhat.com/browse/OSDOCS-5128

From OSDOCS-5128, seems it mostly talks about how to use ccoctl to generate workload managed identity for operators.

credentialsMode should be set to Manual when installer with managed identity.
I think it's better to mention it in all impacted install docs. User can choose to use service principal or workload managed identity on demands to generate secrets for those CredentialRequest when credentialsMode: Manual.
WDYT?

mjpytlak · 2023-08-16T16:24:42Z

I think it's better to mention it in all impacted install docs. User can choose to use service principal or workload managed identity on demands to generate secrets for those CredentialRequest when credentialsMode: Manual.
WDYT?

@jinyunma I should have provided additional details, and I referenced the wrong Jira issue (osdocs-5128) . @jeana-redhat and I have been working closely together on the docs strategy for this feature because it involves updates to both the installer-specific docs (me) and using the CCO utility to create the required manifests (Jeana). We determined that the logical point to break the installer-specific content was what is included in this PR. As detailed in https://issues.redhat.com/browse/OSDOCS-6135, the plan is to include all of the additional CCO steps, including that of setting credentialsMode: Manual, to all of the affected install docs. As detailed in the preview they provided in osdocs-6135, the additional content will be provided in a step-wise fashion.

With the latter in mind, Jeana is going to tag you as part of QE review for their work.

mjpytlak · 2023-08-21T13:51:06Z

@jinyunma With the latter in mind, did you have any additional feedback or are the draft docs good to go? I would like to get this merged, so Jeana can pick up my work.

jinyunma · 2023-08-23T01:14:03Z

@mjpytlak thanks for your detailed explanation.
LGTM

…ervice principal

mjpytlak · 2023-08-24T14:40:01Z

/cherrypick enterprise-4.14

openshift-cherrypick-robot · 2023-08-24T14:40:49Z

@mjpytlak: new pull request created: #63910

Details

In response to this:

/cherrypick enterprise-4.14

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

mjpytlak added the branch/enterprise-4.14 label Jul 27, 2023

mjpytlak added this to the Planned for 4.14 GA milestone Jul 27, 2023

openshift-ci Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 27, 2023

mjpytlak force-pushed the osdocs-5126 branch from 00414c2 to 392dde3 Compare July 31, 2023 21:39

mjpytlak commented Jul 31, 2023

View reviewed changes

mjpytlak force-pushed the osdocs-5126 branch 2 times, most recently from 1d71899 to 6a5ff5a Compare August 2, 2023 16:30

mjpytlak commented Aug 3, 2023

View reviewed changes

mjpytlak force-pushed the osdocs-5126 branch 4 times, most recently from e7c14f1 to da0abd5 Compare August 4, 2023 17:46

r4f4 reviewed Aug 10, 2023

View reviewed changes

openshift-ci Bot assigned r4f4 Aug 10, 2023

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 10, 2023

mjpytlak force-pushed the osdocs-5126 branch from da0abd5 to fef83f1 Compare August 10, 2023 16:14

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Aug 10, 2023

r4f4 reviewed Aug 10, 2023

View reviewed changes

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 10, 2023

mjpytlak force-pushed the osdocs-5126 branch from fef83f1 to b85eb90 Compare August 14, 2023 21:19

openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Aug 14, 2023

mjpytlak force-pushed the osdocs-5126 branch from b85eb90 to 75aad3b Compare August 14, 2023 21:49

mjpytlak added the peer-review-needed Signifies that the peer review team needs to review this PR label Aug 15, 2023

bscott-rh added peer-review-in-progress Signifies that the peer review team is reviewing this PR and removed peer-review-needed Signifies that the peer review team needs to review this PR labels Aug 15, 2023

mjpytlak force-pushed the osdocs-5126 branch from 75aad3b to 2bf81b0 Compare August 15, 2023 13:33

bscott-rh approved these changes Aug 15, 2023

View reviewed changes

mjpytlak force-pushed the osdocs-5126 branch 2 times, most recently from 539cef4 to f09f2de Compare August 24, 2023 12:49

OSDOCS#5126: Using an Azure managed identity as an alternative to a s…

4366dd7

…ervice principal

bscott-rh added peer-review-done Signifies that the peer review team has reviewed this PR and removed peer-review-in-progress Signifies that the peer review team is reviewing this PR labels Aug 24, 2023

mjpytlak force-pushed the osdocs-5126 branch from f09f2de to 4366dd7 Compare August 24, 2023 13:03

mjpytlak merged commit dbdd3e5 into openshift:main Aug 24, 2023

openshift-cherrypick-robot mentioned this pull request Aug 24, 2023

[enterprise-4.14] OSDOCS#5126: Using an Azure managed identity as an alternative to a s… #63910

Merged

xenolinux mentioned this pull request Oct 10, 2023

OCPBUGS#11410: Add required Azure AD permission #65911

Merged

1 task

	**** If you are using a user-assigned managed identity,leave this value blank.
	**** If you are using a user-assigned managed identity, leave this value blank.

		The installation program requires an Azure identity to complete the installation. You can use a service principal.

		Alternatively, you can use a managed identity.

		The installation program requires an Azure identity to complete the installation. You can use either a system-assigned or user-assigned managed identity.

		Alternatively, you can use a service principal.

	The installation program requires access to an Azure service principal or managed identity with the necessary permissions to deploy the the cluster and to maintain the daily operation of it. These permissions must be granted on the Azure subscription that is associated with the identity.
	The installation program requires access to an Azure service principal or managed identity with the necessary permissions to deploy the the cluster and to maintain its daily operation. These permissions must be granted to the Azure subscription that is associated with the identity.

Conversation

mjpytlak commented Jul 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ocpdocs-previewbot commented Jul 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mjpytlak left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjpytlak Jul 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

r4f4 Jul 31, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

r4f4 commented Jul 31, 2023

Uh oh!

mjpytlak Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjpytlak Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjpytlak commented Aug 4, 2023

Uh oh!

r4f4 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjpytlak commented Aug 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

r4f4 left a comment

Choose a reason for hiding this comment

Uh oh!

jinyunma commented Aug 11, 2023

Uh oh!

mjpytlak commented Aug 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci Bot commented Aug 14, 2023

Uh oh!

mjpytlak commented Aug 15, 2023

Uh oh!

bscott-rh left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mjpytlak Aug 23, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jinyunma commented Aug 16, 2023 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mjpytlak commented Jul 27, 2023 •

edited

Loading

ocpdocs-previewbot commented Jul 27, 2023 •

edited

Loading

mjpytlak left a comment •

edited

Loading

mjpytlak Jul 31, 2023 •

edited

Loading

r4f4 Jul 31, 2023 •

edited

Loading

mjpytlak Aug 3, 2023 •

edited

Loading

mjpytlak Aug 3, 2023 •

edited

Loading

mjpytlak commented Aug 10, 2023 •

edited

Loading

mjpytlak commented Aug 11, 2023 •

edited

Loading

mjpytlak Aug 23, 2023 •

edited

Loading

jinyunma commented Aug 16, 2023 •

edited by openshift-ci Bot

Loading

mjpytlak commented Aug 16, 2023 •

edited by openshift-ci Bot

Loading