az ad sp create-for-rbac: flakiness

[ahmetb]

The following cmd sometimes returns error, but the actual AAD app is created (but not the SP):

$ az ad sp create-for-rbac --scopes /subscriptions/xxxxxx/resourcegroups/gggggg/providers/Microsoft.ContainerRegistry/registries/nnnn \
 --role Owner --password "$azure_client_secret" \
 --years 2 --name http://foo-bar-ahmetb/

ERROR: Resource 'ServicePrincipal_00e1c5b1-394e-48ca-a1b7-f39b77635928' does not exist or one of its queried reference-property objects are not present.

When I do az ad sp show --id http://foo-bar-ahmetb/, I got error:

ERROR: service principal 'http://foo-bar-ahmetb/' doesn't exist

but az ad app show --id http://foo-bar-ahmetb/ worked just fine.

So it looks like create-for-rbac probably is running into some replication delay issue or somethign else from AAD and therefore erroring out, possibly having created the AAD App but not the SP.

[ahmetb]

I can repro this better now, even az ad app show fails a few times before actually showing up.

It looks like an eventual consistency issue.

➜  ~ az ad sp create-for-rbac --scopes [...] --role Owner --password "$azure_client_secret" --years 2 --name http://ahmetb-test-1234
ERROR: Resource '2bf6c936-32f4-4074-89e2-e1a9f05573a9' does not exist
or one of its queried reference-property objects are not present.

➜  ~ az ad app show --id 'http://ahmetb-test-1234'
ERROR: Application 'http://ahmetb-test-1234' doesn't exist

➜  ~ az ad app show --id 'http://ahmetb-test-1234'
ERROR: Application 'http://ahmetb-test-1234' doesn't exist

➜  ~ az ad app show --id 'http://ahmetb-test-1234'
{
  "appId": "x-32f4-4074-89e2-e1a9f05573a9",
  "REDACTED": "REDACTED....."
}

[ahmetb]

Sometimes the error is like:

$ az ad sp create-for-rbac ...
ERROR: No matches in graph database for '375b3ba7-fb59-43dc-ac48-fe93cb807026'

in that case, returned ID doesn't show up in az ad app show --id at all.

[yugangw-msft]

There is a known replication limitation that the sp is created, but is not propagated to the pas system where the role assignment gets processed. The SLA is 2 minutes. create-for-rbac avoids this limitation by providing the AD server where the sp got created when submit the request to create the assignment
But your issue reveals what i see it first time that the ad object was not replicated within AAD system itself. Next time do you mind share the correlation id info (using --debug), I can submit an icm ticket for ad team to take look.

[colemickens]

Once again, I file an Issue only to find out Ahmet's reported it days earlier. 👍

@yugangw-msft This, among other issues with AAD, is causing a lot of friction for our users. Do you want to file ICMs against AAD, or do you mind if I go ahead and do it? I can repro the error Ahmet reports frequently.

I'm also seeing this error sometimes: No matches in graph database for 'e2a6833c-9453-47b0-8f06-41083189768d' which is being returned from getObjectsByObjectId call. But I'm not sure why you need that call anyway... when I wrote this flow in Go, it was just a POST (app), POST (sp), POST (rbac) and that was it.

I'm guessing it's the same consistency error inside Graph though, so probably also should result in an ICM.

[yugangw-msft]

@colemickens
Please go ahead to file icm ticket.
For getObjectsByObjectId, it is not used in create-for-rbac, but used in role assignment list to resolve the logical names of the principal from the object ids, including service principal or user principal. Not doing this, interactive users won't understand the assignees. This failure is caused by the same AAD issue.

[colemickens]

Thanks for the quick reply @yugangw-msft. I'll follow up with AAD via ICM.

[jmspring]

Seeing this quite regularly...

jims@dockeropolis:~$ az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/04f7ec88-8e28-41ed-8537-5e17766001f5"
Resource 'ServicePrincipal_6e519a4b-be36-4fc1-8117-855eb6721431' does not exist or one of its queried reference-property objects are not present.
jims@dockeropolis:~$ az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/04f7ec88-8e28-41ed-8537-5e17766001f5"
Resource '7615a9b8-3e78-41ef-b5d4-bf951c750d6f' does not exist or one of its queried reference-property objects are not present.

[colemickens]

@yugangw-msft I don't feel like we can afford to wait for Azure Active Directory to fix this. Is it possible to use the session key header on consecutive calls to Graph so that we can fix this flakiness for our users in a shorter time frame? Thanks.

[yugangw-msft]

@colemickens, I will get to it soon. Note, we already used session key on graph calls, so to mitigate it further we might have to use retry.
BTW, is there any pattern you have observed that it is easier to see a repro? Or it is just completely random?

[colemickens]

I can't tell a pattern, unfortunately. I can run it repeatedly back to back and it fails more often than not. At least, that was the case the last time I tried.

I didn't realize you were passing the session key in the graph calls as well as the arm/rbac call. That motivates me to raise the issue further internally. Thanks for letting me know.

Unfortunately, I think retrying might be a good idea short-term. Thanks again for all of the work here.

[yugangw-msft]

We need to get to the bottom of it.
I am injecting more trace information, so that on failures I will dump out the request-id/correlation-id as warning before raise the exception. If i can get a repro, I will go ahead and open an ICM ticket; otherwise I will still commit the tracing, so you can use it.

[jmspring]

For me as well, it fails near 100% of the time. When it does work, I can get 3-5 successful calls in a row. This is mostly on a ubuntu VM on a Mac on my home (based in CA) 50Mbps symmetric network connection.

…

-j

Sent from my iThingy

On Dec 6, 2016, at 10:14, Yugang Wang ***@***.***> wrote: We need to get to the bottom of it. I am injecting more trace information, so that on failures I will dump out the request-id/correlation-id as warning before raise the exception. If i can get a repro, I will go ahead and open an ICM ticket; otherwise I will still commit the tracing, so you can use it. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[yugangw-msft]

@jmspring, I have got a repro. It looks like being out of corpnet makes easier to repro. I have submitted a ICM ticket