update from 1.9.0 to 1.9.1 causes kiota_authentication_azure._exceptions.HTTPError: Valid url scheme and host required

[cedriclecoz]

Hi all,

one of my CI jobs checking user status in azure started failing yesterday following upgrade of the kiota python packages.
I've pinned it back down to 1.9.0 so am not stuck but I'd like to know if I am doing something wrong in my script.

Environement:

• Linux and Mac.
• Python 3.12.5
  -- microsoft-kiota-abstractions==1.9.1
  -- microsoft-kiota-authentication-azure==1.9.1
  -- microsoft-kiota-http==1.9.1
  -- microsoft-kiota-serialization-form==1.9.1
  -- microsoft-kiota-serialization-json==1.9.1
  -- microsoft-kiota-serialization-multipart==1.9.1
  -- microsoft-kiota-serialization-text==1.9.1
  -- msgraph-core==1.1.2
  -- msgraph-sdk==1.5.4
  -- urllib3==2.2.2

Traceback (most recent call last):
  File "/Users/clecoz/work/cmf_lib/cmflib_main/cmflib/test_kiota.py", line 52, in <module>
    asyncio.run(_get_user_status('clecoz'))
  File "/Users/clecoz/.pyenv/versions/3.12.8/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/Users/clecoz/.pyenv/versions/3.12.8/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clecoz/.pyenv/versions/3.12.8/lib/python3.12/asyncio/base_events.py", line 686, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/Users/clecoz/work/cmf_lib/cmflib_main/cmflib/test_kiota.py", line 38, in _get_user_status
    response = await client.users.get(request_configuration=request_configuration)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clecoz/.pyenv/versions/ldap_3.12.8/lib/python3.12/site-packages/msgraph/generated/users/users_request_builder.py", line 72, in get
    return await self.request_adapter.send_async(request_info, UserCollectionResponse, error_mapping)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clecoz/.pyenv/versions/ldap_3.12.8/lib/python3.12/site-packages/kiota_http/httpx_request_adapter.py", line 186, in send_async
    response = await self.get_http_response_message(request_info, parent_span)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clecoz/.pyenv/versions/ldap_3.12.8/lib/python3.12/site-packages/kiota_http/httpx_request_adapter.py", line 595, in get_http_response_message
    await self._authentication_provider.authenticate_request(
  File "/Users/clecoz/.pyenv/versions/ldap_3.12.8/lib/python3.12/site-packages/kiota_abstractions/authentication/base_bearer_token_authentication_provider.py", line 50, in authenticate_request
    token = await self.access_token_provider.get_authorization_token(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/clecoz/.pyenv/versions/ldap_3.12.8/lib/python3.12/site-packages/kiota_authentication_azure/azure_identity_access_token_provider.py", line 78, in get_authorization_token
    raise exc
kiota_authentication_azure._exceptions.HTTPError: Valid url scheme and host required

Quick test script I wrote to investigate: following works with 1.9.0 but give above error with 1.9.1

#!/usr/bin/env python3

import asyncio
from msgraph import GraphServiceClient
from msgraph.generated.users.users_request_builder import UsersRequestBuilder
from msgraph.generated.groups.groups_request_builder import GroupsRequestBuilder
from azure.identity.aio import ClientSecretCredential



DEFAULT_GRAPH = 'https://graph.microsoft.com/.default'
DEFAULT_TENANT = 'tenant_redacted'

username='uname_redacted'
password='password_redacted'


credentials = ClientSecretCredential(DEFAULT_TENANT, username, password)
scopes=[DEFAULT_GRAPH]

client = GraphServiceClient(credentials=credentials, scopes=scopes)


async def _get_user_status(userid:str) -> dict:
    query_params = UsersRequestBuilder.UsersRequestBuilderGetQueryParameters(
        search=[
            f'("displayName:{userid}" OR "mail:{userid}" OR "userPrincipalName:{userid}"\
                    OR "givenName:{userid}" OR "surName:{userid}" OR "otherMails:{userid}")'
        ],
        filter="accountEnabled eq true",
    )
    request_configuration = (
        UsersRequestBuilder.UsersRequestBuilderGetRequestConfiguration(
            query_parameters=query_params,
        )
    )
    request_configuration.headers.add("ConsistencyLevel", "eventual")

    response = await client.users.get(request_configuration=request_configuration)
    if response.value:
        user = response.value[0]
        print(f"{userid} found in AD. Principal: {user.user_principal_name}, "
                f"Id: {user.id}, Display: {user.display_name}")
        return {
                "display": user.display_name,
                "mail": user.mail,
                "id": user.id,
                "principal": user.user_principal_name
               }
    print(f"{userid} not found in AD")


asyncio.run(_get_user_status('clecoz'))

Thanks,
Cedric.

[adam-blanchard]

Hi everyone.

We're also facing this issue even with simple get user by id. Our data engineering pipelines started to fail today, and the only environment change I've identified is the upgrading of this dependency from 1.9.0 to 1.9.1.

Same env setup as Cedric, and the code below is resulting in HTTPError: Valid url scheme and host required .

I will try pinning the dependency to 1.9.0 as a workaround.

import asyncio
from azure.identity.aio import ClientSecretCredential
from msgraph import GraphServiceClient

TENANT_ID = '<REDACTED>'
CLIENT_ID = '<REDACTED>'
CLIENT_SECRET = '<REDACTED>'

credential = ClientSecretCredential(TENANT_ID, CLIENT_ID, CLIENT_SECRET)
scopes = ['https://graph.microsoft.com/.default']
client = GraphServiceClient(credentials=credential, scopes=scopes)

user = await client.users.by_user_id('<REDACTED>').get()

HTTPError: Valid url scheme and host required
File <command-3714762163119403>, line 1
----> 1 user = await client.users.by_user_id('<REDACTED>').get()
      2 print(user)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-333e6112-c98d-42d6-a5c5-76f6b0493c1c/lib/python3.11/site-packages/msgraph/generated/users/item/user_item_request_builder.py:160, in UserItemRequestBuilder.get(self, request_configuration)
    157     raise Exception("Http core is null") 
    158 from ...models.user import User
--> 160 return await self.request_adapter.send_async(request_info, User, error_mapping)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-333e6112-c98d-42d6-a5c5-76f6b0493c1c/lib/python3.11/site-packages/kiota_http/httpx_request_adapter.py:186, in HttpxRequestAdapter.send_async(self, request_info, parsable_factory, error_map)
    183     parent_span.record_exception(REQUEST_IS_NULL)
    184     raise REQUEST_IS_NULL
--> 186 response = await self.get_http_response_message(request_info, parent_span)
    188 response_handler = self.get_response_handler(request_info)
    189 if response_handler:
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-333e6112-c98d-42d6-a5c5-76f6b0493c1c/lib/python3.11/site-packages/kiota_http/httpx_request_adapter.py:595, in HttpxRequestAdapter.get_http_response_message(self, request_info, parent_span, claims)
    592 if claims:
    593     additional_authentication_context[self.CLAIMS_KEY] = claims
--> 595 await self._authentication_provider.authenticate_request(
    596     request_info, additional_authentication_context
    597 )
    599 request = self.get_request_from_request_information(
    600     request_info, _get_http_resp_span, parent_span
    601 )
    602 resp = await self._http_client.send(request)
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-333e6112-c98d-42d6-a5c5-76f6b0493c1c/lib/python3.11/site-packages/kiota_abstractions/authentication/base_bearer_token_authentication_provider.py:50, in BaseBearerTokenAuthenticationProvider.authenticate_request(self, request, additional_authentication_context)
     47     request.headers = HeadersCollection()
     49 if not request.headers.contains(self.AUTHORIZATION_HEADER):
---> 50     token = await self.access_token_provider.get_authorization_token(
     51         request.url, additional_authentication_context
     52     )
     53     if token:
     54         request.headers.add(f'{self.AUTHORIZATION_HEADER}', f'Bearer {token}')
File /local_disk0/.ephemeral_nfs/envs/pythonEnv-333e6112-c98d-42d6-a5c5-76f6b0493c1c/lib/python3.11/site-packages/kiota_authentication_azure/azure_identity_access_token_provider.py:78, in AzureIdentityAccessTokenProvider.get_authorization_token(self, uri, additional_authentication_context)
     76     exc = HTTPError("Valid url scheme and host required")
     77     span.record_exception(exc)
---> 78     raise exc
     80 if parsed_url.scheme != "https" and (parsed_url.hostname not in self.LOCALHOST_STRINGS):
     81     span.set_attribute(self.IS_VALID_URL, False)

[adam-blanchard]

I have just done some more digging and pinning the package microsoft-kiota-http to version 1.9.0 has resolved the issue. Looks like microsoft-kiota-http 1.9.1 introduced the bug.

[adam-blanchard]

Confirmed with my team that pinning microsoft-kiota-http to 1.9.0 is a functioning workaround.

[lucas-bremond]

On a similar vibe, from 1.9.1, the following

from azure.identity.aio import DefaultAzureCredential
from msgraph_core import AzureIdentityAuthenticationProvider

AzureIdentityAuthenticationProvider(
    credentials=DefaultAzureCredential(),
)

now raises a ValueError: Allowed host value cannot contain 'https://' or 'http://' prefix.

Freezing microsoft-kiota-abstractions==1.9.0 solves the issue...

[andrueastman]

Hey everyone,
After investigations, this issue is related to microsoftgraph/msgraph-sdk-python-core#818 together with changes made in #459.

In python 3.11, the formating of enum values to strings does not result in the enum value being set but the property name being used. This resulted in the httpclient.baseurl being set to something like NationalClouds.Global/APIVersion.v1/ rather than https://graph.microsoft.com/v1.0/. This change was unnoticed/inconsequenstial till the fix made in #459 that made the adapter use the base_url from the http client.

This issue was fortunately already been resolved in microsoftgraph/msgraph-sdk-python-core#820 and therefore the solution here it to ensure that your dependency tree has msgraph-core version 1.2.1 or greater.

https://github.com/microsoftgraph/msgraph-sdk-python-core/releases/tag/v1.2.1