Allow custom api versions in MSGraphAsyncOperator

[dabla]

We also use the MSGraphAsyncOperator to trigger Fabric pipelines, unfortunately the MS Fabric REST API doesn't follow the same versioning convention like MS Graph or PowerBI. In MS Graph and PowerBI, the version is v1.0, while in Fabric it is v1. So in order to be able to also invoke MS Fabric endpoints, we had to define the absolute URL instead of the relative one in order to make it work, as with relative url's we where unable to specify v1 as api_version. This PR solves this issue.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[dabla]

@potiuk Could this one also be merged? It allows using custom API versions, everything seems fine and already tested this solution in our local environment.

[potiuk]

Of course. It's just a metter of making tests pass.

[potiuk]

You will need to make some tests db -tests and apparently power-bi connection is not created in older versions of airlfow (why would they be - so some test compat code should be added - creating the connection to make the tests pass for 2.8/2.9.

[dabla]

You will need to make some tests db -tests and apparently power-bi connection is not created in older versions of airlfow (why would they be - so some test compat code should be added - creating the connection to make the tests pass for 2.8/2.9.

Still, weird those tests start failing now while those where merged by another PR.

[potiuk]

Still, weird those tests start failing now while those where merged by another PR.

What I see now is that "test_powerbi" fails - which it looks it needs "power_bi" connection from the DB - so it should be 'db test" - i am not sure if see other "merged" things that influence it ?

[dabla]

You will need to make some tests db -tests and apparently power-bi connection is not created in older versions of airlfow (why would they be - so some test compat code should be added - creating the connection to make the tests pass for 2.8/2.9.

Should I just create a Connection instance like for example in test_mssql.py at top of the file?

PYMSSQL_CONN = Connection(
    conn_type="mssql", host="ip", schema="share", login="username", password="password", port=8081
)

[dabla]

@potiuk I would also like to have this PR merged for the next providers release as this fixes also fixes proxy issue with PowerBIDatasetRefreshOperator and allows custom api versions to be specified.

[potiuk]

Tests failing though

[dabla]

Tests failing though

Yes I'm working on it ;)

[dabla]

@potiuk I think the PR is ready now

[potiuk]

Indeed.

[dabla]

Indeed.

Thanks @potiuk

[jscheffl]

Hi @dabla I just checked some tests and realized that this test added in this PR always needs 30 seconds - presumably because of the TIMEOUT being defined =30 above.

This is not really acceptable and I propose to use a better method such that the pytest does not run 30 seconds.

Can you please improve this? THANKS.

If you want to test the runtime, you can check selectively via breeze testing tests --test-type "Providers[microsoft]" -- providers/tests/microsoft/azure/triggers/test_powerbi.py::TestPowerBITrigger::test_powerbi_trigger_run_timeout

[dabla]

I’m willing to look at it but I think this test was originally added by @ambika-garg as this is regarding the PowerBi operator, not the MSGraphOperator. Those tests were probably updated by me due to the change of api version definitions.

[ambika-garg]

Hi @jscheffl, Thank you for reviewing it. I will look into this test.

[dabla]

Hi @jscheffl, Thank you for reviewing it. I will look into this test.

Thank you @ambika-garg for looking into it.