[Text Analytics] "Invalid URL '$(js-textanalytics-test-service-endpoint)/text/analytics/v3.2-preview.2/analyze': No schema supplied" in Public, USGOV and China clouds

[v-xuto]

We are running live Tests against other clouds like US Gov and Azure China Cloud. The goal is to check whether new azure sdk package work with other clouds or not.

Error Description:
When running the test test_analyze.py::TestAnalyze::test_disable_service_logs in weekly pipeline, it runs failed and the error message is shown as following. In addition, I ran this test in the local and nightly pipelines, and the result of both is pass. For more details on this issue, please check here:

Expected Behavior:
This service request is a valid url.
Maybe we need to confirm whether the value of the environment variable js-textanalytics-test-service-endpoint (textanalytics/tests.yml) in weekly pipeline is correct and exists.

@benbp, @jameszliao-msft, @lmazuel, @meeraharidasa , @mayurid, @abhahn for notification.

[benbp]

These appear to be references to a static test resource via devops variables. Any time these are here it means that the tests will not work by default for sovereign clouds, and the static test resource needs to either be replicated across the sovereign clouds by the owning team OR updated to be provisioned dynamically. FYI @kristapratico

The reason you are seeing the current error (where it fails to expand the variable) is because this variable group has not been authorized for use by the weekly pipeline (but doing so won't fix the underlying issue).

[v-xuto]

@benbp So how are we currently going to fundamentally solve this problem? Do you have any idea? Or shall we first authorize this variable group to the weekly pipeline for sovereign testing?

[kristapratico]

These vars do refer to static test resources which are only available in the Public cloud (service constraint) - so provisioning dynamically will not quite work yet for sovereign clouds. I believe you could authorize the weekly pipeline to use the variable group since it will just use an endpoint/key combo from the Public cloud vars for these specific tests. Otherwise we can skip these tests in sovereign clouds for now.

[benbp]

@kristapratico I'd rather not make cross-cloud service requests if we can avoid it (from a test coverage/isolation standpoint). Would you be able to help copy over the static resources to the UsGov and China clouds? I can help with access.

[kristapratico]

@kristapratico I'd rather not make cross-cloud service requests if we can avoid it (from a test coverage/isolation standpoint). Would you be able to help copy over the static resources to the UsGov and China clouds? I can help with access.

@benbp That's fair. The issue is that these static resources pertain to features of the service that only are supported in westus2 and westeurope at the moment. The particular endpoint/key passed with env vars has trained and deployed models under the account in the Public cloud which we test from. We're unable to create these models in different clouds until the service supports it.

[benbp]

@kristapratico ah ok, in that case I'd say we should skip these tests in sovereign clouds until the feature set is supported.

[v-xuto]

@kristapratico @benbp Thanks for your help. I will skip these tests related to variable group (like js-single-category-classify-project-name) in UsGov and China clouds.

1. Please help authorize the weekly pipeline to use this variable group in textanalytics/tests.yml.
2. I will skip the following tests in UsGov and China, please help confirm whether it is correct or missing.

• test_analyze.py::TestAnalyze::test_disable_service_logs
• test_analyze.py::TestAnalyze::test_single_category_classify
• test_analyze.py::TestAnalyze::test_multi_category_classify
• test_analyze.py::TestAnalyze::test_recognize_custom_entities
• test_analyze.py::TestAnalyze::test_generic_action_error_no_target
• test_analyze.py::TestAnalyze::test_action_errors_with_targets
• test_analyze.py::TestAnalyze::test_action_job_failure
• test_analyze_async.py::TestAnalyzeAsync::test_disable_service_logs
• test_analyze_async.py::TestAnalyzeAsync::test_single_category_classify
• test_analyze_async.py::TestAnalyzeAsync::test_multi_category_classify
• test_analyze_async.py::TestAnalyzeAsync::test_recognize_custom_entities
• test_analyze_async.py::TestAnalyzeAsync::test_generic_action_error_no_target
• test_analyze_async.py::TestAnalyzeAsync::test_action_errors_with_targets
• test_analyze_async.py::TestAnalyzeAsync::test_action_job_failure

[kristapratico]

@v-xuto it might be best to skip on the absence of textanalytics_custom_text_endpoint since the env var associated is used in all the tests that will need to be skipped.

1. Like Ben said, we shouldn't need to authorize the variable group for this pipeline since we don't want to mix cross-cloud requests. Let me know if you're still running into issues with this, though.
2. That set of tests looks correct to me.

[benbp]

@kristapratico because the variables are still referenced in tests.yml, devops will try to download them hence the access error.

I recommend we build out a subscription configuration overlay for text analytics and migrate these values out of the variable group, that way we know the values won't be defined at all for sovereign tests and the test skips can handle that appropriately. I can take this on.