Skip to content

Should run full test if there is API, SDK or CLI version bump  #329

@jiasli

Description

@jiasli

Currently, for SDK or API version bump, only incremental tests are run. This frequently leads to dev branch Batched CI failure because the new API or SDK conflicts with other modules, mainly due to API version mismatch, such as Azure/azure-cli#21681 (comment):

https://dev.azure.com/azure-sdk/public/_build/results?buildId=1466412&view=logs&j=ad51cf76-b294-5bde-777e-be77cc5f3050&t=0e02ab23-4098-514d-f7b9-ab422251bbb2

2022-03-29T02:16:29.3378803Z         except CannotOverwriteExistingCassetteException as ex:
2022-03-29T02:16:29.3379238Z >           raise AssertionError(ex)
2022-03-29T02:16:29.3380350Z E           AssertionError: Can't overwrite existing cassette ('/opt/az/lib/python3.8/site-packages/azure/cli/command_modules/monitor/tests/latest/recordings/test_metric_alert_for_sql_database_scope.yaml') in your current record mode ('once').
2022-03-29T02:16:29.3382083Z E           No match for the request (<Request (PUT) https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli_test_metric_alert_v1_2000001/providers/Microsoft.Sql/servers/clitestservermatricalertA000002/databases/cliautomationdb01?api-version=2021-05-01-preview>) was found.
2022-03-29T02:16:29.3383114Z E           Found 1 similar requests with 1 different matcher(s) :
2022-03-29T02:16:29.3383519Z E           
2022-03-29T02:16:29.3384713Z E           1 - (<Request (PUT) https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli_test_metric_alert_v1_2000001/providers/Microsoft.Sql/servers/clitestservermatricalertA000002/databases/cliautomationdb01?api-version=2021-02-01-preview>).
2022-03-29T02:16:29.3385863Z E           Matchers succeeded : ['method', 'scheme', 'host', 'port', 'path']
2022-03-29T02:16:29.3386340Z E           Matchers failed :
2022-03-29T02:16:29.3386940Z E           _custom_request_query_matcher - assertion failure :
2022-03-29T02:16:29.3387367Z E           None
2022-03-29T02:16:29.3387562Z 
2022-03-29T02:16:29.3388186Z /opt/az/lib/python3.8/site-packages/azure/cli/testsdk/base.py:299: AssertionError
2022-03-29T02:16:29.3389118Z ----------------------------- Captured stdout call -----------------------------
2022-03-29T02:16:29.3389583Z None
2022-03-29T02:16:29.3390202Z ------------ generated xml file: /azure_cli_test_result/monitor.xml ------------
2022-03-29T02:16:29.3390794Z =========================== short test summary info ============================
2022-03-29T02:16:29.3391462Z FAILED tests/latest/test_monitor_metric_alert_scenarios.py::MonitorTests::test_metric_alert_for_sql_database_scope

The solution is to run full test if contents in these files are changed:

  • src/azure-cli-core/azure/cli/core/profiles/_shared.py
  • src/azure-cli/requirements.py3.Darwin.txt
  • src/azure-cli/requirements.py3.Linux.txt
  • src/azure-cli/requirements.py3.windows.txt

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions