Skip to content

{CI} Enable incremental test on commnad modules only #12657

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
haroldrandom merged 9 commits into from
Mar 23, 2020
Merged

{CI} Enable incremental test on commnad modules only #12657

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
ae7e1fa
Init
Mar 18, 2020
4a3c1a7
echo
Mar 18, 2020
6678dee
Add env in azure-pipeline.yml
Mar 19, 2020
5f1c49f
exit 0 when no tests
Mar 19, 2020
267079a
test to remove REDUCE=True
Mar 19, 2020
72250a2
remove Reduce on name
Mar 19, 2020
82f2e9a
deal with CI stage
Mar 19, 2020
c1d633f
fix bug
Mar 19, 2020
a590a30
dedup test_paths
Mar 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 11 additions & 7 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,8 +398,8 @@ jobs:
- bash: ./scripts/ci/test_profile_integration.sh
displayName: 'Run Integration Test against Profiles'

- job: RunAutomationReducedPython3_6
displayName: Run Automation Reduced Python 3.6
- job: RunAutomationPython3_6
displayName: Run Automation Python 3.6
dependsOn: BuildPythonWheel
timeoutInMinutes: 90

Expand All @@ -423,7 +423,7 @@ jobs:
targetType: 'filePath'
filePath: ./scripts/ci/test_automation.sh
env:
REDUCE_SDK: 'True'
ADO_PULL_REQUEST_TARGET_BRANCH: $(System.PullRequest.TargetBranch)

- job: RunAutomationReduced20190301
displayName: Run Automation Reduced Python 3.6, Profile 2019-03-01
Expand Down Expand Up @@ -451,6 +451,7 @@ jobs:
env:
REDUCE_SDK: 'True'
AZURE_CLI_TEST_TARGET_PROFILE: '2019-03-01'
ADO_PULL_REQUEST_TARGET_BRANCH: $(System.PullRequest.TargetBranch)

- job: RunAutomation20180301
displayName: Run Automation, Profile 2018-03-01
Expand All @@ -477,9 +478,10 @@ jobs:
filePath: ./scripts/ci/test_automation.sh
env:
AZURE_CLI_TEST_TARGET_PROFILE: '2018-03-01'
ADO_PULL_REQUEST_TARGET_BRANCH: $(System.PullRequest.TargetBranch)

- job: RunAutomationReducedPython3_8
displayName: Run Automation Reduced Python 3.8
- job: RunAutomationPython3_8
displayName: Run Automation Python 3.8
dependsOn: BuildPythonWheel
timeoutInMinutes: 90

Expand All @@ -503,7 +505,7 @@ jobs:
targetType: 'filePath'
filePath: ./scripts/ci/test_automation.sh
env:
REDUCE_SDK: 'True'
ADO_PULL_REQUEST_TARGET_BRANCH: $(System.PullRequest.TargetBranch)

- job: RunAutomationReduced20190301Python3_8
displayName: Run Automation Reduced Python 3.8, Profile 2019-03-01
Expand Down Expand Up @@ -531,6 +533,7 @@ jobs:
env:
REDUCE_SDK: 'True'
AZURE_CLI_TEST_TARGET_PROFILE: '2019-03-01'
ADO_PULL_REQUEST_TARGET_BRANCH: $(System.PullRequest.TargetBranch)

- job: TestExtensionsLoading
displayName: Test Extensions Loading
Expand Down Expand Up @@ -578,7 +581,8 @@ jobs:
targetType: 'filePath'
filePath: ./scripts/ci/test_automation.sh
env:
AZURE_CLI_TEST_TARGET_PROFILE: '2018-03-01'
AZURE_CLI_TEST_TARGET_PROFILE: '2018-03-01'
ADO_PULL_REQUEST_TARGET_BRANCH: $(System.PullRequest.TargetBranch)

- job: BuildHomebrewFormula
displayName: Build Homebrew Formula
Expand Down
78 changes: 78 additions & 0 deletions tools/automation/tests/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,70 @@
TEST_INDEX_FORMAT = 'testIndex_{}.json'


def _separator_line():
print('-' * 100)


def _extract_module_name(path):
_CORE_NAME_REGEX = re.compile(r'azure-cli-(?P<name>[^/\\]+)[/\\]azure[/\\]cli')
_MOD_NAME_REGEX = re.compile(r'azure-cli[/\\]azure[/\\]cli[/\\]command_modules[/\\](?P<name>[^/\\]+)')
_EXT_NAME_REGEX = re.compile(r'.*(?P<name>azext_[^/\\]+).*')

for expression in [_MOD_NAME_REGEX, _CORE_NAME_REGEX, _EXT_NAME_REGEX]:
match = re.search(expression, path)
if not match:
continue
return match.groupdict().get('name')
raise ValueError('unexpected error: unable to extract name from path: {}'.format(path))


def _extract_modified_files(target_branch=os.environ.get('ADO_PULL_REQUEST_TARGET_BRANCH')):
ado_raw_env_replacement = '$(System.PullRequest.TargetBranch)'

if target_branch == ado_raw_env_replacement:
# in ADO env but not in PR stage

# dummy file name src/azure-cli-core/azure/cli/core/__init__.py
return [os.path.join('src', 'azure-cli-core', 'azure', 'cli', 'core', '__init__.py')]
else:
qualified_target_branch = 'origin/{}'.format(target_branch)

cmd_tpl = 'git --no-pager diff --name-only --diff-filter=ACMRT {} -- src/'
cmd = cmd_tpl.format(qualified_target_branch)

modified_files = check_output(cmd, shell=True).decode('utf-8').split('\n')
modified_files = [f for f in modified_files if len(f) > 0]

_separator_line()
if modified_files:
print('modified files in src/ :', '\n'.join(modified_files))
else:
print('no modified files in src/, no need to run automation test')

return modified_files


def _extract_top_level_modified_modules():
modified_modules = set()

modified_files = _extract_modified_files()

for file_path in modified_files:
try:
mod = _extract_module_name(file_path)
modified_modules.add(mod)
except ValueError:
continue

_separator_line()
if modified_modules:
print('related top level modules:', list(modified_modules))
else:
print('no related top level modules modified, no need to run automation test')

return modified_modules


def extract_module_name(path):
mod_name_regex = re.compile(r'azure[/\\]cli[/\\]([^/\\]+)')
ext_name_regex = re.compile(r'.*(azext_[^/\\]+).*')
Expand All @@ -42,6 +106,20 @@ def execute(args):
output('Running in CI Mode')
selected_modules = [('All modules', 'azure.cli', 'azure.cli'),
('CLI Linter', 'automation.cli_linter', 'automation.cli_linter')]

modified_modules = _extract_top_level_modified_modules()
if any(base_mod in modified_modules for base_mod in ['core', 'testsdk', 'telemetry']):
pass # if modified modules contains those 3 modules, run all tests
else:
test_paths = set()
for mod in modified_modules:
try:
test_paths.add(os.path.normpath(test_index[mod]))
except KeyError:
display("no tests found in module: {}".format(mod))
args.tests = test_paths

selected_modules = []
Comment on lines +113 to +122
Copy link
Member

@fengzhou-msft fengzhou-msft Mar 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the below code, both selected_modules and args.tests are set, but here you cleaned selected_modules instead of setting to modified_modules. What's the difference?

azure-cli/tools/automation/tests/__init__.py

Lines 133 to 157 in a590a30

else:
# Otherwise run specific tests
args.tests = args.tests or []
# Add any tests from file
if args.src_file:
with open(args.src_file, 'r') as f:
for line in f.readlines():
line = line.strip('\r\n')
line = line.strip('\n')
if line not in args.tests:
args.tests.append(line)
test_paths = []
selected_modules = []
for t in args.tests:
try:
test_path = os.path.normpath(test_index[t])
mod_name = extract_module_name(test_path)
test_paths.append(test_path)
if mod_name not in selected_modules:
selected_modules.append(mod_name)
except KeyError:
display("Test '{}' not found.".format(t))
continue
selected_modules = filter_user_selected_modules_with_tests(selected_modules, args.profile)
args.tests = test_paths

Copy link
Contributor Author

@haroldrandom haroldrandom Mar 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The selected_modules is a list of tuple which consist of 3 elements, like:

azure-cli/tools/automation/tests/__init__.py

Lines 43 to 44 in 450f751

selected_modules = [('All modules', 'azure.cli', 'azure.cli'),
('CLI Linter', 'automation.cli_linter', 'automation.cli_linter')]

The last element of the tuple is a Python Namesapce.

In the incremental test case, I identify module from modified files and then extract module name from it.
So the modified_modules is a list of Python module name like core, vm, storage.
Then I extract corresponding tests from testIndex.json which is generated from get_test_index().

In the runner:

azure-cli/tools/automation/tests/main.py

Lines 15 to 38 in 450f751

def run_tests(modules, parallel, run_live, tests):
if not modules and not tests:
display('No tests set to run.')
sys.exit(1)
display("""
=============
Run Tests
=============
""")
if modules:
display('Modules: {}'.format(', '.join(name for name, _, _ in modules)))
# set environment variable
if run_live:
os.environ['AZURE_TEST_RUN_LIVE'] = 'True'
test_paths = tests or [p for _, _, p in modules]
display('Drive test by nosetests')
runner = get_nose_runner(parallel=parallel, process_timeout=3600 if run_live else 600)
results = runner([path for path in test_paths])
return results, []

tests has higher priority than the module.
So, when I have the extract test need to run, no need to provide selected_module.

elif not (args.tests or args.src_file):
# Default is to run with modules (possibly via environment variable)
if os.environ.get('AZURE_CLI_TEST_MODULES', None):
Expand Down
2 changes: 1 addition & 1 deletion tools/automation/tests/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ def run_tests(modules, parallel, run_live, tests):

if not modules and not tests:
display('No tests set to run.')
sys.exit(1)
sys.exit(0)

display("""
=============
Expand Down