{CI} Enable incremental test on commnad modules only

[haroldrandom]

Scenario of modified files:

1. Modified files in src/azure-cli-core/azure/cli/core ([CI] Incremental test on modifying core #12658) - 83mins
2. Modified files in src/azure-cli-telemetry/ ([CI] test on modifying telemetry #12662) - 53mins
3. Modified files in src/azure-cli/azure/cli/command_modules/network ([CI] incremental test on modifying network test #12661) take 11 mins.
4. Modified files in srv/azure-cli/azure/cli/command_modules/vm ([CI] incremental test on modifying vm test #12664) take 11 mins.
5. Modified files in src/azure-cli-testsdk/ ([CI] Incremental test on modifying testsdk #12666) - 48mins
6. Modified files in vm and network ([CI] Incremental test on vm and network #12665) - 14mins.
7. Modified files in network and storage ([CI] Incremental Test on command modules: network and storage #12676), - 14 mins.

---

This checklist is used to make sure that common guidelines for a pull request are followed.

• The PR title and description has followed the guideline in Submitting Pull Requests.

• I adhere to the Command Guidelines.

[yonzhan]

incremental test

[qianwens]

[fengzhou-msft]

In the below code, both selected_modules and args.tests are set, but here you cleaned selected_modules instead of setting to modified_modules. What's the difference?

azure-cli/tools/automation/tests/__init__.py

Lines 133 to 157 in a590a30

	else:
	# Otherwise run specific tests
	args.tests = args.tests or []
	# Add any tests from file
	if args.src_file:
	with open(args.src_file, 'r') as f:
	for line in f.readlines():
	line = line.strip('\r\n')
	line = line.strip('\n')
	if line not in args.tests:
	args.tests.append(line)
	test_paths = []
	selected_modules = []
	for t in args.tests:
	try:
	test_path = os.path.normpath(test_index[t])
	mod_name = extract_module_name(test_path)
	test_paths.append(test_path)
	if mod_name not in selected_modules:
	selected_modules.append(mod_name)
	except KeyError:
	display("Test '{}' not found.".format(t))
	continue
	selected_modules = filter_user_selected_modules_with_tests(selected_modules, args.profile)
	args.tests = test_paths

[haroldrandom]

The selected_modules is a list of tuple which consist of 3 elements, like:

azure-cli/tools/automation/tests/__init__.py

Lines 43 to 44 in 450f751

	selected_modules = [('All modules', 'azure.cli', 'azure.cli'),
	('CLI Linter', 'automation.cli_linter', 'automation.cli_linter')]

The last element of the tuple is a Python Namesapce.

In the incremental test case, I identify module from modified files and then extract module name from it.
So the modified_modules is a list of Python module name like core, vm, storage.
Then I extract corresponding tests from testIndex.json which is generated from get_test_index().

In the runner:

azure-cli/tools/automation/tests/main.py

Lines 15 to 38 in 450f751

	def run_tests(modules, parallel, run_live, tests):
	if not modules and not tests:
	display('No tests set to run.')
	sys.exit(1)
	display("""
	=============
	Run Tests
	=============
	""")
	if modules:
	display('Modules: {}'.format(', '.join(name for name, _, _ in modules)))
	# set environment variable
	if run_live:
	os.environ['AZURE_TEST_RUN_LIVE'] = 'True'
	test_paths = tests or [p for _, _, p in modules]
	display('Drive test by nosetests')
	runner = get_nose_runner(parallel=parallel, process_timeout=3600 if run_live else 600)
	results = runner([path for path in test_paths])
	return results, []

tests has higher priority than the module.
So, when I have the extract test need to run, no need to provide selected_module.