Skip to content

[CI] Onboard pytest and deprecate nose for Automation Test#13153

Merged
haroldrandom merged 21 commits intodevfrom
onboard-pytest-to-automation-test
Apr 27, 2020
Merged

[CI] Onboard pytest and deprecate nose for Automation Test#13153
haroldrandom merged 21 commits intodevfrom
onboard-pytest-to-automation-test

Conversation

@haroldrandom
Copy link
Contributor

@haroldrandom haroldrandom commented Apr 24, 2020 •
edited
Loading

Description

  1. Onboard new azdev to run automation test.
  2. Onboard pytest to run Automation Test
  3. Deprecate nose test framework

Testing Guide

  1. test incremental test on network package {CI} test network for onboarding pytest on CI #13173,
    1.1 it ran 132 tests under profile latest
    1.2 it ran 71 tests under profile 2018
  2. test incremental test on compute package {CI} test vm for onboarding pytest on CI #13175, it ran all tests under latest profile for vm.
  3. test incremental test on storage package {CI} test storage for onboarding pytest on CI #13176, it ran all tests under latest profile for storage.
  4. test incremental test on keyvault package {CI} test keyvault for onboarding pytest on CI #13177, it ran all tests under latest profile for keyvault
  5. test incremental test on resource {CI} test resource for onboarding pytest on CI #13178, it ran all tests under latest profile for resource
  6. test core {CI} test core for onboarding pytest on CI #13179,
    6.1 it rall all 1936 tests under profile latest
    6.2 it ran all 610 tests under profile 2019-03-01-hybrid
    6.3 it ran all 635 tests under profile 2018-03-01-hybrid
    6.4 it ran all 314 tests under profile 2017-03-09-profile
  7. test network {CI} test network for onboarding pytest on CI for 2018-03-01-hybrid profile #13181 for profile 2018-03-01-hybrid, it ran all tests including core as design.
  8. test vm {CI} test vm for onboarding pytest on CI for 2019-03-01-hybrid profile #13182 for profile 2019-03-01-hybrid, it ran all tests including core as desgin.
  9. test network and vm {CI} test network, vm for onboarding pytest on CI for latest profile #13185 for profile latest, it ran all vm and network tests.

History Notes

This checklist is used to make sure that common guidelines for a pull request are followed.

@haroldrandom haroldrandom self-assigned this Apr 24, 2020
@haroldrandom haroldrandom added the CI CI label Apr 24, 2020
@haroldrandom haroldrandom added this to the S169 - For Build milestone Apr 24, 2020
@haroldrandom haroldrandom changed the title Onboard pytest to automation test [CI] Onboard pytest to automation test Apr 24, 2020
@yonzhan yonzhan requested review from jsntcy and mmyyrroonn April 24, 2020 10:36
@yonzhan
Copy link
Collaborator

yonzhan commented Apr 24, 2020

add to S169

@haroldrandom
Copy link
Contributor Author

haroldrandom commented Apr 26, 2020

/azp run

@azure-pipelines
Copy link

azure-pipelines bot commented Apr 26, 2020

Azure Pipelines successfully started running 2 pipeline(s).

@haroldrandom haroldrandom force-pushed the onboard-pytest-to-automation-test branch from 55ee863 to 685f67d Compare April 26, 2020 05:38
@haroldrandom haroldrandom changed the title [CI] Onboard pytest to automation test [CI] Onboard pytest for automation test on CI Apr 26, 2020
@haroldrandom haroldrandom changed the title [CI] Onboard pytest for automation test on CI [CI] Onboard pytest and deprecate nose for automation test on CI Apr 26, 2020
@haroldrandom haroldrandom changed the title [CI] Onboard pytest and deprecate nose for automation test on CI [CI] Onboard pytest and deprecate nose for Automation Test on CI Apr 26, 2020
@haroldrandom haroldrandom marked this pull request as ready for review April 26, 2020 07:54
mmyyrroonn
mmyyrroonn reviewed Apr 26, 2020
View reviewed changes
azure-pipelines.yml
displayName: Automation Test (Profile Latest)
dependsOn: BuildPythonWheel
timeoutInMinutes: 90
timeoutInMinutes: 120
Copy link
Contributor

@mmyyrroonn mmyyrroonn Apr 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should it be incremental?

Copy link
Contributor Author

@haroldrandom haroldrandom Apr 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pytest take more time to run all tests (1936), so, have to increase this timeout limit. And Will be fixed in the future to optimize the time cost.

Copy link
Member

@fengzhou-msft fengzhou-msft Apr 26, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the time cost now with pytest?

Copy link
Contributor Author

@haroldrandom haroldrandom Apr 26, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1H 49m from this build from this PR #13179, increased by 20 mins

mmyyrroonn
mmyyrroonn reviewed Apr 26, 2020
View reviewed changes
qianwens
qianwens reviewed Apr 26, 2020
View reviewed changes
azure-pipelines.yml Outdated
displayName: Run Automation, Profile 2018-03-01
dependsOn: BuildPythonWheel
pip install -q azdev
azdev --version
Copy link
Member

@qianwens qianwens Apr 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

duplicate scripts in different profile, please wrap them in a script file or common variable in yml?

Copy link
Contributor Author

@haroldrandom haroldrandom Apr 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I searched and found the ADO template only works for steps not for job. So, I moved those similar steps into a single file .azure-pipelines/template/automation_test.yml.

qianwens
qianwens reviewed Apr 26, 2020
View reviewed changes
azure-pipelines.yml
filePath: ./scripts/ci/test_automation.sh
env:
AZURE_CLI_TEST_TARGET_PROFILE: '2018-03-01'
ADO_PULL_REQUEST_TARGET_BRANCH: $(System.PullRequest.TargetBranch)
Copy link
Member

@qianwens qianwens Apr 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why cannot we reuse these two environment?

Copy link
Contributor Author

@haroldrandom haroldrandom Apr 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They have became parameters while instantiating the template .azure-pipelines/template/automation_test.yml.

qianwens
qianwens reviewed Apr 26, 2020
View reviewed changes
fengzhou-msft
fengzhou-msft reviewed Apr 26, 2020
View reviewed changes
qianwens
qianwens approved these changes Apr 27, 2020
View reviewed changes
Copy link
Member

@qianwens qianwens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@haroldrandom
Copy link
Contributor Author

haroldrandom commented Apr 27, 2020

/azp run

@azure-pipelines
Copy link

azure-pipelines bot commented Apr 27, 2020

Azure Pipelines successfully started running 2 pipeline(s).

@haroldrandom
Copy link
Contributor Author

haroldrandom commented Apr 27, 2020 •
edited
Loading

The actual output on ADO portal shows all jobs are passed in Attemp 2 but on Github it shows the result of Attemp 1
image

@haroldrandom haroldrandom changed the title [CI] Onboard pytest and deprecate nose for Automation Test on CI [CI] Onboard pytest and deprecate nose for Automation Test Apr 27, 2020
@haroldrandom haroldrandom merged commit d261ede into dev Apr 27, 2020
@haroldrandom haroldrandom deleted the onboard-pytest-to-automation-test branch April 27, 2020 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jsntcy jsntcy Awaiting requested review from jsntcy

3 more reviewers

@mmyyrroonn mmyyrroonn mmyyrroonn left review comments

@fengzhou-msft fengzhou-msft fengzhou-msft left review comments

@qianwens qianwens qianwens approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@haroldrandom haroldrandom

Labels

CI CI

Projects

None yet

Milestone

S169 - For Build

Development

Successfully merging this pull request may close these issues.

5 participants

@haroldrandom @yonzhan @qianwens @mmyyrroonn @fengzhou-msft