Initial LISAv3 via Pytest review#1065
Closed
andyleejordan wants to merge 84 commits into
Closed
Conversation
This is a _working_ test.
Extend Fabric’s Connection class with a new (and simple) command “cat” which return the value of a remote file as a string. Setup a Config for the Connection when creating it that echoes every command, disables the stdin forwarding (since we’re running under Pytest), and fixes the PATH since the remote commands don’t run under a login shell.
This could be extended to instead deploy a host of the specified distro. Most likely we’ll want a command-line parameter that the fixture uses to create a Node with the given requirements, and then tests will be skipped if their requirements aren’t met. Further more, the mark here is very simple. It can instead take keyword arguments, which would map to our metadata.
As supplying types for these would be supremely annoying.
Accidentally eliminated Fabric’s default overrides of Invoke by supplying my own config based on `invoke.Config` to Fabric. Oops.
Running with ‘-n 1’ takes 10 seconds, or 3 seconds for each feature in sequence plus overhead. Running with ‘-n 8’ takes 4 seconds, 3 seconds for each in parallel plus overhead.
andyleejordan
force-pushed
the
andschwa/initial-pytest-review
branch
from
d9e5fac to
31f0fa0
Compare
squirrelsc
reviewed
Nov 4, 2020
squirrelsc
reviewed
Nov 6, 2020
squirrelsc
reviewed
Nov 6, 2020
andyleejordan
force-pushed
the
andschwa/initial-pytest-review
branch
2 times, most recently
from
e400dcb to
b5eb4cc
Compare
squirrelsc
reviewed
Nov 9, 2020
squirrelsc
reviewed
Nov 9, 2020
Example from `make test`:
```
Created target: set() / {'platform': 'Azure', 'image': 'citrix:netscalervpx-130:netscalerbyol:latest', 'sku': 'Standard_DS1_v2'}
Created target: set() / {'platform': 'Azure', 'image': 'audiocodes:mediantsessionbordercontroller:mediantvirtualsbcazure:latest', 'sku': 'Standard_DS1_v2'}
Created target: set() / {'platform': 'Azure', 'image': 'credativ:Debian:9:9.0.201706190', 'sku': 'Standard_DS1_v2'}
Created target: set() / {'platform': 'Azure', 'image': 'github:github-enterprise:github-enterprise:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'xdp'} / {'platform': 'Azure', 'image': 'citrix:netscalervpx-130:netscalerbyol:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'xdp'} / {'platform': 'Azure', 'image': 'audiocodes:mediantsessionbordercontroller:mediantvirtualsbcazure:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'xdp'} / {'platform': 'Azure', 'image': 'credativ:Debian:9:9.0.201706190', 'sku': 'Standard_DS1_v2'}
Created target: {'xdp'} / {'platform': 'Azure', 'image': 'github:github-enterprise:github-enterprise:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'gpu'} / {'platform': 'Azure', 'image': 'citrix:netscalervpx-130:netscalerbyol:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'gpu'} / {'platform': 'Azure', 'image': 'audiocodes:mediantsessionbordercontroller:mediantvirtualsbcazure:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'gpu'} / {'platform': 'Azure', 'image': 'credativ:Debian:9:9.0.201706190', 'sku': 'Standard_DS1_v2'}
Created target: {'gpu'} / {'platform': 'Azure', 'image': 'github:github-enterprise:github-enterprise:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'rdma'} / {'platform': 'Azure', 'image': 'citrix:netscalervpx-130:netscalerbyol:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'rdma'} / {'platform': 'Azure', 'image': 'audiocodes:mediantsessionbordercontroller:mediantvirtualsbcazure:latest', 'sku': 'Standard_DS1_v2'}
Created target: {'rdma'} / {'platform': 'Azure', 'image': 'credativ:Debian:9:9.0.201706190', 'sku': 'Standard_DS1_v2'}
Created target: {'rdma'} / {'platform': 'Azure', 'image':
'github:github-enterprise:github-enterprise:latest', 'sku':
'Standard_DS1_v2'}
```
andyleejordan
force-pushed
the
andschwa/initial-pytest-review
branch
from
992e951 to
ac019f9
Compare
Instead of “Azure” which of course doesn’t work in CI.
The old-style was recently deprecated: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/
Member
Author
|
Closing this in preference of #1107! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I created the branch
pytest/mainand enabled branch protections on it so all code goes through review. This PR supersedes #1044 which was an initial demo. There are TODOs throughout, which I'd like to work with @lpereira to prioritize and work-on to finish this proof-of-concept, but I'd like to get what does exist reviewed before it goes any further. I have attempted to document design decisions and alternatives considered inpytest/README.md, and have setup some basic CI with a self-test and semantic analsysis.The smoke test (when Azure is working, which it hasn't been today) demonstrates a runtime-configured matrix of images to test with the criteria given to me. The LIS test was my first test for the initial proof-of-concept, and I'd like to use @lpereira's XML->Pytest generator to lift-and-shift as many LISAv2 tests as possible. Anything that just runs existing Bash scripts can be made to run with this we believe practically automatically. The XDP test is a work-in-progress because it requires two NICs, which is a similar problem as deploying two nodes for the
ntttcptest, but these are both obviously tractable problems that simply require small investment in scripting aroundaz. I know @squirrelsc's big concern is enabling a YAML playbook-based test selection and environment creation, but as can be seen by the parameterization of the smoke test, this is also a tractable problem.I borrowed the CoC and contributing guidelines from
mainbranch, from where we can and should pull more in.Please give this a thorough review. The implementation is still <200 lines of code, with another ~100 for the written tests, so it should be very straightforward to read. I would suggest using the file filter to look at just
*.py.