Rework WP.com e2e tests to remove custom test harness and run in CI#1126
Rework WP.com e2e tests to remove custom test harness and run in CI#1126
Conversation
3f4fc46 to
5083059
Compare
5083059 to
dcc4085
Compare
crazytonyli
left a comment
There was a problem hiding this comment.
I'm not sure if I asked this before (I forgot the answer if I did). Any reason we don't run this test suite like other tests, using cargo test?
The "ignore write tests" part of this PR seems to be similar to cargo features.
wp_com_e2e/README.md
Outdated
| ### Run read-only tests (default) | ||
|
|
||
| ```bash | ||
| ./target/debug/wp_com_e2e |
There was a problem hiding this comment.
Should we use cargo to run the tests? Like cargo run --quiet --bin wp_com_e2e -- [cli-options]?
This is pretty much what this PR does, but it maintains the ability to build the binary and run it externally. |
|
Sorry, I don't quite follow. We can use commands like
Why do we need to build an executable binary for this end-to-end test suite? By "externally", do you mean it's used in somewhere else? |
The idea behind the crate is that it can be compiled into a binary that could be run from Docker or as part of WP.com's CI process to validate that a given PR doesn't break API client compatibility. |
Summary
libtest-mimicfor a cargo-test-like CLI experienceTest plan
cargo build -p wp_com_e2eto verify the refactored test runner compiles./target/debug/wp_com_e2e --listto verify test discovery works./target/debug/wp_com_e2ewithWP_COM_API_KEYset to verify read-only tests passChanges
The main refactor replaces the ad-hoc test execution in
wp_com_e2ewith libtest-mimic, providing:--list,--ignored,--include-ignored,--exact,--nocapture)TestContextfor API client reuse across testsThe CI addition runs the read-only e2e tests on the
androidqueue after installing Rust toolchains.We'll run the new tests as not-required for now to see how it goes and will make them required for a PR to land in the future.