Skip to content

Rework WP.com e2e tests to remove custom test harness and run in CI#1126

Merged
jkmassel merged 3 commits intotrunkfrom
use/rust-tests
Jan 29, 2026
Merged

Rework WP.com e2e tests to remove custom test harness and run in CI#1126
jkmassel merged 3 commits intotrunkfrom
use/rust-tests

Conversation

@jkmassel
Copy link
Contributor

@jkmassel jkmassel commented Jan 27, 2026

Summary

  • Refactor WordPress.com e2e tests to use libtest-mimic for a cargo-test-like CLI experience
  • Add WordPress.com e2e tests to the Buildkite CI pipeline

Test plan

  • Run cargo build -p wp_com_e2e to verify the refactored test runner compiles
  • Run ./target/debug/wp_com_e2e --list to verify test discovery works
  • Run ./target/debug/wp_com_e2e with WP_COM_API_KEY set to verify read-only tests pass
  • Verify the new Buildkite pipeline step executes successfully

Changes

The main refactor replaces the ad-hoc test execution in wp_com_e2e with libtest-mimic, providing:

  • Standard test CLI flags (--list, --ignored, --include-ignored, --exact, --nocapture)
  • Test filtering by name/pattern
  • Separation of read-only vs write tests (write tests are ignored by default)
  • Shared TestContext for API client reuse across tests
  • Dynamic test generation based on API responses (per-site, per-conversation tests)

The CI addition runs the read-only e2e tests on the android queue after installing Rust toolchains.

We'll run the new tests as not-required for now to see how it goes and will make them required for a PR to land in the future.

@jkmassel jkmassel force-pushed the use/rust-tests branch 4 times, most recently from 3f4fc46 to 5083059 Compare January 27, 2026 17:18
@jkmassel jkmassel changed the title WP.com e2e test Rework WP.com e2e tests to remove custom test harness and run in CI Jan 27, 2026
@jkmassel jkmassel requested a review from crazytonyli January 27, 2026 18:08
@jkmassel jkmassel marked this pull request as ready for review January 27, 2026 18:09
Copy link
Contributor

@crazytonyli crazytonyli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if I asked this before (I forgot the answer if I did). Any reason we don't run this test suite like other tests, using cargo test?

The "ignore write tests" part of this PR seems to be similar to cargo features.

### Run read-only tests (default)

```bash
./target/debug/wp_com_e2e
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use cargo to run the tests? Like cargo run --quiet --bin wp_com_e2e -- [cli-options]?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point! Adjusted in 37d479e

@jkmassel
Copy link
Contributor Author

jkmassel commented Jan 28, 2026

Any reason we don't run this test suite like other tests, using cargo test?

This is pretty much what this PR does, but it maintains the ability to build the binary and run it externally.

@crazytonyli
Copy link
Contributor

Sorry, I don't quite follow. We can use commands like cargo test -p wp_api_integration_tests to run other tests, but we can't cargo test -p wp_com_e2e.

it maintains the ability to build the binary and run it externally.

Why do we need to build an executable binary for this end-to-end test suite? By "externally", do you mean it's used in somewhere else?

@jkmassel
Copy link
Contributor Author

Why do we need to build an executable binary for this end-to-end test suite? By "externally", do you mean it's used in somewhere else?

The idea behind the crate is that it can be compiled into a binary that could be run from Docker or as part of WP.com's CI process to validate that a given PR doesn't break API client compatibility.

@jkmassel jkmassel merged commit 75d9ae9 into trunk Jan 29, 2026
23 checks passed
@jkmassel jkmassel deleted the use/rust-tests branch January 29, 2026 20:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants