From c795b68810d357c7841d91cf64af8a35b46ada1a Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 3 Jun 2025 16:57:29 +0100 Subject: [PATCH 1/3] Remove documentation surrounding MSC build tags We no longer use them as of #666 --- ONBOARDING.md | 8 ++------ README.md | 26 +++++++++++++++++--------- 2 files changed, 19 insertions(+), 15 deletions(-) diff --git a/ONBOARDING.md b/ONBOARDING.md index 49949c98..5a37ad02 100644 --- a/ONBOARDING.md +++ b/ONBOARDING.md @@ -185,10 +185,6 @@ To conditionally skip an entire *file* based on the homeserver being run, add a ```go // +build !dendrite_blacklist ``` -You can also do this based on features for MSC tests (which means you must run Complement *with* this tag for these tests *to run*): -```go -// +build msc_2836 -``` See [GH Actions](https://github.com/matrix-org/complement/blob/master/.github/workflows/ci.yaml) for an example of how this is used for different homeservers in practice. ### Why do we use `t.Errorf` sometimes and `t.Fatalf` other times? @@ -210,11 +206,11 @@ For Goland: * Under "Run"->"Edit Configurations..."->"Edit Configuration Templates..."->"Go Test", and add `COMPLEMENT_BASE_IMAGE=complement-dendrite:latest` to "Environment" * Then you can right-click on any test file or test case and "Run ". - + ### How do I make the linter checks pass? Use [`goimports`](https://pkg.go.dev/golang.org/x/tools/cmd/goimports) to sort imports and format in the style of `gofmt`. - + Set this up to run on save in VSCode as follows: - File -> Preferences -> Settings. - Search for "Format On Save" and enable it. diff --git a/README.md b/README.md index 5b2cc332..ff0fb58d 100644 --- a/README.md +++ b/README.md @@ -140,24 +140,32 @@ for an example of how to do this correctly. To get started developing Complement tests, see [the onboarding documentation](ONBOARDING.md). -### Build tags +### Build tags (test blacklisting) + +Complement uses build tags to exclude tests for each homeserver implementation. +Build tags are comments at the top of the file that look like: -Complement uses build tags to include or exclude tests for each homeserver. Build tags are comments at the top of the file that look -like: ```go -// +build msc2403 +// +build !dendrite_blacklist ``` -We have tags for MSCs (the above is in `msc2403_test.go`) as well as general blacklists for a homeserver implementation e.g Dendrite, -which has the name `dendrite_blacklist`. These are implemented as inverted tags such that specifying the tag results in the file not -being picked up by `go test`. For example, `apidoc_presence_test.go` has: + +These are implemented as inverted tags, such that specifying the tag results in +the file not being picked up by `go test`. This serves as a way to exclude +known-broken tests per implementation. + +For example, `apidoc_presence_test.go` has: + ```go // +build !dendrite_blacklist ``` + and all Dendrite tests run with `-tags="dendrite_blacklist"` to cause this file to be skipped. You can run tests with build tags like this: + ``` -COMPLEMENT_BASE_IMAGE=complement-synapse:latest go test -v -tags="synapse_blacklist,msc2403" ./tests/... +COMPLEMENT_BASE_IMAGE=complement-synapse:latest go test -v -tags="synapse_blacklist" ./tests/... ``` -This runs Complement with a Synapse HS and ignores tests which Synapse doesn't implement, and includes tests for MSC2403. + +This runs Complement with a Synapse HS and ignores tests which Synapse doesn't implement. ## Why 'Complement'? From 08eefdd59946f56c8c42ff600b6208541ec05d55 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 3 Jun 2025 17:09:43 +0100 Subject: [PATCH 2/3] Document test writing for unstable MSCs --- README.md | 47 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/README.md b/README.md index ff0fb58d..324e2b61 100644 --- a/README.md +++ b/README.md @@ -167,6 +167,53 @@ COMPLEMENT_BASE_IMAGE=complement-synapse:latest go test -v -tags="synapse_blackl This runs Complement with a Synapse HS and ignores tests which Synapse doesn't implement. +### Writing tests for unstable MSCs + +Complement is frequently used to test homeserver implementations of unstable +MSCs. As these features/changes often become stable eventually and for +convenience, this repo accepts such tests. + +Tests for a given MSC should be placed in a new directory under `tests/`. For +example, to write tests for MSC9999, create a directory at `tests/msc9999`. + +This creates a new go "package", and tests contained within will not be run +unless explicitly noted. A package directory should contain the following +files: + +``` +tests/msc9999 +├── main_test.go +└── msc9999_test.go +``` + +where `main_test.go` sets up Complement and indicates that this is a package +containing tests: + +```go +package tests + +import ( + "testing" + + "github.com/matrix-org/complement" +) + +func TestMain(m *testing.M) { + complement.TestMain(m, "msc9999") +} +``` + +and `msc9999_test.go` contains your actual tests. See existing `tests/msc*` +directories for examples. + +You can create additional files to separate and organise logical chunks of +tests. Just be sure each file is named `*_test.go` for `go test` to find it. + +Once an MSC is accepted, the tests should be migrated out of the `msc*` +directory, as the MSC is now considered stable. Consider adding the tests to +the blacklist of other homeserver implementations (see above section) if they +don't yet implement the new changes described by the MSC. + ## Why 'Complement'? Because **M***C* = **1** - **M** From 715c765e14a139c08e52bd926b75e417cca068f3 Mon Sep 17 00:00:00 2001 From: Andrew Morgan Date: Tue, 3 Jun 2025 17:13:31 +0100 Subject: [PATCH 3/3] Add currently-known blacklist tags So those adding new tests know what to blacklist out of the box. --- README.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 324e2b61..63c31e7e 100644 --- a/README.md +++ b/README.md @@ -159,7 +159,8 @@ For example, `apidoc_presence_test.go` has: // +build !dendrite_blacklist ``` -and all Dendrite tests run with `-tags="dendrite_blacklist"` to cause this file to be skipped. You can run tests with build tags like this: +and all Dendrite tests run with `-tags="dendrite_blacklist"` to cause this file +to be skipped. You can run tests with build tags like this: ``` COMPLEMENT_BASE_IMAGE=complement-synapse:latest go test -v -tags="synapse_blacklist" ./tests/... @@ -167,6 +168,13 @@ COMPLEMENT_BASE_IMAGE=complement-synapse:latest go test -v -tags="synapse_blackl This runs Complement with a Synapse HS and ignores tests which Synapse doesn't implement. +The currently known blacklist tags are: + +* `synapse_blacklist` +* `dendrite_blacklist` +* `conduit_blacklist` +* `conduwuit_blacklist` + ### Writing tests for unstable MSCs Complement is frequently used to test homeserver implementations of unstable