Skip to content

Fix flaking tests#292

Merged
jacekbogdanski merged 3 commits intomasterfrom
t/4.20.0-tests
Sep 29, 2022
Merged

Fix flaking tests#292
jacekbogdanski merged 3 commits intomasterfrom
t/4.20.0-tests

Conversation

@MMMalik
Copy link
Copy Markdown
Contributor

@MMMalik MMMalik commented Sep 28, 2022

No description provided.

@MMMalik
Copy link
Copy Markdown
Contributor Author

MMMalik commented Sep 29, 2022

Hi @jacekbogdanski,

as you noted, unit tests have recently started failing. There were no code nor tests changes since the last time CI was green, so it's really disturbing that this has happened. The only differences I noticed between these CI runs are:

  • Chrome version (used to be 103, now it's 105)
  • CKEditor version (bumped from 4.19.1 to 4.20.0)
  • Possibly different versions of some dev dependencies (hard to track since there is no lock file)

I can reproduce failing tests locally as well.

I don't think it's related to Chrome versions (but I haven't tested though) and it's certainly not related to CKEditor versions (I've tested multiple older versions with the same result). My bet is that a newer version of some dependency must have been installed which introduced some subtle bugs to our setup. This is possible since the setup we use is quite exotic (Karma + Chrome + testing-library + async runtime loading of CKEditor dependencies) and proved to be "flaky" in the past.

While debugging I noticed that our testing setup now started to have troubles with proper cleanup of CKEditor instances between runs. This led to some inconsistencies in the assertions. I couldn't pinpoint the exact cause of that though.

Following mitigations have been applied:

  • Removed beforeEach check that asserted cleanup of editor instances. Instances get cleaned up eventually but they might not get cleaned up before next test starts.
  • Adjusted a few expectations in useCKEditor tests.
  • Refactored tests in registerEditorEventHandler to not rely on useCKEditor. That was another source of "flakiness". Now a mock is passed instead of a real editor instance.

The CI is green again ✅. E2E tests pass correctly and I've additionally tested samples manually. I haven't noticed any issues.

As a long term solution we might discuss a different approach to testing.

@MMMalik MMMalik changed the title [WIP] Fix flaking tests Fix flaking tests Sep 29, 2022
@jacekbogdanski jacekbogdanski self-assigned this Sep 29, 2022
@jacekbogdanski jacekbogdanski merged commit 254552d into master Sep 29, 2022
@jacekbogdanski jacekbogdanski deleted the t/4.20.0-tests branch September 29, 2022 11:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants