fix(cypress): Fix failing/flaky E2E tests by codyml · Pull Request #22460 · apache/superset

codyml · 2022-12-20T00:31:25Z

SUMMARY

This PR attempts to fix some failing/flaky Cypress tests:

Horizontal FilterBar > should spot changes in "more filters" and apply their values (fail 23%, flake 22% on master last 7 days)
Drill to detail modal > Tier 1 charts > Modal actions > paginates (fail 8%, flake 30% on master last 7 days)
Charts list > Cross-referenced dashboards > should show the newly added dashboards in a tooltip (fail 0%, flake 80% on master last 7 days)
Horizontal FilterBar > should go from vertical to horizontal and the opposite (fail 3%, flake 22% on master last 7 days)
Horizontal FilterBar > should show all default actions in horizontal mode (fail 0%, flake 13% on master last 7 days)
Horizontal FilterBar > should display newly added filter (fail 2%, flake 13% on master last 7 days)

The code changes include:

Replacing .then with .should calls in drilltodetail.test.ts to allow for retries within the callback
Adding a timed .wait in nativeFilters.test.ts to give the filter bar orientation menu time to change orientation after scrolling into position
Switching the target for .type from the input's parent to the input itself in horizontal filter bar tests, as the input's parent was at times not considered "visible" by Cypress
Making Cypress wait for the confirmation toast to appear after saving a chart to a dashboard before navigating away from the page

TESTING INSTRUCTIONS

Check that E2E tests pass on CI, and feel free to re-run a few times to try to catch any extra issues
Run Cypress E2E tests locally several times with throttling enabled, specifically looking for issues in:
- dashboard/drilltodetail.test.ts
- dashboard/nativeFilters.test.ts
- chart_list/list.test.ts

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

…lters" and apply their values.'

codecov · 2022-12-20T01:24:13Z

Codecov Report

Merging #22460 (f1f0a91) into master (858796b) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #22460   +/-   ##
=======================================
  Coverage   66.89%   66.89%           
=======================================
  Files        1850     1850           
  Lines       70701    70701           
  Branches     7750     7750           
=======================================
  Hits        47292    47292           
  Misses      21393    21393           
  Partials     2016     2016

Flag	Coverage Δ
javascript	`53.86% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

villebro

LGTM, thanks for taking on this challenge, as CI has been pretty painful lately ❤️

villebro · 2022-12-20T10:11:50Z

    const { body } = res;
    const dashboardId = body.result.id;
-    const allFilters: Record<string, any>[] = [];
+    const allFilters: Record<string, unknown>[] = [];


I believe these are Filters:

import { Filters } from '@superset-ui/core'; const allFilters: Record<string, Filters>[] = [];

Hmm, I tried this but it gave me a "Module '"@superset-ui/core"' has no exported member 'Filter'." error. I also didn't see anywhere else in superset-frontend/cypress-base that imports from @superset-ui/core – do you know if these packages are supposed to be isolated for any reason?

geido

LGTM!

kgabryje · 2022-12-20T12:56:05Z

        cy.get('.ant-pagination-item')
          .should('have.length', 6)
-          .then($pages => {
+          .should($pages => {


Cool, I didn't know that should waits and then doesn't!

Me either, this PR was a learning experience!

kgabryje · 2022-12-20T13:00:25Z


 function setFilterBarOrientation(orientation: 'vertical' | 'horizontal') {
  cy.getBySel('filterbar-orientation-icon').click();
+  cy.wait(250);


Can we add timeout to getBySel instead of this wait? Like cy.getBySel('dropdown-selectable-info', { timeout: 250 })

Actually, not sure which method do we prefer in Superset, so feel free to leave it as is 🙂

Nevermind, I didn't notice that it's merged 😄

Haha no worries! The issue seemed to be that the loading of the page caused some scrolling action while the menu was opening, which disrupted Cypress's ability to click on the menu item even though it is able to find it, so this .wait hopefully will let the scrolling finish in more cases before Cypress tries to click on the menu item.

EugeneTorap · 2022-12-21T10:37:22Z

Hi @codyml! I have problem with chart_list/filter.test.ts and dashboard_list/filter.test.ts

EugeneTorap · 2022-12-21T13:09:33Z

@codyml Can you see my #22489 PR and check failed cypress CI?

codyml added 5 commits December 19, 2022 21:55

Fix typing lint warning.

9757c7f

Adjust filter selection logic to fix 'should spot changes in "more fi…

4bdba37

…lters" and apply their values.'

Add wait to give dropdown time to reposition.

de8e46f

Change then to should to enable retries.

9b01795

Wait for chart save confirmation before proceeding.

f1f0a91

codyml requested review from geido and jinghua-qa as code owners December 20, 2022 00:31

pull-request-size Bot added the size/S label Dec 20, 2022

villebro approved these changes Dec 20, 2022

View reviewed changes

geido approved these changes Dec 20, 2022

View reviewed changes

geido merged commit 21b56b2 into apache:master Dec 20, 2022

kgabryje reviewed Dec 20, 2022

View reviewed changes

This was referenced Jan 3, 2023

chore: Migrate /superset/search_queries to API v1 #22579

Merged

chore: Migrate /superset/queries/<last_updated_ms> to API v1 #22611

Merged

mistercrunch added the 🚢 2.1.3 First shipped in 2.1.3 label Feb 18, 2024

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 2.1.0 First shipped in 2.1.0 and removed 🚢 2.1.3 First shipped in 2.1.3 labels Mar 13, 2024

qfcwell pushed a commit to qfcwell/superset that referenced this pull request May 12, 2026

fix(cypress): Fix failing/flaky E2E tests (apache#22460)

362de00

Conversation

codyml commented Dec 20, 2022

SUMMARY

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Uh oh!

codecov Bot commented Dec 20, 2022

Codecov Report

Uh oh!

villebro left a comment

Choose a reason for hiding this comment

Uh oh!

villebro Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

codyml Dec 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geido left a comment

Choose a reason for hiding this comment

Uh oh!

kgabryje Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

codyml Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

kgabryje Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

kgabryje Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

kgabryje Dec 20, 2022

Choose a reason for hiding this comment

Uh oh!

codyml Dec 20, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

EugeneTorap commented Dec 21, 2022

Uh oh!

EugeneTorap commented Dec 21, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codyml Dec 20, 2022 •

edited

Loading

codyml Dec 20, 2022 •

edited

Loading