Fix test suite #1114

cpascual · 2020-05-27T10:00:27Z

The test suite is still too fuzzy, and the py35-qt5 target had to be quarantined.

This PR does a large refactoring of the tests (mostly the taurus/qt ones) to improve the situation. With it, the testsuite seems finally stable for all the targets. As a summary, here is what changed:

refactor all the qt tests to use pytest-qt (this seemed to fix many of the "random" crashes during test execution). And deprecate old unittest-based qt test utilities and TimeOut DS.
force to use PyTango >=9.3.1 for all targets (I had to provide my own conda package of PyTango v9.3.1 which was not officially available, see ab6888b )
Add some pytest-based utilities for new tests
Fix a few bugs in the code that affect the tests (8efbe87 , 14d94b8, e48a2d9, dcb6831, )
Remove taurus usages of deprecated APIs of dependencies (1d44c7a, 46ddb2c, b5173ad, 964b18f, 6cb42ff, 4aaf110, 5bc24ad) and configure the tests to treat future occurrences as errors (66e4d32)
change the tango docker images to use a mirror from those of ska-docker to avoid availability issues of ska's registry

Also, this PR removes the allowed-to-fail tests (python2 ones and flake8) from travis matrix (to avoid wasting time). They can be run locally and/or re-enabled at any moment

Add a fixture for launching TangoSchemeTest for a test

Parameterizable pytest test for setting models sequentially This is the Pytest equivalent to GenericWidgetTestCase.test10_SetModelsSequentially

Implement a context manager to facilitate checking deprecation errors in pytest tests

Leave only some relevant tests for quicker debugging of py35-qt5 testing issues

Re-enable test_form_itemFactory after refactoring it to use the qtbot fixture from pytest-qt

Refactor (to use qtbot) and re-enable the following: - test_form_itemFactory_selection - test_form_cwidget_bck_compat - test_form_itemFactory_loading

Try to avoid problems with tango in atexit() by forcing a previous unsubscribe in the test itself

- use cpascual conda forge (TEMPORARY) - set minimum version in tox.ini

py35-qt5 py36-qt5 py37-qt5 py37-ps2

This reverts commit abdf6ee.

This reverts commit f619202.

Use pytest-qt and parameterize. Also avoid segfaults in pyside2 tests due to PYSIDE-683 bug

The last refactoring introduced a failure in the last paramterization of test_TaurusValueCombobox. Try to fix it

The introduction of pytest-qt triggered several hidden issues in the PySide2 tests. Fix (or avoid) them

- Use pytest-qt - Skip if using PySide2 (to avoid segfault)

The taurus.external.qt.QtCore shim simply aliases pyqtProperty to Property when using the PySide2 binding. This produces segfaults during exit of tests if Property is called with the doc kwarg. Workaround by implementing pyqtProperty as a wrapper of Property that avoids passing the doc kwarg.

TaurusGui.loadConfiguration fails if the QApplication is not a TaurusApplication. This breaks the taurusgui test. Protect against this case

Reimplement using pytest and pytest-qt. The skipped tests are not ported.

Reimplement using pytest and pytest-qt.

Use pytest.mark.skipif instead of pytest.mark.skipIf

The *WidgetTestCase classes were created to help with Qt tests based on unittest. Since we moved all the Qt test to pytest-qt, we no longer want them (they are less robust and don't play well with the pytest-qt based tests) Mark them as deprecated and skip any test based on them.

Remove last usages of taurus.qt.qtgui.test.base in taurus code

TaurusLabel does not currently update its text when updating the format. Fix this by forcing a controllerUpdate() call whenever resetFormat is called. This was likely the cause of some of the fuzzyness in previous tests.

The "value" and "w_value" FgRoles do not raise deprecation warnings. Adapt the tests accordingly

The test_class_format is fuzzy because not always the format is reset (probably a race condition). Force an explicit format reset to avoid it.

For some reason, the check_taurus_deprecations context manager is not working ok when PySide2 is used via taurus.external.qt (maybe it has to do with the installed log manager?) As a workaround use xfail in these cases.

test_tauruslabel_text is not properly isolated and when it is called with FgRole="quality" it fails in PySide2 if it has been previously called with another FgRole. Workaround this by moving it to the first position.

Since python3.6 we get DeprecationWarning for invalid scape sequences. Fix them by marking the string literals as "raw"

Thread.isAlive was deprecated in 2.7 and is removed in 3.9/ Avoid it in our code

Since python3.3, Sequence and Mapping should be imported from collections.abc instead of from collections. Change code accordingly, keeping bck-compat for python2.7

Use logging.warning instead

Ensure file is closed.

AttributeTestCase.write_read_attribute() issues a pint warning. Refactor it to avoid unnecessary downcasting of Quantities to numpy arrays.

Usage of the VideoImageCodec raises the following deprecation warning in py3: "DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead" Replace fromstring -> frombuffer accordingly.

Ignore 3rd-party issued deprecation warnings and treat all warnings issued by taurus as errors (tests should fail if taurus emits a warning)

cpascual · 2020-06-12T17:30:22Z

any @taurus-org/integrators daring to have a look at this PR? ;)
It would be nice to integrate it soon to unblock the road for the other changes needed for the Jul20 release

tiagocoutinho · 2020-06-17T10:49:05Z

Ok, for what is worth I did an overview and it LGTM.

Also ran the tests locally (all except epics): Ran 565 tests in 6.247s

The CI likes it also go for it! Do you press the button or should I?

cpascual · 2020-06-17T11:18:21Z

Thanks @tiagocoutinho !

Do you press the button or should I?

do the honors yourself!

Carlos Pascual added 4 commits May 25, 2020 19:18

Add taurus_test_ds pytest fixture

0be8624

Add a fixture for launching TangoSchemeTest for a test

Add taurus_widget_set_models generic test

806cb0f

Parameterizable pytest test for setting models sequentially This is the Pytest equivalent to GenericWidgetTestCase.test10_SetModelsSequentially

Add check_taurus_deprecations CM

5651212

Implement a context manager to facilitate checking deprecation errors in pytest tests

[TEMPORARY] Disable of some test for debugging

6635f94

Leave only some relevant tests for quicker debugging of py35-qt5 testing issues

cpascual self-assigned this May 27, 2020

Carlos Pascual added 9 commits May 27, 2020 12:05

Disable newly added pytest tests

0679773

Add pytest-qt to tox.ini dependencies

bfc8305

Refactor and re-enable test_form_itemFactory

17d7e6e

Re-enable test_form_itemFactory after refactoring it to use the qtbot fixture from pytest-qt

Refactor and re-enable remaining form tests

4579516

Refactor (to use qtbot) and re-enable the following: - test_form_itemFactory_selection - test_form_cwidget_bck_compat - test_form_itemFactory_loading

set model to None as a tear down

26de611

Try to avoid problems with tango in atexit() by forcing a previous unsubscribe in the test itself

TEMPORARY Add debugging prints

f619202

TEMPORARY Disable ouput capture in tests

718b2a7

TEMPORARY: move debug print

abdf6ee

Use pytango >= 9.3 for all tox targets

ab6888b

- use cpascual conda forge (TEMPORARY) - set minimum version in tox.ini

cpascual force-pushed the fix_test_py3qt5 branch from ab6888b to 3d97c40 Compare June 2, 2020 12:09

cpascual added this to the Jul20 milestone Jun 2, 2020

cpascual added bug tests labels Jun 2, 2020

Carlos Pascual added 12 commits June 3, 2020 13:19

re-enable some travis targets

ef444c5

py35-qt5 py36-qt5 py37-qt5 py37-ps2

Revert "TEMPORARY: move debug print"

26a3102

This reverts commit abdf6ee.

Revert "TEMPORARY Add debugging prints"

f96c18b

This reverts commit f619202.

Refactor TaurusCombobox tests

48b8402

Use pytest-qt and parameterize. Also avoid segfaults in pyside2 tests due to PYSIDE-683 bug

Try fixing failure in combobox test

b5fdc75

The last refactoring introduced a failure in the last paramterization of test_TaurusValueCombobox. Try to fix it

Fix failures in py37-pyside2 tests

b2524c2

The introduction of pytest-qt triggered several hidden issues in the PySide2 tests. Fix (or avoid) them

Refactor test_jdraw_parser

2de66a2

- Use pytest-qt - Skip if using PySide2 (to avoid segfault)

re-enable doc travis target

9685e47

re-enable output capturing in pytest

3024abc

Do not assume TaurusApplication in taurusgui.py

14d94b8

TaurusGui.loadConfiguration fails if the QApplication is not a TaurusApplication. This breaks the taurusgui test. Protect against this case

Register gui widget in test_paneldescription

ae18436

Carlos Pascual added 16 commits June 9, 2020 20:51

Refactor taurusbase tests

6e0acf2

Reimplement using pytest and pytest-qt. The skipped tests are not ported.

Refactor ui tests

e6f466f

Reimplement using pytest and pytest-qt.

Refactor taurusvalue tests

147bb03

Reimplement using pytest and pytest-qt.

Fix typo in pytest skipif marks

4f0a771

Use pytest.mark.skipif instead of pytest.mark.skipIf

Remove left usages of taurus.qt.qtgui.test.base

23ec894

Remove last usages of taurus.qt.qtgui.test.base in taurus code

Update label controller on format reset

dcb6831

TaurusLabel does not currently update its text when updating the format. Fix this by forcing a controllerUpdate() call whenever resetFormat is called. This was likely the cause of some of the fuzzyness in previous tests.

Correct deprecation check for label test

611fa16

The "value" and "w_value" FgRoles do not raise deprecation warnings. Adapt the tests accordingly

Fix fuzziness in test_class_format

8a129bd

The test_class_format is fuzzy because not always the format is reset (probably a race condition). Force an explicit format reset to avoid it.

Workaround: Allow fails in depr tests for PySide2

e71762a

For some reason, the check_taurus_deprecations context manager is not working ok when PySide2 is used via taurus.external.qt (maybe it has to do with the installed log manager?) As a workaround use xfail in these cases.

Workaround: improper isolation on label tests

4addee3

test_tauruslabel_text is not properly isolated and when it is called with FgRole="quality" it fails in PySide2 if it has been previously called with another FgRole. Workaround this by moving it to the first position.

Avoid invalid escape sequence warnings

1d44c7a

Since python3.6 we get DeprecationWarning for invalid scape sequences. Fix them by marking the string literals as "raw"

replace Thread.isAlive --> is_alive

46ddb2c

Thread.isAlive was deprecated in 2.7 and is removed in 3.9/ Avoid it in our code

Use collections.abc for ABCs

b5173ad

Since python3.3, Sequence and Mapping should be imported from collections.abc instead of from collections. Change code accordingly, keeping bck-compat for python2.7

Avoid using deprecated logging.warn

964b18f

Use logging.warning instead

Avoid file not closed warning

6cb42ff

Ensure file is closed.

cpascual force-pushed the fix_test_py3qt5 branch from 43ecbde to 6cb42ff Compare June 11, 2020 15:38

cpascual changed the title ~~[WIP] Fix test py3qt5~~ [WIP] Fix test suite Jun 11, 2020

Carlos Pascual added 3 commits June 12, 2020 18:37

Avoid warning in write_read_attribute tests

4aaf110

AttributeTestCase.write_read_attribute() issues a pint warning. Refactor it to avoid unnecessary downcasting of Quantities to numpy arrays.

Add pytest.ini to consider warnings as errors

66e4d32

Ignore 3rd-party issued deprecation warnings and treat all warnings issued by taurus as errors (tests should fail if taurus emits a warning)

cpascual force-pushed the fix_test_py3qt5 branch from 0cb35aa to 66e4d32 Compare June 12, 2020 16:41

cpascual changed the title ~~[WIP] Fix test suite~~ Fix test suite Jun 12, 2020

cpascual mentioned this pull request Jun 15, 2020

Config loaders #1116

Draft

3 tasks

tiagocoutinho merged commit e3c9050 into taurus-org:develop Jun 17, 2020

cpascual deleted the fix_test_py3qt5 branch June 18, 2020 14:23

cpascual mentioned this pull request Jun 22, 2020

Remove/reduce warnings during tests #1040

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix test suite #1114

Fix test suite #1114

Uh oh!

cpascual commented May 27, 2020 •

edited

Loading

Uh oh!

cpascual commented Jun 12, 2020

Uh oh!

tiagocoutinho commented Jun 17, 2020

Uh oh!

cpascual commented Jun 17, 2020 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Fix test suite #1114

Fix test suite #1114

Uh oh!

Conversation

cpascual commented May 27, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cpascual commented Jun 12, 2020

Uh oh!

tiagocoutinho commented Jun 17, 2020

Uh oh!

cpascual commented Jun 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cpascual commented May 27, 2020 •

edited

Loading

cpascual commented Jun 17, 2020 •

edited

Loading