EDIT: See my comment below for a table of current test failures on main.
Description
When trying to fix an issue with test failures run against PRs (e.g. #1088 identifies a breakage with /merge), I had multiple integration test failures from a relatively simple PR against main (#1106 pinning a dependency to the version before a recent release).
To troubleshoot this, I made a simpler PR against main (#1107 editing CONTRIBUTORS), which also has integration test failures.
Here is a table showing the number of times each test failured over 4 runs of the integration tests on these two 2 PRs.
| test |
#1106 |
#1107 |
| test_app_relation_destroy_block_until_done |
4 |
4 |
| test_deploy_bundle_with_multiple_overlays_with_include_files |
4 |
4 |
| test_deploy_bundle_with_overlay_as_argument |
4 |
4 |
| test_wait_for_idle_more_units_than_needed |
3 |
3 |
| test_upgrade_local_charm - juju... |
2 |
4 |
| test_wait_for_idle_with_not_enough_units |
2 |
2 |
| test_unit_annotations - asyncio.excep... |
2 |
0 |
| test_action - juju.errors.JujuA... |
1 |
1 |
| test_upgrade_local_charm_resource |
0 |
2 |
| test_attach_resource - asyncio.except... |
0 |
1 |
It would be ideal if the tests were deterministic.
Short of fixing the tests themselves, it would be good if the current state of the tests was prominently documented in contributing guidelines -- which test failures are likely to just be the tests being flaky and shouldn't block a merge. A separate issue can then be opened against that list to fix the individual tests.
Urgency
Blocker for our release
Python-libjuju version
main?
Juju version
the version the github workflow is using (doesn't look like versioning for juju etc is printed during test setup)
Reproduce / Test
Run integration tests on main.
EDIT: See my comment below for a table of current test failures on main.
Description
When trying to fix an issue with test failures run against PRs (e.g. #1088 identifies a breakage with
/merge), I had multiple integration test failures from a relatively simple PR against main (#1106 pinning a dependency to the version before a recent release).To troubleshoot this, I made a simpler PR against main (#1107 editing CONTRIBUTORS), which also has integration test failures.
Here is a table showing the number of times each test failured over 4 runs of the integration tests on these two 2 PRs.
It would be ideal if the tests were deterministic.
Short of fixing the tests themselves, it would be good if the current state of the tests was prominently documented in contributing guidelines -- which test failures are likely to just be the tests being flaky and shouldn't block a merge. A separate issue can then be opened against that list to fix the individual tests.
Urgency
Blocker for our release
Python-libjuju version
main?
Juju version
the version the github workflow is using (doesn't look like versioning for juju etc is printed during test setup)
Reproduce / Test
Run integration tests on
main.