Running multiple integration tests script

[HakkyuKim]

Script that runs multiple integration tests locally and aggregates the result. Implementation is referenced from flutter-plugin-tools. Please see the help message(./tools/run_integration_test.py -h) for details oh how to use the script.

Contributes to #148

Example Command:

# exclude plugins that don't have tests
# exclude failing tests
./tools/run_integration_test.py

Output:

============= TEST RESULT =============
FAILED: wearable_rotary
Missing integration tests (use --exclude if this is intentional).
FAILED: audioplayers
Missing integration tests (use --exclude if this is intentional).
FAILED: webview_flutter
flutter-tizen test integration_test failed, see the output above for details.
SUCCEEDED: network_info_plus
FAILED: package_info_plus
Timeout expired. The test may need more time to finish.
If you expect the test to finish before timeout, check if the tests 
require device screen to be awake or if they require manually 
clicking the UI button for permissions.
SUCCEEDED: battery
FAILED: image_picker
No tests ran.
SUCCEEDED: wifi_info_flutter
FAILED: camera
Timeout expired. The test may need more time to finish.
If you expect the test to finish before timeout, check if the tests 
require device screen to be awake or if they require manually 
clicking the UI button for permissions.
SUCCEEDED: sensors_plus
SUCCEEDED: flutter_tts
SUCCEEDED: path_provider
SUCCEEDED: device_info
SUCCEEDED: shared_preferences
SUCCEEDED: sensors
FAILED: integration_test
Timeout expired. The test may need more time to finish.
If you expect the test to finish before timeout, check if the tests 
require device screen to be awake or if they require manually 
clicking the UI button for permissions.
SUCCEEDED: url_launcher
SUCCEEDED: device_info_plus
SUCCEEDED: share
FAILED: video_player
Timeout expired. The test may need more time to finish.
If you expect the test to finish before timeout, check if the tests 
require device screen to be awake or if they require manually 
clicking the UI button for permissions.
SUCCEEDED: messageport
SUCCEEDED: battery_plus
SUCCEEDED: connectivity
SUCCEEDED: connectivity_plus
FAILED: package_info
Timeout expired. The test may need more time to finish.
If you expect the test to finish before timeout, check if the tests 
require device screen to be awake or if they require manually 
clicking the UI button for permissions.
SUCCEEDED: wakelock
FAILED: permission_handler
flutter-tizen test integration_test failed, see the output above for details.
SUCCEEDED: share_plus

Currently each plugin test will all run on the same connected device, and it cannot skip tests for plugins that aren't supported with that device's profile.

[swift-kim]

How is this related to the flutter_plugin_tools package? What's common between your implementation and flutter_plugin_tools?

[HakkyuKim]

@swift-kim
The flutter_plugin_tools package is a command line app and it has several features besides running integration tests. This python script referenced implementations of some commands related to running integration tests:

• command line interface design: argument naming, descriptions, and constraints, ...
• aggregation strategy: Defining test result states into success, fail, and skip. Implementations on what and how log messages are displayed.
• test policy: Which condition is considered a test failure? For example, plugin without integration tests is a test failure unless explicitly specifying it with the --exclude option.
• CI consideration: Adopting --run-on-changed-packages option for PR triggers.

[bbrto21]

Thanks for making such a great tool!
But while trying this out, I found something weird!

flutter tool logs

flutter-tizen drive --driver=test_driver/integration_test.dart --target=integration_test/path_provider_test.dart
Running "flutter pub get" in example...                            546ms
The platform profile is used for signing.                               
Building a Tizen application in debug mode...                      28.8s
✓ Built build/tizen/tpk/org.tizen.path_provider_tizen_example-1.0.0.tpk (25.8MB).
Installing build/tizen/tpk/org.tizen.path_provider_tizen_example-1.0.0.tpk...        24.4s
VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:45413/sCDkqKWjWBg=/
VMServiceFlutterDriver: Isolate found with number: 1779345267457467
VMServiceFlutterDriver: Isolate is paused at start.
VMServiceFlutterDriver: Attempting to resume isolate
VMServiceFlutterDriver: Connected to Flutter application.
All tests passed.

dlogs

...
I/ConsoleMessage(18728): flutter: 00:03 +6 -7: (tearDownAll)
I/ConsoleMessage(18728): flutter: 00:03 +7 -7: Some tests failed.

[HakkyuKim]

@bbrto21
I've checked the test code in path_provider and the results in the dlog is correct, the integration test shouldn't have passed. It seems that the drive command doesn't return the correct results. I'm currently not sure what's causing this. I'll look into it.

[HakkyuKim]

Although the problem mentioned by @bbrto21 is not caused by this script, the problem should be fixed for the script to work correctly. I'll work on that first.

flutter-tizen/flutter-tizen#168

[HakkyuKim]

With flutter-tizen/flutter-tizen#177, it's now possible to display correct results of integration_test using the command flutter-tizen test integration_test. The script is updated to use this command instead of flutter-tizen drive. Thanks @bbrto21 for noticing the problem, and @swift-kim for providing solutions to make the test command work.

Some plugins fail integration tests because they either require the screen to be awake or require manually clicking the UI button for permission requests. These are currently handled by setting timeouts.

[bbrto21]

LGTM

[swift-kim]

Some nitpicks. Overall looks good to me!