Running integration tests with CI service

[HakkyuKim]

I tried several ways to run emulators in CI server for integration tests but concluded that it just cannot be done without digging into how Tizen emulators work under the hood. I did manage to run the emulator, but sdb can't find it: https://cirrus-ci.com/task/5968653813284864 (test branch: https://github.com/HakkyuKim/plugins/tree/cirrus).

Look for:

em-cli launch --name flutter_emulator
Launch command : "/home/tizen/tizen-studio/platforms/tizen-4.0/common/emulator/bin/emulator.sh" --conf "/home/tizen/tizen-studio-data/emulator/vms/flutter_emulator/vm_launch.conf" -j "/home/tizen/tizen-studio/jdk/bin/java"
The VM is launched
...

sdb devices
List of devices attached 

So instead, I'm considering running integration tests on our machine that's connected to physical targets using Azure Pipelines self-hosted agent.

There are a few considerations:

• Corporate policy regarding network security (I'll start asking around next week)
• Pricing
  According to pricing for Azure DevOps, we need at minimum "User License: Basic Plan", which is free for the first 5 users.
• Cost of maintenance
  The host machine and the Tizen devices attached to it.

Personally I think it's the best option to do CI for plugin integration tests and possibly future engine regression tests. Please share any ideas or suggestions.

TODOs:

• write test tool script (reference: flutter-plugin-tools) Running multiple integration tests script #150
• fix broken integration tests or exclude them in CI task: video_players, integration_test, etc Add integration test workflow #169 [permission_handler] Allow emulators to pass integration test #182
• write azure pipelinesgithub workflow script Add integration test workflow #169
• think about how to let the tool know about unsupported device profiles of each plugin Support testing on multiple targets #191

[swift-kim]

Regarding the pricing, we don't need to buy user licenses. We just need to add a self-hosted agent which costs $15 per month.

[HakkyuKim]

Pull requests will now trigger integration tests through internal server(self-hosted runner). Currently only a single self-hosted runner performs testing on wearable-5.5 emulator for all pull requests. The number of runners and testing platforms(device profiles) will be improved sometime next week. In the meantime, we may check how the system performs in our current development process.