From a7a47eb76ee6a7bdd239525805339688cda75611 Mon Sep 17 00:00:00 2001 From: Matan Lurey Date: Wed, 14 Feb 2024 10:55:20 -0800 Subject: [PATCH 1/8] Re-organize scenario_app documentation a bit. --- testing/scenario_app/README.md | 94 +++--------- testing/scenario_app/android/README.md | 28 ++++ testing/scenario_app/bin/README.md | 5 + .../bin/android_integration_tests.dart | 139 +++++++++++++----- testing/scenario_app/ios/README.md | 50 +++++++ testing/scenario_app/lib/README.md | 16 ++ 6 files changed, 220 insertions(+), 112 deletions(-) create mode 100644 testing/scenario_app/android/README.md create mode 100644 testing/scenario_app/bin/README.md create mode 100644 testing/scenario_app/ios/README.md create mode 100644 testing/scenario_app/lib/README.md diff --git a/testing/scenario_app/README.md b/testing/scenario_app/README.md index ce80b24246356..6ebeb56c88c40 100644 --- a/testing/scenario_app/README.md +++ b/testing/scenario_app/README.md @@ -1,85 +1,31 @@ # Scenario App -This folder contains e2e integration tests for the engine in conjunction with a -fake dart:ui framework running in JIT or AOT. +This package simulates a Flutter app that uses the engine (`dart:ui`) only, +in conjunction with Android and iOS-specific embedding code that simulates the +use of the engine in a real app (such as plugins and platform views). -It intentionally has no dependencies on the Flutter framework or tooling, such -that it should be buildable as a presubmit or postsubmit to the engine even in -the face of changes to Dart or dart:ui that require upstream changes in the -Flutter tooling. +To run the tests, you will need to build the engine with the appropriate +configuration. The [`run_android_tests.sh`](run_android_tests.sh) and +[`run_ios_tests.sh`](run_ios_tests.sh) are then used to run the tests on a +connected device or emulator. -## Adding a New Scenario +See also: -Create a new subclass of [Scenario](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/lib/src/scenario.dart#L9) -and add it to the map in [scenarios.dart](https://github.com/flutter/engine/blob/db4d423ad9c6dad373618712690acd06b0a385fd/testing/scenario_app/lib/src/scenarios.dart#L22). -For an example, see [animated_color_square.dart](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/lib/src/animated_color_square.dart#L15), -which draws a continuously animating colored square that bounces off the sides -of the viewport. +- [`bin/`](bin/), the entry point for running Android integration tests. +- [`lib/`](lib/), the Dart code and instrumentation for the scenario app. +- [`ios/`](ios/), the iOS-side native code and tests. +- [`android/`](android/), the Android-side native code and tests. -Then set the scenario from the Android or iOS app by calling "set_scenario" on -platform channel. +## Running a smoke test on Firebase TestLab -## Running for iOS - -Build the `ios_debug_sim_unopt` engine variant, and run +To run the smoke test on Firebase TestLab test, build `android_profile_arm64`, +and run [`./ci/firebase_testlab.py`](../../ci/firebase_testlab.py), or pass +`--variant` to run a different configuration. ```sh -./run_ios_tests.sh +# From the root of the engine repository +$ ./ci/firebase_testlab.py --variant android_debug_arm64 ``` -in your shell. - -To run or debug in Xcode, open the xcodeproj file located in -`/ios_debug_sim_unopt/scenario_app/Scenarios/Scenarios.xcodeproj`. - -### iOS Platform View Tests - -For PlatformView tests on iOS, you'll also have to edit the dictionaries in -[AppDelegate.m](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/ios/Scenarios/Scenarios/AppDelegate.m#L29) and [GoldenTestManager.m](https://github.com/flutter/engine/blob/db4d423ad9c6dad373618712690acd06b0a385fd/testing/scenario_app/ios/Scenarios/ScenariosUITests/GoldenTestManager.m#L25) so that the correct golden image can be found. Also, you'll have to add a [GoldenPlatformViewTests](https://github.com/flutter/engine/blob/5d9509ae056b04c30295df27f201f31af9777842/testing/scenario_app/ios/Scenarios/ScenariosUITests/GoldenPlatformViewTests.h#L18) in [PlatformViewUITests.m](https://github.com/flutter/engine/blob/af2ffc02b72af2a89242ca3c89e18269b1584ce5/testing/scenario_app/ios/Scenarios/ScenariosUITests/PlatformViewUITests.m). - -If `PlatformViewRotation` is failing, make sure Simulator app Device > Rotate Device Automatically -is selected, or run: - -```bash -defaults write com.apple.iphonesimulator RotateWindowWhenSignaledByGuest -int 1 -``` - -### Generating Golden Images on iOS - -Screenshots are saved as -[XCTAttachment](https://developer.apple.com/documentation/xctest/activities_and_attachments/adding_attachments_to_tests_and_activities?language=objc)'s. -If you look at the output from running the tests you'll find a path in the form: -`/Users/$USER/Library/Developer/Xcode/DerivedData/Scenarios-$HASH`. -Inside that directory you'll find -`./Build/Products/Debug-iphonesimulator/ScenariosUITests-Runner.app/PlugIns/ScenariosUITests.xctest/` which is where all the images that were -compared against golden reside. - -## Running for Android - -### Integration tests - -For emulators running on a x64 host, build `android_debug_unopt_x64` using -`./tools/gn --android --unoptimized --goma --android-cpu=x64`. - -Then, launch the emulator, and run `./testing/scenario_app/run_android_tests.sh android_debug_unopt_x64`. - -If you wish to build a different engine variant, make sure to pass that variant to the script `run_android_tests.sh`. - -If you make a change to the source code, you would need to rebuild the same engine variant. - -### Smoke test on FTL - -To run the smoke test on Firebase TestLab test, build `android_profile_arm64`, and run -`./flutter/ci/firebase_testlab.py`. If you wish to test a different variant, e.g. -debug arm64, pass `--variant android_debug_arm64`. - -### Updating Gradle dependencies - -If a Gradle dependency is updated, lockfiles must be regenerated. - -To generate new lockfiles, run: - -```bash -cd android/app -../../../../../third_party/gradle/bin/gradle generateLockfiles -``` +> ![NOTE] +> These instructions were not verified at the time of writing/refactoring. diff --git a/testing/scenario_app/android/README.md b/testing/scenario_app/android/README.md new file mode 100644 index 0000000000000..3880b4c7618e1 --- /dev/null +++ b/testing/scenario_app/android/README.md @@ -0,0 +1,28 @@ +# Scenario App: Android Tests + +As mentioned in the [top-level README](../README.md), this directory contains +the Android-specific native code and tests for the [scenario app](../lib). To +run the tests, you will need to build the engine with the appropriate +configuration. + +For example, `android_debug_unopt` or `android_debug_unopt_arm64` was built, +run: + +```sh +# From the root of the engine repository +$ ./testing/run_android_tests.sh android_debug_unopt + +# Or, for arm64 +$ ./testing/run_android_tests.sh android_debug_unopt_arm64 +``` + +## Updating Gradle dependencies + +If a Gradle dependency is updated, lockfiles must be regenerated. + +To generate new lockfiles, run: + +```bash +cd android/app +../../../../../third_party/gradle/bin/gradle generateLockfiles +``` diff --git a/testing/scenario_app/bin/README.md b/testing/scenario_app/bin/README.md new file mode 100644 index 0000000000000..76737692bd957 --- /dev/null +++ b/testing/scenario_app/bin/README.md @@ -0,0 +1,5 @@ +# `android_integration_tests` runner + +This directory contains code specific to running Android integration tests. + +See [`android_integration_tests.dart`](android_integration_tests.dart). diff --git a/testing/scenario_app/bin/android_integration_tests.dart b/testing/scenario_app/bin/android_integration_tests.dart index 3332de5f7f81d..2964aaa91bf80 100644 --- a/testing/scenario_app/bin/android_integration_tests.dart +++ b/testing/scenario_app/bin/android_integration_tests.dart @@ -16,8 +16,44 @@ import 'utils/logs.dart'; import 'utils/process_manager_extension.dart'; import 'utils/screenshot_transformer.dart'; -const int tcpPort = 3001; - +/// Runs the Android integration tests on a connected device or emulator. +/// +/// The tests are uploaded and run on the device using `adb`, and screenshots +/// are captured and compared using Skia Gold (if available, for example on CI). +/// +/// ## Usage +/// +/// ```sh +/// dart bin/android_integration_tests.dart \ +/// --adb ../third_party/android_tools/sdk/platform-tools/adb \ +/// --out-dir ../out/android_debug_unopt_arm64 +/// ``` +/// +/// ## Debugging +/// +/// When debugging, you can use the `--smoke-test` argument to run a single test +/// by class name, which can be useful to verify the setup. +/// +/// For example, to run the `EngineLaunchE2ETest` test: +/// +/// ```sh +/// dart bin/android_integration_tests.dart \ +/// --adb ../third_party/android_tools/sdk/platform-tools/adb \ +/// --out-dir ../out/android_debug_unopt_arm64 \ +/// --smoke-test dev.flutter.scenarios.EngineLaunchE2ETest +/// ``` +/// +/// ## Additional arguments +/// +/// - `--use-skia-gold`: Use Skia Gold to compare screenshots. Defaults to true +/// when running on CI, and false otherwise (i.e. when running locally). If +/// set to true, [isSkiaGoldClientAvailable] must be true. +/// +/// - `--enable-impeller`: Enable Impeller for the Android app. Defaults to +/// false, which means that the app will use Skia as the graphics backend. +/// +/// - `--impeller-backend`: The Impeller backend to use for the Android app. +/// Defaults to 'vulkan'. Only used when `--enable-impeller` is set to true. void main(List args) async { final ArgParser parser = ArgParser() ..addOption( @@ -58,9 +94,13 @@ void main(List args) async { final bool useSkiaGold = results['use-skia-gold'] as bool; final String? smokeTest = results['smoke-test'] as String?; final bool enableImpeller = results['enable-impeller'] as bool; - final _ImpellerBackend? impellerBackend = _ImpellerBackend.tryParse(results['impeller-backend'] as String?); + final _ImpellerBackend? impellerBackend = + _ImpellerBackend.tryParse(results['impeller-backend'] as String?); if (enableImpeller && impellerBackend == null) { - panic(['invalid graphics-backend', results['impeller-backend'] as String? ?? '']); + panic([ + 'invalid graphics-backend', + results['impeller-backend'] as String? ?? '' + ]); } await _run( outDir: outDir, @@ -82,6 +122,8 @@ void main(List args) async { ); } +const int _tcpPort = 3001; + enum _ImpellerBackend { vulkan, opengles; @@ -107,7 +149,10 @@ Future _run({ const ProcessManager pm = LocalProcessManager(); if (!outDir.existsSync()) { - panic(['out-dir does not exist: $outDir', 'make sure to build the selected engine variant']); + panic([ + 'out-dir does not exist: $outDir', + 'make sure to build the selected engine variant' + ]); } if (!adb.existsSync()) { @@ -118,15 +163,22 @@ Future _run({ final String logcatPath = join(scenarioAppPath, 'logcat.txt'); final String screenshotPath = join(scenarioAppPath, 'screenshots'); final String apkOutPath = join(scenarioAppPath, 'app', 'outputs', 'apk'); - final File testApk = File(join(apkOutPath, 'androidTest', 'debug', 'app-debug-androidTest.apk')); + final File testApk = File( + join(apkOutPath, 'androidTest', 'debug', 'app-debug-androidTest.apk')); final File appApk = File(join(apkOutPath, 'debug', 'app-debug.apk')); if (!testApk.existsSync()) { - panic(['test apk does not exist: ${testApk.path}', 'make sure to build the selected engine variant']); + panic([ + 'test apk does not exist: ${testApk.path}', + 'make sure to build the selected engine variant' + ]); } if (!appApk.existsSync()) { - panic(['app apk does not exist: ${appApk.path}', 'make sure to build the selected engine variant']); + panic([ + 'app apk does not exist: ${appApk.path}', + 'make sure to build the selected engine variant' + ]); } // Start a TCP socket in the host, and forward it to the device that runs the tests. @@ -134,39 +186,41 @@ Future _run({ // for the screenshots. // On LUCI, the host uploads the screenshots to Skia Gold. SkiaGoldClient? skiaGoldClient; - late ServerSocket server; + late ServerSocket server; final List> pendingComparisons = >[]; await step('Starting server...', () async { - server = await ServerSocket.bind(InternetAddress.anyIPv4, tcpPort); - stdout.writeln('listening on host ${server.address.address}:${server.port}'); + server = await ServerSocket.bind(InternetAddress.anyIPv4, _tcpPort); + stdout + .writeln('listening on host ${server.address.address}:${server.port}'); server.listen((Socket client) { - stdout.writeln('client connected ${client.remoteAddress.address}:${client.remotePort}'); - client.transform(const ScreenshotBlobTransformer()).listen((Screenshot screenshot) { + stdout.writeln( + 'client connected ${client.remoteAddress.address}:${client.remotePort}'); + client.transform(const ScreenshotBlobTransformer()).listen( + (Screenshot screenshot) { final String fileName = screenshot.filename; final Uint8List fileContent = screenshot.fileContent; log('host received ${fileContent.lengthInBytes} bytes for screenshot `$fileName`'); assert(skiaGoldClient != null, 'expected Skia Gold client'); late File goldenFile; try { - goldenFile = File(join(screenshotPath, fileName))..writeAsBytesSync(fileContent, flush: true); + goldenFile = File(join(screenshotPath, fileName)) + ..writeAsBytesSync(fileContent, flush: true); } on FileSystemException catch (err) { panic(['failed to create screenshot $fileName: $err']); } log('wrote ${goldenFile.absolute.path}'); if (isSkiaGoldClientAvailable) { final Future comparison = skiaGoldClient! - .addImg(fileName, goldenFile, - screenshotSize: screenshot.pixelCount) - .catchError((dynamic err) { - panic(['skia gold comparison failed: $err']); - }); + .addImg(fileName, goldenFile, + screenshotSize: screenshot.pixelCount) + .catchError((dynamic err) { + panic(['skia gold comparison failed: $err']); + }); pendingComparisons.add(comparison); } - }, - onError: (dynamic err) { + }, onError: (dynamic err) { panic(['error while receiving bytes: $err']); - }, - cancelOnError: true); + }, cancelOnError: true); }); }); @@ -180,7 +234,8 @@ Future _run({ }); await step('Starting logcat...', () async { - final int exitCode = await pm.runAndForward([adb.path, 'logcat', '-c']); + final int exitCode = + await pm.runAndForward([adb.path, 'logcat', '-c']); if (exitCode != 0) { panic(['could not clear logs']); } @@ -204,21 +259,25 @@ Future _run({ }); await step('Get API level of connected device...', () async { - final ProcessResult apiLevelProcessResult = await pm.run([adb.path, 'shell', 'getprop', 'ro.build.version.sdk']); + final ProcessResult apiLevelProcessResult = await pm + .run([adb.path, 'shell', 'getprop', 'ro.build.version.sdk']); if (apiLevelProcessResult.exitCode != 0) { panic(['could not get API level of the connected device']); } - final String connectedDeviceAPILevel = (apiLevelProcessResult.stdout as String).trim(); + final String connectedDeviceAPILevel = + (apiLevelProcessResult.stdout as String).trim(); final Map dimensions = { 'AndroidAPILevel': connectedDeviceAPILevel, - 'GraphicsBackend': enableImpeller ? 'impeller-${impellerBackend!.name}' : 'skia', + 'GraphicsBackend': + enableImpeller ? 'impeller-${impellerBackend!.name}' : 'skia', }; log('using dimensions: ${json.encode(dimensions)}'); skiaGoldClient = SkiaGoldClient( outDir, dimensions: { 'AndroidAPILevel': connectedDeviceAPILevel, - 'GraphicsBackend': enableImpeller ? 'impeller-${impellerBackend!.name}' : 'skia', + 'GraphicsBackend': + enableImpeller ? 'impeller-${impellerBackend!.name}' : 'skia', }, ); }); @@ -237,21 +296,24 @@ Future _run({ }); await step('Reverse port...', () async { - final int exitCode = await pm.runAndForward([adb.path, 'reverse', 'tcp:3000', 'tcp:$tcpPort']); + final int exitCode = await pm.runAndForward( + [adb.path, 'reverse', 'tcp:3000', 'tcp:$_tcpPort']); if (exitCode != 0) { panic(['could not forward port']); } }); await step('Installing app APK...', () async { - final int exitCode = await pm.runAndForward([adb.path, 'install', appApk.path]); + final int exitCode = + await pm.runAndForward([adb.path, 'install', appApk.path]); if (exitCode != 0) { panic(['could not install app apk']); } }); await step('Installing test APK...', () async { - final int exitCode = await pm.runAndForward([adb.path, 'install', testApk.path]); + final int exitCode = + await pm.runAndForward([adb.path, 'install', testApk.path]); if (exitCode != 0) { panic(['could not install test apk']); } @@ -264,11 +326,9 @@ Future _run({ 'am', 'instrument', '-w', - if (smokeTestFullPath != null) - '-e class $smokeTestFullPath', + if (smokeTestFullPath != null) '-e class $smokeTestFullPath', 'dev.flutter.scenarios.test/dev.flutter.TestRunner', - if (enableImpeller) - '-e enable-impeller', + if (enableImpeller) '-e enable-impeller', if (impellerBackend != null) '-e impeller-backend ${impellerBackend.name}', ]); @@ -290,7 +350,8 @@ Future _run({ final int exitCode = await pm.runAndForward([ adb.path, 'reverse', - '--remove', 'tcp:3000', + '--remove', + 'tcp:3000', ]); if (exitCode != 0) { panic(['could not unforward port']); @@ -298,14 +359,16 @@ Future _run({ }); await step('Uinstalling app APK...', () async { - final int exitCode = await pm.runAndForward([adb.path, 'uninstall', 'dev.flutter.scenarios']); + final int exitCode = await pm.runAndForward( + [adb.path, 'uninstall', 'dev.flutter.scenarios']); if (exitCode != 0) { panic(['could not uninstall app apk']); } }); await step('Uinstalling test APK...', () async { - final int exitCode = await pm.runAndForward([adb.path, 'uninstall', 'dev.flutter.scenarios.test']); + final int exitCode = await pm.runAndForward( + [adb.path, 'uninstall', 'dev.flutter.scenarios.test']); if (exitCode != 0) { panic(['could not uninstall app apk']); } diff --git a/testing/scenario_app/ios/README.md b/testing/scenario_app/ios/README.md new file mode 100644 index 0000000000000..3c5444b16eec9 --- /dev/null +++ b/testing/scenario_app/ios/README.md @@ -0,0 +1,50 @@ +# Scenario App: iOS Tests + +As mentioned in the [top-level README](../README.md), this directory contains +the iOS-specific native code and tests for the [scenario app](../lib). To run +the tests, you will need to build the engine with the appropriate configuration. + +For example, `ios_debug_sim_unopt` or `ios_debug_sim_unopt_arm64` was built, +run: + +```sh +# From the root of the engine repository +$ ./testing/run_ios_tests.sh +``` + +To run or debug in Xcode, open the xcodeproj file located in +`/ios_debug_sim_unopt/scenario_app/Scenarios/Scenarios.xcodeproj`. + +## CI Configuration + +See [`ci/builders/mac_unopt.json`](../../../../ci/builders/mac_unopt.json), and +grep for `run_ios_tests.sh`. + +## iOS Platform View Tests + +For PlatformView tests on iOS, edit the dictionaries in +[AppDelegate.m](Scenarios/Scenarios/AppDelegate.m) and +[GoldenTestManager.m](Scenarios/ScenariosUITests/GoldenTestManager.m) so that +the correct golden image can be found. Also, add a +[GoldenPlatformViewTests](Scenarios/ScenariosUITests/GoldenPlatformViewTests.h) +in [PlatformViewUITests.m](Scenarios/ScenariosUITests/PlatformViewUITests.m). + +If `PlatformViewRotation` is failing, make sure +`Simulator app Device > Rotate Device Automatically` is selected, or run: + +```bash +defaults write com.apple.iphonesimulator RotateWindowWhenSignaledByGuest -int 1 +``` + +## Generating Golden Images on iOS + +Screenshots are saved as +[XCTAttachment](https://developer.apple.com/documentation/xctest/activities_and_attachments/adding_attachments_to_tests_and_activities?language=objc)'s. + +A path in the form of +`/Users/$USER/Library/Developer/Xcode/DerivedData/Scenarios-$HASH` will be +printed to the console. + +Inside that directory there is a directory +`./Build/Products/Debug-iphonesimulator/ScenariosUITests-Runner.app/PlugIns/ScenariosUITests.xctest/` +which is where all the images that were compared against golden reside. diff --git a/testing/scenario_app/lib/README.md b/testing/scenario_app/lib/README.md new file mode 100644 index 0000000000000..0f9c016ce5fca --- /dev/null +++ b/testing/scenario_app/lib/README.md @@ -0,0 +1,16 @@ +# `package:scenario_app` + +This package simulates a Flutter app that uses the engine (`dart:ui`) only. It +is used to test the engine in isolation from the rest of the Flutter framework +and tooling. + +## Adding a New Scenario + +Create a new subclass of [Scenario](src/scenario.dart) and add it to the map +in [scenarios.dart](src/scenarios.dart). For an example, see +[animated_color_square.dart](src/animated_color_square.dart), which draws a +continuously animating colored square that bounces off the sides of the +viewport. + +Then set the scenario from the Android or iOS app by calling `set_scenario` on +platform channel `driver`. From 6b95e94f405c658f9683416d12f13fe75c46d7ab Mon Sep 17 00:00:00 2001 From: Matan Lurey Date: Wed, 14 Feb 2024 11:20:26 -0800 Subject: [PATCH 2/8] Add matrix and change gradle instructions. --- testing/scenario_app/android/README.md | 28 +++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/testing/scenario_app/android/README.md b/testing/scenario_app/android/README.md index 3880b4c7618e1..9274cbcd9c7af 100644 --- a/testing/scenario_app/android/README.md +++ b/testing/scenario_app/android/README.md @@ -16,13 +16,27 @@ $ ./testing/run_android_tests.sh android_debug_unopt $ ./testing/run_android_tests.sh android_debug_unopt_arm64 ``` -## Updating Gradle dependencies +## CI Configuration -If a Gradle dependency is updated, lockfiles must be regenerated. +See [`ci/builders/linux_android_emulator.json`](../../../ci/builders/linux_android_emulator.json) +, and grep for `run_android_tests.sh`. -To generate new lockfiles, run: +The following matrix of configurations is tested on the CI: -```bash -cd android/app -../../../../../third_party/gradle/bin/gradle generateLockfiles -``` +| API Version | Graphics Backend | Skia Gold | Rationale | +| ----------- | ------------------- | -------------------------------------- | ---------------------------------------------------------- | +| 28 | Skia | [Link][skia-gold-skia-28] | Older Android devices (without `ImageReader`) on Skia. | +| 28 | Impeller (OpenGLES) | [Link][skia-gold-impeller-opengles-28] | Older Android devices (without `ImageReader`) on Impeller. | +| 34 | Skia | [Link][skia-gold-skia-34] | Newer Android devices on Skia. | +| 34 | Impeller (OpenGLES) | [Link][skia-gold-impeller-opengles-34] | Newer Android devices on Impeller with OpenGLES. | +| 34 | Impeller (Vulkan) | [Link][skia-gold-impeller-vulkan-34] | Newer Android devices on Impeller. | + +[skia-gold-skia-28]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D28%26GraphicsBackend%3Dskia&negative=true&positive=true +[skia-gold-impeller-opengles-28]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D28%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true +[skia-gold-skia-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dskia&negative=true&positive=true +[skia-gold-impeller-opengles-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true +[skia-gold-impeller-vulkan-34]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D34%26GraphicsBackend%3Dimpeller-vulkan&negative=true&positive=true + +## Updating Gradle dependencies + +See [Updating the Embedding Dependencies](../../../tools/cipd/android_embedding_bundle/README.md). From 9bad8515f7b7fa0a7bfdeda503042d18f516afb3 Mon Sep 17 00:00:00 2001 From: Matan Lurey Date: Wed, 14 Feb 2024 12:30:23 -0800 Subject: [PATCH 3/8] ++ --- testing/scenario_app/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/testing/scenario_app/README.md b/testing/scenario_app/README.md index 6ebeb56c88c40..d1110f2aa6ddd 100644 --- a/testing/scenario_app/README.md +++ b/testing/scenario_app/README.md @@ -1,5 +1,7 @@ # Scenario App +[![GitHub Issues or Pull Requests by label](https://img.shields.io/github/issues/flutter/flutter/e%3A%20scenario-app)](https://github.com/flutter/flutter/issues?q=is%3Aopen+is%3Aissue+label%3A%22e%3A+scenario-app%22) + This package simulates a Flutter app that uses the engine (`dart:ui`) only, in conjunction with Android and iOS-specific embedding code that simulates the use of the engine in a real app (such as plugins and platform views). From 5012e1fd5dabb8a27784ec06c2e6fdbff18c6066 Mon Sep 17 00:00:00 2001 From: Matan Lurey Date: Wed, 14 Feb 2024 12:31:01 -0800 Subject: [PATCH 4/8] ++ --- testing/scenario_app/bin/android_integration_tests.dart | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/testing/scenario_app/bin/android_integration_tests.dart b/testing/scenario_app/bin/android_integration_tests.dart index 2964aaa91bf80..6c7532e5cd3c7 100644 --- a/testing/scenario_app/bin/android_integration_tests.dart +++ b/testing/scenario_app/bin/android_integration_tests.dart @@ -48,10 +48,10 @@ import 'utils/screenshot_transformer.dart'; /// - `--use-skia-gold`: Use Skia Gold to compare screenshots. Defaults to true /// when running on CI, and false otherwise (i.e. when running locally). If /// set to true, [isSkiaGoldClientAvailable] must be true. -/// +/// /// - `--enable-impeller`: Enable Impeller for the Android app. Defaults to /// false, which means that the app will use Skia as the graphics backend. -/// +/// /// - `--impeller-backend`: The Impeller backend to use for the Android app. /// Defaults to 'vulkan'. Only used when `--enable-impeller` is set to true. void main(List args) async { From 991b1b0772e1cbcf41c8e58d1957c67f5117d1d7 Mon Sep 17 00:00:00 2001 From: Matan Lurey Date: Wed, 14 Feb 2024 12:35:13 -0800 Subject: [PATCH 5/8] Revert formatting. --- .../bin/android_integration_tests.dart | 95 +++++++------------ 1 file changed, 35 insertions(+), 60 deletions(-) diff --git a/testing/scenario_app/bin/android_integration_tests.dart b/testing/scenario_app/bin/android_integration_tests.dart index 6c7532e5cd3c7..e66efbe3d5db9 100644 --- a/testing/scenario_app/bin/android_integration_tests.dart +++ b/testing/scenario_app/bin/android_integration_tests.dart @@ -94,13 +94,9 @@ void main(List args) async { final bool useSkiaGold = results['use-skia-gold'] as bool; final String? smokeTest = results['smoke-test'] as String?; final bool enableImpeller = results['enable-impeller'] as bool; - final _ImpellerBackend? impellerBackend = - _ImpellerBackend.tryParse(results['impeller-backend'] as String?); + final _ImpellerBackend? impellerBackend = _ImpellerBackend.tryParse(results['impeller-backend'] as String?); if (enableImpeller && impellerBackend == null) { - panic([ - 'invalid graphics-backend', - results['impeller-backend'] as String? ?? '' - ]); + panic(['invalid graphics-backend', results['impeller-backend'] as String? ?? '']); } await _run( outDir: outDir, @@ -149,10 +145,7 @@ Future _run({ const ProcessManager pm = LocalProcessManager(); if (!outDir.existsSync()) { - panic([ - 'out-dir does not exist: $outDir', - 'make sure to build the selected engine variant' - ]); + panic(['out-dir does not exist: $outDir', 'make sure to build the selected engine variant']); } if (!adb.existsSync()) { @@ -163,22 +156,15 @@ Future _run({ final String logcatPath = join(scenarioAppPath, 'logcat.txt'); final String screenshotPath = join(scenarioAppPath, 'screenshots'); final String apkOutPath = join(scenarioAppPath, 'app', 'outputs', 'apk'); - final File testApk = File( - join(apkOutPath, 'androidTest', 'debug', 'app-debug-androidTest.apk')); + final File testApk = File(join(apkOutPath, 'androidTest', 'debug', 'app-debug-androidTest.apk')); final File appApk = File(join(apkOutPath, 'debug', 'app-debug.apk')); if (!testApk.existsSync()) { - panic([ - 'test apk does not exist: ${testApk.path}', - 'make sure to build the selected engine variant' - ]); + panic(['test apk does not exist: ${testApk.path}', 'make sure to build the selected engine variant']); } if (!appApk.existsSync()) { - panic([ - 'app apk does not exist: ${appApk.path}', - 'make sure to build the selected engine variant' - ]); + panic(['app apk does not exist: ${appApk.path}', 'make sure to build the selected engine variant']); } // Start a TCP socket in the host, and forward it to the device that runs the tests. @@ -186,41 +172,39 @@ Future _run({ // for the screenshots. // On LUCI, the host uploads the screenshots to Skia Gold. SkiaGoldClient? skiaGoldClient; - late ServerSocket server; + late ServerSocket server; final List> pendingComparisons = >[]; await step('Starting server...', () async { server = await ServerSocket.bind(InternetAddress.anyIPv4, _tcpPort); - stdout - .writeln('listening on host ${server.address.address}:${server.port}'); + stdout.writeln('listening on host ${server.address.address}:${server.port}'); server.listen((Socket client) { - stdout.writeln( - 'client connected ${client.remoteAddress.address}:${client.remotePort}'); - client.transform(const ScreenshotBlobTransformer()).listen( - (Screenshot screenshot) { + stdout.writeln('client connected ${client.remoteAddress.address}:${client.remotePort}'); + client.transform(const ScreenshotBlobTransformer()).listen((Screenshot screenshot) { final String fileName = screenshot.filename; final Uint8List fileContent = screenshot.fileContent; log('host received ${fileContent.lengthInBytes} bytes for screenshot `$fileName`'); assert(skiaGoldClient != null, 'expected Skia Gold client'); late File goldenFile; try { - goldenFile = File(join(screenshotPath, fileName)) - ..writeAsBytesSync(fileContent, flush: true); + goldenFile = File(join(screenshotPath, fileName))..writeAsBytesSync(fileContent, flush: true); } on FileSystemException catch (err) { panic(['failed to create screenshot $fileName: $err']); } log('wrote ${goldenFile.absolute.path}'); if (isSkiaGoldClientAvailable) { final Future comparison = skiaGoldClient! - .addImg(fileName, goldenFile, - screenshotSize: screenshot.pixelCount) - .catchError((dynamic err) { - panic(['skia gold comparison failed: $err']); - }); + .addImg(fileName, goldenFile, + screenshotSize: screenshot.pixelCount) + .catchError((dynamic err) { + panic(['skia gold comparison failed: $err']); + }); pendingComparisons.add(comparison); } - }, onError: (dynamic err) { + }, + onError: (dynamic err) { panic(['error while receiving bytes: $err']); - }, cancelOnError: true); + }, + cancelOnError: true); }); }); @@ -234,8 +218,7 @@ Future _run({ }); await step('Starting logcat...', () async { - final int exitCode = - await pm.runAndForward([adb.path, 'logcat', '-c']); + final int exitCode = await pm.runAndForward([adb.path, 'logcat', '-c']); if (exitCode != 0) { panic(['could not clear logs']); } @@ -259,25 +242,21 @@ Future _run({ }); await step('Get API level of connected device...', () async { - final ProcessResult apiLevelProcessResult = await pm - .run([adb.path, 'shell', 'getprop', 'ro.build.version.sdk']); + final ProcessResult apiLevelProcessResult = await pm.run([adb.path, 'shell', 'getprop', 'ro.build.version.sdk']); if (apiLevelProcessResult.exitCode != 0) { panic(['could not get API level of the connected device']); } - final String connectedDeviceAPILevel = - (apiLevelProcessResult.stdout as String).trim(); + final String connectedDeviceAPILevel = (apiLevelProcessResult.stdout as String).trim(); final Map dimensions = { 'AndroidAPILevel': connectedDeviceAPILevel, - 'GraphicsBackend': - enableImpeller ? 'impeller-${impellerBackend!.name}' : 'skia', + 'GraphicsBackend': enableImpeller ? 'impeller-${impellerBackend!.name}' : 'skia', }; log('using dimensions: ${json.encode(dimensions)}'); skiaGoldClient = SkiaGoldClient( outDir, dimensions: { 'AndroidAPILevel': connectedDeviceAPILevel, - 'GraphicsBackend': - enableImpeller ? 'impeller-${impellerBackend!.name}' : 'skia', + 'GraphicsBackend': enableImpeller ? 'impeller-${impellerBackend!.name}' : 'skia', }, ); }); @@ -296,24 +275,21 @@ Future _run({ }); await step('Reverse port...', () async { - final int exitCode = await pm.runAndForward( - [adb.path, 'reverse', 'tcp:3000', 'tcp:$_tcpPort']); + final int exitCode = await pm.runAndForward([adb.path, 'reverse', 'tcp:3000', 'tcp:$_tcpPort']); if (exitCode != 0) { panic(['could not forward port']); } }); await step('Installing app APK...', () async { - final int exitCode = - await pm.runAndForward([adb.path, 'install', appApk.path]); + final int exitCode = await pm.runAndForward([adb.path, 'install', appApk.path]); if (exitCode != 0) { panic(['could not install app apk']); } }); await step('Installing test APK...', () async { - final int exitCode = - await pm.runAndForward([adb.path, 'install', testApk.path]); + final int exitCode = await pm.runAndForward([adb.path, 'install', testApk.path]); if (exitCode != 0) { panic(['could not install test apk']); } @@ -326,9 +302,11 @@ Future _run({ 'am', 'instrument', '-w', - if (smokeTestFullPath != null) '-e class $smokeTestFullPath', + if (smokeTestFullPath != null) + '-e class $smokeTestFullPath', 'dev.flutter.scenarios.test/dev.flutter.TestRunner', - if (enableImpeller) '-e enable-impeller', + if (enableImpeller) + '-e enable-impeller', if (impellerBackend != null) '-e impeller-backend ${impellerBackend.name}', ]); @@ -350,8 +328,7 @@ Future _run({ final int exitCode = await pm.runAndForward([ adb.path, 'reverse', - '--remove', - 'tcp:3000', + '--remove', 'tcp:3000', ]); if (exitCode != 0) { panic(['could not unforward port']); @@ -359,16 +336,14 @@ Future _run({ }); await step('Uinstalling app APK...', () async { - final int exitCode = await pm.runAndForward( - [adb.path, 'uninstall', 'dev.flutter.scenarios']); + final int exitCode = await pm.runAndForward([adb.path, 'uninstall', 'dev.flutter.scenarios']); if (exitCode != 0) { panic(['could not uninstall app apk']); } }); await step('Uinstalling test APK...', () async { - final int exitCode = await pm.runAndForward( - [adb.path, 'uninstall', 'dev.flutter.scenarios.test']); + final int exitCode = await pm.runAndForward([adb.path, 'uninstall', 'dev.flutter.scenarios.test']); if (exitCode != 0) { panic(['could not uninstall app apk']); } From f98b52254071c7ef2890c7897332048dfb5a9098 Mon Sep 17 00:00:00 2001 From: Matan Lurey Date: Wed, 14 Feb 2024 13:07:27 -0800 Subject: [PATCH 6/8] ++ --- testing/scenario_app/README.md | 14 ++++++- testing/scenario_app/android/README.md | 14 +++---- testing/scenario_app/bin/README.md | 37 +++++++++++++++++- .../bin/android_integration_tests.dart | 39 +------------------ testing/scenario_app/lib/README.md | 16 -------- 5 files changed, 56 insertions(+), 64 deletions(-) delete mode 100644 testing/scenario_app/lib/README.md diff --git a/testing/scenario_app/README.md b/testing/scenario_app/README.md index d1110f2aa6ddd..2a3cfc5636b0a 100644 --- a/testing/scenario_app/README.md +++ b/testing/scenario_app/README.md @@ -6,8 +6,7 @@ This package simulates a Flutter app that uses the engine (`dart:ui`) only, in conjunction with Android and iOS-specific embedding code that simulates the use of the engine in a real app (such as plugins and platform views). -To run the tests, you will need to build the engine with the appropriate -configuration. The [`run_android_tests.sh`](run_android_tests.sh) and +The [`run_android_tests.sh`](run_android_tests.sh) and [`run_ios_tests.sh`](run_ios_tests.sh) are then used to run the tests on a connected device or emulator. @@ -31,3 +30,14 @@ $ ./ci/firebase_testlab.py --variant android_debug_arm64 > ![NOTE] > These instructions were not verified at the time of writing/refactoring. + +## Adding a New Scenario + +Create a new subclass of [Scenario](lib/src/scenario.dart) and add it to the map +in [scenarios.dart](lib/src/scenarios.dart). For an example, see +[animated_color_square.dart](lib/src/animated_color_square.dart), which draws a +continuously animating colored square that bounces off the sides of the +viewport. + +Then set the scenario from the Android or iOS app by calling `set_scenario` on +platform channel `driver`. diff --git a/testing/scenario_app/android/README.md b/testing/scenario_app/android/README.md index 9274cbcd9c7af..ec4f1857bc665 100644 --- a/testing/scenario_app/android/README.md +++ b/testing/scenario_app/android/README.md @@ -23,13 +23,13 @@ See [`ci/builders/linux_android_emulator.json`](../../../ci/builders/linux_andro The following matrix of configurations is tested on the CI: -| API Version | Graphics Backend | Skia Gold | Rationale | -| ----------- | ------------------- | -------------------------------------- | ---------------------------------------------------------- | -| 28 | Skia | [Link][skia-gold-skia-28] | Older Android devices (without `ImageReader`) on Skia. | -| 28 | Impeller (OpenGLES) | [Link][skia-gold-impeller-opengles-28] | Older Android devices (without `ImageReader`) on Impeller. | -| 34 | Skia | [Link][skia-gold-skia-34] | Newer Android devices on Skia. | -| 34 | Impeller (OpenGLES) | [Link][skia-gold-impeller-opengles-34] | Newer Android devices on Impeller with OpenGLES. | -| 34 | Impeller (Vulkan) | [Link][skia-gold-impeller-vulkan-34] | Newer Android devices on Impeller. | +| API Version | Graphics Backend | Skia Gold | Rationale | +| ----------- | ------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------- | +| 28 | Skia | [Android 28 + Skia][skia-gold-skia-28] | Older Android devices (without `ImageReader`) on Skia. | +| 28 | Impeller (OpenGLES) | [Android 28 + Impeller OpenGLES][skia-gold-impeller-opengles-28] | Older Android devices (without `ImageReader`) on Impeller. | +| 34 | Skia | [Android 34 + Skia][skia-gold-skia-34] | Newer Android devices on Skia. | +| 34 | Impeller (OpenGLES) | [Android 34 + Impeller OpenGLES][skia-gold-impeller-opengles-34] | Newer Android devices on Impeller with OpenGLES. | +| 34 | Impeller (Vulkan) | [Android 34 + Impeller Vulkan][skia-gold-impeller-vulkan-34] | Newer Android devices on Impeller. | [skia-gold-skia-28]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D28%26GraphicsBackend%3Dskia&negative=true&positive=true [skia-gold-impeller-opengles-28]: https://flutter-engine-gold.skia.org/search?left_filter=AndroidAPILevel%3D28%26GraphicsBackend%3Dimpeller-opengles&negative=true&positive=true diff --git a/testing/scenario_app/bin/README.md b/testing/scenario_app/bin/README.md index 76737692bd957..7964d6e5960cc 100644 --- a/testing/scenario_app/bin/README.md +++ b/testing/scenario_app/bin/README.md @@ -2,4 +2,39 @@ This directory contains code specific to running Android integration tests. -See [`android_integration_tests.dart`](android_integration_tests.dart). +The tests are uploaded and run on the device using `adb`, and screenshots are +captured and compared using Skia Gold (if available, for example on CI). + +## Usage + +```sh +dart bin/android_integration_tests.dart \ + --adb ../third_party/android_tools/sdk/platform-tools/adb \ + --out-dir ../out/android_debug_unopt_arm64 +``` + +## Debugging + +When debugging, you can use the `--smoke-test` argument to run a single test +by class name, which can be useful to verify the setup. + +For example, to run the `EngineLaunchE2ETest` test: + +```sh +dart bin/android_integration_tests.dart \ + --adb ../third_party/android_tools/sdk/platform-tools/adb \ + --out-dir ../out/android_debug_unopt_arm64 \ + --smoke-test dev.flutter.scenarios.EngineLaunchE2ETest +``` + +## Additional arguments + +- `--use-skia-gold`: Use Skia Gold to compare screenshots. Defaults to true + when running on CI, and false otherwise (i.e. when running locally). If + set to true, [isSkiaGoldClientAvailable] must be true. + +- `--enable-impeller`: Enable Impeller for the Android app. Defaults to + false, which means that the app will use Skia as the graphics backend. + +- `--impeller-backend`: The Impeller backend to use for the Android app. + Defaults to 'vulkan'. Only used when `--enable-impeller` is set to true. diff --git a/testing/scenario_app/bin/android_integration_tests.dart b/testing/scenario_app/bin/android_integration_tests.dart index e66efbe3d5db9..98b8b7abcb39c 100644 --- a/testing/scenario_app/bin/android_integration_tests.dart +++ b/testing/scenario_app/bin/android_integration_tests.dart @@ -16,44 +16,7 @@ import 'utils/logs.dart'; import 'utils/process_manager_extension.dart'; import 'utils/screenshot_transformer.dart'; -/// Runs the Android integration tests on a connected device or emulator. -/// -/// The tests are uploaded and run on the device using `adb`, and screenshots -/// are captured and compared using Skia Gold (if available, for example on CI). -/// -/// ## Usage -/// -/// ```sh -/// dart bin/android_integration_tests.dart \ -/// --adb ../third_party/android_tools/sdk/platform-tools/adb \ -/// --out-dir ../out/android_debug_unopt_arm64 -/// ``` -/// -/// ## Debugging -/// -/// When debugging, you can use the `--smoke-test` argument to run a single test -/// by class name, which can be useful to verify the setup. -/// -/// For example, to run the `EngineLaunchE2ETest` test: -/// -/// ```sh -/// dart bin/android_integration_tests.dart \ -/// --adb ../third_party/android_tools/sdk/platform-tools/adb \ -/// --out-dir ../out/android_debug_unopt_arm64 \ -/// --smoke-test dev.flutter.scenarios.EngineLaunchE2ETest -/// ``` -/// -/// ## Additional arguments -/// -/// - `--use-skia-gold`: Use Skia Gold to compare screenshots. Defaults to true -/// when running on CI, and false otherwise (i.e. when running locally). If -/// set to true, [isSkiaGoldClientAvailable] must be true. -/// -/// - `--enable-impeller`: Enable Impeller for the Android app. Defaults to -/// false, which means that the app will use Skia as the graphics backend. -/// -/// - `--impeller-backend`: The Impeller backend to use for the Android app. -/// Defaults to 'vulkan'. Only used when `--enable-impeller` is set to true. +// If you update the arguments, update the documentation in the README.md file. void main(List args) async { final ArgParser parser = ArgParser() ..addOption( diff --git a/testing/scenario_app/lib/README.md b/testing/scenario_app/lib/README.md deleted file mode 100644 index 0f9c016ce5fca..0000000000000 --- a/testing/scenario_app/lib/README.md +++ /dev/null @@ -1,16 +0,0 @@ -# `package:scenario_app` - -This package simulates a Flutter app that uses the engine (`dart:ui`) only. It -is used to test the engine in isolation from the rest of the Flutter framework -and tooling. - -## Adding a New Scenario - -Create a new subclass of [Scenario](src/scenario.dart) and add it to the map -in [scenarios.dart](src/scenarios.dart). For an example, see -[animated_color_square.dart](src/animated_color_square.dart), which draws a -continuously animating colored square that bounces off the sides of the -viewport. - -Then set the scenario from the Android or iOS app by calling `set_scenario` on -platform channel `driver`. From e49b765f6d614cf8295f5ae7127738cf0e82ee41 Mon Sep 17 00:00:00 2001 From: Matan Lurey Date: Wed, 14 Feb 2024 13:09:20 -0800 Subject: [PATCH 7/8] Update testing/scenario_app/ios/README.md Co-authored-by: Jenn Magder --- testing/scenario_app/ios/README.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/testing/scenario_app/ios/README.md b/testing/scenario_app/ios/README.md index 3c5444b16eec9..972625056d403 100644 --- a/testing/scenario_app/ios/README.md +++ b/testing/scenario_app/ios/README.md @@ -4,12 +4,17 @@ As mentioned in the [top-level README](../README.md), this directory contains the iOS-specific native code and tests for the [scenario app](../lib). To run the tests, you will need to build the engine with the appropriate configuration. -For example, `ios_debug_sim_unopt` or `ios_debug_sim_unopt_arm64` was built, +For example, after building `ios_debug_sim_unopt` (to run on Intel Macs) or `ios_debug_sim_unopt_arm64` (to run on ARM Macs), run: ```sh # From the root of the engine repository -$ ./testing/run_ios_tests.sh +$ ./testing/run_ios_tests.sh ios_debug_sim_unopt +``` +or +```sh +# From the root of the engine repository +$ ./testing/run_ios_tests.sh ios_debug_sim_unopt_arm64 ``` To run or debug in Xcode, open the xcodeproj file located in From d811ec9949b81261c35d3bdad5ce7481a2fff035 Mon Sep 17 00:00:00 2001 From: Matan Lurey Date: Wed, 14 Feb 2024 13:24:13 -0800 Subject: [PATCH 8/8] Update testing/scenario_app/README.md Co-authored-by: Gray Mackall <34871572+gmackall@users.noreply.github.com> --- testing/scenario_app/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/testing/scenario_app/README.md b/testing/scenario_app/README.md index 2a3cfc5636b0a..470eb320fb116 100644 --- a/testing/scenario_app/README.md +++ b/testing/scenario_app/README.md @@ -28,7 +28,7 @@ and run [`./ci/firebase_testlab.py`](../../ci/firebase_testlab.py), or pass $ ./ci/firebase_testlab.py --variant android_debug_arm64 ``` -> ![NOTE] +> [!NOTE] > These instructions were not verified at the time of writing/refactoring. ## Adding a New Scenario