-
Notifications
You must be signed in to change notification settings - Fork 6k
Add custom test plugin that supports screenshot tests #12079
Conversation
lib/web_ui/dev/environment.dart
Outdated
| io.stderr.writeln('Command-line options not passed to Environment.'); | ||
| io.exit(1); | ||
| } | ||
| if (_environment == null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return _environment ??= Environment(); if you wanna be concise :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
lib/web_ui/dev/environment.dart
Outdated
|
|
||
| /// Contains various environment variables, such as common file paths and command-line options. | ||
| Environment get environment { | ||
| if (Environment.commandLineArguments == null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This check already exists in the Environment constructor. Is there a reason for duplicating it here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
| final bool isDebug = options['debug']; | ||
| final List<String> targets = options['target']; | ||
|
|
||
| final io.File self = io.File.fromUri(io.Platform.script); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks to me like all you need to compute here is the engine src dir. Everything else can be a getter that's derived from engine src.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! That cleaned it up significantly.
| _expect(allSourceFiles.isNotEmpty, 'Dart source listing of ${environment.webUiRootDir.path} must not be empty.'); | ||
|
|
||
| final List<String> allDartPaths = allSourceFiles.map((f) => f.path).toList(); | ||
| print(allDartPaths.join('\n')); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
lib/web_ui/dev/test.dart
Outdated
| .whereType<io.File>() | ||
| .where((f) => f.path.endsWith('.dart') || f.path.endsWith('.js')) | ||
| .where((f) { | ||
| if (!f.path.endsWith('.dart') || f.path.endsWith('.js')) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the .js check necessary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! Copy'n'paste error. The condition should've been fully inverted here. Fixed.
| // Run screenshot tests one at a time. | ||
| for (String testFilePath in screenshotTestFiles) { | ||
| await _runTestBatch(<String>[testFilePath], concurrency: 1, expectFailure: false); | ||
| _checkExitCode(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use the return value of _runTestBatch instead of relying on the indirect communication between the two functions through io.exitCode?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately no. This relies on the package:test's test runner API, which communicates test success and failures by setting the exit code. It kind of makes sense, since on CI you communicate via process exit codes as well. Otherwise, if some place in the code accidentally catches an exception or ignores the returned exit code the CI will incorrectly pass.
|
Ok, this is good to review. In the meantime I'll fix the licenses. |
|
|
||
| // TODO(yjbanov): do not fail Cirrus builds. They currently fail because Chrome produces | ||
| // different pictures. We need to pin Chrome versions and use a fuzzy image | ||
| // comparator. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would look at what flutter_goldens is doing with package:image
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea. Looking at it right now.
jonahwilliams
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if it works, then LGTM
git@github.com:flutter/engine.git/compare/7ea9884ab00e...5b94c8a git log 7ea9884..5b94c8a --no-merges --oneline 2019-09-11 mklim@google.com Revert "Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits)" (flutter/engine#12223) 2019-09-11 iska.kaushik@gmail.com Do not generate kernel platform files on topaz tree (flutter/engine#12222) 2019-09-11 hterkelsen@users.noreply.github.com Don't disable toString in release mode for dart:ui classes (flutter/engine#12204) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/linux-amd64 from 7gDBN... to u7Q31... (flutter/engine#12221) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia d96ef09317d6..120e7d6766e4 (2 commits) (flutter/engine#12220) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/mac-amd64 from vDk46... to _nS67... (flutter/engine#12219) 2019-09-11 iska.kaushik@gmail.com Namespace patched SDK names to not conflict with Topaz (flutter/engine#12218) 2019-09-11 bkonyi@google.com Roll src/third_party/dart ca7baa4013..4d5e15abde (29 commits) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia 14318c140949..d96ef09317d6 (2 commits) (flutter/engine#12216) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia b23a4f9d9442..14318c140949 (2 commits) (flutter/engine#12215) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia 26ac0467cb4c..b23a4f9d9442 (2 commits) (flutter/engine#12214) 2019-09-11 bkonyi@google.com Roll src/third_party/dart 7bbfd532de..ca7baa4013 (3 commits) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/linux-amd64 from R1yqu... to 7gDBN... (flutter/engine#12212) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/mac-amd64 from spUG2... to vDk46... (flutter/engine#12210) 2019-09-11 chinmaygarde@google.com Roll buildroot and Fuchsia toolchain and unblock the Fuchsia/Linux autoroller manually. (flutter/engine#12209) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia 4fe30e15c06c..26ac0467cb4c (2 commits) (flutter/engine#12207) 2019-09-11 chinmaygarde@google.com Only build the x64 variant of Fuchsia on the try-jobs. (flutter/engine#12206) 2019-09-10 hterkelsen@users.noreply.github.com Don't load Roboto by default (flutter/engine#12205) 2019-09-10 bkonyi@google.com Roll src/third_party/dart 300c3333d1..7bbfd532de (5 commits) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 66d8006c2bb1..4fe30e15c06c (11 commits) (flutter/engine#12202) 2019-09-10 yjbanov@google.com add a convenience CLI tool for building and testing web engine (flutter/engine#12197) 2019-09-10 jason-simmons@users.noreply.github.com [flutter_runner] Generate symbols for the Dart VM profiler (flutter/engine#12048) 2019-09-10 yjbanov@google.com Add custom test plugin that supports screenshot tests (flutter/engine#12079) 2019-09-10 chinmaygarde@google.com Move the Fuchsia tryjob into a its own step and disable LTO. (flutter/engine#12190) 2019-09-10 bkonyi@google.com Roll src/third_party/dart 62f78a7abb..300c3333d1 (6 commits) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia b88894c8811b..66d8006c2bb1 (5 commits) (flutter/engine#12178) 2019-09-10 jason-simmons@users.noreply.github.com [flutter_runner] Port the accessibility bridge from Topaz (flutter/engine#12054) 2019-09-10 liyuqian@google.com Smooth out iOS irregular input events delivery (flutter/engine#11817) 2019-09-10 bkonyi@google.com Roll src/third_party/dart ccb6ba948b..62f78a7abb (3 commits) 2019-09-10 jonahwilliams@google.com Make ImageShader implement Shader for web ui (flutter/engine#12161) 2019-09-10 bkonyi@google.com Roll src/third_party/dart 2e8d912848..ccb6ba948b (30 commits) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 9e5c47936b17..b88894c8811b (3 commits) (flutter/engine#12151) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 1bf30ce852e0..9e5c47936b17 (2 commits) (flutter/engine#12129) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 8cae1e95a23b..1bf30ce852e0 (2 commits) (flutter/engine#12106) 2019-09-10 skia-flutter-autoroll@skia.org Roll fuchsia/clang/mac-amd64 from H1Qjc... to HfPKR... (flutter/engine#12088) 2019-09-10 chinmaygarde@google.com Don't launch the observatory by default on each embedder unit-test invocation. (flutter/engine#12087) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia c2d84bfa7421..8cae1e95a23b (4 commits) (flutter/engine#12086) 2019-09-10 bkonyi@google.com Roll src/third_party/dart fb14babf59..2e8d912848 (65 commits) 2019-09-10 chinmaygarde@google.com Guard availability of user notification related methods to iOS 10.0 (flutter/engine#12084) 2019-09-09 bmparr@google.com Add capability to add AppDelegate as UNUserNotificationCenterDelegate (flutter/engine#9864) 2019-09-09 ferhat@gmail.com Add GradientRadial paintStyle implementation (flutter/engine#12081) 2019-09-09 hterkelsen@users.noreply.github.com Don't quote generic font families (flutter/engine#12080) 2019-09-09 skia-flutter-autoroll@skia.org Roll src/third_party/skia 28d40b2e7ade..c2d84bfa7421 (3 commits) (flutter/engine#12082) 2019-09-09 magder@google.com Remove ENABLE_BITCODE from Scenarios test app (flutter/engine#11839) 2019-09-09 skia-flutter-autoroll@skia.org Roll src/third_party/skia 4f2674da4bbc..28d40b2e7ade (4 commits) (flutter/engine#12077) ...
git@github.com:flutter/engine.git/compare/7ea9884ab00e...bbb1f12 git log 7ea9884..bbb1f12 --no-merges --oneline 2019-09-12 50503328+flar@users.noreply.github.com Adjust iOS frame start times to match the platform info (flutter/engine#11802) 2019-09-12 skia-flutter-autoroll@skia.org Roll src/third_party/skia 50f377e275c3..7c47d41067d4 (3 commits) (flutter/engine#12231) 2019-09-12 50503328+flar@users.noreply.github.com Revert "Manage iOS contexts separately (#12078)" (flutter/engine#12233) 2019-09-11 gw@gwright.org.uk Manage iOS contexts separately (flutter/engine#12078) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia 120e7d6766e4..50f377e275c3 (7 commits) (flutter/engine#12224) 2019-09-11 mklim@google.com Revert "Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits)" (flutter/engine#12223) 2019-09-11 iska.kaushik@gmail.com Do not generate kernel platform files on topaz tree (flutter/engine#12222) 2019-09-11 hterkelsen@users.noreply.github.com Don't disable toString in release mode for dart:ui classes (flutter/engine#12204) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/linux-amd64 from 7gDBN... to u7Q31... (flutter/engine#12221) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia d96ef09317d6..120e7d6766e4 (2 commits) (flutter/engine#12220) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/mac-amd64 from vDk46... to _nS67... (flutter/engine#12219) 2019-09-11 iska.kaushik@gmail.com Namespace patched SDK names to not conflict with Topaz (flutter/engine#12218) 2019-09-11 bkonyi@google.com Roll src/third_party/dart ca7baa4013..4d5e15abde (29 commits) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia 14318c140949..d96ef09317d6 (2 commits) (flutter/engine#12216) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia b23a4f9d9442..14318c140949 (2 commits) (flutter/engine#12215) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia 26ac0467cb4c..b23a4f9d9442 (2 commits) (flutter/engine#12214) 2019-09-11 bkonyi@google.com Roll src/third_party/dart 7bbfd532de..ca7baa4013 (3 commits) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/linux-amd64 from R1yqu... to 7gDBN... (flutter/engine#12212) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/mac-amd64 from spUG2... to vDk46... (flutter/engine#12210) 2019-09-11 chinmaygarde@google.com Roll buildroot and Fuchsia toolchain and unblock the Fuchsia/Linux autoroller manually. (flutter/engine#12209) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia 4fe30e15c06c..26ac0467cb4c (2 commits) (flutter/engine#12207) 2019-09-11 chinmaygarde@google.com Only build the x64 variant of Fuchsia on the try-jobs. (flutter/engine#12206) 2019-09-10 hterkelsen@users.noreply.github.com Don't load Roboto by default (flutter/engine#12205) 2019-09-10 bkonyi@google.com Roll src/third_party/dart 300c3333d1..7bbfd532de (5 commits) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 66d8006c2bb1..4fe30e15c06c (11 commits) (flutter/engine#12202) 2019-09-10 yjbanov@google.com add a convenience CLI tool for building and testing web engine (flutter/engine#12197) 2019-09-10 jason-simmons@users.noreply.github.com [flutter_runner] Generate symbols for the Dart VM profiler (flutter/engine#12048) 2019-09-10 yjbanov@google.com Add custom test plugin that supports screenshot tests (flutter/engine#12079) 2019-09-10 chinmaygarde@google.com Move the Fuchsia tryjob into a its own step and disable LTO. (flutter/engine#12190) 2019-09-10 bkonyi@google.com Roll src/third_party/dart 62f78a7abb..300c3333d1 (6 commits) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia b88894c8811b..66d8006c2bb1 (5 commits) (flutter/engine#12178) 2019-09-10 jason-simmons@users.noreply.github.com [flutter_runner] Port the accessibility bridge from Topaz (flutter/engine#12054) 2019-09-10 liyuqian@google.com Smooth out iOS irregular input events delivery (flutter/engine#11817) 2019-09-10 bkonyi@google.com Roll src/third_party/dart ccb6ba948b..62f78a7abb (3 commits) 2019-09-10 jonahwilliams@google.com Make ImageShader implement Shader for web ui (flutter/engine#12161) 2019-09-10 bkonyi@google.com Roll src/third_party/dart 2e8d912848..ccb6ba948b (30 commits) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 9e5c47936b17..b88894c8811b (3 commits) (flutter/engine#12151) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 1bf30ce852e0..9e5c47936b17 (2 commits) (flutter/engine#12129) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 8cae1e95a23b..1bf30ce852e0 (2 commits) (flutter/engine#12106) 2019-09-10 skia-flutter-autoroll@skia.org Roll fuchsia/clang/mac-amd64 from H1Qjc... to HfPKR... (flutter/engine#12088) 2019-09-10 chinmaygarde@google.com Don't launch the observatory by default on each embedder unit-test invocation. (flutter/engine#12087) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia c2d84bfa7421..8cae1e95a23b (4 commits) (flutter/engine#12086) 2019-09-10 bkonyi@google.com Roll src/third_party/dart fb14babf59..2e8d912848 (65 commits) 2019-09-10 chinmaygarde@google.com Guard availability of user notification related methods to iOS 10.0 (flutter/engine#12084) 2019-09-09 bmparr@google.com Add capability to add AppDelegate as UNUserNotificationCenterDelegate (flutter/engine#9864) ...
git@github.com:flutter/engine.git/compare/7ea9884ab00e...bbb1f12 git log 7ea9884..bbb1f12 --no-merges --oneline 2019-09-12 50503328+flar@users.noreply.github.com Adjust iOS frame start times to match the platform info (flutter/engine#11802) 2019-09-12 skia-flutter-autoroll@skia.org Roll src/third_party/skia 50f377e275c3..7c47d41067d4 (3 commits) (flutter/engine#12231) 2019-09-12 50503328+flar@users.noreply.github.com Revert "Manage iOS contexts separately (flutter#12078)" (flutter/engine#12233) 2019-09-11 gw@gwright.org.uk Manage iOS contexts separately (flutter/engine#12078) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia 120e7d6766e4..50f377e275c3 (7 commits) (flutter/engine#12224) 2019-09-11 mklim@google.com Revert "Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits)" (flutter/engine#12223) 2019-09-11 iska.kaushik@gmail.com Do not generate kernel platform files on topaz tree (flutter/engine#12222) 2019-09-11 hterkelsen@users.noreply.github.com Don't disable toString in release mode for dart:ui classes (flutter/engine#12204) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/linux-amd64 from 7gDBN... to u7Q31... (flutter/engine#12221) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia d96ef09317d6..120e7d6766e4 (2 commits) (flutter/engine#12220) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/mac-amd64 from vDk46... to _nS67... (flutter/engine#12219) 2019-09-11 iska.kaushik@gmail.com Namespace patched SDK names to not conflict with Topaz (flutter/engine#12218) 2019-09-11 bkonyi@google.com Roll src/third_party/dart ca7baa4013..4d5e15abde (29 commits) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia 14318c140949..d96ef09317d6 (2 commits) (flutter/engine#12216) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia b23a4f9d9442..14318c140949 (2 commits) (flutter/engine#12215) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia 26ac0467cb4c..b23a4f9d9442 (2 commits) (flutter/engine#12214) 2019-09-11 bkonyi@google.com Roll src/third_party/dart 7bbfd532de..ca7baa4013 (3 commits) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/linux-amd64 from R1yqu... to 7gDBN... (flutter/engine#12212) 2019-09-11 skia-flutter-autoroll@skia.org Roll fuchsia/sdk/core/mac-amd64 from spUG2... to vDk46... (flutter/engine#12210) 2019-09-11 chinmaygarde@google.com Roll buildroot and Fuchsia toolchain and unblock the Fuchsia/Linux autoroller manually. (flutter/engine#12209) 2019-09-11 skia-flutter-autoroll@skia.org Roll src/third_party/skia 4fe30e15c06c..26ac0467cb4c (2 commits) (flutter/engine#12207) 2019-09-11 chinmaygarde@google.com Only build the x64 variant of Fuchsia on the try-jobs. (flutter/engine#12206) 2019-09-10 hterkelsen@users.noreply.github.com Don't load Roboto by default (flutter/engine#12205) 2019-09-10 bkonyi@google.com Roll src/third_party/dart 300c3333d1..7bbfd532de (5 commits) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 66d8006c2bb1..4fe30e15c06c (11 commits) (flutter/engine#12202) 2019-09-10 yjbanov@google.com add a convenience CLI tool for building and testing web engine (flutter/engine#12197) 2019-09-10 jason-simmons@users.noreply.github.com [flutter_runner] Generate symbols for the Dart VM profiler (flutter/engine#12048) 2019-09-10 yjbanov@google.com Add custom test plugin that supports screenshot tests (flutter/engine#12079) 2019-09-10 chinmaygarde@google.com Move the Fuchsia tryjob into a its own step and disable LTO. (flutter/engine#12190) 2019-09-10 bkonyi@google.com Roll src/third_party/dart 62f78a7abb..300c3333d1 (6 commits) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia b88894c8811b..66d8006c2bb1 (5 commits) (flutter/engine#12178) 2019-09-10 jason-simmons@users.noreply.github.com [flutter_runner] Port the accessibility bridge from Topaz (flutter/engine#12054) 2019-09-10 liyuqian@google.com Smooth out iOS irregular input events delivery (flutter/engine#11817) 2019-09-10 bkonyi@google.com Roll src/third_party/dart ccb6ba948b..62f78a7abb (3 commits) 2019-09-10 jonahwilliams@google.com Make ImageShader implement Shader for web ui (flutter/engine#12161) 2019-09-10 bkonyi@google.com Roll src/third_party/dart 2e8d912848..ccb6ba948b (30 commits) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 9e5c47936b17..b88894c8811b (3 commits) (flutter/engine#12151) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 1bf30ce852e0..9e5c47936b17 (2 commits) (flutter/engine#12129) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia 8cae1e95a23b..1bf30ce852e0 (2 commits) (flutter/engine#12106) 2019-09-10 skia-flutter-autoroll@skia.org Roll fuchsia/clang/mac-amd64 from H1Qjc... to HfPKR... (flutter/engine#12088) 2019-09-10 chinmaygarde@google.com Don't launch the observatory by default on each embedder unit-test invocation. (flutter/engine#12087) 2019-09-10 skia-flutter-autoroll@skia.org Roll src/third_party/skia c2d84bfa7421..8cae1e95a23b (4 commits) (flutter/engine#12086) 2019-09-10 bkonyi@google.com Roll src/third_party/dart fb14babf59..2e8d912848 (65 commits) 2019-09-10 chinmaygarde@google.com Guard availability of user notification related methods to iOS 10.0 (flutter/engine#12084) 2019-09-09 bmparr@google.com Add capability to add AppDelegate as UNUserNotificationCenterDelegate (flutter/engine#9864) ...
PlatformPluginthat spins up a server waiting for test to request a screenshot.dev/test.dartwith anArgParserfor future extension into a proper developer tool for the team.--debugoption to launch Chrome in debug mode, a--targetoption to choose a single test to run rather than all tests, and--shardoption to choose a subset of tests to run.Limitations: