[sample-tester] Collect app size and build duration statistics. #7517

rolfbjarne · 2019-11-29T11:23:02Z

No description provided.

rolfbjarne · 2019-11-29T11:23:53Z

do-not-merge since this is still a work in progress.

monojenkins · 2019-11-29T11:23:56Z

xamarin-release-manager · 2019-11-29T11:52:31Z

🔥 Jenkins job (on internal Jenkins) failed in stage(s) 'Running XM tests on '10.11'' 🔥 : hudson.AbortException: Xamarin.Mac tests on macOS 10.11 failed (xammac_tests)

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.
🔥 Xamarin.Mac tests on 10.11 failed: Xamarin.Mac tests on macOS 10.11 failed (xammac_tests) 🔥

spouliot · 2019-11-29T13:39:53Z

tests/sampletester/ProcessHelper.cs

+		var watch = Stopwatch.StartNew ();
+		var failed = false;
+		try {
+			AssertRunProcess ("msbuild", sb.ToArray (), TimeSpan.FromMinutes (5), Configuration.SampleRootDirectory, environment_variables, "build");


Adding

environment_variables ["MTOUCH_ENV_OPTIONS"] = "--timestamp --timestamp --timestamp --timestamp";

would produce more data, from mtouch itself.

I'll add the timestamp data as well.

Although it ends up confusing/incorrect when building app extensions, since those are built in parallel with the main app: https://gist.github.com/rolfbjarne/570cb3301d35173467a17826f559feb9

spouliot · 2019-11-29T13:43:28Z

tests/sampletester/ProcessHelper.cs

+				xml.WriteAttributeString ("configuration", configuration);
+				if (!failed) {
+					xml.WriteAttributeString ("duration", watch.ElapsedTicks.ToString ());
+					xml.WriteAttributeString ("duration-formatted", watch.Elapsed.ToString ());


Can it be more granular, e.g. --timestamp above and the data from msbuild /v:diag task times ?

If we see an increase we won't be able to know what's causing it without running both the current and older builds. The older ones might become hard (e.g. new Xcode/OS requirement) to reproduce.

spouliot · 2019-11-29T13:46:14Z

tests/sampletester/ProcessHelper.cs

+			};
+			var xml = XmlWriter.Create (logfile, xmlSettings);
+			xml.WriteStartDocument (true);
+			xml.WriteStartElement ("perf-data");


Do we gather the versions (mono, SDK...) somewhere ?

Also is there a baseline project (to compare across different bots hardware) ? or is that leaved for later (analysis) ?

Do we gather the versions (mono, SDK...) somewhere ?

Indirectly, through the tested hash, but I'll make it explicit as well.

Also is there a baseline project (to compare across different bots hardware)

That's still a TODO 😄

…PTIONS.

…diagnostic msbuild log.

monojenkins · 2019-11-29T19:38:53Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-11-29T20:08:46Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-02T13:41:45Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-02T14:11:34Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-02T18:27:22Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-02T18:56:21Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-02T23:23:37Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-03T01:18:14Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-03T12:46:40Z

Build success
ℹ️ Skipped execution

monojenkins · 2019-12-13T22:26:11Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-13T22:56:52Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-13T22:58:34Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-13T23:28:07Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-13T23:54:32Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-14T00:23:53Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-14T00:24:55Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-14T00:54:36Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-14T01:00:28Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-14T01:30:21Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-16T07:46:24Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-16T08:21:19Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-16T08:34:37Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-16T09:09:05Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-16T10:49:10Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-16T11:31:36Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

monojenkins · 2019-12-16T11:54:40Z

Build success
ℹ️ Skipped execution

xamarin-release-manager · 2019-12-16T12:35:42Z

✅ Jenkins job (on internal Jenkins) succeeded

✅ Build succeeded
✅ Packages:

✅ API Diff (from stable)
✅ API Diff (from PR only) (no change)
✅ Generator Diff (no change)
ℹ️ Test run skipped: Not running tests here because they're run on public Jenkins.

[sample-tester] Collect app size and build duration statistics.

473ff14

spouliot reviewed Nov 29, 2019

View reviewed changes

rolfbjarne added 4 commits November 29, 2019 14:53

Always publish performance data.

d1cd15d

[mmp] Add support for MMP_ENV_OPTIONS to mirror mtouch's MTOUCH_ENV_O…

20a0955

…PTIONS.

[sample-tester] Make mmp/mtouch show timing information, and get the …

6583117

…diagnostic msbuild log.

Collect task and target info.

d980a43

rolfbjarne requested a review from chamons as a code owner November 29, 2019 19:38

Add more perf data and a baseline test.

192f686

Redo the base test a bit.

0f8b068

More tweaks.

3d2bd4f

Disable pushing of perf data for now.

7e81071

Use subdirectories.

aafc2c7

Reset maccore.

e06e1bf

Enable xamarin.

1dfc8fd

cd

5d58c6b

hackityhack.

85fe9d7

Debug spew.

0a7cff8

Another idea.

37e8779

rolfbjarne added 2 commits December 16, 2019 10:51

Undebug spew.

cc3fbd5

Build me

732f0cf

Paths, paths, paths...

6f5d8e8

rolfbjarne closed this Dec 16, 2019

rolfbjarne deleted the sample-test-statistics branch December 16, 2019 18:23

rolfbjarne mentioned this pull request Dec 16, 2019

[sample-tester] Collect app size and build duration statistics. #7602

Merged

[sample-tester] Collect app size and build duration statistics. #7517

[sample-tester] Collect app size and build duration statistics. #7517

Uh oh!

Conversation

rolfbjarne commented Nov 29, 2019

Uh oh!

rolfbjarne commented Nov 29, 2019

Uh oh!

monojenkins commented Nov 29, 2019

Uh oh!

xamarin-release-manager commented Nov 29, 2019

Uh oh!

spouliot Nov 29, 2019

Choose a reason for hiding this comment

Uh oh!

rolfbjarne Nov 29, 2019

Choose a reason for hiding this comment

Uh oh!

spouliot Nov 29, 2019

Choose a reason for hiding this comment

Uh oh!

spouliot Nov 29, 2019

Choose a reason for hiding this comment

Uh oh!

spouliot Nov 29, 2019

Choose a reason for hiding this comment

Uh oh!

rolfbjarne Nov 29, 2019

Choose a reason for hiding this comment

Uh oh!

monojenkins commented Nov 29, 2019

Uh oh!

xamarin-release-manager commented Nov 29, 2019

Uh oh!

monojenkins commented Dec 2, 2019

Uh oh!

xamarin-release-manager commented Dec 2, 2019

Uh oh!

monojenkins commented Dec 2, 2019

Uh oh!

xamarin-release-manager commented Dec 2, 2019

Uh oh!

monojenkins commented Dec 2, 2019

Uh oh!

xamarin-release-manager commented Dec 3, 2019

Uh oh!

monojenkins commented Dec 3, 2019

Uh oh!

monojenkins commented Dec 13, 2019

Uh oh!

xamarin-release-manager commented Dec 13, 2019

Uh oh!

monojenkins commented Dec 13, 2019

Uh oh!

xamarin-release-manager commented Dec 13, 2019

Uh oh!

monojenkins commented Dec 13, 2019

Uh oh!

xamarin-release-manager commented Dec 14, 2019

Uh oh!

monojenkins commented Dec 14, 2019

Uh oh!

xamarin-release-manager commented Dec 14, 2019

Uh oh!

monojenkins commented Dec 14, 2019

Uh oh!

xamarin-release-manager commented Dec 14, 2019

Uh oh!

monojenkins commented Dec 16, 2019

Uh oh!

xamarin-release-manager commented Dec 16, 2019

Uh oh!

monojenkins commented Dec 16, 2019

Uh oh!

xamarin-release-manager commented Dec 16, 2019

Uh oh!

monojenkins commented Dec 16, 2019

Uh oh!

xamarin-release-manager commented Dec 16, 2019

Uh oh!