Setup new dotnet new android measurements using Mono or CoreCLR runtimes

[matouskozak]

Switching MAUI-Android (and .NET Android) scenarios to use latest .NET SDK + dotnet10 feed, the steps are:

1. Install the latest .NET SDK: .dotnet-install.sh --channel 10.0 --quality daily
2. Using the latest feed (dotnet10), find the latest packs using dotnet package search.
3. Put the latest packs to a rollback file and install maui workload.

Adding new jobs:

• dotnet new android running CoreCLR.
• dotnet new maui (Android) running CoreCLR.
• dotnet new maui-blazor (Android) running CoreCLR.

Internal pipeline run: https://dev.azure.com/dnceng/internal/_build/results?buildId=2684265

TODO:

• Remove added android feeds from Nuget.config (should be added by maestro)

Out of scope

• Fix flaky memory consumption measurements for .NET Android and .NET MAUI Android [android] Fix flaky Android memory consumption measurements #4826

---

#4754

[caaavik-msft]

Looks good to me. One question I have, are any of these changes (e.g. the _MSBuildArgs) being made because they are coupled to the runtime repository itself or would you want them set even if we decided to run against an older version of the runtime?

As part of the recent changes we made to move the yaml from the runtime to the performance repository, I'd like to know if any of the things we moved over were better living in the runtime repository. If so, we can look at restructuring where these live to better couple related changes.

[matouskozak]

One question I have, are any of these changes (e.g. the _MSBuildArgs) being made because they are coupled to the runtime repository itself.

They are not coupled with the dotnet/runtime repository but rather with dotnet/android that use them to select/configure appropriate runtime (Mono vs CoreCLR).

if we decided to run against an older version of the runtime

We won't be able to run CoreCLR Android perf jobs against older runtime because the runtime doesn't support it. For Mono, it would work as is.

I'd like to know if any of the things we moved over were better living in the runtime repository.

I think it is ideal that all this now lives in dotnet/performance as it doesn't necessary depend on any custom/local builds of runtime. Also it is much simpler to debug now as these specific jobs all run on a single pipeline.

[ivanpovazan]

Why do we only care about TTFD measurement here? Or why don't we measure both TTID and TTFD?

[matouskozak]

I don't know. I just saw that it is passing the --use-fully-drawn-time so I added it to the scenario name to make it clear. I can probably add TTID as well if this is something we want.

[matouskozak]

@LoopedBard3 do you remember why for blazor we are using --use-fully-drawn-time and not the default startup measurement config?

[LoopedBard3]

I don't recall anything specifically. Maybe TTID doesn't include some of the blazor startup. Regardless, I don't see any reason not to include both.

[ivanpovazan]

@matouskozak

There might be some issues with the approach of dotnet workload config --update-mode manifests especially around servicing releases, because the manifests will include a reference to a new servicing release (like atm 9.0.4) which is not publicly available and we don't know the exact feed, so the workload installation would fail with:

Workload installation failed: One or more errors occurred. (Version 9.0.4 of package microsoft.net.runtime.monoaotcompiler.task is not found in NuGet feeds ...

Will experiment a bit with this but I think we might need to the the Option 2 from your proposal in: #4754

[LoopedBard3]

This looks really great! I am excited to have what seems to be a more robust solution for building the apps. I left one comment but am also wondering if this supports 9.0 and 8.0 mobile app runs. It seems that it currently sets up the runs to only run main.

[matouskozak]

I left one comment but am also wondering if this supports 9.0 and 8.0 mobile app runs.

Yes, currently works just on main. I think it could be extended to other branches. We would just search a different dotnet feed I think and make it more parametrized. However, I don't see much value now in having measurements on other branches but we could extend it in the future if needed.

[Copilot]

Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.

Files not reviewed (1)

• eng/performance/maui_scenarios_android.proj: Language not supported

Comments suppressed due to low confidence (1)

eng/pipelines/sdk-perf-jobs.yml:405

• [nitpick] The pipeline uses 'codeGenType' (PascalCase) while the corresponding code in run_performance_job.py references 'codegen_type' (lowercase). Consider using consistent naming conventions.

codeGenType: ProfiledAOT

[Copilot]

The function 'extract_latest_dotnet_feed_from_nuget_config' uses 'ET' without an import. Please add 'import xml.etree.ElementTree as ET' at the top of the file.

[Copilot]

The regex usage 're.match' requires the 're' module but no import is present. Please add 'import re' at the beginning of the file.

[ivanpovazan]

This looks neat, thank you!

The only thoughts left (not for this PR) are:

1. It seems like the rollback file could be provided by some other service rather than us manually creating it as part of performance runs
2. Think about a way of doing performance runs on specific releases, versions i.e., answering a question: what is the performance of our preview.X release - and with that freeze/tag such measurements for reference

Other than that, really nice work!

[matouskozak]

Think about a way of doing performance runs on specific releases, versions i.e., answering a question: what is the performance of our preview.X release - and with that freeze/tag such measurements for reference

We can do that locally by using manually installed dotnet and providing --readonly-dotnet --has-workload arguments to the pre.py.

[caaavik-msft]

LGTM, just the one minor comment on the yaml condition