Create common infra to build MSBuild tasks specific to a repo

[dagood]

Some repos build MSBuild tasks as part of their product build: Arcade should explicitly support that. There are also cases where we want to add new custom MSBuild tasks to specific repos:

• We might not understand the task's requirements well enough to put it in Arcade yet, where subsequent changes always (?) have to be backwards compatible.
• The task might only make sense for that repo.

Making this a common infrastructure means it will be easy to create a new task for any repo, even if that repo hasn't built MSBuild tasks ever before. Then, that task can graduate it to Arcade when it makes sense.

[weshaggard]

One of the goals/principles we want to have is to allow repo's to be able to just open up the root solution and build, while not all repo's will be able to support this (i.e. corefx) we want this principle to hold for most repos. In that model it will be difficult to have a pre-build step that builds tasks that are needed to build the repo. In my opinion if we support this it should be an exceptional case and not first-class as I don't think we want folks to usually do it.

[markwilkie]

I was thinking only of build tasks/packages - nothing pre-build. This idea is to try and avoid having Arcade be a dumping ground for packages which "might" be useful for others.

Does that change your opinion @weshaggard?

[dagood]

Wes is talking about how these tasks/packages have to be built as a pre-build step so that they can be used during the build. Maybe you're thinking of having each task-producing repo have a "side" build that produces the tool packages, and then they're pulled down from the internet--that seems even worse to me.

@weshaggard, maybe this is mitigated by having it built into the infrastructure? Any repo that doesn't have any tasks to build shouldn't have to do anything pre-build, and if any repo does have the requirement that nothing happen pre-build, it just shouldn't add any custom tasks and get them into Arcade instead. With it being a first class feature, there also isn't maintenance work to add/remove the build task creation infrastructure whenever tasks are added for the first time or removed.

[markwilkie]

Oh I see what @weshaggard was getting at.

@dagood - your thinking aligns with mine. Basically, just add the functionality to the build, and then if warranted, we can pull it out as a package and put it into Arcade. This can also be managed to a degree with a clear "bar" for when things get into Arcade.

[dagood]

I think the bar is an issue--for repos that can't have a pre-build step, they need somewhere to put their prototype/prerelease tasks, and Arcade is the only place for them. As long as the bar's flexible and there's a way to mark these packages, maybe that's ok?

[weshaggard]

For repo's like corefx this pre-build step isn't a huge deal because we require a full build of the repo before you can work in VS but that isn't true for most repo's. I'm curious to get @tmat and @jaredpar's take on this. I'm generally not a fan of producing tools that are used to build the repo in a pre-build step in the same repo. I would like arcade to be that repo, or perhaps another repo (hopefully not :().

[tmat]

I am not fan of pre-build step either. The root problem here seems to be the inability to build and use a task "dynamically". But we do have a similar ability - the build can run a C# script. We even ship csi task that runs a specified C# script . There are some issues there that need to be fixed to make it smooth, but that can be done.
@dagood would customizing build using C# scripts instead of tasks address your needs?

[dagood]

That's an interesting idea. Seems to me it'll work as long as we can pull down extra libraries from NuGet packages and use them in the C# build scripts.

[tmat]

@dagood Currently C# scripts do not support NuGet references (it's in the backlog, so maybe someday then will). However, that should be easy to work around. You can add those package references to the containing project and then access the binaries directly from the NuGet cache.

[dagood]

Sounds reasonable to me as long as it's clear how to do it. (I'm not familiar enough with the modern CLI to know if I have a "containing project" in the DotNet-Orchestration-Utilities repo I have in mind.) Opened #97 for C# build scripts, closing this.

[weshaggard]

I think C# scripts are a resonable bridge for repo's to have custom logic specific to them but it isn't a great on-ramp for putting tasks into arcade. I still think we want most of this code to be in a task library overtime so repo's still need a place to put those and I still think arcade is that place.

[tmat]

Scripts are good for one-off extensibility. If you need to customize one task specific to the repo start with a script. If it turns out to be useful across multiple repos, then convert it to a task and put it to Arcade. My 2 cents.