Need to figure out what to do with ILLink in source-build

[eerhardt]

Today, we turn off ILLink in corefx

source-build/repos/corefx.proj

Line 6 in ac7159e

<BuildCommand>$(ProjectDirectory)/build$(ShellExtension) $(BuildArguments) -- /p:ILLinkTrimAssembly=false</BuildCommand>

But we are now dependent on ILLink with #376.

We should either not use ILLink at all. Or we should embrace it everywhere and figure out how to build it from source (it depends on Mono.Cecil), and then use that source-built version to build .NET Core.

/cc @weshaggard @dagood @dseefeld

[tmds]

@eerhardt I know corefx is using illink for avoiding zeroing stackalloced memory. Is it is used for other things?

[eerhardt]

dotnet/coreclr appears to be using it to build System.Private.CoreLib - probably for the same reasons corefx does.

I'm not 100% sure where else. @dagood - I didn't see any reasoning in #376 why ILLink.Tasks was added. Can you explain why the dependency got added here?

[weshaggard]

@erozenfeld @stephentoub any thoughts on implications of turning this off our source-builds?

[stephentoub]

appears to be using it to build System.Private.CoreLib - probably for the same reasons corefx does.

Yes, both coreclr and corefx use the linker to trim away dead code and to avoid zero'ing locals. (Though it appears from your initial comment the trimming was turned off for corefx... I wasn't aware of that. I know I've had cases in recent memory where things got trimmed away, so I'm not sure what's going on, maybe it's a very recent change? Or... maybe that's just for source build?)

Implications of turning off the linker would be larger binaries and less-efficient code; in particular for corelib, the latter was very measurable in various microbenchmarks around things like string processing, integer formatting and parsing, etc., things involving sizeable stackallocs.

Once https://github.com/dotnet/csharplang/blob/master/proposals/skip-localsinit.md is available in the compiler, we can use that instead of the linker, at least for the locals zero'ing.

[eerhardt]

Dumb question about trimming dead code - now that we’ve brought back all the APIs in netstandard2.0, can’t we just delete any remaining dead code? Or at least #if it out for .NET Core?

[stephentoub]

can’t we just delete any remaining dead code?

For the most part, yes, and it's in progress:
https://github.com/dotnet/corefx/issues/17905

[erozenfeld]

Yes, ILLink is currently used only for trimming code and clearing initlocals. Note that some dead things can't be removed from source but can be removed by ILLink: e.g., inlined constants, private constructors. ILLink is also a place for any other future IL-to_IL transformations that either can't be done by Roslyn or can be done by Roslyn but may benefit from first testing them in ILLink (like we did with clearing initlocals). Having ILLink in the corefx and coreclr pipelines is convenient for that.

@sbomer and I have been building ILLink.Tasks from source (including Cecil) and publishing the packages. We can help with integrating that into source-build if that is what is required. Is source build not allowed to use any packages?

[erozenfeld]

@jeffschwMSFT @jkotas

[dagood]

@dagood - I didn't see any reasoning in #376 why ILLink.Tasks was added. Can you explain why the dependency got added here?

With the BuildTools and CoreCLR update, the package not existing made CoreCLR fail, and the priority was getting CoreCLR working. I didn't get a deep understanding of why it wasn't needed before and was after.

[tmds]

@eerhardt @dagood My understanding from this issue is that ILLink packages are being fetched, but they are not being used. Is that correct?

[dagood]

My understanding is we're at a weird middle ground because I didn't know CoreFX had it disabled: CoreCLR uses it, but CoreFX doesn't.

We should be able to disable it in CoreCLR as well and remove it again, with the perf cost stephentoub mentioned. (I don't know the build arg to do this.) If it's ok to have it as a prebuilt, we could enable it for CoreFX so that we're at least using what we already have.

[jkotas]

The performance cost to disable it in CoreCLR is prohibitive. It will cause performance regressions that are not going to be acceptable. (Unless we pick up the compiler with SkipLocalsInitAttribute and start using this feature.)

[dagood]

[@erozenfeld] @sbomer and I have been building ILLink.Tasks from source (including Cecil) and publishing the packages. We can help with integrating that into source-build if that is what is required.

Unless there are alternative ways to avoid the perf impact, that is what we need to do to get rid of the prebuilt dependency, yeah. Thanks for the offer. Here's a doc on the basics of the source-build repo if you want to see the general expectations ahead of time--it doesn't seem to me like we have bandwidth to get that in our imminent releases though: https://github.com/dotnet/source-build/blob/master/Documentation/2.1-update-process.md.

/cc @dleeapho

[@erozenfeld] Is source build not allowed to use any packages?

Check out https://github.com/dotnet/source-build#goals for why we don't want prebuilt (downloaded) dependencies in source-build. This guideline in particular: https://fedoraproject.org/wiki/Packaging:Guidelines#No_inclusion_of_pre-built_binaries_or_libraries. The "area-prebuilt-reduction" label tracks exceptions we're aiming to get rid of.

[@stephentoub] I know I've had cases in recent memory where things got trimmed away, so I'm not sure what's going on, maybe it's a very recent change? Or... maybe that's just for source build?

Yep, just source-build. /p:ILLinkTrimAssembly=false is set when building CoreFX from source.