Project Cracker Output and Behavior Issues

[cloudRoutine]

The ProjectCracker is a constant source of frustration for us while developing Ionide tools and for our users (although they rarely realize it) whether it's heisenbugs resolving project settings to pass to the compiler service or just falling short on a simple project evaluation

Personally I'm not a fan of how the project cracker runs msbuild in the background and starts creating files in a directory as soon as a project has been evaluated to provide editor services.
from

to

upon opening TextUtil.fs link to repository screencap'd

Furthermore the utility of the FSharp.Compiler.Service.ProjectCrackerTool.exe could use some improvement

λ» .\FSharp.Compiler.Service.ProjectCrackerTool.exe -h
{"LogOutput@":"At least two arguments required.","Options@":[],"ProjectFile@":"","ReferencedProjectOptions@":[]}

It shouldn't be necessary to read the src code to figure out that in order to get a text log of the cracked information you need to input

λ» .\FSharp.Compiler.Service.ProjectCrackerTool.exe FSharpTest.fsproj false --text
{ProjectFile = "FSharpTest.fsproj";
 Options =
  [|"--simpleresolution"; "--noframework";
    "--out:C:\Users\jared\Desktop\fsexp\clones\ionide-error-repro\FSharpTest\bin\Debug\FSharpTest.dll";
    "--doc:C:\Users\jared\Desktop\fsexp\clones\ionide-error-repro\FSharpTest\bin\Debug\FSharpTest.XML";
    "--subsystemversion:6.00"; "--highentropyva+"; "--fullpaths"; "--flaterrors";
    "--target:library"; "--define:DEBUG"; "--define:TRACE"; "--debug+";
    "--optimize-"; "--tailcalls-"; "--debug:full"; "--platform:anycpu";
    "-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\FSharp\.NETFramework\v4.0\4.4.0.0\FSharp.Core.dll";
    "-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6\mscorlib.dll";
    "-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6\System.Core.dll";
    "-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6\System.dll";
    "-r:C:\Program Files (x86)\Reference Assemblies\Microsoft\Framework\.NETFramework\v4.6\System.Numerics.dll";
    "C:\Users\jared\Desktop\fsexp\clones\ionide-error-repro\FSharpTest\AssemblyInfo.fs";
    "C:\Users\jared\Desktop\fsexp\clones\ionide-error-repro\FSharpTest\TextUtil.fs"|];
 ReferencedProjectOptions = [||];
 LogOutput = "";}

to get a proper output, yet at the same time the following input will fail

λ» .\FSharp.Compiler.Service.ProjectCrackerTool.exe FSharpTest.fsproj --text
{ProjectFile = "";
 Options = [||];
 ReferencedProjectOptions = [||];
 LogOutput = "At least two arguments required.";}

and really to get the output the commandline args can be

λ» .\FSharp.Compiler.Service.ProjectCrackerTool.exe FSharpTest.fsproj ItDoesntMatterWhatYouInput --text

instead of requiring false, which isn't the most intuitive behavior.
Another issue with output from an input array the tool can't properly process

{"LogOutput@":"At least two arguments required.","Options@":[],"ProjectFile@":"","ReferencedProjectOptions@":[]}

is that it implies an argument order different from what it actually accepts.

If the project cracker is going to build projects (which again I don't think it should) it should at least build them properly

λ» .\FSharp.Compiler.Service.ProjectCrackerTool.exe FSharpTest.fsproj false Configuration Release --text

shouldn't result in

where it's copying the references to the build output directory but not the actual dll produced as the result of the build.

When logging is enabled not being able to set the diagnostic level to something besides Microsoft.Build.Framework.LoggerVerbosity.Diagnostic is a nuisance.

There also doesn't seem to be a way to cancel a build being executed by the project cracker, unless I missed that somewhere.

@rneatherway thoughts?

[rneatherway]

Hi @cloudRoutine, sorry for the delayed reply, I have been away.

The technique used by the project cracker is the same as the one used by the Visual F# tools produced by Microsoft, and that is why I implemented it that way. I agree that it is fragile and frustrating, and I'm not happy about it either. There are a couple of things that make it harder for us than for them:

• They assume they are bundled with Visual Studio and therefore know the MSBuild version they should be compiled against, whereas I tried to support both VS2012 and VS2015, which led to difficulties with the MSBuild resolution. This is why it is out of process, otherwise you are pinned to the version used by the hosting process.
• They know all the internal undocumented magic (like setting the BuildingInVisualStudio property!).

I didn't design the cracker tool to be used directly from the command line, that is just a debugging aid, and that's why the command line options are not intuitive. Making those better would obviously be a good thing. I was also planning to allow selecting the verbosity, which would require a major version bump in FCS as it would change the API.

As the cracker uses MSBuild internally, this is the only 100% accurate way to determine the correct references, and again this is just what the Visual F# tools do (unless they've changed). If you look at the code you will see that it injects a fake null compiler so that the code is not actually compiled (which would be much slower) and that is why the dll doesn't end up in the output directory. It might be possible to avoid some of the other copying by selecting a more specific MSBuild target but I couldn't achieve that even when following the Visual F# tools code in detail.

I hope that context is useful to explain how things are this way. I don't plan to work on the project cracker anymore as I don't find it enjoyable or rewarding. I'm happy to answer questions and share knowledge that I learned along the way though!

[cloudRoutine]

Working with all of this msbuild stuff is pretty terrible. I use xlinq to pull the data out in forge, but like you said it's not perfect and I always need to fix something new with it 💫

If you're not going to work on the project cracker anymore, how is it going to be improved any further than its current state?

[rneatherway]

Working with all of this msbuild stuff is pretty terrible. I use xlinq to pull the data out in forge, but like you said it's not perfect and I always need to fix something new with it 💫

For simple cases that certainly works, but the more complicated the project file the more hand coding you have to do. Still, forge is a pretty cool tool and so it's obviously working well.

If you're not going to work on the project cracker anymore, how is it going to be improved any further than its current state?

I think a new maintainer will need to step up for that component.

[dsyme]

@cloudRoutine All those things are definitely real, huge problems with the F# compiler/project reliance on MSBuild

It all fundamentally come from the current need to rely on MSBuild APIs to do very basic work in the F# ecosystem (project file interpretation), which itself comes from the systematic reliance on the fsproj file format (and separately on the use of MSBuild assembly resolution for incomplete assembly specifications by the F# compiler, a poison I'm currently working on isolating and then extracting, e.g. here: dotnet/fsharp#1593)

Any use of MSBuild brings a huge range of problems - the project cracker is just some bandaging over the fundamental problem. It is useful enough for what it does - drive MSBuild and interrogate it - but it is MSBuild itself doing the bad business of creating files, and which has an API which is significantly hard to drive. The fact that the implementation of MSBuild can't be packaged as a bunch of copied DLLs like all other normal software components (at least for .NET Framework MSBuild) is another real problem.

The ProjectCracker is definitely simple enough that someone who knows how to drive the MSBuild API or read the MSBuild docs in depth could make improvements to it (e.g. cancellation) - it just takes some calls to the MSBuid APIs and puts them out-of-process. The code is nice and for what it does it has been a very important step forward. But it doesn't solve the fundamental problem: the F# community very badly need a library that cracks project files reliably (without any of the problems above, and preferably without going out-of-process). Or they need a new project format.

Someone who writes (or finds) an accurate, compatible MSBuild evaluator sufficient for real-world project file cracking, available as plain F# code, would be a community hero (as @rneatherway has been for contributing and maintaining the project cracker so far :) )

[7sharp9]

@dsyme Ive recently had to delve into msbuild to retrieve a list of hard dll references from a project files and its fairly easy to retrieve with minimal code. Ideally I would like to replicate the algorithm to find the references so it can be applied whenever we want.

[dsyme]

With the advent of the new .NET SDK 2.0 prject file format the use of the project cracker is gradually declining. It's good for what it is and has served us well, but I'm closing out old project cracker bugs as won't fix. We can reactivate if really urgent.

[georgewfraser]

For anyone who stumbles across this issue who is trying to do project-cracker-like-things, I found this project to be very useful and reliable for analyzing .fsproj and .csproj files: https://github.com/daveaglick/Buildalyzer