.NET Core installation types for 3.0#58
Conversation
vitek-karas
requested review from
KathleenDollard,
johnbeisner,
leecow and
richlander
vitek-karas
changed the title
jeffschwMSFT
left a comment
jeffschwMSFT
left a comment
There was a problem hiding this comment.
Thanks for writing this up.
I am happy to see custom install locations coming to Linux (cc @leecow) which I believe was a requirement from one of our Linux distro partners.
| * :new: Linux - `/usr/share/dotnet` | ||
| * :new: macOS - `/usr/local/share/dotnet` | ||
|
|
||
| *Note: The above means that on Linux/macOS in 2.2 the multi-level lookup was effectively non-functional, as there were no search paths for it. Part of the 3.0 changes is to change that and make Linux/macOS work similarly to Windows in this case as well.* |
There was a problem hiding this comment.
We have been talking about phasing-out multi-level lookup, e.g.: https://github.com/dotnet/core-setup/issues/3606#issuecomment-373458531 . Have we changed our opinion?
There was a problem hiding this comment.
That issue is more of an SDK issue - if the user would invoke the app via dotnet exec it would behave as expected. The problem is with dotnet run which first finds SDK and then runs the app in the "context" of the SDK (using the same dotnet.exe and frameworks).
I think we will have some form of multi-level lookup going forward. If it's an environment variable or not is a separate discussion I think. I'll mention that issue in the doc.
If we end up not doing anything in this area in .NET Core 3.0 we're left where we're today which is very confusing. On Windows it works, on Linux/macOS it doesn't, for no good reason. The code mentions that "we don't have a global location" on Linux but
- That's not true already as we're using these global locations on Linux/macOS in
apphostto search forhostfxr. - We want the globally registered location for Linux as it's required by distros which will install to custom locations.
There was a problem hiding this comment.
we will have some form of multi-level lookup going forward.
Multilevel-lookup stitching multiple global locations is fine, I think.
I think the problematic part of the multilevel-lookup is that it stitches your local unzipped SDK/runtime with the globally installed ones. It makes the locally unzipped SDK/runtime scenario very confusing.
FWIW, I have DOTNET_MULTILEVEL_LOOKUP=0 set in the global environment on my Windows machine today. Multilevel-lookup never does what I want, but I do realize that I may be special (running preview builds by just unziping them, etc.).
There was a problem hiding this comment.
So do you think we should leave the current behavior where on Windows it does something but on Linux/macOS it's effectively ignored?
There was a problem hiding this comment.
I think that the problem with multi-level lookup is that it mixes local unzipped installs with global ones. If it stopped doing that, it would be fine with me (on both Windows and non-Windows).
Would something like this work (on both Windows and non-Windows)?
- For
dotnetrun from official global install, stitch together all global locations. (We would need a bread crumb that indentifies official global install.) - For
dotnetrun from local unzipped install, look in the directory wheredotnetresides orDOTNET_ROOTpoints to only.
There was a problem hiding this comment.
I spot checked aspnet/AspNetCore repo and it disables it just like all the other repos... interesting. I mean it makes perfect sense for repos to isolate themselves, for predictability.
I don't know why we introduced it in the first place... I will try to find who did it and figure out the reasoning. Unfortunately this a well document feature even externally. And also unfortunately it's on by default, so users may rely on it without even knowing about it.
There was a problem hiding this comment.
I don't know why we introduced it in the first place.
I suspect it was to make the original dnx & project.json workflows work that are history by now
There was a problem hiding this comment.
The multi-level lookup for frameworks was introduced in dotnet/core-setup#1241 and the SDK counterpart then in dotnet/core-setup#1568. It has been introduced first in .NET Core 2.0.
Unfortunately it seems that most people involved in the original work have left... I guess the best bet is @eerhardt and @nguerrera .
Short version: We're looking for thoughts on basically changing the DOTNET_MULTILEVEL_LOOKUP to off by default (possibly with some minor tweaks to the exact behavior).
There was a problem hiding this comment.
I'll try to summarize some history as I remember it...
When first implemented, multi-level lookup searched several more levels. It looked in some well known offset from CWD and in a well known offset from user profile. When I learned about this, I reached out questioning the wisdom of CWD, and after several discussions, it was determined that both CWD and user profile should be removed. This left x-plat with no other locations to search. At the time, there was no apphost and no code that tried a hard-coded location in /usr/ as a global location x-plat. (Actually, originally, dotnet on PATH was global location x-plat I think, but this was also deemed problematic and useless if folks put a different location on PATH.)
The user profile scenario case was intended for this workflow that we have yet to deliver as the convenience of the experience never seems to get past other concerns including security:
- Repo build/init script pulls down version repo needs to user profile (dotnet-install even defaults to this location IIRC)
- Global.json in repo points to (1)
- From there, double clicking on .sln in repo makes VS use (1) without setting any environment variables prior to VS launch
Essentially, this allows a zero admin install / zero global configuration: just clone, build, launch in IDE however you like.
User profile was used in place of a private location inside the repo so that repos could share. Once you built one repo with that version, you didn't need to re-download and waste space. This was particularly interesting in a world of 50 ASP.NET repos.
I believe that ASP.NET ended up using the closest equivalent to this under the degenerate multi-level lookup (dotnet dir + global location) by setting PATH once on their machines to the user location. It has probably changed again since then. Later on, putting dotnet on PATH alone was not good enough with the appearance of apphosts and DOTNET_ROOT. You now have to set both PATH and DOTNET_ROOT to have a full experience in the IDE.
I still think we're missing this scenario, and I do think it would be nice to go after some variant of it. Environment variables as per-repo configuration are really painful. We get a ton of negative feedback about how hard it is to use a private install in the IDE. No matter how many instructions are put in the repo docs, they are simply not read. Anything other than clone/build/open in VS leads to upset contributors.
We have informally talked about the alternate possibility of putting a PATH to dotnet location in global.json, which repo build would arrange to download to. For this to actually work as a replacement for the original vision, it would have to be respected as DOTNET_ROOT. I suspect we're going to have about the same difficulty there as with the original multi-level lookup vision. :(
cc @davidfowl
There was a problem hiding this comment.
I agree that multi-level lookup interaction "unzip and use directly from the unzipped location" is extremely surprising behavior. I have spent a ton of time helping people getting tripped up by it. I also turn off multi-level lookup globally on my machines. Where we are now, we are getting leftover surprises from a feature that ended up being crippled to the point of not providing its original intent.
I also agree that searching multiple global, privileged, registered locations should not be something we need to disable by default.
| * `./dotnet.exe` - the "muxer" which is used to execute apps and issue dotnet CLI commands. | ||
| * `./host/fxr/<version>/hostfxr.dll` - this library contains all the logic to resolve framework references and select SDK. Regardless of application, the latest available version is used. | ||
| * `./shared` - contains shared frameworks. | ||
| * `./Microsoft.NETCore.App/<version>` - the root framework which contains the .NET Core runtime. This framework contains `hostpolicy.dll` which is the entry point called by the `hostfxr` to run the app. Applications can choose which version to use. |
There was a problem hiding this comment.
This could use some wordsmithing for clarity.
There was a problem hiding this comment.
@KathleenDollard can you please suggest some improvements, the "wordsmithing" is probably the most difficult part of this whole thing for me. Thanks in advance!
|
|
||
| The biggest problem is with the app-contained hosts which we can't service in any way. So introducing new install types or changing the default location can be breaking for apps using old hosts. | ||
|
|
||
| The problem already exists since the `apphost` has been shipped in .NET Core 2.1 with support for framework dependent apps. Fortunately this feature is not widely used yet. |
There was a problem hiding this comment.
It is widely used in global tools.
It's not how many tools there are, but how many people are using those tools.
But I don't see any breaking changes proposed.
There was a problem hiding this comment.
I added the global tools case.
Please note that the problem is not breaking changes in the apphost, but rather the effect of introducing new install locations in the future.
If I have an apphost 2.2 app on a clean machine which only installs into a custom location (registered in registery) as can happen in the 3.0 case. Even if that app has "roll forward on major" and should run on the 3.0 just fine, it won't, because it won't find it.
Since in 3.0 majority of apps will have apphost, if we in 4.0 we introduce a new install location this problem will be greatly exagerated. The urgency comes from this fact. It would be better if we can agree on set of install locations now, which we think should work well in the future. Then 3.0 apphost should work going forward.
I know that we can't predict the future, but this is an attempt to mitigate the risk of the above mentioned problem in the future. (And also to write down the existing and desired behavior somewhere, since we don't seem to have that anywhere really).
| * Linux - `/usr/share/dotnet` | ||
| * macOS - `/usr/local/share/dotnet` | ||
|
|
||
| In each case with the `<version>` subdirectory, the latest available version as per SemVer 2.0 rules will be selected. Note that the algorithm picks the first location which exists, and only then looks for the `/host/fxr/<version>` subdirectory, and if that doesn't exist, or the library is missing it will fail. |
There was a problem hiding this comment.
Does this mean...
Machine is working with some earlier versions of .NET Core in the default global location. User decides to put a new .NET Core into a user or global registered location. Now, do apps previously working against the earlier versions of .NET Core fail?
There was a problem hiding this comment.
They should not fail. Two parts to the solution:
- the host components (specifically
hostfxr) must be 100% backward compatible. So new .NET Core'shostfxrshould work just fine with old shared frameworks/SDKs. - the installer should not allow install to a custom location if there already is .NET Core installed in the default location. This behavior has already been speced and discussed I just forgot to mention it in this document. I added that.
Based on feedback I'll remove the per-user install type completely - it has too many issues.
|
|
||
| In .NET Core 3.0 the situation will be very different. Framework dependent `apphost` is the default for apps. Scenarios requiring the other hosts are also likely to be relatively common (COM, IJW, native hosting). So the hosts which we ship with .NET Core 3.0 will be spread across all machines running .NET Core (not just development but production as well) without any ability to service them directly. | ||
|
|
||
| This means that .NET Core 3.0 is the last chance to get the algorithms for install locations modified in "breaking ways". Future release will very likely have to be 100% backward compatible in this regard. |
There was a problem hiding this comment.
Is this true? Later it says that apphost always looks next to itself first.
It made sense when I read it, then it seemed the reason we think it is our last chance is the scenario least likely to encounter changes.
There was a problem hiding this comment.
I tried to explain the problem here: #58 (comment)
The catch is that
apphostdoes look next to itself first when searching forhostfxr, BUT- framework dependent
apphostwill not find thehostfxrnext to itself - and this is the default layout for 3.0 apps.
So in fact in 3.0 the most common case will be apphost which finds the hostfxr in some globally registered location. That's why I think we better get that search algorithm right.
| * It ignores `DOTNET_ROOT` environment variable. | ||
|
|
||
| #### Framework and SDK search | ||
| I think the muxer should use `DOTNET_ROOT` to resolve frameworks and SDKs just like all the other hosts (the actual implementation mechanism aside). The current discrepancy is simply unexpected and makes certain scenarios unnecessarily complex (for example VS can't really use private installs because of this, since it always uses the muxer from path). |
There was a problem hiding this comment.
Note that VS worked fine with private installs when they were put on PATH (ignoring the whole discussion about how folks want a solution that doesn't change any env vars). Until apphost became defaullt for F5 in 3.0 and now you need DOTNET_ROOT as well (exacerbating the problem of needing to set env vars).
This has been a topic of much debate in the past. People have argued various things:
-
Should muxer respect DOTNET_ROOT as you propose, but also:
-
Should muxer export DOTNET_ROOT to children. One thing that people find surprising is
/some/private/dotnet Xlaunchingdotnet Yand the latter can't find framework or SDK in /some/private. -
Should CLI do (2) if muxer won't.
The problem with DOTNET_ROOT is that it is a hammer. Setting it can fix some things and break others. The fact that you initiated a process chain with some dotnet probably means that the frameworks and sdks in that location should be considered. We know from experience that folks consider this surprising when it isn't. On the other hand, there can be other things in the chain that need the global locations. For example, in a build, I run a global tool built against 2.1 and I have 2.1 globally, but I also run a freshly built 3.0 apphost in the same process chain and I only have 3.0 in a private location.
For this reason, we have been reluctant to guess and set DOTNET_ROOT. Except that /some/private/dotnet run was completely broken by apphost, so now we set it in dotnet run but that has been thought of as a hack pending the outcome here.
|
Based on the discussion I made these changes:
Please take another look if this new proposal is something we can agree on for .NET Core 3.0. |
| * `HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\<arch>\InstallLocation` - where `<arch>` is one of the `x86`, `x64`, `arm` or `arm64`. | ||
| Note that this registry key is "redirected" that means that 32-bit processes see different copy of the key then 64bit processes. So it's important that both installers and the host access only the 32-bit view of the registry. | ||
| * Linux and macOS - the installer will store the install location in a text file | ||
| * `/etc/dotnet/install_location.conf` - the file will contain a single line which is the path. |
There was a problem hiding this comment.
Would it be better if we put more settings than just a single one per file? There are other settings that should go into /etc/dotnet (think any env var we have today). Having one file per setting seems excessive.
There was a problem hiding this comment.
It would be interesting to know which variables might be useful. The only ones which might make at least some sense to define globally would probably be:
DOTNET_RUNTIME_IDDOTNET_SHARED_STORE- we're trying to move away from this.DOTNET_MULTILEVEL_LOOKUP- this has no effect on Linux
We could change it for the sake future extensibility:
File name: install.conf (better ideas? Trying to reflect what we have on Windows in registry at least a little bit for consistency. If needed we could add installed SDKs into it, and the hostfxr version)
Content in the .ini format:
#comment
InstallLocation=<path>For now we would probably implement a VERY simplistic .ini parser, just ignore comments and split on = sign.
There was a problem hiding this comment.
Basically anything that is an environment variable check today:
- DOTNET_CLI_TELEMETRY_OPTOUT
- DOTNET_SKIP_FIRST_TIME_EXPERIENCE
- DOTNET_DISABLE_MULTICOREJIT
- DOTNET_CLI_UI_LANGUAGE
See also - https://github.com/dotnet/cli/issues/2960
There was a problem hiding this comment.
Good point - I didn't realize that this would apply to SDK as well.
What about the filename/format, does that sound reasonable?
There was a problem hiding this comment.
I think that file name and format sounds reasonable. Just be sure to allow spaces around the = as well:
InstallLocation = <path>There was a problem hiding this comment.
In other words, I agree 1 setting per file is excessive, but keeping the settings that are read by unserviceable apphost code separate seems wise.
There was a problem hiding this comment.
@eerhardt would you be OK with the approach proposed by @nguerrera?
Keep the things which are required to run apps separate from everything else and as simple as possible...
(in which case I would also like to keep the simple text file with first line meaning the location, everything else ignored for now).
There was a problem hiding this comment.
If we go this way, I would consider using a less suggestive name than the .conf extension: /etc/dotnet/install_location. The .conf extension and the lack of location are suggestive that we would put more things in there, but I think that is probably the only thing the apphost will read. The rest can be read from code we can service in a different file, right?
There was a problem hiding this comment.
Another possible setting that could go in this file is the DOTNET_RUNTIME_ID setting, in cases where a user needs to override this and can't/won't use an environment variable.
Looking for more cases of the host reading environment variables:
https://github.com/dotnet/core-setup/search?q=getenv&unscoped_q=getenv
CORE_BREADCRUMBS and CORE_SERVICING could be others.
I'm OK with having a split between the "runtime" config and "SDK" config, especially considering if the SDK config needs a more complex format. But from a customer's perspective, it feels a bit arbitrary, and I don't know if all users understand the difference.
I think my biggest concern is not having 1 file per setting. But I agree we don't need all settings in one file.
There was a problem hiding this comment.
Even for the runtime/host settings, there's a distinction between what must be read by apphost code sitting in customer app and what can be read by serviceable code. This is why I think we should keep the thing read by apphost distinct and as simple as possible.
|
/cc @omajid |
Changed the format of the configuration file on Linux/macOS to use the `.ini` format.
sdmaclea
left a comment
sdmaclea
left a comment
There was a problem hiding this comment.
Document seems to be consistent with current design.
Keep /etc/dotnet/install.conf as simple as possible.
Avoid changing muxer
| * :new: global registered install location, in the `/host/fxr/<version>` subdirectory | ||
| **We've committed to support this on Windows for .NET Core 3.0** | ||
| * Windows - `HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\<arch>\InstallLocation` (32-bit view) | ||
| * Linux/macOS - `/etc/dotnet/install.conf` |
There was a problem hiding this comment.
It would be great if we can also have this on Linux for 3.0 (https://github.com/dotnet/designs/issues/64).
There was a problem hiding this comment.
We can help with implementing this on Linux if it's a question of devel resources.
There was a problem hiding this comment.
In terms of implementation my current plan is for us to implement it in the hosts first. The installers which install to the default location don't necessarily need to write the file as the hosts will work without that. I would like even the default location to use the config file just so that all cases are consistent, but it's not absolutely necessary, so will probably happen later on.
Obviously if some other distro wanted to install to a non-default location its installers would have to write the file for the system to work.
There was a problem hiding this comment.
Obviously if some other distro wanted to install to a non-default location its installers would have to write the file for the system to work.
The default location that is put into the host doesn't match with where source-build Fedora packages place dotnet (https://github.com/dotnet/designs/issues/64). So we would very much like to provide a config file with the 3.0 packages that makes framework dependent applications work without setting DOTNET_ROOT.
There was a problem hiding this comment.
That's exactly the goal of this proposal. I'll ping you when we have something working... would very much appreciate if you could test it then.
There was a problem hiding this comment.
+1 we can easily sync (through one of the many Microsoft-RedHat regular channels) and help with Linux specific implementation of just about anything, just give us a nudge if there is something! =)
There was a problem hiding this comment.
The implementation has been merged and is now part of the latest runtime/SDK packages. You can get the latest here: https://github.com/dotnet/core-sdk
I verified that downloading the tarball into a custom location and adding the etc/dotnet/install_location file with a single line pointing to the custom location makes it work. That is when I create a new app with it and then run the executable directly (the apphost) without setting any environment variables, it runs and it does so against the new custom location.
Can you please give it a try in your environment?
I guess the most interesting case would be to verify that it works when you put it into the desired distro-specific location and add the install location file. And then validate that it is possible to run apps with the executable directly without any other special environment settings.
There was a problem hiding this comment.
This works.
I've downloaded the sdk and used it to create a FDD linux-x64 application.
Trying to run it gives:
A fatal error occurred. The required library libhostfxr.so could not be found.
If this is a self-contained application, that library should exist in [/tmp/console2/bin/Debug/netcoreapp3.0/linux-x64/publish/].
If this is a framework-dependent application, install the runtime in the global location [/usr/share/dotnet] or use the DOTNET_ROOT environment variable to specify the runtime location.
Creating /etc/dotnet/install_location and adding the path to folder containing dotnet.
Hello World!
There was a problem hiding this comment.
If this is a framework-dependent application, install the runtime in the global location [/usr/share/dotnet] or use the DOTNET_ROOT environment variable to specify the runtime location.
Maybe we should update that error message to also point people to /etc/dotnet/install_location?
There was a problem hiding this comment.
@eerhardt Good suggestion - dotnet/core-setup#6546
14 participants
Description of various supported installation types (and locations) for .NET Core 3.0 runtime and SDK. Proposals for 3.0 behavior to create consistency between platforms.
Important for 3.0 as we're shipping new hosts which don't have a servicing solution.