-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fallback host rid is broken on non-portable builds #81654
Comments
Tagging subscribers to this area: @dotnet/area-infrastructure-libraries Issue DetailsMicrosoft's portable builds perform a fall-back to the portable rid when the This doesn't work for non-portable builds because the information for the portable rid is missing in For example, below is the full "runtimes": {
"fedora.37-x64": [
"fedora.37",
"fedora-x64",
"fedora",
"linux-x64",
"linux",
"unix-x64",
"unix",
"any",
"base"
] For source-build, there should be an exact match with the rid found here and the
In these two cases, the Microsoft build will continue to work, while the source-build builds start failing. We should include this information in the non-portable builds so the fallback works. cc @ViktorHofer @ericstj @am11 @omajid
|
I think we should work towards breaking free of Some context: In .NET 7, work was put to remove hard dependency on procfs (
|
The behavior described in this issue is exactly that: a non-portable build should be able to consume portable assets (per fallback RID) independent of what's in |
What we discussed in NuGet/Home#5862 (especially NuGet/Home#5862 (comment)) touches on the bigger picture and would solve the uber issue instead of band-aiding broken source build scenarios like this N+1 scenario. Unfortunately, I don't think anyone currently considers picking this up from the backlog.
As I don't know much about the host fallback, can you please elaborate or link to how that fallback works today? Does a linux-x64 built host automatically fallback to "linux-x64" but then fails to find the associated parent rids because that information is missing from the graph? In respect to this ask, which fallback RID would we choose when source building for "fedora.37-x64"? Anyway, this sounds like we are again working around the lack of a more flexible RID selection algorithm which presumably could supports floating versions. That isn't a problem per-se but if this will continue to seriously impact source build partners, we might instead or additionally strongly advocate for fixing the above linked issue. |
Yes, I've been part of this discussion. Early on, I was advocating to make the rids 'semantical'. After @richlander put out the Simplify RID Model design, I think this much simpler model may be enough. With that model, non-portable rids are for distinguishing source-built assets from portable ones (cfr dotnet/source-build#2932 (comment)). What's proposed here is not a band-aid or a workaround. It's in line with this simplified model.
The fallback is what makes Microsoft's build pick |
Sure. The only difference is how we get there. I was proposing that we give this control to distros rather than owning ourselves. Ideally, it can be done by removing all calls to
/etc/os-release for good.
|
@am11 it's an interesting idea, you should write it up in an issue. I'd like to scope this to bring the non-portable build behavior in-line with the portable build when |
Rather than changing @ViktorHofer the code for the fallback is here: runtime/src/native/corehost/hostpolicy/deps_format.cpp Lines 126 to 136 in 58719ec
It falls back to the corresponding portable rid defined here: runtime/src/native/corehost/hostmisc/pal.h Lines 78 to 90 in 58719ec
Information about this portable rid is not present in a source-built The fix is either to add the fallback rid to |
I spent some time yesterday thinking about this issue and what you propose is exactly what I imagined would be the right solution. Instead of finding a portable RID, just make sure that the target RID is honored in all scenarios (direct match and host fallback). cc @vitek-karas |
Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov Issue DetailsMicrosoft's portable builds perform a fall-back to the portable rid when the This doesn't work for non-portable builds because the information for the portable rid is missing in For example, below is the full "runtimes": {
"fedora.37-x64": [
"fedora.37",
"fedora-x64",
"fedora",
"linux-x64",
"linux",
"unix-x64",
"unix",
"any",
"base"
] For source-build, there should be an exact match with the rid found here and the
In these two cases, the Microsoft build will continue to work, while the source-build builds start failing. We should include this information in the non-portable builds so the fallback works. cc @ViktorHofer @ericstj @am11 @omajid
|
cc @elinor-fung |
That was what I was thinking we would do when we changed the fallback RIDs (but didn't finish it unfortunately). This obviously assumes that the |
I can look into making this change, if you want.
Yes, it does. See the first comment, it shows what is generated for Fedora 37. |
That would be greatly appreciated. We're not experts in the source-build problem space, so someone who is would be perfect :-) |
I'm probably missing something basic from the start. I see "no mention of Also, It seems like |
@richlander when the host doesn't recognize the rid in We'll make the fallback work on non-portable by changing the fallback rid to match the non-portable rid. The bottom line is that both portable and non-portable builds will be able to use portable assets independent of what's in @vitek-karas you can assign this to me. |
@agocke and I discussed this yesterday and Andy had a great point to make. There are two changes at play here:
With source-built SDK for fedora, the RID for the SDK is going to be I don't know what is the current behavior of source-built SDK on fedora, but given this issue I would suspect it produces the app either with the portable host (falls back to I also don't know what happens on source-built SDK if the app is build with Andy has a great point that the host is special as it should not target the RID of the source-built target, but instead should try to be as portable as possible so that the app produced by the source-built SDK can run on as many targets as possible. That's obviously assuming that is the intention - I don't know for sure. |
Ah, yes. There is a Under For "linux-x64": [
"linux",
"unix-x64",
"unix",
"any",
"base"
]
Not one that is supposed to be working/supported. |
Thank you. That helps me better understand the issue. |
How do you know what the non-portable RID is? I'm assuming that means the value isn't available in |
The non-portable rid is determined by calling Its value is stored in the .NET installation. |
That helps. Which build of .NET do you use to call that API? I'm seeing a chicken/egg challenge. Separately, I see the fully specified For context, @elinor-fung and I are trying to define all the requirements so that we can propose an updated plan. |
The expectation is that
I agree, there is little you can assume if your environment is |
That helps. On the first question, I was asking something more specific. Do you rely on the last-known-good Fedora build to call that API during source-build or do you use the "just built" runtime to generate those assets? It doesn't really matter. I'm just curious. |
Source-build calls it on whatever sdk you are using for the build: https://github.com/dotnet/dotnet/blob/903545fbfc0c80d94803eeaf73f22ae1a1de290d/Directory.Build.props#L161. The packages on Fedora are built using an sdk that was already built from source. |
Thanks. What I expected. Here's a brief sense of what @elinor-fung and I are talking about. We need some more info still. Some quick context: # cat /usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.2/Microsoft.NETCore.App.deps.json | wc -l
2025
# cat /usr/share/dotnet/shared/Microsoft.NETCore.App/7.0.2/Microsoft.NETCore.App.deps.json | jq .runtimes | wc -l
1289 That's a ridiculous number of lines! Plan:
Using Fedora as the example, the algorithm would probe for these assets:
As we discussed, there are other possible RIDs to probe for. It isn't obvious that any of them are broadly useful. |
The algorithmic model enables the runtime to consume The main use-case for the specific rid ( |
I think you are right. @jkotas was providing similar feedback, yesterday. Now that I think more about it, I have an alternate proposal. The algorithm should probe for:
Let's step back. This whole scenario is for portable apps. RID specific apps (FDD or SCD) don't have this need, at least not for the host. So, should we support these districts specific RIDs? Yes, but in a narrow way.
If we had that, I think we could freeze the RID graph and over time make it optional and then remove it. This is similar to the original proposal, but provides more of a story (while narrow) for these distro specific RIDs. It also better defines what portable apps are for. Fair? |
That's not 100% true. If the app acts as a host for plugins, the plugins may need some kind of RID fallback graph (or algorithmic alternative). For example, even if I build the host app for |
Interesting. Now I'm wondering how does / should it work with fake RIDs, e.g.
Either distro-RID-specific plugins are not in use or distros which switched to |
That sounds pretty rough. I don't think we should support distro-specific plugins. That seems like way too niche a scenario. |
I agree that distro specific plugins are unlikely... but distro specific app with cross-plat plugins (so plugin which has windows/linux/mac specific assets) seems very possible. I mean our plugin sample app does that. And since the plugins are using the same RID resolution code as the app itself, even distro specific apps will need to be able to figure out what |
Yes. I'm suggesting that it is OK to support portable RIDs for that scenario. |
It also occurs to me if we adopt this plan that |
The proposal has the host (distro-specific or not) supporting portable RIDs - via the algorithmic model - in all scenarios, right? (Hence distro-specific app and cross-plat plugins should still work.) |
I just realized that the host itself does not print the RID at all - only the SDK. Although it does rely on the host, since that is what |
We could support distro-specific RIDs with portable apps, too. It's just a question of whether we really have a scenario for it. I'd rather see that be a thing for:
That seems like it would cover the bases. Also, wouldn't it be wonderful if .NET became more popular on Linux and devs started putting distro-specific libraries into package managers instead of NuGet. portable content in NuGet and distro-specific content in the distro archive. That seems like the sweet-spot. Another option is to make portable apps with distro-specific runtime folders opt-in. There could be a property for |
I agree. The non-portable rid is used as 'namespace' for identifying source build artifacts on the distro. That namespace is defined and used by the distro .NET build. I doesn't need to mean anything elsewhere.
The host already supports consuming portable rids without the algorithm model. The algorithmic model (that dynamically makes the |
note: the scalability issue is higher on distros that release frequent with short support cycles, like Fedora, than on long supported distros, like RHEL. |
There is an aspect of the rid graph that wasn't discussed yet: compatibility. |
Microsoft's portable builds perform a fall-back to the portable rid when the
/etc/os-release/
rid is not known.This doesn't work for non-portable builds because the information for the portable rid is missing in
Microsoft.NETCore.App.deps.json
.For example, below is the full
runtimes
section of a Fedora 37 build. It does not include a section forlinux-x64
(which is the fallback host rid).For source-build, there should be an exact match with the rid found here and the
/etc/os-release
rid.That may not be the case:
/etc/os-release
rid is unexpected (like in System.PlatformNotSupportedException: System.IO.Ports is currently only supported on Windows. on Archlinux #79196).fedora.38-x64
is unknown).In these two cases, the Microsoft build will continue to work, while the source-build builds start failing.
We should include this information in the non-portable builds so the fallback works.
cc @ViktorHofer @ericstj @am11 @omajid
The text was updated successfully, but these errors were encountered: