stabilize Strict Provenance and Exposed Provenance APIs

[RalfJung]

Given that RFC 3559 has been accepted, t-lang has approved the concept of provenance to exist in the language. So I think it's time that we stabilize the strict provenance and exposed provenance APIs, and discuss provenance explicitly in the docs:

// core::ptr
pub const fn without_provenance<T>(addr: usize) -> *const T;
pub const fn dangling<T>() -> *const T;
pub const fn without_provenance_mut<T>(addr: usize) -> *mut T;
pub const fn dangling_mut<T>() -> *mut T;
pub fn with_exposed_provenance<T>(addr: usize) -> *const T;
pub fn with_exposed_provenance_mut<T>(addr: usize) -> *mut T;

impl<T: ?Sized> *const T {
    pub fn addr(self) -> usize;
    pub fn expose_provenance(self) -> usize;
    pub fn with_addr(self, addr: usize) -> Self;
    pub fn map_addr(self, f: impl FnOnce(usize) -> usize) -> Self;
}

impl<T: ?Sized> *mut T {
    pub fn addr(self) -> usize;
    pub fn expose_provenance(self) -> usize;
    pub fn with_addr(self, addr: usize) -> Self;
    pub fn map_addr(self, f: impl FnOnce(usize) -> usize) -> Self;
}

impl<T: ?Sized> NonNull<T> {
    pub fn addr(self) -> NonZero<usize>;
    pub fn with_addr(self, addr: NonZero<usize>) -> Self;
    pub fn map_addr(self, f: impl FnOnce(NonZero<usize>) -> NonZero<usize>) -> Self;
}

I also did a pass over the docs to adjust them, because this is no longer an "experiment". The ptr docs now discuss the concept of provenance in general, and then they go into the two families of APIs for dealing with provenance: Strict Provenance and Exposed Provenance. I removed the discussion of how pointers also have an associated "address space" -- that is not actually tracked in the pointer value, it is tracked in the type, so IMO it just distracts from the core point of provenance. I also adjusted the docs for with_exposed_provenance to make it clear that we cannot guarantee much about this function, it's all best-effort.

There are two unstable lints associated with the strict_provenance feature gate; I moved them to a new strict_provenance_lints feature since I didn't want this PR to have an even bigger FCP. ;)

@rust-lang/opsem Would be great to get some feedback on the docs here. :)
Nominating for @rust-lang/libs-api.

Part of #95228.

FCP comment

[rustbot]

r? @Mark-Simulacrum

rustbot has assigned @Mark-Simulacrum.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rustbot]

The Miri subtree was changed

cc @rust-lang/miri

Portable SIMD is developed in its own repository. If possible, consider making this change to rust-lang/portable-simd instead.

cc @calebzulawski, @programmerjake

[5225225]

One thing I'm not entirely sure about is: if we ever plan to support actual targets (as opposed to just running under miri) that cannot implement expose_provenance, what does it do on those targets? Is a library that depends on expose_provenance conditionally sound, depending on the target?

Personally I would prefer the functions to simply not exist if they can't act according to the docs, akin to target specific atomic types. That way, if a user of a library tries to run the library on a target that doesn't support expose, they get told upfront that the library can't be used.

Not that I would expect that to come up much, since as far as I know every current target can implement expose, and anyone running code on CHERI or similar would need to audit their libraries anyways (since as is also a problem, and you can't really make that a compile error under CHERI). So the functions existing nearly everywhere would be fine.

[RalfJung]

One thing I'm not entirely sure about is: if we ever plan to support actual targets (as opposed to just running under miri) that cannot implement expose_provenance, what does it do on those targets?

I'm not sure. But no such target is supported right now, and as casts have the exact same issue, so it seems largely orthogonal to the stabilization discussion to me.

Personally I would prefer the functions to simply not exist if they can't act according to the docs, akin to target specific atomic types. That way, if a user of a library tries to run the library on a target that doesn't support expose, they get told upfront that the library can't be used.

I think I'd prefer that, too (and doing the same with as casts as well under CHERI). But that's off-topic in this PR, I would argue.

[5225225]

Makes sense, as long as this isn't stabilizing the fact that these functions will always exist on all (even future) targets, it's fine as-is, I was unsure if that's what it meant. My bad!

[dtolnay]

@rust-lang/libs-api:
@rfcbot fcp merge

We have discussed this API extensively over the past couple years, notably the numerous rounds of iteration on #117658, and #122935.

[VorpalBlade]

Also asked on zulip, but posting here as well because I find zulip confusing (and so it doesn't get lost):

Maybe this is a stupid question, but when I look at the actual implementations of the strict provenance APIs, they have a lot of todo comments noting that they should be intrinsics instead. So currently the API doesn't really do anything?

Should it really be stabilised then? What if when actually implementing the "magic" in the compiler you find out that it doesn't work with the API as stabilised?

In particular, if the underlying LLVM semantics aren't fully figured out, should Rust really stabilise the high level API before that?

[jrtc27]

Just for some context here: @jrtc27 is an actual CHERI contributor. We're not like, hypothesizing if CHERI would be cool with these APIs existing, those folks have been roped in on many of these discussions along the way. If they're happy with what we're doing, then any concerns raised on their behalf are dubious at best.

How do they propose to implement exposed provenances on CHERI then?

It won’t be implemented.

[jrtc27]

This is quite wrong. Capabilities are 128-bit values, with the bounds next to the address.

I think the model you describe is equivalent to mine for the purposes of this discussion.

For the purposes of this discussion, maybe. But your model has gaping security issues and huge overheads, so calling it CHERI is disingenuous, confusing and risks people thinking that those are real issues with CHERI. It is generally best to present a faithful approximation as a model rather than something quite different that happens to have similar properties within the context of this discussion.

[kytans]

Ralf's explanation is the correct one. CHERI is a breaking change to the way people write C/Rust/whatever code (even if it's compatible with the vast majority of code).

This may be true for C which has been designed decades ago, but I think that Rust should reasonably strive for something like CHERI (and in general any predictable future execution environment change) to NOT be a breaking change, and in fact be something that is seamless to adopt, which means not encouraging crates to depend on APIs that are planned to not be supported there.

One possibility is to design with API with CHERI-like systems in mind; another is to discourage its direct usage and instead have crates rely on a wrapper crate that uses the API when available and does something like the global interval tree map construction on CHERI-like architectures.

I think it might be better to directly support the API everywhere since it avoids the burden of having people learn about the third-party crate and avoids the risk of them using the API directly when they shouldn't.

[RalfJung]

For some embedded hardware it is literally impossible to program it in a CHERI-compatible way. So I'm afraid your utopia is not going to happen. Instead we're doing what we generally do in Rust: we give people a nice tool that covers >90% of the usecases (and fully works on CHERI), and then for the few cases where that's not enough we give people the tools they require to Get The Job Done, even if that means handing them a loaded gun. The Strict Provenance API has been designed with CHERI in mind, and this PR will improve the Rust ecosystem support for CHERI by pushing people towards using that API.

This is the wrong thread to suggest that Rust should be 100% CHERI compatible. This PR does not add any fundamentally new CHERI-incompatible operation to Rust, it just gives a new name to an existing such operation (namely the as casts between pointers and integers). If you would like to suggest that Rust should force all people to write CHERI-compatible code, even against the advice of the CHERI people themselves, please write an RFC where you spell out why you think it is a good idea to force embedded people to write C code (which is what they are going to do if Rust doesn't let them do what they need to do).

But meanwhile, please stop derailing this PR, or I'll have to lock it to contributors.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[bors]

☔ The latest upstream changes (presumably #131635) made this pull request unmergeable. Please resolve the merge conflicts.

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

This will be merged soon.

[RalfJung]

@traviscross @dtolnay @scottmcm @CAD97 you all left approving reviews, would one of you like to tell bors about the good news? ;)

[bjorn3]

Too late, @rust-log-analyzer gave some bad news ;)

[dtolnay]

@bors r+

[bors]

📌 Commit 56ee492 has been approved by dtolnay

It is now in the queue for this repository.