pub(restricted) item

[pnkfelix]

Summary

Expand the current pub/non-pub categorization of items with the ability to say "make this item visible solely to a (named) module tree."

The current crate is one such tree, and would be expressed via: pub(crate) item. Other trees can be denoted via a path employed in a use statement, e.g. pub(a::b) item, or pub(super) item.

(accepted) RFC text

rendered draft

[oli-obk]

I don't find this to be a convincing example. You could just leave out the pub(a) and the example would work right now just as expected. Adding impls inside submodules is imo the correct way to say it's only accessible in there. The example should probably be showing two submodules accessing the same function. But then it doesn't fit here, but to to the unresolved question point.

[pnkfelix]

You're right, this was not a good example. I will try to fix.

[sfackler]

I currently emulate this with traits defined at the level that needs visibility, but it's a lot of annoying boilerplate: https://github.com/sfackler/rust-postgres/blob/master/src/lib.rs#L1437-L1487

[petrochenkov]

Note, that reachability analysis is already performed by the privacy pass. Its results are used by rustdoc, dead code lint, stability and link-time-reachability passes, but not by privacy itself.
It's also less buggy than it was.

[ticki]

I like the idea, but I'm not sure about the syntax, you propose.

[petrochenkov]

I like the general direction, although pub(arbitrary_path) feels a bit to powerful than needed and pub(crate) may be enough. A crate is usually moderately sized and maintained by a team or one person and it's never a problem to modify or reorganize it, or make something public or private.
I also feel that the current system is pretty good already and the need in this extension is not especially dire.

I mostly have concerns about the implementation side, the current privacy system is not well formalized and three its main components - privacy checker itself, private-in-public checker and reachability analysis often work based on different assumptions and contradict to each other. Together with this and the new macro privacy system from @nrc it may bring some chaos.
So, I'd prefer to roll this out in parts:
Make the parts of the current system coherent and document them well -> Macro privacy -> pub(crate) -> pub(path) if necessary

Also, fixing rust-lang/rust#30079 or relaxing the rules to allow

mod m {
    struct S;
    mod n {
        // Allowed, but you can't reexport `f` out of `m` without making `S` public
        pub fn f(arg: super::S) {}
    }
}

will probably involve implementing some rudimentary form of pub(restricted) from this RFC, so I still see it as a good long term direction.

[petrochenkov]

It would be really nice to try making pub(super) (or pub(crate)) a default for pub instead of pub(universe). At least it is in line with the general "private by default" ideology. It would also allow reachability analysis to make more things internal at link time. But it would break the world unfortunately.

[nrc]

We could have pub(::) instead of pub(crate), this would mean only having a single syntax, at the expense of grotesque ugliness.

[pnkfelix]

my eyes!

[oli-obk]

alternatively crate could be allowed at the beginning of paths, thus making this a single syntax

[pnkfelix]

But then people will complain about having more than one way to do things (in this case, writing absolute paths) right?

[withoutboats]

Big 👍 on something in this direction. I've repeatedly felt contorted trying to make items public to the rest of my crate without exposing them outside the crate.

Maybe I haven't worked on large enough libraries, but I am not sure that anything more powerful than pub(crate) or maybe pub(super) is really needed. I wonder if we couldn't just add a keyword for that instead of going for the heavy duty solution. It would be kind of strange to use priv for something less private than default, but it is still a reserved word.

It would be really nice to try making pub(super) (or pub(crate)) a default for pub instead of pub(universe).

👍 on this if Rust ever increments the major version number.

[nrc]

I'm strongly against this alternative - I think it is ill-advised to make crates more significant than they already are - we would end up with two layers of modularisation. I prefer to keep crates as essentially file-system artefacts.

[pnkfelix]

Obviously I agree with @nrc; otherwise this RFC would have been written to just suggest pub(crate) on its own.

Of course we can see there are people on the other side of this debate; I've seen two commenters already state that they think pub(crate) may suffice on its own.

---

@nrc I am curious if you can come up with other use-cases for pub(path) beyond the hypothetical crate-merging refactoring tool that I sketched in the text of this subsection.

• Well, of course there is the obvious "some crates are large enough to warrant privacy controls within the crate, that are more fine-grained than just pub/pub(crate)/private

---

I actually rather likes the proposal from @petrochenkov that we might roll this out in stages, with pub(crate) very early on, and adding pub(path) later after convincing the community at large that it is a good thing too.

Except I might advocate putting in pub(crate) before we tighten up the rules for privacy at large, for the precise reason described in the RFC: giving people the ability to express pub(crate) will encourage them to fix their code to say something closer to what they mean now and stop exploiting bugs in the privacy system to express such structures, which will give us more freedom to tighten up privacy (and tell them to use pub(crate) in the warnings emitted during warning cycle period).

[petrochenkov]

before we tighten up the rules for privacy at large

I don't expect noticeable breakage from privacy in the future, most practically significant holes are closed now. Regarding the existing private_in_public warnings, I don't think pub(crate) will help much, but e.g. relaxing rules on trait bounds or alias substitution certainly will.

[withoutboats]

I think it is ill-advised to make crates more significant than they already are - we would end up with two layers of modularisation.

I agree that this is not ideal, but how is this not already the case? Coherence rules operate at the crate rather than module level, extern crate declarations and cargo dependencies are in themselves not transparent, etc. There is already a pretty big difference in the relation between modules inside my crate and relations with modules outside of it.

[petrochenkov]

With regards to privacy, crates are also important, as units of compilation - some code or metadata can potentially be omitted from the compiled crate if it can be proven that it is not accessible from other crates.

[bstrie]

@nrc I think the battle against crates as a separate unit of modularization has long been lost. The crate vs. mod distinction already matters for circular dependencies, inlining, and incremental recompilation. I don't see any danger to embracing this distinction even further.

@pnkfelix I have indeed seen people asking for "crate-level visibility" for years now (I'd be surprised if there isn't an old RFC for it somewhere), so paring down this proposal to merely pub(crate) may very well be viable. And in that case you can even consider simplifying the syntax to just pub crate or crate pub or something.

[pnkfelix]

@bstrie

paring down this proposal to merely pub(crate) may very well be viable. And in that case you can even consider simplifying the syntax to just pub crate or crate pub or something.

I am pretty strongly opposed to adopting a syntax that isn't future-extensible to pub(path) ... pub in crate <item> (which then supports e.g. pub in path <item>) might work, though visually I think I prefer the parentheses.

[arielb1]

The typeck point-of-view is that associated items are as private as their containing traits and that all types are basically public. I am not sure about the motivation for the no-privates-in-public rule (even reading the comments does not make that clear) - we should clarify what properties we want from privacy.

[arielb1]

From my typeck POV, the NPIP/RFC136 rule feels to me rather lintish - it is not sure exactly what it is trying to prevent. Local reasoning on reachability is always kind of odd - you need to be sure nobody exposes your API in some indirect way - but having some block on direct uses would be fine.

[aturon]

@arielb1 Just to check, have you seen Felix's related blog post? It tries to address at least some of the questions you're raising.

[arielb1]

@aturon

Yes I've read it.

[huonw]

It seems somewhat unfortunate that pub fn can't be written as pub(X) fn for some X (like extern fn == extern "C" fn). Although it looks like pub fn foo should be pretty similar to pub(super) fn foo, except maybe that if the module containing foo is pub then foo can be used outside super?

What happens if the restriction is to a subpath? e.g. is the call to foo legal?

pub(module) fn foo() {}

fn bar() {
    foo() // is foo visible here?
}

mod module {}

[pnkfelix]

@huonw

It seems somewhat unfortunate that pub fn can't be written as pub(X) fn for some X

Hmm, you're right; if I'm going to allow pub (self) item as a synonym for no pub at all, then one would think I should also provide a form that one could feed into the pub(_) that stands for the universe of all crates. (pub(pub) ? Oh man, that is truly terrible, nearly as bad as @nrc's pub(::) ...)

What happens if the restriction is to a subpath? e.g. is the call to foo legal?

See the unresolved question section on this topic.

[petrochenkov]

@arielb1

we should clarify what properties we want from privacy

In addition to user-oriented properties, I'd like to have the next guarantee:
If a type is private or pub(crate)/pub(path), then all its methods (inherent or from traits) can have internal linkage (these methods shouldn't also be used from inlinable code, but it's separate story).
It's mostly true today with exception of some cases where type inference is too smart.

Update: Ideally this should be done based on reachability and not pub annotations, but the rule "type inference can not result in unreachable types" would look kind of strange compared to the rule "type inference can not result in private types". Or we can calculate somehow which exactly private types can be reached through type inference (I have no idea how to do this), add them to the reachable set and abandon the privacy-based reasoning completely.

[diwic]

There is no example with struct impls, but I assume they are covered as well?

I've hit this in my bindings a few times, I wanted to do:

mod ctl_int {
    pub struct ElemInfo(*mut alsa::snd_ctl_elem_info_t);
    impl ElemInfo {
        // Accessible in this crate, but not other crates
        pub($crate) handle(&self) -> *mut alsa::snd_ctl_elem_info_t { self.0 }
        // Other public methods accessible to other crates
    }
}
pub use ctl_int::ElemInfo;

Now I'm instead doing this workaround:

mod ctl_int {
    pub struct ElemInfo(*mut alsa::snd_ctl_elem_info_t);
    pub fn elem_info_ptr(a: &ElemInfo) -> *mut alsa::snd_ctl_elem_info_t { a.0 }
    impl ElemInfo {
        // Other public methods accessible to other crates
    }
}
pub use ctl_int::ElemInfo;

Edit: Changed a to self

[petrochenkov]

@diwic

mod ctl_int {
    pub struct ElemInfo(*mut alsa::snd_ctl_elem_info_t);
    impl ElemInfo {
        // Other public methods accessible to other crates
    }
}
pub use ctl_int::ElemInfo;

impl ElemInfo {
    // Accessible in this crate, but not other crates
    fn handle(&self) -> *mut alsa::snd_ctl_elem_info_t { a.0 };
}

?

[diwic]

@petrochenkov - that doesn't work because self.0 is private. Which is expected. Potentially,

pub struct ElemInfo(pub($crate) *mut alsa::snd_ctl_elem_info_t);

...could work, but it leaks the internal layout of ElemInfo to the entire crate, so an accessor method is better.

[blaenk]

Huge 👍

I'm very glad and grateful you came up with this @pnkfelix . I've run into situations where this would be very useful and much more straightforward than the contortions that are required today, but I had this (what I hope to be mis)conception that changes to item visibility rules/functionality would be very difficult to convince the team about. Glad it's happening from the inside! 😁

[mdinger]

Comment at the top says I and foo are exported. Not sure why bar is also exported.

[pnkfelix]

i added fn bar to the example later (just so "many places" that semisecret would be called did actually mean "more than one place"), but forgot to update the comment above the mod a.

[liigo]

to describe a crate-scoped-item:
protected item?
internal item?
crate item?

protected mod xxx should be sufficient, imho.

[nixpulvis]

I was giving something along the lines of pub(...) for a more general stability annotation. Since we should only care about an item's stability when it's public. So how the syntax here could tie into a syntax for that might be something to consider. For example pub in a::b as stable as an offhand idea.

[nikomatsakis]

How does this interact with glob reexports? Glob reexports currently reexport only pub(universe) items.

Did we ever clarify this in particular? Longer term, I would expect glob re-exports to re-export all things that are visible to the "re-exporter" (with the same privacy). However, our current rules about shadowing make it a problematic breaking change to add these semantics, because use super::* would now import private items. So we could either continue to do imports so long as the pub keyword is used -- which sort of violates my mental model, in which no keyword is short for pub(self) -- or just keep the current model of "only do pub (with no restriction)" and then expand things all at once. I don't care that much except that long term I'd like to go for the more expansive model. But it seems like the best thing short term is to only have glob re-exports affect pub with no restriction.

Some possible solutions to #1422 (comment) are discussed in rust-lang/rust#31783

But I should go make sure I'm caught up on this thread...

[nodakai]

What happens if we want to expose a name to multiple modules? pub(krate1::m11, krate2::m21) ? Yuck.

[petrochenkov]

I'd prefer to start with the strict rules (i.e. the answer "No"). They fit better into the whole picture and can always be relaxed if necessary (I doubt it will be necessary).

[nikomatsakis]

Huzzah! The @rust-lang/lang team has decided to accept this RFC. With respect to unresolved questions -- such as the strictness of certain rules -- I think we tend towards the stricter interpretation, but also intend to answer such questions after gaining some experience with an implementation.

[nikomatsakis]

Tracking issue: rust-lang/rust#32409

[nikomatsakis]

@pnkfelix I think you still had some minor clarifications that you planned to make?

[glaebhoerl]

cc @glaebhoerl -- not sure if you've been following this, but now would be a good time to leave your thoughts.

Welp. This has been sitting in my notifications for so long that I never noticed the cc.

Anyway:

• As far as Ban private items in public APIs #136 is concerned, I never figured out what I really felt was the right way to formulate the rules I wanted (which was a major contributing factor to the fact that we ended up with completely different ones), and the details are even less in my memory now, but I do know that I didn't intend it to be any kind of "global privacy (or publicity) inference" scheme -- that's something I deliberately wanted to avoid. It was always my goal for it to be based solely on simple, local, compositional rules.

  For instance, in @nikomatsakis's "Problem 0", it wouldn't check at the definition site of outer_fn, "are there any pub uses of this in outer modules?", to determine whether or not it is allowed to refer to the privately-declared OuterStruct in its signature. mod inner is exposing pub outer_fn to its parent module (outer), and what to do with it from there (whether or not to expose it further) is entirely outer's concern. The privacy checker sees that the type OuterStruct is public to the parent module, so the definition is accepted.

  If in fact there were a pub use self::inner::outer_fn in outer, then the check for whether this pub use is allowed -- whether the types in the signature of item being re-exported are public to its parent module -- would happen at that point. (I think this might be equivalent to substituting the signature of the pub use's target in place of the pub use, and re-checking it there.) The privacy checker would see that OuterStruct is not public to outer's parent module, and so the pub use would be rejected. (Likewise, if it were pub mod inner instead of private, the relevant test would be, "Is mod outer allowed to declare this module public? Are the types in the signatures of its public items public to outer's parent module?" -- whether or not its parent module (or maybe I should say owner) declares it public is not inner's concern.)

• But that's water under the bridge. I don't have any well-defined feelings about this pub(restricted) RFC, or even about whether the referencing-private-items rules in their new form are worth keeping. The new syntax that has to be learned seems undesirable, but I suppose that's banal and obvious.

[pnkfelix]

@nikomatsakis I'll try to update doc today

[mitchmindtree]

An alternative idea for the syntax:

pub::foo item (rather than pub(foo) item).

This syntax might be more intuitively associable with paths than the proposed syntax?

[Xion]

I know I'm late to the party, but I just read this RFC and noticed the section where the author isn't sure about precedents to the suggested solution.

The visibility system in Bazel (the build system) is basically identical to the this pub scoping, with some additional features that are mostly listed in the "Why not more ambitious?" section. For more information, see here.

[iqualfragile]

nomenclature idea: lets make "pub" be short for publish, if you don't specify where you publish it to everywhere, when you restrict it then you just publish your function/struct/member to them.