Encapsulation of project with private keyword

[Akirathan]

Motivation - encapsulation

Currently, all the entities (modules, types, methods, constructors, fields) from a project/library can be imported and used, even the entities that are meant to be internal to the library. For example, Standard.Table.Internal.Vector_Builder.Vector_Builder can be easily imported and used with:

import Standard.Table.Internal.Vector_Builder.Vector_Builder
main =
    Vector_Builder.from_vector [1,2,3] ++ [4,5]

One has to explicitly import this internal type with the aforementioned import statement, it cannot be imported directly from Standard.Table library by from Standard.Table import Vector_Builder, as it is not exported from the library's Main module. But it can be imported and used nonetheless. We could also skip the import statement, and use it directly via FQN.

We want to be strict about this and forbid users any kind of usage of internal entities.

Requirements

• Be able to hide an entity on demand. By hiding, we mean that the entity cannot be directly imported, and that it cannot be used via FQN.
  • i.e. an entity shall be hidden both during compile time (project compilation), and during runtime.
  • Entity being hidden at runtime implies that it does not have any entry in the Suggestion database, therefore, no entry in the Component browser.
• Be able to import all public symbols from a library with from Library import all.
• Be able to import a selected set of symbols from a library with from Library import Symbol_1, Symbol_2, ....
• Import a public symbol directly with import Library.Public_Module.Public_Type.
• Use a public symbol via FQN: Library.Public_Module.Public_Type.

Proposals that were already dismissed

There has been a long internal discussion in the Engine and Libs team. So far, two proposals and one prototype have been made:

• Jaroslav's proposal to differentiate between logical and physical layout
  • An associated, now declined, Hide non-exported stuff from other libraries #6905.
• Pavel's proposal to Make everything project-private and the associated presentation with examples.

The Pavel's proposal was dismissed because it clashes with Wojciech's vision of everything being public by default and private on demand, and also because it is unnecessarily difficult to expose an entity.

Jaroslav's proposal was dismissed by Wojciech after they had a chat about it.

In this discussion, let's only discuss the syntax and semantics of the private keyword that shall be added to the language, as suggested by @wdanilo. To better understand the dismissed proposals, see the links.

Introduction of private keyword

Let's introduce a private keyword. By prepending (syntax rules discussed below) private keyword` to an entity, we declare it as project private. A project-private entity is an entity that can be imported and used in the same project, but cannot be imported nor used in different projects. Note that it is not desirable to declare the entities as module private, as that would be too restrictive, and would prevent library authors using the entity within the project.

From now on, let's consider project-private and private synonymous, and public as an entity that is not private.

Syntax

All the entities, except modules, shall be declared private by prepending them with private keyword. Declaring a module as private shall be done be writing the private keyword at the very beginning of the module, before all the import statements, ignoring all the comments before. Fields cannot have private keyword, only constructors.

Semantics

Submodules

A hierarchy of submodules cannot mix public and private modules. In other words, if a module is public, its whole subtree must be public as well. For example, having a public module A and private submodule A.B is forbidden and shall be reported as an error during compilation.

Example

For an example, let's consider a few sources in a simple library Lib and its usage from a temporary source tmp.enso. Compiler and runtime failures are described in comments in the code snippets.

Lib/src/Pub_Type.enso:

type Pub_Type
  Constructor field
  private priv_method self = ...
  pub_method self = self.field.to_text

private type Priv_Type

Lib/src/Methods.enso:

pub_stat_method x y = x + y
private priv_stat_method x y = x - y

Lib/src/Internal/Helpers.enso:

# Mark the whole module as private
private

# OK to import private types in the same project
import project.Pub_Type.Priv_Type

Lib/src/Main.enso:

import project.Pub_Type.Pub_Type
export project.Pub_Type.Pub_Type

import project.Pub_Type.Priv_Type # OK - we can import private types in the same project.
export project.Pub_Type.Priv_Type # Failes at compile time - re-exporting private types is forbidden.

tmp.enso:

from Lib import Pub_Type
import Lib.Pub_Type.Priv_Type # Fails during compilation
import Lib.Methods

main =
  # This constructor is not private, we can use it here.
  obj = Pub_Type.Constructor field=42
  obj.field # OK - Constructor is public, therefore, field is public
  obj.priv_method # Runtime failure - priv_method is private
  Pub_Type.priv_method self=obj # Runtime failure
  obj.pub_method # OK

  Lib.Pub_Type.Priv_Type # Fails at runtime - accessing private types via FQN is forbidden

  Methods.pub_stat_method 1 2 # OK
  Methods.priv_stat_method # Fails at runtime

Checks

There shall be two checks. One check during compilation, that can be implemented as a separate compiler pass, and that will ensure that no private entity is re-exported (exported from a module that is different from the module inside which the entity is defined) and that for every type it holds that either all the constructors are public or all the constructors are private

The second check shall be done during the method/name resolution step. This step happens at runtime, before a method is called. After the method is resolved, there shall be no further checks, so that the peak performance is not affected.

Performance impact

The performance hit on compilation time is minimal, as there are already dozens of different compiler passes. Moreover, in the new compiler pass we shall check only imports and exports statements, no other IR.

The performance hit on runtime, during method resolution, is minimal as well, because it can be as easy as additional lookup in a hash map. Peak performance will not be affected at all, as there are no further checks after method resolution.

Overcoming encapsulation

Sometimes it is useful to be able to access internal entities. Testing is the most obvious example. Let's introduce a new CLI flag to the Engine launcher called --disable-private-check, which will disable all the private checks during compilation and method resolution.

Other notes

• A private module implies that all the entities defined within are private
• A private type implies that all the constructors, methods and fields defined within are private
• A private constructor implies private fields defined in that constructor.

Edit (2023-07-17)

Based on @GregoryTravis's observation in https://github.com/orgs/enso-org/discussions/7088#discussioncomment-6233948: All the types shall be either open (all constructors and all fields are public) or closed (all constructors and all fields are private), but nothing in between.

The related changes to the specification:

• Do not allow the private keyword for fields.
• Extend the compile-time check to ensure that are constructors are either public or all constructors are private for a type.

cc @jdunkerley @radeusgd @GregoryTravis @JaroslavTulach @hubertp @4e6

[GregoryTravis]

What if we allowed private on constructors, but not on individual constructor fields? This would be a restriction, but I think it would provide more benefit.

Constructors/fields are made private when we don't want the application programmer access, because that would put the burden on them to maintain the invariants of the time. There are usually two cases:

• (Open) Users are allowed to create and pattern-match freely on the types; the only restriction is that the constructor arguments are the right type
• (Closed) The type represents an opaque value that must be created by the module, and the module handles the maintenance of internal invariants

Allowing individual fields to be private is half-way between these two.

Open types are simple to create and simple to use, and we use them unless we have a reason not to.
Closed types have more internal requirements, and we create smart constructors (new) and accessors, because we need this to maintain internal invariances.

Open types have a clear interface: you can put anything here, and you can read it afterwards.
Closed types have a clear interface: the methods define what you can do.

A partially-closed type is more confusing for the user; they have to understand the internals of the type to understand why certain fields are private. In a fully closed type, smart constructors and others methods define the interface.
Once we decide a type isn't open, perhaps we might as well do the extra work to make it fully closed. It's not that much more work, but it may well be easier for the reader to understand.

I think the syntax is a little awkward. I understand why it makes sense to not require parentheses for a keyword, but it still reads initially like there is a field called private:

Constructor pub_field private priv_field

The elegance of the constructor format, borrowed from languages like Haskell, works in part because modifiers (such as visibility) are elsewhere in the file, which leaves the constructor in a form that's easy to read.

[radeusgd]

I'd just prefer the transparent and opaque terms or something else.

open and closed are already loaded with meaning in the type theoretic world and I feel like re-using that with different semantics can confuse people (for example some languages have the open/sealed dichotomy of whether a class can be inherited by classes from outside of some compilation units).

[jdunkerley]

I'm not sure about the opaque / transparent names but appreciate naming us hard.

To advocate for private in front of constructors - it feels fairly rare we would have multiple constructors in these cases and I feel that is most natural syntax in the single cases.

And to clarify opaque/private would still be ignored within same library?

[radeusgd]

To advocate for private in front of constructors - it feels fairly rare we would have multiple constructors in these cases and I feel that is most natural syntax in the single cases.

I think you are right. IMO we can just enforce that either all constructors are private or all are public, but keep this syntax, it will likely be simplest.

And to clarify opaque/private would still be ignored within same library?

That is my understanding, yes.

[JaroslavTulach]

enforce that either all constructors are private or all are public

+1

A private constructor does not imply private fields (defined in that constructor).

When a type has private constructors it shouldn't be possible to decompose its instances anyhow from other modules. E.g. no pattern matching on constructors and no accessors to its fields.

Btw. adding new public constructor to an existing type while evolving a library is an incompatible change. Originally written case of on instances of such type will no longer match all branches and yield execution errors at places where they used to cover all constructors branches. As such I'd be very careful to expose such decomposable types in APIs.

[Akirathan]

I have just updated the description to match the conclusion from this thread.

[Akirathan]

This whole discussion was added to the repo documentation in a30993a

[Akirathan]

I will try to reflect all the future changes in this discussion in the aforementioned documentation.

[Akirathan]

I have just created these tasks that track the progress of implementation:

• Implement private methods #8834
• Implement private types #8835
• Implement private constructors & deconstructors/getters for types #8836

The implementation order of these issues is to be specified.