Pure/isolated function concept

[jclark]

I think there is a concept of what might be called a pure function that needs to be worked out. The basic idea is that pure functions have limited access to global state. They can access mutable state via parameters but their access to module-level variables is restricted. A Listener would be able to safely use multiple threads on resource methods that are pure.

This relates to a number of different things that we have discussed:

• service concurrency Service concurrency #116
• when is right to treat a function value as immutable, and thus able to be a member of a frozen container? relates to issue Restriction on annotation values to anydata #70
• compile-time functions
• functions used within queries
• lock statement Stabilize lock statement #357
• transactions Stabilize transactions support #267
• sessions

There is a close relationship to immutability: I suspect that just as our solution to immutability is not purely static (because that is excessively burdensome on the programmer), so our approach to functions will also need to be at least partly dynamic.

[jclark]

D has something like this

https://dlang.org/spec/function.html#pure-functions

[jclark]

Need to think about how this relates to function as a service: the key thing is to be able to separate out the state from the logic so that the runtime can manage the state.

[jclark]

There's now a proposal.

[rdhananjaya]

I think we need to extend the isolated analysis into type definitions since object/record construction could execute user-defined behavior as default values. Maybe mandate those functions to be isolated.

int x = 10;

type Foo record {
    int i = foo(); 
}

function foo() returns int {
     x= 20;
     return 33;
}

isolate function bar () {
    Foo f = {}; // this might cause a side effect. // A problem case.
}

isolate function baz (Foo f) {
    f.i = 10; // OK
}

Putting down a few thoughts I had while reading the proposal, these are influenced by experiences I had working with the taint analyzer.

Say when the Ballerina program is first developed foo() was not side-effecting (inferred to be isolated), but after some time someone changes something and now the foo() is inferred to be a side-effecting function and the compiler now surprise the developer with an error saying something like invalid record literal, a default value may cause the state to change in an isolated context.

I think it's really difficult to analyze the side effects caused by functions passed in as values:

Q1. When a function pointer is passed in as an argument to the function and execution of that function pinter could mutate something somewhere, are those mutable bits part of the mutable local storage graph of the isolated function?

And since the arguments are runtimes values, except for simple cases it's difficult to detect which function is being passed in as an argument. (I think same is true when the function value is passed via a closure or module variable)

Another thing to note is that majority of the improvements/bug fixes to this isolated function analyzer will break user code,

[jclark]

The design of isolated feature is different in fundamental ways from the taint feature.

On your first point, my inclination to mandate that expressions specifying defaults for record fields be isolated.

On your second point, the key difference is that isolated is part of the type of the function. If an isolated function f gets passed a function g as a parameter, f cannot call g unless g is isolated.

If the user declares a function as isolated, then it's an error unless the compiler can verify that it is indeed isolated. Improvements to the compiler will thus cause things that were errors to cease to be errors.

[rdhananjaya]

I was under the impression that inferred isolatedness of callee functions is used when analyzing a function that is explicitly marked isolated.

Thank you for the clarification.

[jclark]

The proposal is explicit that this is not the case:

if a function is explicitly declared as isolated, then all functions it calls must also be explicitly declared as isolated

[jclark]

@MaryamZi When you implement this, can you please have a look whether it will cause problems in the current code base to require closures specifying default values to be isolated?

[jclark]

Need to think about how isolated works for arrow functions, since there is no syntax for explicitly declaring them as isolated.

[jclark]

Right, new C is implicitly calling C's init function. If there isn't an explicit init function defined, then the compiler synthesizes one that's isolated, has no arguments and returns ().

[MaryamZi]

@MaryamZi When you implement this, can you please have a look whether it will cause problems in the current code base to require closures specifying default values to be isolated?

When mandating default value closures to be isolated we came across an issue in the standard library, where configuration values read via functions from the ballerina/config module are used as default values for certain fields. These functions are not isolated, and thus with the new restriction they cannot be used as default values, and will require some changes to maintain the same behaviour.

public type ConnectionPool record {|
    // `15` is the default if not specified as a config
    int maxOpenConnections = config:getAsInt("b7a.sql.pool.maxOpenConnections", 15);
    int minIdleConnections = config:getAsInt("b7a.sql.pool.minIdleConnections", 15);
|};

Instead of requiring record default value closures to be isolated, can we disallow in an isolated function the construction of values which call non-isolated closures in the process?

For example, where foo is not isolated

type Bar record {
   int i = foo();
};

function foo() returns int {}

can we allow the above definition and make it an error

1. if a Bar value is constructed in an isolated function

isolated function baz() {
   Bar b1 = {}; // error, since a non-isolated closure will be called

   Bar b2 = {i: 1}; // error, since a default value closure is non-isolated 
}

or

2. if a Bar value is constructed in an isolated function resulting in the non-isolated closure being called

isolated function baz() {
   Bar b1 = {}; // error, since a non-isolated closure will be called

   Bar b2 = {i: 1}; // OK, since the non-isolated closure will not be called
}

I guess we will have similar concerns with objects too.

[jclark]

The config:getXyz functions definitely have to be isolated. I don't know what the implementation looks like, but it might need the ability for isolated functions to access isolated module level variables within a lock. A simple isolated implementation would be for the start-up code to load the config into a readonly final variable, then the config:getXyz function would just access this. So conceptually I see no need for them not to be isolated.

I don't think your suggestion is a good approach, because modules need to have an explicit interface, and there's no way for records to declare which fields' defaults have an isolated closure.

The specific case of config would also be dealt with by #41.

[MaryamZi]

The config:getXyz functions definitely have to be isolated. I don't know what the implementation looks like, but it might need the ability for isolated functions to access isolated module level variables within a lock. A simple isolated implementation would be for the start-up code to load the config into a readonly final variable, then the config:getXyz function would just access this. So conceptually I see no need for them not to be isolated.

Yes, we had a discussion regarding this and decided that the config functions can be marked as isolated, where the config registry can be considered as an isolated module level variable.

I don't think your suggestion is a good approach, because modules need to have an explicit interface, and there's no way for records to declare which fields' defaults have an isolated closure.

Yeah, agreed.

The specific case of config would also be dealt with by #41.

[jclark]

I have moved isolated module-level variables and objects to a separate issue, so we can keep this just for the basic isolated functions concept, which has been implemented already.

[jclark]

Spec changes needed:

• Fix grammar to allow isolated
• Explain what isolated means for a function type (in Types section)
• Explain requirements for function bodies when type is isolated (probably in Function Definition section)
• Explain that for methods, self is treated like parameter for purposes of isolation
• Explain requirements for external functions
• Restriction on records default closures to be isolated
• How new, init and object initialization work with isolated
• How isolated is inferred within a module
• Inferring isolated for anonymous functions without explicit type (e.g. x => x + 1).
• Declare langlib functions as isolated (issue Declared langlib functions as isolated #596)
• Semantics of @isolatedParam
• Concept of a closure being isolated wrt an invocation (only need 1 level up, I think) and inferring this (move to Closures need to be isolated wrt a stack frame #602)
• Deal with defaults for functions arguments
• Add isolated to Swan Lake changes section

[jclark]

We can deal with function argument defaults straightforwardly: when a function is declared as isolated, the requirements for the function body apply also to the expressions for the defaults. This is consistent with the handling of initialisers for object fields and init.

[jclark]

I pushed the isolatedness wrt a stack frame problem (mentioned in #145 (comment)) to #602.