handle_default() implementation without generics/inference

[JMLX42]

Hi,

I'm writing a Solidity backend to create smart contracts from the Catala code (cf #143).

One of the issues I'm facing is the implementation of handle_default() in the Solidity runtime.

The OCaml implementation relies on type inference. The Python implementation relies on generics.

Thus, target languages that have neither type inference nor generics cannot implement handle_default().

Is there any way to workaround this?

• Try to use address(this0.call(...) to emulate try/catch using if.
• Workaround the fact there is no way to define local functions.
• Implement an inline version of handle_default().

[JMLX42]

@denismerigoux just suggested on Twitter via DM to inline the implementation of handle_default(). I'll try to see how that can work.

[denismerigoux]

You can inline it either directly at the level of your specific backend or it could be inlined on demand (with a CLI flag) during the translation from default calculus to lambda calculus.

[JMLX42]

You can inline it either directly at the level of your specific backend

I'm not sure how I would handle all the possible (parameter) permutations at the top level. Do you suggest to generate a function overload for each possible type?

[denismerigoux]

If you inline it directly in your backend generation, you can turn handle_default [e1; e2; e3] e_just e_cons into:

let exception_acc = None in 
let current_exception = match Some (try e1) with EmptyError -> None 
let exception_acc = match exception_acc, current_exception with 
  | None, x -> x 
  | Some x, None -> Some x 
  | Some _, Some _ -> panic!
in
(* same thing with e2 and e3 *) ...
match exception_acc with 
| Some x -> x
| None -> if e just then e_cons else raise EmptyError 

[JMLX42]

let current_exception = match Some (try e1) with EmptyError -> None 

The problem is then to have the "option" idiom ( Some / None).

I tried to do something like this:

struct T { bool __fixme; }

error ConflictError();

error EmptyError();

function handle_default(
        function() returns (T memory)[] calldata exceptions,
        function() returns (T memory) just,
        function() returns (T memory) cons
    ) pure returns (T memory)
{
    T memory acc;
    bool acc_is_none = true;
    
    for (uint i = 0; i < exceptions.length; i++) {
        T memory new_val;
        bool new_val_is_none = true;
        
        try exceptions[i]() returns (T memory val) {
            new_val = val;
            new_val_is_none = false;
        } catch {
            new_val_is_none = true;
        }
        if (acc_is_none) {
            acc = new_val;
        } else if (!acc_is_none && new_val_is_none) {
            // pass
        } else if (!acc_is_none && !new_val_is_none) {
            revert ConflictError();
        }
    }
    if (acc_is_none) {
        if (just()) {
            return cons();
        } else {
            revert EmptyError();
        }
    } else {
        return acc;
    }
}

But I got two errors:

TypeError: Try can only be used with external function calls and contract creation calls.
  --> contracts/catala.sol:26:13:
   |
26 |         try exceptions[i]() returns (T memory val) {
   |             ^^^^^^^^^^^^^^^

That's a very big limitation on try: https://docs.soliditylang.org/en/develop/control-structures.html#try-catch

The try keyword has to be followed by an expression representing an external function call or a contract creation (new ContractName()).

https://docs.soliditylang.org/en/develop/control-structures.html#external-function-calls

Functions can also be called using the this.g(8); and c.g(2); notation, where c is a contract instance and g is a function belonging to c. Calling the function g via either way results in it being called “externally”, using a message call and not directly via jumps.

TypeError: Type struct T memory is not implicitly convertible to expected type bool.
  --> contracts/catala.sol:41:13:
   |
41 |         if (just()) {
   |             ^^^^^^

The Some idiom is missing here, since just is typed as function() returns (T memory) and T is not Option or something like that.

[JMLX42]

The Some idiom is missing here, since just is typed as function() returns (T memory) and T is not Option or something like that.

@denismerigoux so my conclusion is that even if we do inline handle_default(), there is some target language requirement regarding an Option type/idiom. And AFAIK it cannot be implemented without type erasure or generics/templates.

So we're back to square one.

[denismerigoux]

What you can't do Option in Solidity ? You don't even have some kind of null or undefined value you can test for ?

[denismerigoux]

Why do your functions return memory everywhere ? Catala programs are completely pure, you shouldn't need to pass memory around. I'm afraid I don't know enough about Solidity yet to help you on that.

[JMLX42]

Why do your functions return memory everywhere ?

@denismerigoux I know, but the Solidity compiler was complaining about calldata. So I did not try to make this any better.

@denismerigoux what are the types held by an option/the template type here?

Here is an example of the generated Solidity code:

Any local_var_7;
        
        function local_var_7(Any _) {
            return individual_2(Unit());
        }
        function local_var_9(Any _) {
            return true;
        }
        Any local_var_9;
        
        function local_var_11(Any _) {
            function local_var_13(Any _) {
                return false;
            }
            Any local_var_13;
            
            function local_var_15(Any _) {
                revert EmptyError;
            }
            Any local_var_15;
            
            return handle_default([], local_var_13, local_var_15);
        }

If we forget about the Any _ param which is never used, it looks like handle_default() is templated by function () returns (Any). What are the possible values for Any here? Is it limited to:

• TUnit
• TMoney
• TInt
• TRat
• TDate
• TDuration
• TBool

or can it be other types as well (such as a function, or a TTuple, ...)?

[denismerigoux]

Yeah handle_default can return anything, including a tuple, a structure, an enum or a function. It is truly polymorphic.

[JMLX42]

@denismerigoux what do you think about trying to implement option with a (is_some: bool, value: T) tuple?

function handle_default(
        function() returns (T memory)[] calldata exceptions,
        function() returns (bool, T memory) just,
        function() returns (T memory) cons
    ) pure returns (T memory)
{
    // ...

    if (acc_is_none) {
        bool is_some;
        T memory val;
        (is_some, val) = just();
        if (!is_some) {
            return cons();
        } else {
            revert EmptyError();
        }
    } else {
        return acc;
    }
}

The Solidity compiler is fine with that part.

Now if that works I have to deal with the remaining "external function call" problem (cf #153 (comment))...

[denismerigoux]

Yeah it sounds it could work. But you have to carefully review your backend translation, and these sorts of workarounds could introduce subtle bugs. The Solidity typechecking seems very weird from what you're experiencing.

[JMLX42]

The Solidity typechecking seems very weird from what you're experiencing.

@denismerigoux are you referring to the memory vs calldata thing? It's only because of that same limitation on try: since it expects an external call, it enforces memory instead of calldata. And that constraints goes up to the handle_default() params.

calldata everywhere if fine otherwise I think.

[JMLX42]

Potentially related issues:

• Typed errors / exceptions ethereum/solidity#7877
• templates / generics ethereum/solidity#869
• Enums with data / sum datatype / algebraic datatypes ethereum/solidity#909

We might be able to implement exceptions in a more suitable way using the Yul assembly language:

ethereum/solidity#1505 (comment)

[JMLX42]

Possible workaround using tuple return values:

https://ethereum.stackexchange.com/a/78563

pragma solidity ^0.5.0;

import "./Token.sol";

/**
 * @dev This contract showcases a simple Try-catch call in Solidity
 */
contract Example {
    Token public token;
    uint256 public lastAmount;

    constructor(Token _token) public {
        token = _token;
    }

    event TransferFromFailed(uint256 _amount);

    function tryTransferFrom(address _from, address _to, uint256 _amount) public returns(bool returnedBool, uint256 returnedAmount) {
        lastAmount = _amount; // We can query this after transferFrom reverts to confirm that the whole transaction did NOT revert
        // and the changes we made to the state are still present.

        (bool success, bytes memory returnData) =
            address(token).call( // This creates a low level call to the token
                abi.encodePacked( // This encodes the function to call and the parameters to pass to that function
                    token.transferFrom.selector, // This is the function identifier of the function we want to call
                    abi.encode(_from, _to, _amount) // This encodes the parameter we want to pass to the function
                )
            );
        if (success) { // transferFrom completed successfully (did not revert)
            (returnedBool, returnedAmount) = abi.decode(returnData, (bool, uint256));
        } else { // transferFrom reverted. However, the complete tx did not revert and we can handle the case here.
            // I will emit an event here to show this
            emit TransferFromFailed(_amount);
        }
    }
}

[JMLX42]

https://forum.openzeppelin.com/t/a-brief-analysis-of-the-new-try-catch-functionality-in-solidity-0-6/2564

Before Solidity 0.6, the only way of simulating a try/catch was to use low level calls such as call, delegatecall and staticcall. Here’s a simple example on how you’d implement some sort of try/catch in a function that internally calls another function of the same contract:

pragma solidity <0.6.0;

contract OldTryCatch {

    function execute(uint256 amount) external {

        // the low level call will return `false` if its execution reverts
        (bool success, bytes memory returnData) = address(this).call(
            abi.encodeWithSignature(
                "onlyEven(uint256)",
                amount
            )
        );

        if (success) {
            // handle success            
        } else {
            // handle exception
        }
    }

    function onlyEven(uint256 a) public {
        // Code that can revert
        require(a % 2 == 0, "Ups! Reverting");
        // ...
    }
}

First of all, as mentioned, the new feature is exclusively available for external calls. If we want to use the try / catch pattern with internal calls within a contract (as in the first example), we can still use the method described previously using low-level calls or we can make use of this global variable to call an internal function as if it was an external call.

[JMLX42]

First of all, as mentioned, the new feature is exclusively available for external calls. If we want to use the try / catch pattern with internal calls within a contract (as in the first example), we can still use the method described previously using low-level calls or we can make use of this global variable to call an internal function as if it was an external call.

We can emulate a more classic try/catch like this:

bool success;
do // try
{
  (success, bytes memory returnData) = address(this).call(
    abi.encodeWithSignature(
      "onlyEven(uint256)",
      amount
    )
  );
  if (!success) {
    break;
  }
}
while (false)
if (!success) // catch
{
  // ...
}

[denismerigoux]

This looks absolutely horrible, at least I wouldn't trust this in a smart contract :) There is an intern that started this month and who's working on replacing exceptions in the generated code with more classic if/then/else, perhaps you could benefit from that in Solidity.

[JMLX42]

This looks absolutely horrible, at least I wouldn't trust this in a smart contract :)

Yep. More of a "proof of concept" than a production solution for sure.

There is an intern that started this month and who's working on replacing exceptions in the generated code with more classic if/then/else, perhaps you could benefit from that in Solidity.

@denismerigoux sounds perfect! Removing the need for exceptions is a big step forward in lowering the target language feature-set requirements. Any chance we can have an issue/PR to follow that progress? Maybe even test it?

[denismerigoux]

You should lookout for a PR from @lIlIlIlIIIIlIIIllIIlIllIIllIII in the coming weeks. I'll tag it here once it appears.

[denismerigoux]

The PR is #158, it's about to get merged and the related issue is #83

[JMLX42]

The PR is #158, it's about to get merged and the related issue is #83

@denismerigoux If #158 implements the necessary changes, can we close this issue?

[JMLX42]

There is an intern that started this month and who's working on replacing exceptions in the generated code with more classic if/then/else, perhaps you could benefit from that in Solidity.

@denismerigoux correct me if I'm wrong but it looks like Catala still heavily relies on exceptions despite #158 .

Here is an example taken from allocations_familiales.py:

def enfant_le_plus_age(enfant_le_plus_age_in:EnfantLePlusAgeIn):
    enfants = enfant_le_plus_age_in.enfants_in
    try:
        def temp_le_plus_age(_:Unit):
            def temp_le_plus_age_1(potentiel_plus_age_1:Enfant, potentiel_plus_age_2:Enfant):
                def temp_le_plus_age_2(potentiel_plus_age:Enfant):
                    return potentiel_plus_age.date_de_naissance
                def temp_le_plus_age_3(potentiel_plus_age_1:Enfant):
                    return potentiel_plus_age_1.date_de_naissance
                if (temp_le_plus_age_3(potentiel_plus_age_1) <
                    temp_le_plus_age_2(potentiel_plus_age_2)):
                    return potentiel_plus_age_1
                else:
                    return potentiel_plus_age_2
            return list_reduce(temp_le_plus_age_1,
                               Enfant(identifiant = integer_of_string("-1"),
                               obligation_scolaire = SituationObligationScolaire(SituationObligationScolaire_Code.Pendant,
                               Unit()),
                               remuneration_mensuelle = money_of_cents_string("0"),
                               date_de_naissance = date_of_numbers(2999,12,31),
                               prise_en_charge = PriseEnCharge(PriseEnCharge_Code.EffectiveEtPermanente,
                               Unit()),
                               a_deja_ouvert_droit_aux_allocations_familiales = False,
                               beneficie_titre_personnel_aide_personnelle_logement = False),
                               enfants)
        def temp_le_plus_age_4(_:Unit):
            return True
        temp_le_plus_age_5 = handle_default(SourcePosition(filename="examples/allocations_familiales/prologue.catala_fr",
                                            start_line=80, start_column=12,
                                            end_line=80, end_column=23,
                                            law_headings=["Allocations familiales",
                                            "Champs d'applications",
                                            "Prologue"]), [],
                                            temp_le_plus_age_4,
                                            temp_le_plus_age)
    except EmptyError:
        temp_le_plus_age_5 = dead_value
        raise NoValueProvided(SourcePosition(filename="examples/allocations_familiales/prologue.catala_fr",
                                             start_line=80, start_column=12,
                                             end_line=80, end_column=23,
                                             law_headings=["Allocations familiales",
                                             "Champs d'applications",
                                             "Prologue"]))
    le_plus_age = temp_le_plus_age_5
    return EnfantLePlusAge(le_plus_age = le_plus_age)

[JMLX42]

Exceptions are apparently elided/removed when using the --avoid_exceptions option.

But then the python target does not work:

9595a6a87d50:/workspaces/catala$ ./_build/install/default/bin/catala python --avoid_exceptions examples/allocations_familiales/allocations_familiales.catala_fr 
catala: internal error, uncaught exception:
        File "compiler/shared_ast/expr.ml", line 733, characters 17-23: Assertion failed
        Raised at Shared_ast__Expr.make_app.(fun) in file "compiler/shared_ast/expr.ml", line 733, characters 17-29
        Called from Shared_ast__Expr.fold_marks in file "compiler/shared_ast/expr.ml", line 189, characters 13-55
        Called from Shared_ast__Expr.make_app in file "compiler/shared_ast/expr.ml", line 723, characters 4-382
        Called from Lcalc__Compile_without_exceptions.translate_expr.(fun) in file "compiler/lcalc/compile_without_exceptions.ml", line 332, characters 10-175
        Called from Stdlib__List.fold_left in file "list.ml", line 121, characters 24-34
        Called from Lcalc__Compile_without_exceptions.translate_and_hoist in file "compiler/lcalc/compile_without_exceptions.ml", line 195, characters 15-37
        Called from Stdlib__List.map in file "list.ml", line 92, characters 20-23
        Called from Lcalc__Compile_without_exceptions.translate_and_hoist in file "compiler/lcalc/compile_without_exceptions.ml", line 250, characters 6-48
        Called from Lcalc__Compile_without_exceptions.translate_expr.(fun) in file "compiler/lcalc/compile_without_exceptions.ml", line 309, characters 19-44
        Called from Lcalc__Compile_without_exceptions.translate_scope_let in file "compiler/lcalc/compile_without_exceptions.ml", line 482, characters 21-66
        Called from Lcalc__Compile_without_exceptions.translate_scope_body in file "compiler/lcalc/compile_without_exceptions.ml", line 512, characters 27-58
        Called from Lcalc__Compile_without_exceptions.translate_code_items.(fun) in file "compiler/lcalc/compile_without_exceptions.ml", line 530, characters 16-63
        Called from Shared_ast__Scope.fold_map in file "compiler/shared_ast/scope.ml", line 94, characters 20-34
        Called from Lcalc__Compile_without_exceptions.translate_code_items in file "compiler/lcalc/compile_without_exceptions.ml", line 517, characters 4-653
        Called from Lcalc__Compile_without_exceptions.translate_program in file "compiler/lcalc/compile_without_exceptions.ml", line 570, characters 6-79
        Called from Driver.driver in file "compiler/driver.ml", line 304, characters 16-71
        Called from Cmdliner_term.app.(fun) in file "cmdliner_term.ml", line 24, characters 19-24
        Called from Cmdliner_eval.run_parser in file "cmdliner_eval.ml", line 34, characters 37-44

[JMLX42]

But then the python target does not work:

@denismerigoux after looking at the stack trace, it looks like this is not specific to the python target.

[denismerigoux]

@adelaett another thing for you to debug :)