Contract API

[sanity]

/// Verify that the state is valid, given the parameters. This will be used before a peer
/// caches a new State.
fn validate_state(parameters : &[u8], state : &[u8]) -> bool;

/// Verify that a delta is valid - at least as much as possible. The goal is to prevent DDoS of
/// a contract by sending a large number of invalid delta updates. This allows peers
/// to verify a delta before forwarding it.
fn validate_delta(parameters : &[u8], delta : &[u8]) -> bool;

enum UpdateResult {
  VALID_UPDATED, VALID_NO_CHANGE, INVALID,
}
/// Update the state to account for the state_delta, assuming it is valid
fn update_state(parameters : &[u8], state : &mut Vec<u8>, state_delta : &[u8]) -> UpdateResult;

/// Generate a concise summary of a state that can be used to create deltas
/// relative to this state. This allows flexible and efficient state synchronization between peers.
fn summarize_state(parameters : &[u8], state : &[u8]) -> [u8]; // Returns state summary

/// Generate a state_delta using a state_summary from the current state. Tthis along with 
/// summarize_state() allows flexible and efficient state synchronization between peers.
fn get_state_delta(parameters : &[u8], state : &[u8], state_summary : &[u8]) -> [u8]; // Returns state delta

fn update_state_summary(parameters : &[u8], state_summary : &mut Vec<u8>) -> UpdateResult;

struct Related {
  contract_hash : [u8],
  parameters : [u8],
}
/// Get other contracts that should also be updated with this state_delta
fn get_related_contracts(parameters : &[u8], state : &[u8], state_delta : &[u8])
    -> Vec<Related>;

Note that the Related struct implies the following approach to generating a contract hash:

contract_hash = hash(hash(contract_wasm) + parameters)

The reason is so that a contract_hash can be created with only a hash of the contract_wasm, not the entire contract_wasm. This should reduce contract sizes.

[iduartgomez]

Couple issues:

• Wasm is stateless, how are changes to values in contracts persisted? (Leaving aside synchronization/race conditions problems for now, as that is an other whole can of worms that will have to be tackled following some strategy.) WASI is supposed to add a series of interfaces that will give access inside wasm modules to file-like objects with some limitations etc. but besides that, afaik wasm modules are just pure computation units with a clear start and finish and inputs and outputs.

An option here is to force the contract to return serialized state after computation and this would be handled by the node runtime.

• No mention on how contracts are published on the network in the https://github.com/freenet/locutus/blob/main/docs/architecture.md document. We are lacking a "Publish" operation along "subscribe" or am I missing something? "Put" is just modification to values of existing contracts, not a publishing action.

[sanity]

Wasm is stateless, how are changes to values in contracts persisted?

My thought was that the update_value function would mutate the value parameter, the mutated value would be stored after update_value returns true by the calling code.

If wasm functions can't be passed mutable parameters then update_value will need to return the updated value.

No mention on how contracts are published on the network in the https://github.com/freenet/locutus/blob/main/docs/architecture.md document. We are lacking a "Publish" operation along "subscribe" or am I missing something?

This will be handled by the put operation which is passed both the contract and a value - this allows every node participating in the put to verify the value against the contract.

[iduartgomez]

Suggestion for making this more idiomatic:

type ContractKeyResult<T> = Result<T, ContractKeyError>;

struct ContractKeyError {
  /// original PUT value
  value: Vec<u8>,
  kind: ErrorKind
} 

enum ErrorKind {
 ...
}

trait ContractKey {
  /// Determine whether this value is valid for this contract
  fn validate_value(value : &[u8]) -> bool;

  /// Determine whether this value is a valid update for this contract. If it is, return the modified value,
  /// else return error and the original value.
  fn update_value(value: Vec<u8>, value_update: &[u8]) -> ContractKeyResult<Vec<u8>>;

  /// Obtain any other related contracts for this value update. Typically used to ensure 
  /// update has been fully propagated.
  fn related_contracts(value_update : &[u8]) -> Vec<ContractKey>;

  /// Extract some data from the value and return it.
  ///
  /// eg. `extractor` might contain a byte range, which will be extracted
  /// from the value and returned.
  fn extract(extractor : &[u8], value: &[u8]) -> Vec<u8>;
}

[sanity]

Wanted to get a few thoughts down:

• Contracts can be used to store data, but conceptually they're also similar to "channels" because they can also be used to send data in realtime
• The contract implementation must guarantee that a given set of value_updates will result in the same eventual value, irrespective of what order the value_updates are applied in, in other words they must be commutative

There are three possibilities for a value_update:

1. The value_update is invalid for this contract, it should be disregarded and it may hurt the reputation of the peer it was received from
2. The value_update is valid but has already been applied to this contract value so the value doesn't change
3. The value_update is valid and hasn't already been applied to the contract value is updated

Note: The current ContractRuntime implementation doesn't provide a way to distinguish between 2 and 3, ContractUpdateResult might need to be an enum rather than a Result because 2 isn't really an "error".

Questions:

• Valid value_updates need to be relayed to other interested peers, either peers closer to the contract than this peer, or peers which are subscribed to the contract - but should we relay a value update that we've already seen? If we do then there is the risk of flooding the network with duplicate value updates, but if not the update may not reach interested peers that haven't seen it yet.

[iduartgomez]

Currently update_value returns the current (latest) valid value, regardless of it being changed or not (so returns a valid result in the cases 2 and 3 above and should be the same as it's order invariant). Do you see any value (no pun intended haha) in distinguishing them? A priori I don't see it makes much of a difference. The error case of the result in this case would be the first posibility above (and probably in the future other "runtime related" errors).

Regarding your question, we don't know if a value update has been already seen right now from outside the contract, if so this information should be communicated or computed somehow (hashing the value and if it success it should show in the commit log). I see pros and cons here, it could potentially remove noise from the network, but on the other side add to the compute/memory footprint of the node... Although this would be highly specific to the contract (it may very well cheaper to avoid running the contract again with a value that already was observer instead of blindly broadcasting it).

My first initial intuition is that is worth trying to deduplicate beforehand, but this will be limited and expire since we are not gonna keep an unlimited history of those updates.

Contracts can be used to store data, but conceptually they're also similar to "channels" because they can also be used to send data in realtime

Correct. We could consider values like packets sent, specially if we consider that those can be partial (is not strictly required to send the whole value when you are putting, you can send just one "partial" value that then can be ingested by the contract and incorporated).

[sanity]

Do you see any value (no pun intended haha) in distinguishing them?

I think the main motivation would be that it would allow deduplication of updates without needing to maintain some other record of them.

My first initial intuition is that is worth trying to deduplicate beforehand, but this will be limited and expire since we are not gonna keep an unlimited history of those updates.

Agreed, however, if contracts report whether a value has already been applied then it becomes an implicit record of what updates have already been made to the contract. This would allow for update deduplication without the need to maintain an explicit list of which updates we've seen - the contract value is the record.

is not strictly required to send the whole value when you are putting, you can send just one "partial" value that then can be ingested by the contract and incorporated)

Is a "partial" value the same as a value_update? The concepts seem very similar.

Another question that occurred to me: Should a contract support "merging" two values?

The idea would be that let's say a value - V1 - contains updates U1, U3, and U4, but another node had a value V2 which contained updates U1, U2, and U3.

If these nodes were to somehow discover that they had different values for the same contract, seems like it would be useful for them to combine their knowledge, which would mean "merging" their values.

On the positive side, this seems like it would help ensure that updates reach the peers that need them. On the negative side, hopefully the network will be very good at getting updates where they need to be - so this situation may not happen frequently enough to matter.

[sanity]

Another thought, perhaps instead of:

fn related_contracts(value_update : &[u8]) -> Vec<ContractKey>;

We modify update_value as follows:

fn update_value(value: Vec<u8>, value_update: &[u8], store : &mut Store) -> ContractKeyResult<Vec<u8>>;

Where store can be used to push updates related to this one to other contracts:

impl Store {
  fn update_value(&mut self, contract : &Contract, value : &Vec<u8>);
}

[sanity]

struct TwitterValue {
  tweets : Vec<Tweet>, // Stores the most recent 5 tweets for this contract
}

struct TwitterUpdate {
   tweets : Vec<Tweet>, // Stores the most recent 5 tweets for this contract
}

fn update_value(value : TwitterValue, update : TwitterUpdate) -> TwitterValue {
  let newValueTweets = (value.tweets + update.tweets).sort().take_last_n(5);
  TwitterValue { tweets : newValueTweets} 
}

struct TweetExtractor {
  get_tweets : Vec<TweetID>,
}

fn extract(query : &[u8], value : &TwitterValue) -> ExtractedTweet {
  ExtractedTweets {
    tweets = value.tweets.filter(query),
  }
}

Question: Combine TwitterValue and TwitterUpdate, so every TwitterUpdate is a TwitterValue, and multiple values can be "merged" to create a new TwitterValue - meeting the criteria that values are commutative - doesn't matter what order they're merged it.

[sanity]

Rewrote issue based on discussion.

[sanity]

Removed the get function as we're dropping the query feature - requestors will always get the entire value.

[iduartgomez]

Last proposal:

trait Contract {

    // Validation functions
    fn validate_state(parameters : &Parameters, state : &State) -> bool;
    fn validate_message(parameters : &Parameters, message : &Message) -> bool;

    /// if the state has been changed persist the changes 
    fn update_state(parameters : &Parameters, state : &mut State, message : &Message) -> bool;

    /// relay this message to each of the contract + parameters tuple given back by the function
    fn related_contracts(parameters : &Parameters, message : &Message) -> Vec<(Contract, Parameters)>;

    // node layer:
    // These functions facilitate more efficient synchronization of state between peers
    fn summarize_state(parameters : &Parameters, state : &State) -> StateSummary;

    fn get_state_delta(parameters : &Parameters, state : &State, delta_to : &StateSummary) -> StateDelta;

    fn apply_state_delta(parameters : &Parameters, state : &mut State, delta : &StateDelta) -> bool;

    // appliction layer:
    fn request(parameters : &Parameters, query : &Query) -> Result<Response, Error>;
}

[iduartgomez]

@sanity I kind of forgot why we hav a validate_state function, since when you have a state it has to be always valid (since you cannot update an invalid state).

Also maybe a suggestion, we can remove validate_message and have something like:

    /// if the state has been changed persist the changes 
    fn update_state(parameters : &Parameters, state : &mut State, message : &Message) -> Result<bool, _>;
    

The idea here would be that you validate if the message is valid (if is invalid you return Err(_)), and if is valid you attempt to update (in case it was duplicate you would return Ok(false), otherwise Ok(true)). It's probably more optimal for many contracts that first validating and then updating and you can do it all in the same pass.

[sanity]

@iduartgomez I think the motivation for validate_state is if you receive the contract state from another peer (perhaps to cache it) then you'll need to validate it.

Re: combining the validate_message functionality into update_state, I'm trying to think - there may be reasons you'd want to validate a message without applying it - perhaps if you're just forwarding a message to another peer. We don't want peers forwarding invalid messages because this could be used as a denial-of-service attack against a particular contract.

[sanity]

Contract functions:

Validate State

fn validate_state() -> bool;

Memory layout:

0 : Parameter length (p) : u32
4 : Parameters : [u8; p]
4+p : State length (s) : u32
8+p : State : [u8; s]
12+p+s  ... : Available memory

Returns true if and only if state is valid.

Validate Delta

fn validate_delta() -> bool;

Memory layout:

0 : Parameter length (p) : u32
4 : Parameters : [u8; p]
4+p : Delta length (d) : u32
8+p : Delta : [u8; d]
8+p+d  ... : Available memory

Returns true if and only if the delta is valid given the contract and
parameters.

Update State

fn update_state() -> UpdateResult;

enum UpdateResult {
  VALID_UPDATED, VALID_NO_CHANGE, INVALID,
}

Memory layout:

0 : Parameter length (p) : u32
4 : Parameters : [u8; p]
4+p : State length (s) : u32
8+p : State : [u8; s]
8+p+s : Delta length (d) : u32
12+p+s : Delta : [u8; d]
12+p+s+d  ... : Available memory

The State should be updated to incorporate the delta if valid. Note that the delta can be appended to the state without
copying by increasing the state length to include the delta.

Summarize State

fn summarize_state();

Memory layout:

0 : Parameter length (p) : u32
4 : Parameters : [u8; p]
4+p : State length (s) : u32
8+p : State : [u8; s]
12+p+s+d  ... : Available memory

The state summary should be placed after the State by the function, preceded by its length in bytes in the usual manner.

Get State Delta

fn get_state_delta();

Memory layout:

0 : Parameter length (p) : u32
4 : Parameters : [u8; p]
4+p : State length (s) : u32
8+p : State : [u8; s]
8+p+s : State summary length (y) : u32
12+p+s : State summary : [u8; y]
12+p+s+y  ... : Available memory

The state delta should be placed after the state summary, preceded by its length in the usual way.

Possible additional functions

fn related_contracts();

Questions

• Would it make more sense to put the lengths of all blocks at the beginning of the memory?
• Is there a better notation to use for the memory layouts?

[sanity]

I've rewritten the original issue description with the latest proposal.

[iduartgomez]

I think the first iteration of the contract API is complete and we can refine it further with additional issues like:

• Contract interaction #167