GraphQL Mutation Support

[dOrgJelli]

EDIT: The below "spec" is outdated, please see comment below for the latest specification.

Do you want to request a feature or report a bug?
feature

What is the feature request?
GraphQL Mutation support, aka write semantics.

When a user queries the mutation, the resolver would execute the corresponding function. These functions would be defined by the protocol developer, and would include all of the logic we're used to seeing in a typical JS wrapper for a smart contract protocol: processing and fetching data, signature requests, external service interactions (IPFS, etc), multiple transactions.

For example, imagine we have an exchange with a token whitelist:

type Exchange @entity {
  id: ID!
  tokens: [Token!]!

  // new functionality
  proposeToken(
    token: Token!,
    description: String
  ): TokenProposal
}

type Token @entity {
  id: ID!
  symbol: String
  name: String
}

type TokenProposal @entity {
  id: ID!
  token: Token!
  descriptionHash: String!
  votesFor: BigInt!
  votesAgainst: BigInt!
  passed: Boolean
}

The proposeToken(...) mutation above would be tied to a backing function that could implement the following logic:

1. Ensures the token isn't already listed.
2. Sanitizes the token metadata (name, symbol), telling the user ahead of time if there are name conflicts.
3. Upload the token metadata to IPFS.
4. Upload the proposal description to IPFS.
5. Create a new transaction payload for the createProposal(...) function on the Exchange contract, with arguments: token address, token metadata hash, proposal description hash.
6. Once the user signs, the transaction is submitted.
7. Await until txReceipt is received.
8. Parse events, extract the TokenProposal.id emitted from the contract, and query the store for the full entity.
9. return TokenProposal to the user.

And the psuedo code:

// Note: more info on the runtime environment
//       of this function further down.
function proposeToken(
  this: Exchange,
  token: Token,
  description: string
): TokenProposal {

  // off-chain sanitization
  if (store.get("Token", token.id)) {
    throw Error("Token already listed.");
  }

  const res = sanitizer(token.name, token.symbol);
  if (res) {
    throw Error(`Token Metadata Invalid: ${res.message}`);
  }

  // IPFS Interactions
  const { name, symbol } = token;
  const metadataHash = IPFS.add(JSON.stringify({ name, symbol}));
  const descHash = IPFS.add(description);

  // transacting
  const exchange = ExchangeContarct.bind(this.id);
  const receipt = await exchange.createProposal(
    token.id, metadataHash, descHash
  ).send();

  // get the emitted proposal ID
  const proposalId = receipt.events["NewProposal"].args.id;

  // pull from the store every 2 seconds until the entity is
  // available, and timeout after 1 minute.
  return await store.pull("TokenProposal", proposalId, 2, 60);
}

Open questions?
Where should the mutations run?
Ideally we'd be able to support client & server side.

What are the mutation's implemented in, AssemblyScript?
Since most smart contract developers build their wrapper libraries in JS, I think this should be the first supported runtime.

If the mutation runs server side, how can you communicate back to the client for signatures?
This could be done by injecting a custom web3 provider server side, which sends responses to a custom GraphQL client client-side.

How would these run client-side?
I believe the client's resolver can just thunk to the mutation implementation without sending a request to the server.

Exciting implications!?

• Type safe, auto-generated, client APIs for Web3 protocols. The GraphQL community's wonderful work in this area enables this. This would cut down maintenance costs for Web3 developers tremendously.
• Use smart contract protocols in any language, without having to rewrite a wrapper library, since GraphQL has implementations everywhere. If we wanted the mutations to run client side in the scenario, they could be in a WASM module that ran within a language specific runtime.

If you made it this far, thank you :) and apologies for the novel. Pretty stoked for the possibilities here...

[dOrgJelli]

After discussing this with the team, I've created an updated specification + example subgraph & dapp integration here:
https://github.com/dOrgTech/the-graph-mutations-spec

Any and all feedback is welcome either here in the comments, or in the repository linked above!

@Jannis @yanivtal @Zerim

[Zerim]

This is looking great. Feels much closer to the direction we discussed on the call.

Some questions/feedback based on the linked repo.

1. GraphQL Mutations Schema I am on board w/ using a GraphQL schema to define mutations interface as long as it is a separate document as you have shown.

2. Circular dependencies

If a seperate .yaml file isn't introduced, it leads to a weird circular dependency graph subgraph mappings <=> mutations, ideally it'd be unidirectional subgraph mappings <= mutations.

I don't understand what is meant by this. "Mappings" are what we call the WASM code that performs ETL in a way that conforms to the data model schema.

Or are you extending the usage of "mappings" to also include WASM code that executes in response to mutations? If so, I would use a different term here.

3. Server Side Mutations I agree the APIs should be the same, though I think server side mutations are of secondary importance to our primary use case, where mutations will generate transactions to be signed by the user's dapp browser or wallet on the client.

4. "Javascript" vs "Node" Resolver Kind. I think we should do some research into how blockchains which support Javascript smart contracts, such as Loom or Cosmos, enforce determinism across different runtimes/ operating systems and let that influence how we describe the target. I.e, should we run in strict mode, allow floating point math, target ES5, etc.

5. Custom Provider

the resolvers could choose to infer the Web3 provider for the user if none is set, but the option to explicitly set one is necessary in my opinion (see "Decisions Made" section below).

I'm unconvinced here. We want subgraphs to be usable across many platforms/ browsers/ devices, etc. If we let the subgraph developer force a custom provider I feel like we are more likely to make the mutations break for some set of users. I think we should define the Web3 provider API as part of our mutations API and take responsibility for maintaining compatibility and abstracting away any inconsistencies in provider implementations from different dapp browsers or wallets w/ our query engine that @Jannis is building.

6. Mutation functions

async createGravatar(_root, args, context) {
      ...
    }

Since mutation handlers mainly transfer user intent into an Ethereum transaction, they don't all need to be async as far as I can tell. It's interesting to think about letting mutation handler chain together several transactions using async or Promise semantics, but I don't think that is the MVP use case.

7. Adding resolvers to app

import { resolvers, setWeb3Provider } from "mutation-resolvers-package"

setWeb3Provider("...")

const client = new ApolloClient({
  ...
  resolvers
}

It's weird to me that both the resolvers and setWeb3Provider are coming directly from a package installed via npm. I would expect the setWeb3Provider to come from the query engine, or whatever package we ship to end users, but I would expect the resolvers to come from the subgraph definition, perhaps by using a helper from the query engine package to read/instantiate the mutation resolvers from a subgraph manifest.

It seems like you start to address this later in the README, but not sure how this reconciles w/ the code above:

graph mutations codegen will codegen types from the schema for the mutation resolvers to use (TypeScript, etc).

This is all the feedback I have for now. Overall this direction looks great. I'll let @yanivtal or @Jannis chime in with any additional feedback.

[dOrgJelli]

Re: @Zerim

2. Circular dependencies
   This is the circular dependency I was referencing...

"subgraph mappings => mutations"
subgraph.yaml

mutations:
  file: ./mutations/mutations.graphql
  resolvers:
    kind: javascript
    package: ./mutations/package.json

"subgraph mappings <= mutations"
mutations/mutations.graphql

type Mutations {
  updateGravatarName(
    displayName: String!
  ): Gravatar
}

^^^ Notice how we reference the schema of the subgraph mappings?

The proposed improvement on this was the introduce a new .yaml file within the mutations project, and remove the reference to the mutations from the subgraph.yaml file:
mutations/mutations.yaml

specVersion: 0.0.2
mutations:
  file: ./src/mutations/mutations.graphql
  baseSchema: ./node_modules/gravatar-subgraph/schema.graphql
  resolvers:
    kind: javascript
    package: ./src/mutations/package.json

The cool part about this approach is that you could potentially support publishing mutations for existing subgraphs.

4. "Javascript" vs "Node" Resolver Kind
   I agree this sounds ideal to have deterministic execution within the browser, but isn't this out of scope, given that we're trying to measure up to the same quality bar any other node package measures up to? For example, these mutations are essentially trying to replace any web3 JS library that's already out there, which aren't deterministic. For example: https://github.com/makerdao/dai.js

5. Custom Provider
   I'm a bit confused on what scenario you're designing around, are you designing around this...

// App.js
setWeb3Provider(myWeb3Provider)

or this...

// mutation.js
let defaultWeb3Provider = ...

export function setWeb3Provider(provider) {
  // do nothing, force my own default
}

6. Mutation functions
   Agreed, they could simply return an await-able object and not await with in the mutation function. In my example implementation, I wanted to query the store after the transaction had gone through, see code snippet here.

7. Adding resolvers to app
   Yes I completely agree, if the app was using the query engine and instantiating the resolvers through there, it would simply just pass the query engine the resolver object (or simply the name of the package?) and the web3 provider would be set within. But I agree that setting the web3 provider in this way is still TBD. Would love to hear @Jannis 's thoughts on how he might go about handling the web3 paradigm.

[dOrgJelli]

I've responded to your comments @Zerim , and it has only left more questions unanswered for me personally. I haven't made any changes to the spec based on your feedback, but would love to hop on a call soon to discuss:

• Web3 Provider Handling
• Deterministic Javascript / Target Env
• Overall design of how the mutations module is defined

[Jannis]

Review from my side (sorry for the last minute comments):

Spec Review <2019-11-07 Thu>

User Story: Protocol Developer

Step 1: Define Mutations

• I like the separate mutations.graphql file. It makes sense to not mix helper
  types used in mutations and subgraph entity types.

  • Since we have @entity directives in schema.graphql, we could still
    manage everything in the same file though.

  • If managed in separate files but merged at GraphQL API generation time, we’d
    have to disallow conflicting names between schema.graphql and
    mutations.graphql.

• Note: the typical root type for mutations is Mutation, not Mutations.

• My biggest question is about the ability to track transactions that are
  created in mutations. How do we make their status queryable and get notified
  of status changes?

  • One idea could be a hook like

    const [ status, mutate ] = useMutation('createGravatar')
       
    if (status.running) {
       ...
    } else {
       ...
    }

    but we’d have to combine that with optimistic updates somehow.

Step 2: Add Mutations To Subgraph Manifest

• kind: javascript makes sense to me.

Step 3: Create The Resolvers’ JavaScript Package

• I think the package should just have a default export, not one called
  resolvers.

• I'd rather not have a setWeb3Provider method. We’ll want to support more
  than one blockchain and Ethereum network. I’d propose that we get providers
  for all of these from the context. That allows us to ingest them easily from
  the outside. E.g.

  let { graphprotocol } = context
  let { providers } = graphprotocol
      
  let web3Mainnet = providers.get('ethereum/mainnet') // or providers['ethereum/mainnet']
  let web3Ropsten = providers.get('ethereum/ropsten')
  let ipfs = providers.get('ipfs')

Step 4: Build & Publish Subgraph

• I think graph deploy (and graph build --ipfs <node>) should upload the
  mutations package to IPFS and store the IPFS hash + the mutation schema in the
  graph-node. People / clients can then download the package from IPFS instead
  of from graph-node.

User Story: Application Developer

Step 2: Add Mutation Resolvers To App

• Create the Graph Protocol context object rather than call setWeb3Provider.

Post MVP Goals

• Haven’t touched the query engine yet. Ideally, the mutations would
  automatically be loaded and merged into the subgraph API schema in the query
  engine.
  • This is similar to the Graph Explorer support / dynamic loading.

[dOrgJelli]

@Jannis thanks so much for this feedback, here are my thoughts:

User Story: Protocol Developer

Step 1: Define Mutations

... How do we make their status queryable and get notified of status changes?

@namesty has been implementing a vanilla GraphQL example of this in the web2app and web2server projects found here (not yet in master). The idea is that this better informs how we can surgically implement this for The Graph without diverging too far from existing GraphQL tooling.

Step 3: Create The Resolvers' Javascript Package

...a default export...

Added to the spec + prototype.

... I'd propose that we get providers for all of these from the context. ...

I've added this to the spec, but it's a bit different than the implementation you described. The reason is that I think it should be up to the mutation developer to decide what libraries they want to use for their providers (ethereum, ipfs, etc). Here is how this works in the updated spec found here:

const resolvers = {
  Mutation: {
    async createGravatar(_root, args, context) {
      // context.thegraph.ethereum
      // context.thegraph.ipfs
      // context.thegraph.datasources.${name} -> address
      ...
    },
    ...
  }
}

const requiredContext = {
  ethereum: (provider) => {
    // these are added to the context.thegraph object
    return new Web3(provider)
  },
  ipfs: (provider) => {
    return new IPFS(provider)
  }
}

export default {
  resolvers,
  requiredContext
}

And in our dApp we...

import gravatarMutations from "gravatar-mutations"
import { initMutations } from "@graphprotocol/mutations-ts"

// 1
const mutations = initMutations(
  gravatarMutations,
  // 2 - used to init our context
  {
    graphnode: process.env.GRAPH_NODE,
    ethereum: process.env.WEB3_PROVIDER,
    ipfs: process.env.IPFS_PROVIDER
  }
)

// 3
const client = new ApolloClient({
  uri: process.env.GRAPH_NODE,
  cache: new InMemoryCache(),
  resolvers: mutations.resolvers, // a
})

The requiredContext object provides 2 functions.

1. Each key in the object is something that's required to be given in the configuration by the dApp developer.
2. Each key is a generator function that creates the object that's to be added to the context, meaning the value returned by requiredContext.ethereum(...) is == the property at context.thegraph.ethereum.

For more info on what initMutations does, read here.

Step 4: Build & Publish Subgraph

... should upload the mutations package to IPFS...

Added to the spec.

User Story: Application Developer

Step 2: Add Mutation Resolvers To App

Create the Graph Protocol context object rather than call setWeb3Provider.

See code snippet above. Full details can be found here.

Post MVP Goals

...query engine...

I've come to the conclusion that the Query Engine & Graph Explorer support both rely on server side execution, would you agree? Reason being is that dynamically loading JS at runtime is... bad... and going about solving this is more energy than it's worth IMO since server side execution seems relatively straight forward to implement if I'm not mistaken.

[dOrgJelli]

An updated version of the spec can be found here. Additionally we've created a vanilla GraphQL application to test things like optimistic updates and resolver status updates, please see the web2app folder. Lastly a rudimentary development roadmap has been started here. It will be fleshed out once the specification gets its seals of approvals and our direction is finalized.

[dOrgJelli]

The specification has been proposed here: graphprotocol/rfcs#10