Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EPIC] Implement loadInBlock and loadRelated and update dependencies #397

Closed
8 tasks done
dimitrovmaksim opened this issue Aug 15, 2023 · 3 comments
Closed
8 tasks done
Labels
bug Something isn't working critical enhancement New feature or request

Comments

@dimitrovmaksim
Copy link
Member

dimitrovmaksim commented Aug 15, 2023

Progress:

Main scope

Extended scope

The Issue

Currently there are two new graph-node functionalities loadInBlock and loadRelated which are not implemented/mocked in Matchstick. This causes matchstick to panic and it prevents some users from properly testing their subgraphs.

loadInBlock

Description:

From my understanding loadInBlock tries to load an entity from some kind of a cache that persists for the time a specific block is being indexed. The entity will be present in the cache only if it was created or updated in that block.

For example event A occurs in block 123, in our handler we create an entity B with ID the address (0x0...1) of the emitter contract C. Later in that block another event A is emitted by contract C. If we try to load the entity using B.loadInBlock("0x0...1") we should get the entity we have created during the first occurrence of the event A.

But if in block 124, contract C has once again emitted event A and we try to load the entity using B.loadInBlock("0x0...1"), the function will return null because the entity is no more in the cache and instead we should try to load it from the database.

Matchstick implementation:

My proposed solution would be to:

  1. Have the default loadInBlock behaviour to return null
  2. Add a mock cashe store
  3. Add a function that mocks the entity in the cache store with user defined
  4. Add a function to remove/clean the cache store

loadRelated

Description:

loadRelated is a function that allows users to load derived entities into their handlers. This was previously not possible in the graph-node, but was possible with Matchstick due a bug in the graph-cli that generated getter (and setter) functions for those fields and the way the Matchstick's own entity store works. Derived fields are virtual fields and do not actually exists in the graph-node database. Instead they are created during query time and are only accessible when querying the subgraph. More info on derived fields https://thegraph.com/docs/en/developing/creating-a-subgraph/#reverse-lookups

Matchstick's own store

Matchstick does not work with a database, but implements its own key/value entity store. In the Matchstick store the derive fields actually exists and they keep the IDs of the child entities, hence why the getter/setter functions worked in Matchstick. While this was helpful for users to test their relations, it caused a lot of issues (especially with handling both String and Byte IDs) and made the code complex.
It will probably be wise to refactor the store and make it function similarly to the graph-node, i.e dynamically construct the derived fields when requested. We already keep the derived relations in a separate derived key/value store, but it may not be needed anymore, because the child entity's name, the parent id and the derivedFrom field should come from the loadRelated function itself, which should make it easier to find it in our entity store.

Matchstick implementation:

Entity Store:

Looking into the function loadRelated(entity: string, id: string, field: string): Array<Entity>; function we may be able to:

  1. Simplify the matchstick store. - Currently we keep all relations into a separate derived mapping. But the function itself should provide us with the relation info we need in order to fetch the child entities from the entities store, so this mapping becomes obsolete as well as this whole module.
  2. We will no longer need to track the relations into the store directly - the derived_fields module will probably become obsolete

logStore():

The potential change to the store will affect what the logStore() function prints to the console. There's several ways to approach it:

  1. Dynamically construct the derived fields when executing logStore()
  2. Add a logEntity(entityId) function that will only print the specified entity and it's relations
  3. Add an optional boolean parameter to logStore() that if set to true will also construct and print the relations
  4. Extend logStore() to accept a set of options {related: true, entityId: 123} that will combine 2. and 3.

Update Matchstick dependencies with the latest graph-node version:

Description:

Matchstick depends on some graph-node functionalities. It hasn't been updated for a while, so will need to do that in Cargo.toml. This may lead to issues that will need to be fixed

Resources:

Matchstick store and derived_fields implementation:

Entity store - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L96
Derived relations store - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L115
Initialising the relations store - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/derived_schema.rs
Save Entity to store - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L503
Saving the derived fields 🤯 - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/derived_fields.rs
Get entity from store - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L478
logStore - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L181
clearStore - https://github.com/LimeChain/matchstick/blob/fix-derived-with-byte-ids/src/context/mod.rs#L190

Matchstick-as

https://github.com/LimeChain/matchstick-as

Matchstick docs

https://github.com/LimeChain/matchstick
https://thegraph.com/docs/en/developing/unit-testing-framework/

Demo-subgraph

https://github.com/LimeChain/demo-subgraph

@dimitrovmaksim dimitrovmaksim added bug Something isn't working enhancement New feature or request critical labels Aug 15, 2023
@azf20
Copy link

azf20 commented Aug 17, 2023

@dimitrovmaksim thanks for this, I think Matchstick may also need updates to support File Data Sources?

@dimitrovmaksim
Copy link
Member Author

@dimitrovmaksim thanks for this, I think Matchstick may also need updates to support File Data Sources?

Yeah, File Data Sources are not currently supported as well

@dimitrovmaksim
Copy link
Member Author

dimitrovmaksim commented Aug 28, 2023

Btw @azf20 just FYI, it seems some changes that were made to the graph-node in order to re-use some modules/functionalities in matchstick have been reverted/moved/removed, so updating the graph-node dependencies to the latest version in matchstick could require a PR to the graph-node as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working critical enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants