[Spike] feat: inclusion

[jannyHou]

Related to #1952

Copy the discussion here:

This PR is a PoC of the inclusion handler proposed by @batjos in comment.

Takes findById as an example method, after adding the code that fetches the included items, TodoList.findById(1, {include: [{relation: 'todos'}]}) returns

The PoC contains:

• A InclusionHandlerFactory is created to return a handler function that fetches the included items.

• When a relation is created, a corresponding handler will be registered in the source repository.

• TBD DONE: retrieve the relation metadata in InclusionHandlerFactory function:

  • return different handler functions according to the relation type
  • find the fkName in the relation definition

• TBD(improve) DONE: Turn the factory to a class, each repository has a InclusionHandler class instance as property:

  • call this._inclusionHandler.register<TargetEntity>(targetRepositoryGetter) to register the relation handler.
  • call this._inclusionHandler.searchHandler(relationName) for each entry in the filter.include.

I also discovered a few problems worth discussion:

• The Filter interface is not designed to describe related entities, see its definition(and Inclusion's definition).

• I feel the inclusion handler's implementation would have overlapping code with relation repository's find method, I will investigate how to leverage those relation repositories when register the handler.

[jannyHou]

From @bajtos

@jannyHou

The Filter interface is not designed to describe related entities, see its definition(and Inclusion's definition).

Good catch, can you work with @raymondfeng to improve the type definition please?

We should eventually take a look at the JSON Schema/OpenAPI parameter schema emitted for filter too.

---

It is important to consider the scenario where we want to fetch multiple source models and include their related models in the result, e.g. TodoList.find(1, {include: [{relation: 'todos'}]}). Under the hood, we must avoid SELECT N+1 issue. Inclusion handler must be able to fetch related models for multiple source-model ids in one query (using inq operator under the hood). The design proposed in #1352 (comment) is already taking that into account, so does the existing implementation in juggler.

[jannyHou]

From @bajtos

This block of code will be repeated a lot, I'd like to see a simpler solution, e.g. a sugar API building on top of InclusionHandlerFactory.

Ideally, the entire block should collapse to a single statement, e.g.

this._registerInclusion('todos', todoRepositoryGetter);

Under the hood, the helper should look up the definition of todos relation to learn about the target model, keyTo, and any other metadata needed. See how _createHasManyRepositoryFactoryFor is implemented.

Also based on the code in juggler, every relation needs a slightly different implementation of the inclusion handler. Please include an example/acceptance test showing how to include models via belongsTo relation to ensure requirements of other relation types were considered.

[jannyHou]

@bajtos

I added the belongsTo handler and it does fetch the parent instance for the child. BUT there is a problem when construct the returned data:

The Todo model only has property todoListId as the foreign key, but not todoList as its parent property. Therefore given the object

const todo = {
    id: 1,
    title: 'a todo item',
    todoListId: 1,
    todoList: {
      id: 1,
      color: red
    }
}

method toEntity(todo) won't convert property todoList and removes it in the constructed todo entity.

[jannyHou]

@bajtos Thank you for the review!

It is important to consider the scenario where we want to fetch multiple source models and include their related models in the result, e.g. TodoList.find(1, {include: [{relation: 'todos'}]}). Under the hood, we must avoid SELECT N+1 issue.

Good stuff, I didn't realize the SELECT N+1 issue before. I changed the handler function to take in an array of fks.

The PoC PR implements the inclusion handler in findById method, so it hasn't really run into the SELECT N+1 issue. We can take care of it when handle inclusion in the find method. (I don't see too much difficult to do a local join). There are a few design level blockers for us to solve:

Inclusion Filter

The Filter interface is not designed to describe related entities, see its definition(and Inclusion's definition).

I was thinking of modify the Inclusion interface, like create a condition based type that returns the entity type for a given relation

export interface Inclusion<MT extends object = AnyObject> {
  relation: string;
  scope?: Filter<returnEntityType<MT>>;
}

But after I found belongsTo only has a fk property but no relation property (see my comment #2124 (comment)), I am wondering do we really need a relation property embedded in the model.

👇

Relation property

Let's continue the above topic. Currently adding a hasMany relation is not essentially equivalent to adding a belongsTo relation in terms of reshaping a model:

• hasMany relation adds a relation property, like todos
• belongsTo relation adds only a fk, like todoListId

Which means a Todo model cannot describe an instance that includes its parent like

{
  id: 1,
  color: red,
  todoListId: 1,
  todoList: {
    id: 1,
    desc: 'a todo list'
  }
}

Is it restricted by the circular model reference issue? If that's the case, I assume simply add a relation property todoList in model todo won't be the solution.

Think it from a more general level, probably we could isolate the relation traverse from entity inclusion:

• IMO it might be more proper to turn relation to be a repository level concept(now it's a model level concept, or kind of a mix), because relation traverse already implies the data to retrieve are persistent, which means a datasource should be specified. By applying a relation, a model's shape shouldn't be changed.

• We could loose including related entities to be a more general topic: given two(or more) entities, can we return a conditional based type at run-time to describe one as the other's property? If possible, this would also solve the self relation issue.

Repository level relation

Seems we preserved the relation definition in model definition for the purpose of supporting the LB3 inclusion. But if we decide to abandon the juggler inclusion system, as I explained above, we don't necessary to keep model level relation decorators anymore.

[raymondfeng]

It can be simplified as:

this._handlers[relationName] = async (
      fks: SID[],
      filter?: Filter<TE>,
    ): Promise<TE[]> => {...}

[raymondfeng]

Maybe we should define a type such as:

export type IncludedItemsFetcher<SID, TE> = (
      fks: SID[],
      filter?: Filter<TE>,
    ) => Promise<TE[]>

[raymondfeng]

The name is a bit confusing as InclusionHandler contains a list of _handlers.

[jannyHou]

yeah it should be plural

[raymondfeng]

It's suspicious to return an Error but later on it checks:

const handler = this._inclusionHandler.findHandler(relation);
    if (!handler) {
      throw new Error('Fetch included items is not supported');
    }

[jannyHou]

yeah the code here should definitely be refined ^

[raymondfeng]

It's suspicious to return an Error but later on it checks:

const handler = this._inclusionHandler.findHandler(relation);
    if (!handler) {
      throw new Error('Fetch included items is not supported');
    }

[raymondfeng]

I like the idea to delegate resolution of inclusion to a list of functions.

[raymondfeng]

I think we can borrow some ideas from https://graphql.org/learn/execution/. It will allow us even define custom relations.

[raymondfeng]

https://www.apollographql.com/docs/graphql-tools/resolvers.html

[bajtos]

Which means a Todo model cannot describe an instance that includes its parent

Good catch!

When building belongsTo relation, I had a gut feeling that it's not a good idea to decorate the foreign key (e.g. Order.prototype.customerId) and don't provide relation property (e.g. Order.prototype.customer), but didn't investigate further. Inclusion of related models is a good use case to drive improvements in the design for belongsTo.

I think the current design is a workaround for TypeScript limitation explained in microsoft/TypeScript#27519

A possible solution is to define a type wrapper that will prevent the circular reference problem at the cost of losing design: type metadata (the metadata will say "Object" instead of the actual type).

export type Related<T> = T | undefined;

class Todo extends Entity {
  // ...

  todoList: Related<TodoList>;
}

class TodoList extends Entity {
  // ...

  todos: Related<Todo>;
}

Note: AFAICT, it's important to use a named type. Replacing Related<Todo> with Todo | undefined does not break the circular reference.

For longer term, I'd like to change the way how relations are defined: relation decorators provide relation metadata only, foreign keys are defined explicitly. For example:

class Todo extends Entity {
  // ...

  @foreignKey(() => TodoList, 'id') // target model, target property
  @property({type: 'number', required: true})
  todoListId: number;

  @belongsTo(()=> TodoList)
  todo: Related<Todo>;
}

[bajtos]

One more idea to consider - perhaps we can define two model classes, the first describing only the properties actually stored in the database, the second including relations?

class Customer extends Entity {
  // ...
}

class CustomerWithRelations extends Customer {
  orders?: Order[]
}

Create/update methods will greatly benefit from this distinction:

• at the moment, it's not possible to update related models together with the source model (e.g. create a Customer and the related Order in one call)
• at the same time, our TypeScript API (and possible OpenAPI spec too) allow relational properties in the create/update/patch input data.

[jannyHou]

Thanks for all the feedback!

I understand that the best outcome of a spike is a concrete solution, but I am afraid the inclusion story is more complicated than we expect, and it would be better to create separate stories to explorer different potential approaches.

Like what @raymondfeng points out, GraphQL's query system has two features we could borrow:

• use type resolver to describe a complicated object and calculate its value
• use different query types to describe the filter of a model

Here is a proposal of the next steps:

• Explorer the approach "Define relation property as a type resolver"
  • The prototype would be:

       import { TodoList } from './TodoList';
       class Todo extends Entity {
          @inclusion()
          TodoList: ()=>TodoList
        }

• Explorer the approach "Use a type wrapper to prevent the circular reference problem"
  • The prototype would be:

        export type Related<T> = T | undefined;
        class Todo extends Entity {
          // ...
          todoList: Related<TodoList>;
        }
        class TodoList extends Entity {
         // ...
          todos: Related<Todo>;
        }

• Explorer "Generate decorator based filter types"
  • Inspired by the query/mutation type in GraphQL, I am trying to define different partial model types described by decorators
  • The prototype:

        // model.ts file
       class Todo extends Entity {
         @property({name: 'id', type: 'string'})
         id: string,
         @property({name: 'desc', type: 'string'})
         desc?: string
         @inclusion()
         TodoList: ()=>TodoList
       }
       // model.query.ts file
       type TodoMutation = Pick<Todo, a union of properties decorated by @property except id>
       type TodoQuery = TodoMutation & Inclusion<Todo>

• Explorer the approach "Define multiple model classes for different purpose"
  • Prototype:

      class Customer extends Entity {
        // ...
      }
    
      class CustomerWithRelations extends Customer {
       orders?: Order[]
      }

[bajtos]

Explorer the approach "Define relation property as a type resolver"
The prototype would be:

   import { TodoList } from './TodoList';
   class Todo extends Entity {
      @inclusion()
      TodoList: ()=>TodoList
    }

How do you envision serializing such structure to JSON, for transmission over HTTP?

Explorer "Generate decorator based filter types"
type TodoQuery = TodoMutation | Inclusion<Todo>

I like that approach! Although I am not sure how will Inclusion<Todo> know what is the property name where to store the target Todo instance?

Also I think you need to use & operator, not |:

type TodoQuery = TodoMutation & Inclusion<Todo>

[jannyHou]

@bajtos

Also I think you need to use & operator, not |

Good catch!

Although I am not sure how will Inclusion know what is the property name where to store the target Todo instance?

That's something I am trying out on local but haven't got it work :(

How do you envision serializing such structure to JSON, for transmission over HTTP?

Hmm could you elaborate more about it? Do you mean infer the JSON schema of a model?

[bajtos]

How do you envision serializing such structure to JSON, for transmission over HTTP?

class Todo extends Entity {
      @inclusion()
      TodoList: ()=>TodoList
    }

Hmm could you elaborate more about it? Do you mean infer the JSON schema of a model?

IIUC your proposal, when we create an instance of Todo model, todo.TodoList will contain a function (the resolver).

When creating the HTTP response body, we want JSON data of the related TodoList instance, e.g.

{
  id: '1',
  desc: 'todo description',
  TodoList: {
    id: '1',
    title: 'list name',
  }
}

My question is how do you envision converting the value from a resolver function into a JSON object? Maybe I misunderstood your proposal?

[jannyHou]

@bajtos good question.

The resolver function should call TodoListRepository.find(filter) to get the related todoList entity, then in the TodoRepository.find() method, we call todoList.toObject() to convert an entity to an object, and assign it to the TodoList property.

But in terms of how we build the type to describe

{
  id: '1',
  desc: 'todo description',
  TodoList: {
    id: '1',
    title: 'list name',
  }
}

That's also my concern 😞 and I am not even sure is it possible to build such a type at run time.
And this is exactly what we could try in the follow up stories.

[b-admike]

I really like the last two proposals from #2124 (comment) (so if we create the follow up stories and had to prioritize them, I'd like to see those two higher in the list) and the idea of moving definition of relations to repository level.

[bajtos]

Another aspect to consider: how are we going to generate JSON Schema and OpenAPI schema for payloads containing both model properties (Customer's id, name, email) and data of related model (Customer's orders, addresses, etc.). At the moment, we are leveraging @property metadata. If we choose a different approach for describing model+relations in TypeScript, then we also need a new way how to obtain JSON Schema for such new types.

[jannyHou]

@bajtos @b-admike Thanks for the feedback! I post your comment with the proposals in a new story #2152. Let's continue the discussion there.

I am closing this PoC PR. Anyone works on the follow up feel free to reopen/cherry-pick the commit if you think useful :)

[AnanthGopal]

Hi Guys,
I need this type of output.
{
id: '1',
desc: 'todo description',
TodoList: {
id: '1',
title: 'list name',
}
}

Did you find any solution for the related model issue?