Request to implement pagination for request forwarding.

[Anjali-NEC]

We know that the community has the following deprecated and to be discontinued the opinion regarding request forwarding (and pagination at the time of request forwarding) using the registrations/context providers functionality.
#3108 (comment)

#3263 (comment)

However, we would like to realize it with Orion as a contact point for information that each company and local government (context provider) has.
We think that using the request forwarding function is effective for the following reasons.

• The data is held only context provider side, and the consumer refers to the latest data each time.
• With the Publish-Subscribe model, it is necessary to store the latest data on the consumer side, and there are problems with the amount of data to be stored and data synchronization.

Orion community has considered two ways to deal with it in the past.
We would like to consider and implement in the direction of the second plan.
#847 (comment)

We devised a way to use no creDate based on the above second plan. please check the diagram below.
If this can be implemented, the context consumer will be able to get all the data of the context providers.
Please consider the implementation.

[mapedraza]

After some internal discussion of the Orion development team, we arrive that there it makes no sense to implements those features inside the Broker. Context registrations codebase present in Orion would need a lot of work to solve all the pending issues and to have this functionalities. Not only pagination, filtering is also a known problem.

Instead, we thought that the best way to implement the functionalities you are proposing to cover the described scenario is to develop a micro service that handle registrations to different Context Providers and serves the information to the Context Consumer. This microservice can be part of the Orion project hosted in the same repository, but could be implemented in a different programming language (such as Node.js or Python).

What we propose would have the following architecture:

In fact, this strategy is pretty similar to the one implemented by MongoDB sharding cluster. The success of such solution in MongoDB proves that it could also be a good alternative in the context management domain. You can check the as reference the mongo documentation here

The advantage of separating this into a different service are severals (our proposal):

• A more friendly programming language can be used for this specific purpose (for example, Node.js or Python)
• It would simplify the development (making independent the registration to the context broker itself)
• It would simplify the codebase of Orion (not being needed to manage the registrations)
• Advanced registration and sorting scenarios can be implemented easily
• It can handle different authentication methods. In several scenarios, an authentication layer protects the access to the Context Broker. Having a service that can handle the authentication.
• It can also act as translation layer between NGSI-V2 and NGSI-LD protocols

The disadvantages of implementing requested features in Orion codebase (your proposal):

• Developing these new functionalities can have an impact on other parts of the system
• Context registrations codebase present in Orion would need a lot of work to solve all the pending issues and to have these functionalities. Not only pagination, filtering is also a known problem. Here a list with the pending issues related with CPrs:
  • Wrong forwarding URL when registrateion URL omits path #3768
  • Fiware Orion Query Forwarding issue #3736
  • Notifications for attributes from CPr #3704
  • Unusual behavior when requesting entities with specific attributes #3688
  • Issue filtering entities with lazy attributes #3676
  • Use milliseconds formats in dates not related with DateTime #3671
  • Notification (and maybe registrations?) endpoint alias #3655
  • NGSIv2 update forwarding sets attribute value to empty string #3647
  • Extra information about partial updates in responses #3499
  • "Complete CPr" NGSIv2 forwarding #3485
  • Basic query and update forwarding cases with registration '.*' pattern #3463
  • CPr forwarding shouldn't match withou ?type (a "dirty" update is got instead) #3271
  • Overoffset doesn't work correctly when CPr is involved #3263
  • NGSIv2 regs: implement status inactive and expired #3108
  • NGSIv2 regs: support registrations expressions #3107
  • Service path vector should be supported for get subs and regs #3100
  • Service path /# is not interpreted as exact match in get subs and regs code #3099
  • Support # ended service paths in registrations #3078
  • PATCH /v2/registrations/<registrationId> #3007
  • Possible crash due to "invalid regular expression for idPattern" #2769
  • Server HTTP header to announce orion version to clients #2480
  • Support query filtering on NGSIv2 query forwarding to CPrs (registered using NGSIv1) #2282
  • Entity type mandatory to find sole entity when forwarding #2247
  • double slash into url in log message using some requests (v1) #2141
  • Use a better code than 404 Not Found in the client response for failing updates forwarded to CPrs #1527
  • Forwarding updateContext with "APPEND" updateAction #1519
  • isPattern true support in updateContext #1503
  • NGSIv1 query and not found CPrs - 200 OK returned in payload - 404 would be better? #1310
  • q= parameter in registrations list #1146
  • Unefficient query forwarding for multi-entity registration #1044
  • Pagination: last page may include entities from previous pages if the entity is both local and remote #1025
  • Fix "v1 leftovers" #959
  • queryContext behaves differently with/without context provider forward #917
  • ContextAttribute values from mongoBackend #882
  • Parallel multi-thread forwards in multi-CPr forwardining scenario #879
  • Pagination trouble with CPr responses  #847
  • Local entity/attribute overlapping with registration unnoticely bypass Context Provider forwarding logic #745
  • Forwarding and geolocation #696
  • Client decides on Uncached/Cached query and update #506

What do you think? Would you like to progress in a solution as we propose?

[Anjali-NEC]

@fgalan @mapedraza

Thank you for your suggestions for Context Shard Manager.
But I want to consider adding features to Orion again.

After some internal discussion of the Orion development team, we arrive that there it makes no sense to implements those features inside the Broker. Context registrations codebase present in Orion would need a lot of work to solve all the pending issues and to have this functionalities. Not only pagination, filtering is also a known problem.

I understand that it is quite difficult to resolve all pending issues.
Would you like to suspend the modification of query filtering this time and narrow down the modification target to pagination (#847) ?
Pagination makes client to get all information available to the context provider through Orion, the contact point for information.
If all the data is available, the client can handle the filtering.

After narrowing down the correction points as shown below, it may be possible to minimize the amount of correction by making corrections on the Orion.

• Targets the getting API of multiple entities. Updates and deletions are excluded from the correction target.
• NGSIv1 ("details": "Count:") and NGSIv2 ("Fiware-Total-Count:) returns the total number of entities of itself and all context providers in.
• Allows the client to get the context broker itself and all context provider entities by specifying limits and offsets.
• Filtering functions are complicated and difficult to modify, so we will not support it this time.

[Anjali-NEC]

@fgalan @mapedraza

Thank you for your suggestions for Context Shard Manager.
But I want to consider adding features to Orion again.

After some internal discussion of the Orion development team, we arrive that there it makes no sense to implements those features inside the Broker. Context registrations codebase present in Orion would need a lot of work to solve all the pending issues and to have this functionalities. Not only pagination, filtering is also a known problem.

I understand that it is quite difficult to resolve all pending issues.
Would you like to suspend the modification of query filtering this time and narrow down the modification target to pagination (https://github.com/telefonicaid/fiware-orion/issues/847) ?
Pagination makes client to get all information available to the context provider through Orion, the contact point for information.
If all the data is available, the client can handle the filtering.

After narrowing down the correction points as shown below, it may be possible to minimize the amount of correction by making corrections on the Orion.

Targets the getting API of multiple entities. Updates and deletions are excluded from the correction target.
NGSIv1 ("details": "Count:") and NGSIv2 ("Fiware-Total-Count:) returns the total number of entities of itself and all context providers in.
Allows the client to get the context broker itself and all context provider entities by specifying limits and offsets.
Filtering functions are complicated and difficult to modify, so we will not support it this time@fgalan @mapedraza

Thank you for your suggestions for Context Shard Manager.
But I want to consider adding features to Orion again.

After some internal discussion of the Orion development team, we arrive that there it makes no sense to implements those features inside the Broker. Context registrations codebase present in Orion would need a lot of work to solve all the pending issues and to have this functionalities. Not only pagination, filtering is also a known problem.

I understand that it is quite difficult to resolve all pending issues.
Would you like to suspend the modification of query filtering this time and narrow down the modification target to pagination (https://github.com/telefonicaid/fiware-orion/issues/847) ?
Pagination makes client to get all information available to the context provider through Orion, the contact point for information.
If all the data is available, the client can handle the filtering.

After narrowing down the correction points as shown below, it may be possible to minimize the amount of correction by making corrections on the Orion.

Targets the getting API of multiple entities. Updates and deletions are excluded from the correction target.
NGSIv1 ("details": "Count:") and NGSIv2 ("Fiware-Total-Count:) returns the total number of entities of itself and all context providers in.
Allows the client to get the context broker itself and all context provider entities by specifying limits and offsets.
Filtering functions are complicated and difficult to modify, so we will not support it this time..

Hi @fgalan @mapedraza

Please provide your response on it. Thanks

[fgalan]

@Anjali-NEC so if I have understand correctly your proposal is not to work in the line described by @mapedraza at #4149 (comment) but focus only in the CPr pagination issue. Is my understanding correct?

In that case, I'd like to clarify what are you exactly proposing, please. Examples may help.

Thus, taking into account your picture:

Let's name CPrs A, B and C (from left to right).

Let's assume we have the following situation in a given moment:

• In CB we have
  • E1 created at time 1
  • E2 created at time 2
  • E3 created at time 3
• In CPra A we have
  • EA1 created at time 1.1
  • EA2 created at time 2.2
  • EA3 created at time 3.3
• In CPra B we have:
  • EB1 created at time 1.2
  • EB2 created at time 2.1
  • EB3 created at time 3.2
• In CPra C we have:
  • EC1 created at time 1.3
  • EC2 created at time 2.3
  • EC3 created at time 3.1

Which entities (order matters) the following queries will return?

• GET /v2/entities
• GET /v2/entities?ordedBy=dateCreated
• GET /v2/entities?offset=2&limit=3
• GET /v2/entities?offset=2&limit=3&orderBy=dateCreated
• GET /v2/entities?offset=5&limit=4
• GET /v2/entities?offset=5&limit=4&orderBy=dateCreated

[Anjali-NEC]

@Anjali-NEC so if I have understand correctly your proposal is not to work in the line described by @mapedraza at #4149 (comment) but focus only in the CPr pagination issue. Is my understanding correct?

Yes your understanding is correct.

In that case, I'd like to clarify what are you exactly proposing, please. Examples may help.

Thus, taking into account your picture:

Let's name CPrs A, B and C (from left to right).

Let's assume we have the following situation in a given moment:

In CB we have
E1 created at time 1
E2 created at time 2
E3 created at time 3
In CPra A we have
EA1 created at time 1.1
EA2 created at time 2.2
EA3 created at time 3.3
In CPra B we have:
EB1 created at time 1.2
EB2 created at time 2.1
EB3 created at time 3.2
In CPra C we have:
EC1 created at time 1.3
EC2 created at time 2.3
EC3 created at time 3.1
Which entities (order matters) the following queries will return?

GET /v2/entities

It will return all the entities of CB, CPra A, CPra B, CPra C based on the registration order of ContextProvider in CB and the registration order of Entities in CP.

If you register in CP in the order of CPra A, CPra B, CPra C. It will return the response in the following order:

{E1, E2, E3, EA1, EA2, EA3, EB1, EB2, EB3, EC1, EC2, EC3}

GET /v2/entities?orderBy=dateCreated

This time, only offset, limit, option=count is modifying target, so orderBy does not work and returns the same result as GET /v2/entities.

{E1, E2, E3, EA1, EA2, EA3, EB1, EB2, EB3, EC1, EC2, EC3}

However, when supporting orderBy in the future, the responses of CB, CPra A, CPra B, CPra C will be sorted by the orderBy condition and returned.

The order of CPra A, CPra B, CPra C is guaranteed on the CP side, so CB just performs a mergesort.
CB does not need to get all CP responses at once. CB can be sorted while getting CP response by pagination.

GET /v2/entities?offset=2&limit=3

The pagination parameters will apply on the basis of the order of CPrs. the limit and offset to be passed to ContextProvider while taking into account the order of ContextProvider registration.

{E3, EA1, EA2}

GET /v2/entities?offset=2&limit=3&orderBy=dateCreated

If orderBy is not implemented, it will return below output:

{E3, EA1, EA2}

It will be {EB1, EC1, E2} in the orderBy implementation.

GET /v2/entities?offset=5&limit=4

This will return below data:

{EA3, EB1, EB2, EB3}

GET /v2/entities?offset=5&limit=4&orderBy=dateCreated

If orderBy is not implemented, it will return below output:

{EA3, EB1, EB2, EB3}

It will be {EB2, EA2, EC2, E3} in the orderBy implementation.

[Anjali-NEC]

Hi @fgalan @mapedraza

We have done some investigation on how the pagination feature laid out in case of request forwarding and how it will be implemented. we have classified some function on the basis of sequence diagram flow and the details of the function which need to be modified for the implementation of pagination in case of request forwarding are given below.

Please find below my solution approach and please let us know if you have any suggestion.

As per my analysis below is the updated sequence diagram regarding the implementation of pagination in request forwarding:

Note: Reference of above image is taken from below link:
https://github.com/telefonicaid/fiware-orion/blob/3.6.0/doc/manuals/devel/mongoBackend.md#mongoquerycontext-sr

Note: Reference of above image is taken from below link:
https://github.com/telefonicaid/fiware-orion/blob/3.6.0/doc/manuals/devel/mongoBackend.md#mongodiscovercontextavailability-sr

Function list in which changes required

As per the current understanding of code and sequence diagram, below table contains the details of modification corresponding to the target functions to implementing the pagination feature in case of request forwarding

File name	Function name	Change point
src/lib/mongoBackend/mongoQueryContext.cpp	mongoQueryContext()	Currently only the default value of the pagination parameters is received. Need to add code for getting the value of limit, offset and count which is provided by the context consumer and manage them.
src/lib/mongoBackend/MongoGlobal.cpp	registrationsQuery()	Need to modify registrationsQuery function to get the exact value of pagination parameters which is pass by context consumer in this function.
src/lib/mongoBackend/mongoDiscoverContextAvailability.cpp	processDiscoverContextAvailability()	Need to add code to pass the value of pagination parameters to the context Provider.
src/lib/mongoBackend/mongoDiscoverContextAvailability.cpp	mongoDiscoverContextAvailability()	Need to change code for the context availability discovery (NGSIv1) operation in order to add the logic for provider ordering is unique and pagination parameters pass on the basis of that.
src/lib/mongoBackend/MongoCommonUpdate.cpp	searchContextProviders()	It uses registrationsQuery() in order to locate Context Providers for forwarding of the update. might be need to change the logic and add code according to the registrationsQuery().
src/lib/mongoBackend/mongoQueryContext.cpp	processGenericEntities()	Need to add code for addition of entities from each contextProvider which is needed in the case of the pagination if limit is not reached with local entities
src/lib/mongoBackend/mongoQueryContext.cpp	addContextProviders()	Need to add code for manage entities from each contextProvider if limit is not reached with local entities
src/lib/ngsi10/QueryContextRequest.cpp	QueryContextRequest()	mongoQueryContext() uses a QueryContextRequest object as input parameter and a QueryContextResponse as output parameter. Its purpose is to build a response object based on a request object and entities and registrations. Need to change in request that it takes the pagination parameters in request and give the response according to the pagination parameters
src/lib/serviceRoutines/postQueryContext.cpp	queryForward()	queryForward() forward the query request to the context provider Need to change that function so that the pagination parameters also forwarded to the context provider
src/lib/ mongoDriver/connectionOperations.cpp	collectionRangedQuery()	collectionRangedQuery() is called in order to do the actual query in the database Currently default value of limit, offset is passed in collectionRangedQuery(). Need to pass the value given by context consumer

[Anjali-NEC]

@fgalan @mapedraza

We have started working on implementing this feature with the shared implementation approach. please let us know if you have any query or concern.

[Anjali-NEC]

@fgalan @mapedraza Gentle Reminder!!

Please let us know the feedback on the shared approach to fix this issue.

[fgalan]

I'm sorry we don't have time to provide feedback on #4149 in a timely manner taking into account the urgency it has for you and that it is not part of the roadmap. Issues that are in the roadmap are the ones in which focus our feedback and review effort.

Thus, go ahead with the implementation of #4149 without waiting for our feedback and we will see the PR when it comes. At that moment we have another feedback point.

[ArqamFarooqui110719]

Hi @fgalan, Is there any plan to review PR of this issue i.e. PR #4275?

[fgalan]

Hi @fgalan, Is there any plan to review PR of this issue i.e. PR #4275?

Yes. It is planned. However, we cannot commit on a date. The review will be done in a best-effort way.

[ArqamFarooqui110719]

Gentle Reminder...
Hi @fgalan, Is there any possibility to review the PR in this month?

[fgalan]

Gentle Reminder... Hi @fgalan, Is there any possibility to review the PR in this month?

I have done a first round or review. Please find my comments in the PR itself.

[ArqamFarooqui110719]

Hi, As both the PRs (#4275 and #4352 ) have been merged, So we can close these issues (#4149 and #4341) now?

[fgalan]

I'd suggest to wait to close this issues until PR #4341 gets merged.

[ArqamFarooqui110719]

As the PR #4341 also got merged, Now we can close the issue #4149 (and other related issues, for e.g. #847)?

[fgalan]

As the PR #4341 also got merged, Now we can close the issue #4149 (and other related issues, for e.g. #847)?

Yes, that issues (along with #1647) will be closed.

However, I have discovered that this other issue, which is related with pagination and CPrs is not solved: #1025 (comment). Please, have a look.