Pagination trouble with CPr responses

[gzarrub]

ContextBroker neither sends the pagination info (offset, limit) to the CPr nor makes any pagination control to CPr responses. So it might be useful to add the pagination parameters when the query is send to the provider.

[fgalan]

Thanks for the feedback! In order to have more information, maybe you could detail a concrete use case or situation in which the problem arises, please?

[gzarrub]

Firstly I thought that the pagination process depended on the CB so my CPr didn't take this process into account.

The query that I tried was similar to this:
http://orion.lab.fiware.org:1026/v1/queryContext?offset=10&limit=50
payload = {"entities":[{"type":"bikerenting","isPattern":"true","id":"Valenbisi.*"}]}

When I saw the response without any pagination restriction, I supposed that maybe the pagination process should depend on the CPr, so I implemented a routine which adds the ability to process queries that include pagination syntax,
http://provider_url/queryContext?offset=10&limit=50. And I tested it by sending the queries to the provider directly.

The problem is that when a query like the previous one is sent to CP, it sends the query to the url which appears at the providingApplication field of the entities registration, but It doesn't add anywhere the pagination info.

[fgalan]

It could happend (not 100% sure but almost) that current Orion version doesn't forward pagination URL params to CPrs. Considering the general case of N CPrs, the expected behaviour should be as follows:

• Client sent queryContext with offset F and limit L. The request is resolved to N CPrs
• Orion forward queryContext using offset 0 and limit L. Thus, as much as N*L elements are received
• The offset F is appliced on the N*L set of elements

[fgalan]

Discussing this with @anabelengp, we have found that the above approach is not valid. Using L in the request to CPr will lead in general to less results in CB at aggregation step. Thus, the set to which CB applies F and L is not the one containing all the possible results.

Two possible alternatives:

• CB recovers a full list in each CPrs (using paginated queries if needed), ordered by creation date. then applies the F and L to that list. This has currently an additional problem: the current NGSI10 API doesn't include creDate.
• Take CPrs as the first level ordering factor, before creDate. For example, ordering the CPrs by URL (being local CB entities as the "0-th CPr") using the details=on to know hoy many entities are in CPrs and do "selective" queries to the subset of needed CPr to solve the client F and L. This solution has its limitations (given that creData is a second level ordering factor, some added/deleted entity could introduce anomalies between paginated query and paginated query), but could work most of the cases assuming that.

[fgalan]

Related issue (in some sense, a "child" of the present one): #1647

[fgalan]

This issue is fixed in PR #4149 so we are going to close it. If we are wrong and it is still not working, please tell us about it (with the detailed steps to reproduce the problem), please.