Kogito - Data Index Support to Query Count

[debu999]

At present data Index only suppports paginated queries and total data based on filters.
The system have to keep hitting the api but is never sure on how many times to make paginated calls.

Can we have a option to return total count for a given filter. This will make the applications depend on it to support more data pull.
At present we have written custom calls to the domain cache and have own logic to parse the filter. It will be very good if we can have count for a given filter returned by DI queries itself.
@nmirasch Please advise on this.

[nmirasch]

Hello @debu999, we have been avoiding this total count queries regarding performance reasons. I guess you can always execute a custom query to get that information from graphql endpoint, as I think you are already doing. Feel free to open a jira with the details about what you really need form data-index in order to discuss this feature request.

[debu999]

https://issues.redhat.com/browse/KOGITO-8242

[debu999]

Hi @nmirasch was there any discussion on this ask.

[cristianonicolai]

@debu999 this is an interesting topic, and there are many ways to solve this problem. Based on our experience with different implementations, including jBPM, we found that the cost of adding a query for the count in every request is not worth it and usually slows down the response times unnecessarily. Basically, we need to do ( generate based on filters ), a count query that gets executed along with the actual filter. This can get complex quite quickly and, as mentioned, slow things down, but more importantly, this count does not give any guarantee to the end user about how many items he might have to go through. The count result can be outdated as soon as it is displayed to the user, so this really makes us question this approach. For our UI tools, we now use a Load More approach. This results in items being added to the page while the user continues to hit load more. This goes until no more results are found, which is similar to what you mentioned, your front end does an extra request that returns nothing. But if you compare this single extra call versus all "pages" that were loaded, and for each one, you added an extra count query, it would still use way fewer resources. Also, at the end of the day, there is so much data that a user can analyse, so we also tried to create smart/pre-defined filters that would help the user navigate through fewer items/rows.

[debu999]

Hi @cristianonicolai ,
Thanks for the insight. What i intended to ask is since we already have option to fetch data and generate filters per query a parameter to call separately for count might help.
Our users usually are interested for a given filter what is the total count of records but only want to to view the top 10-15 with scrolling.

I do understand having count called separately per call might be ineffective but having a separate option to get count for filter might help.

For example if at query time if total count is 30000 we do have subscription to update/create process instances so we can update the count real time.
But getting the count we have to write custom code now despite having the same done by kogito. If there is any thing we can intercept to add this feature will really help keep it simple

[cristianonicolai]

@debu999 ok, so if I understood correctly, you want to be able to execute an aggregator for a specific query, and that would be a separate call to Data Index? That would need to be a separate query, something like ProcessInstanceAggregator where you could use the same filter/where from ProcessInstance, but select different attributes like count.

[debu999]

@debu999 ok, so if I understood correctly, you want to be able to execute an aggregator for a specific query, and that would be a separate call to Data Index? That would need to be a separate query, something like ProcessInstanceAggregator where you could use the same filter/where from ProcessInstance, but select different attributes like count.

Yes it's correct

[debu999]

I feel it will make sense just to add a few aggregation on columns of a filter as all persistence store does provide that out of the box. All we need is to map a give aggregation for a given data type. This can be possibly count/sum to start with.

[cristianonicolai]

@debu999 yep, that sounds reasonable

[debu999]

can there be a simple update to graphql just to return the native db query as a endpoint.

[debu999]

We use data index with mongodb. If we get the db query we will be able to update it for count calls separately.

[debu999]

does it have a jira to start with.

[debu999]

@cristianonicolai is there any option just to expose the filter query so we can modify as need to get the count or other aggregations on top of filter criteria. viz. ABCDomainQuery

[debu999]

@cristianonicolai @nmirasch Is there any tracktion on this request. We are looking for some thing to ensure our apps are providing consistent results.

[debu999]

@pefernan @ricardozanini @nmirasch Please advise

[prerana15]

@nmirasch
Did you get chance to look into this request?
This is the corresponding jira:
https://issues.redhat.com/browse/KOGITO-8242

[nmirasch]

Hello @prerana15 @debu999, sorry for the big delay answering,
just to comment that recently another similar issue has been created, asking for that totalcount information in #2012 Maybe we can define if this is still needed and if it worth to add a new type a type of aggregated data in Graphql and a function to do that aggregation(even it would depend on the DB type, and only for the not domain data at the moment).