Data index incorrectly registered for date field to String

[Pradeepg21]

Bug Details:

As per screenshot attached, we have created bbmn file for createdDate and UpdatedDate as type java.util.Date. Proto file correctly generated for both these field as Date type but in data index it got registered incorrectly with type as String.
Could anyone please check this urgently as it is blocking our development activity?

[debu999]

@cristianonicolai @fjtirado need some help to figure why date field gets registered as string in graphql for data index...

[nmirasch]

@debu999 I guess the register is well done to kogito.Date that is the expected, but It seems the type kogito.Date is not being found correctly. I'm taking a look here, trying to reproduce this. I'll let you know what I find.

[debu999]

Sure. Seems to be something in proto which is not recognized in schema generation.

[nmirasch]

@debu999, @Pradeepg21 Currently the dates included in process variables are added to the graphql schema as Strings. Even in the proto generation the proper marshaling are used to read and write Dates, it remains to add the fields as a Dates in the graphql schema. I've created https://issues.redhat.com/browse/KOGITO-7795 to add the Domain Dates fields mapping to graphql.DateTime instead or Strings..

[debu999]

Looking forward to the fix soon

[Pradeepg21]

Hi @nmirasch , Do you have any update on this?

[debu999]

@nmirasch any update on this.

[debu999]

@nmirasch will there be anything in 1.28.0.Final on this??

[nmirasch]

@debu999 I'm sorry for the delay, as sooner this development will be included in 1.29.

[Pradeepg21]

Hi @nmirasch, has this issue been fixed and getting push with release 1.29? Please keep us posted.
Thanks

[debu999]

@nmirasch is this still on target for 1.29 version
29

[nmirasch]

hi @debu999, the dev is on this PR: https://github.com/kiegroup/kogito-apps/pull/1481 Still trying to merge it once the CI is green. I guess it will be for 1.30 release

[debu999]

@nmirasch we tried using the date filters and seems it's implemented partially. For example in mongodb the data stored for a date field is till iso formated string where as metadata is stored as integer

So only date filter works for kogtoeta data objects but not for variables with date Data type.

Is there any change to be done in kogito runtime to store the date ans number instead.

More over float or double is registered as string in graphql.

[debu999]

@nmirasch
Adding details for the bug as discussed.
Refer the to reproducer at https://github.com/debu999/kogito-learn.git
I have a simple BPMN with a date process variable called dt of type java.util.Date.
Now when we persist it to MongoDB then lets see what happens

For the following payload and businesskey datemongocheck1

{
  "dt": "2022-03-10T01:02:03.123-04:30",
  "abc": "datemongocheck1"
}

I get the response for kogito runtime as

{
  "id": "922c8585-65a0-4499-8bc8-3ca75157e11b",
  "dt": "2022-03-10T11:02:03.123+05:30",
  "abc": "datemongocheck1"
}

Request

Response

Lets look at the data in mongodb for instance from runtime
clearly in context variable we see the details correctly.

{
  "_id": {
    "$oid": "6378e498ff9bb71b7c54f1ee"
  },
  "processType": "BPMN",
  "processId": "dataindexdate",
  "id": "922c8585-65a0-4499-8bc8-3ca75157e11b",
  "businessKey": "datemongocheck1",
  "description": "dataindexdate",
  "state": 1,
  "startDate": "1668867224773",
  "signalCompletion": true,
  "sla": {
    "slaCompliance": 0
  },
  "context": {
    "variable": [
      {
        "name": "abc",
        "dataType": "java.lang.String",
        "value": {
          "@type": "type.googleapis.com/google.protobuf.StringValue",
          "value": "datemongocheck1"
        }
      },
      {
        "name": "dt",
        "dataType": "java.util.Date",
        "value": {
          "@type": "type.googleapis.com/google.protobuf.Timestamp",
          "value": "2022-03-10T05:32:03.123Z"
        }
      }
    ],
    "nodeInstance": [...],
    "iterationLevels": [...]
  },
  "completedNodeIds": [
    "_5458A15F-5450-4380-927C-4D4AED2C48AD"
  ]
}

So far so good from runtime persistence.

Data index version of data...
Do look at the storage in data domain cache for variable which is date vs metadata which is date
we see metadata.lastUpdate and metadata.processinstances.lastUpdate are epoch numbers in the system whereas the variable dt is ISO8601 formatted string. Hence we do not query by that. Something that does not seem normal and might need a tweak during storage.

_id: "922c8585-65a0-4499-8bc8-3ca75157e11b"
_type: "org.doogle.dataindexdate.Dataindexdate"

metadata: Object
    lastUpdate: 1668867224856
    userTasks: [...]
    processInstances: 
      [ 
         { id: "922c8585-65a0-4499-8bc8-3ca75157e11b",
           processId: "dataindexdate",
           processName: "dataindexdate" ,
           state: 1,
           endpoint: "http://localhost:8080//dataindexdate",
           start: 1668867224773,
           lastUpdate: 1668867224856,
           businessKey: "datemongocheck1" }
      ]

dt: "2022-03-10T11:02:03.123+05:30"
abc: "datemongocheck1"

As you can see dt=2022-03-10T11:02:03.123+05:30 while lastUpdate: 1668867224856, and start: 1668867224773
I feel the di query does not work for date time for variables. It might be a mongodb thing, but just wanted to know how to fix this.
domain cache

di query result

{
  notworking: Dataindexdate(where: {dt:{greaterThan:"2022-03-09T00:00:00.000+05:30"}})
  {
    abc
    dt
    id
    metadata{
      lastUpdate
      processInstances{
        id
        lastUpdate
        businessKey
      }
    }
  }
  working: Dataindexdate(where: {metadata: {lastUpdate:{greaterThan:"2022-03-09T00:00:00.000+05:30"}}})
  {
    abc
    dt
    id
    metadata{
      lastUpdate
      processInstances{
        id
        lastUpdate
        start
        businessKey
      }
    }
  }
}

response:

{
  "data": {
    "notworking": [],
    "working": [
      {
        "abc": "datemongocheck1",
        "dt": "2022-03-10T05:32:03.123Z",
        "id": "922c8585-65a0-4499-8bc8-3ca75157e11b",
        "metadata": {
          "lastUpdate": "2022-11-19T14:13:44.856Z",
          "processInstances": [
            {
              "id": "922c8585-65a0-4499-8bc8-3ca75157e11b",
              "lastUpdate": "2022-11-19T14:13:44.856Z",
              "start": "2022-11-19T14:13:44.773Z",
              "businessKey": "datemongocheck1"
            }
          ]
        }
      }
    ]
  }
}

now for some transformation breaks the same in during the query
I do not see issue with transforming the date from epoch. But just that we do not have option to query on the date and the result never show up.

[debu999]

@nmirasch was there any findings on this and any advise on this discrepancy.

[nmirasch]

@debu999 from the dataindex point of view, what we are doing is adding the received variables where dates come as a String into a graphql Date type. Now we can see those fields as Date at graphql, even before they were mapped as Strings, and AFAIK graphql date filters are supposed to work with that kind of fields.
If this issue is related with how they are stored in mongodb, maybe what is needed is to review how that objects are mapped to the model with mongodb persistence regarding the Domain data is mapped different as the predefined entities like ProcessInstance, Jobs or UserTasks.

[debu999]

Is there a way to get the query fired to fetch data from mongodb. I did try with log level TRACE. But no query gets printed. To understand why it's not working might lie in the query being built to fetch data.

[nmirasch]

@debu999 Usually we are using the DEBUG level to print the executed query, but reviewing the code I don't see that is included when executing query in mongodb.

[debu999]

@nmirasch did we get any update on date filter issue for variables

[debu999]

Do we need to look at mongodb storage project

[nmirasch]

@debu999 We have added some logging to mongodb query execution to have more information about what's exactly the problem with that filtering (https://issues.redhat.com/browse/KOGITO-8292)

[debu999]

@nmirasch
find below the details.
As you can see in the di vs the translated the call I got the issue now.
When we try to query any variable via data index it converts the string to long millis and thus the same is breaking in mongo db query.
For some reason the variable is stored as string instead of number as it should be. Can you have a look

{"namespace": "kogito.dataindexdate_domain", "indexFilterSet": false, "parsedQuery": {"input.ldt": {"$gt": 1646928950000}}, "queryHash": "4BD4E2C6", "planCacheKey": "4D07EF24", "maxIndexedOrSolutionsReached": false, "maxIndexedAndSolutionsReached": false, "maxScansToExplodeReached": false, "winningPlan": {"stage": "COLLSCAN", "filter": {"input.ldt": {"$gt": 1646928950000}}, "direction": "forward"}, "rejectedPlans": []}

See the details.

[debu999]

Any update on this @nmirasch