Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve handling edge cases regarding JSONPath / dot notation discrepancies #478

Closed
5 tasks done
ohltyler opened this issue Nov 14, 2024 · 0 comments · Fixed by #481
Closed
5 tasks done

Improve handling edge cases regarding JSONPath / dot notation discrepancies #478

ohltyler opened this issue Nov 14, 2024 · 0 comments · Fixed by #481
Labels
bug Something isn't working v2.19.0

Comments

@ohltyler
Copy link
Member

ohltyler commented Nov 14, 2024

There are some remaining discrepancies in the ML processor transforms regarding dot notation and JSONPath, that can yield different results when rendering examples on the frontend, vs. the execution on the backend. As such, this can result in successful transformations when testing on the frontend, but failures when executing ingest/search. Tracking known discrepancies below:

  • ML search request/response transforms only use JSONPath, and the backend JSONPath package does not support dot notation for arrays. To have feature parity, we should automatically convert any dot notation with arrays (e.g., arrays.0) into dot notation with brackets (e.g., arrays[0])., or see if the functionality to automatically generate the keys, can take in some config to automatically convert to brackets by default, or see if there is some external package that can handle all of this dot -> bracket transformation for us altogether.
  • discrepancies with arrays for single values. Currently, the default JSONPath query command we execute returns an array by default, even if for a discrete/single/definite value. The backend package has a config field called ALWAYS_RETURN_LIST, set to false. We should see if there is a similar config on the frontend side, and/or if we can parse out whether a path is "definite" or not, and do post-processing.
  • enforce output transforms are always using JSONPath as required per the backend.
  • improve tooltip wording to indicate the above discrepancies / limitations
  • keys with dashes in them fail using the nodejs library - for example, given:
{ 
  "field": {
    "sub-field": "value" 
  }
}

and some jsonpath, like $.field.sub-field, will fail to fetch "value". One workaround is to wrap it in brackets with quotes - for example, $.field["sub-field"] will succeed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working v2.19.0
Projects
None yet
1 participant