Add buffered lookahead for Jackson #489
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JSON deserialization requires to look ahead in JSON objects on two occasions:
Property
type, whose variant is defined by the innertype
property.error
property is present and a successful result if thetook
,hits
or a number of other properties are present.The JSON-P API doesn't offer a way to buffer JSON events and replay them, and the initial approach was to parse the JSON stream as a JSON object (map of maps), pick the information we need, and, since the JSON stream had been consumed, serialize it back to create a parser. This is suboptimal but somehow acceptable for small objects.
However, the multisearch response can contain arbitrary large JSON objects containing index documents on which we have to perform a look ahead, and in that case the performance hit can be significant as shown in #471.
This PR introduces the
LookAheadJsonParser
extension to JSON-P'sJsonParser
that expose the look ahead functions we need in the deserizalization framework. And it provides an implementation for Jackson, which is by far the most often used JSON library, based on Jackson'sTokenBuffer
. This allows look ahead to consume the actual parser only until we find the information we need, and buffer the JSON stream in a data structure that can be efficiently replayed.The previous implementation is kept as a fallback for parsers not implementing
LookAheadJsonParser
. It will be improved in a subsequent PR.Fixes #471.