Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FIX: coercion on json pointer values #1178

Conversation

chenqi0805
Copy link
Collaborator

Signed-off-by: Chen [email protected]

Description

This PR fixes coercion on corner cases in jsonpointer value, especially

  • Double value -> convert to float
  • non literal value -> exception

Issues Resolved

Resolves #1003

Check List

  • New functionality includes testing.
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@chenqi0805 chenqi0805 requested a review from a team as a code owner March 11, 2022 16:32
@codecov-commenter
Copy link

codecov-commenter commented Mar 11, 2022

Codecov Report

Merging #1178 (61a8715) into main (6394263) will increase coverage by 0.07%.
The diff coverage is 100.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##               main    #1178      +/-   ##
============================================
+ Coverage     93.81%   93.88%   +0.07%     
- Complexity     1027     1044      +17     
============================================
  Files           136      139       +3     
  Lines          2925     2962      +37     
  Branches        262      265       +3     
============================================
+ Hits           2744     2781      +37     
  Misses          130      130              
  Partials         51       51              
Impacted Files Coverage Δ
...xpression/LiteralTypeConversionsConfiguration.java 100.00% <100.00%> (ø)
...taprepper/expression/ParseTreeCoercionService.java 100.00% <100.00%> (ø)
...h/dataprepper/expression/UnaryNumericOperator.java 100.00% <0.00%> (ø)
.../expression/UnaryNumericOperatorConfiguration.java 100.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6394263...61a8715. Read the comment docs.

private Object resolveJsonPointerValue(final String jsonPointer, final Event event) {
final Object value = event.get(jsonPointer, Object.class);
if (value == null) {
throw new ExpressionCoercionException("Unsupported type for value " + value);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does a null value throw?

I see that there is no clear default value to use in this case. Maybe just return a new Object or have a NullValue class which represents any null values.

Copy link
Collaborator Author

@chenqi0805 chenqi0805 Mar 11, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right now there is no null value support in the grammar literal types so it is impossible for cx to check jsonpointer value is null in conditional expression. We have two options for null appearing in jsonPointer value:

  • throw Exception
  • return null and let each operator complain about the arg type.

I think both should work and I do not favor one over the other. @sbayer55 thoughts?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So in the current design /message == 200 would have to fail if /message is null? I think this is clearly a false expression. I think if we returned a special NullValue class the rest of the code would work as expected. Right?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this code (currently in PR), I think the equals operator would work correctly if the ParseTreeCoercsionService provided a new Object().

https://github.com/opensearch-project/data-prepper/pull/1177/files#diff-083f9ffc9761303a542e0904c2de3bc5ba2ec7ef612278087bf6da29208ab961R34

We could even do something like this:

class NullValue {
  private static NULL_VALUE = new NullValue();

  private NullValue() { }

  public static nullValue() {
    return NULL_VALUE;
  }
}

And return:

return NullValue.nullValue()


@Named
class ParseTreeCoercionService {
private final List<? extends Class<? extends Serializable>> literalTypes = Arrays.asList(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be static.

throw new ExpressionCoercionException("Unsupported type for value " + value);
} else if (literalTypes.contains(value.getClass())) {
return value;
} else if (value instanceof Double) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This approach would be easier to extend if we have other possible types.

private static final Map<Class, Function<Object, Object>> literalTypes;

static {
  literalTypes = new HashMap<>();
  literalTypes.put(String.class, Function.identity());
  literalTypes.put(Boolean.class, Function.identity());
  literalTypes.put(Integer.class, Function.identity());
  literalTypes.put(Float.class, Function.identity());
  literalTypes.put(Double.class, v -> ((Double) value).floatValue());
}

And here:

else if (literalTypes.containsKey(value.getClass())) {
             return literalTypes.get(value.getClass()).apply(value);

But, this is all internal and later refactoring is light. So I don't have a strong opinion on it.

@chenqi0805 chenqi0805 requested a review from dlvenable March 11, 2022 19:19
dlvenable
dlvenable previously approved these changes Mar 11, 2022

@Named
class ParseTreeCoercionService {
private static final Map<Class<? extends Serializable>, Function<Object, Object>> literalTypes;

static {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be moved to a constructor argument and generated in a bean configuration function? Static variable can make classes difficult to unit test.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me.


static {
literalTypes = new HashMap<>();
literalTypes.put(String.class, Function.identity());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are all of these keys needed? Could we use something like

if (literalTypes.hasKey(value.getClass()))
    return literalTypes..get(value.getClass()).apply(value);
else
    return value;

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that will not work for Long type which we want to throw exception. Also, for other custom types, it will just return literal objects which I am not sure if we want to support. e.g.

/code1 == /code2              | {"code1": StatusCode, "code2": StatusCode}

final Event testEvent = createTestEvent(new HashMap<>());
@ParameterizedTest
@MethodSource("provideUnSupportedJsonPointerValues")
void testCoerceTerminalNodeJsonPointerTypeUnSupportedValues(final Object testValue) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could testValue type be changed to Event and create the events in the provideUnSupportedJsonPointerValues method? I think it would make the test more readable.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually the same method is used for both jsonpointer and escape jsonpointer test case in which keys are separate. So providing the value only will probably make it more compact and focused on the difference (testValue). Thought?

@chenqi0805 chenqi0805 requested a review from dlvenable March 11, 2022 21:24
@chenqi0805 chenqi0805 merged commit 02a9050 into opensearch-project:main Mar 11, 2022
@chenqi0805 chenqi0805 deleted the fix/1003-resolve-json-pointer-value-type-in-coercion-service branch March 11, 2022 23:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Data Prepper Expression Evaluator
4 participants