Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add quotes for all values listed in https://yaml.org/type/bool.html - Backport #137 for 2.9 #138

Conversation

GuillaumeSmaha
Copy link
Contributor

Fix #129 - Backport of #137 for 2.9

@GuillaumeSmaha GuillaumeSmaha force-pushed the backport_protect_value_to_avoid_implict_convertion branch from ed5c229 to c46c372 Compare June 28, 2019 04:27
GuillaumeSmaha added a commit to GuillaumeSmaha/openapi-generator that referenced this pull request Jun 28, 2019
 quotes around string value and fix compatibility with parser using yaml 1.1.

Fix OpenAPITools#3196

In spec 1.1, some words without quotes can be implicitly converted to
 a boolean like "yes" or "on" (See https://yaml.org/type/bool.html)
In v3.core.util.Yaml with MINIMIZE_QUOTES enabled, the formatter created
 a YAML but forget to protect some words.

PRs are created to fix this in the formatter:
 - FasterXML/jackson-dataformats-text#137
 - FasterXML/jackson-dataformats-text#138

Until the PRs will be merged and released, this patch fix it by
re-adding quotes.
@GuillaumeSmaha
Copy link
Contributor Author

@cowtowncoder What do think to backport the fix in 2.9.x ?

@cowtowncoder
Copy link
Member

@GuillaumeSmaha at this point I'd like to minimize backporting of fixes for stability -- so although this is not necessarily high-risk fix, I am not planning to backport it.

In related news, I just released 2.10.0.pr1 which while not production release, is not a snapshot and can be tested, even used.

@GuillaumeSmaha
Copy link
Contributor Author

Thank you, I just wanted to know.
I am closing this PR.

@cowtowncoder
Copy link
Member

Sure, that's a valid question.

But also: I am really interested in getting feedback -- so if you (or someone you know) can run existing regression tests for some system that is using Jackson 2.9, that'd be great. I need to post to mailing lists about this, as well as write a blog post about new features of course. But this is good time to figure out minor compatibility issues that are easy to resolve before 2.10.0 final.

@GuillaumeSmaha
Copy link
Contributor Author

My aim was to fix this issue in OpenAPI generator project.
I will try 2.10.0 and give you a feedback about it.

@GuillaumeSmaha
Copy link
Contributor Author

GuillaumeSmaha commented Jul 20, 2019

@cowtowncoder I tried on swagger-core, here my branch: https://github.com/GuillaumeSmaha/swagger-core/tree/upgrade_fasterxml_jackson

My first problem was that I forgot yo upgrade snakeyaml from 1.18 to 1.24.
After that, I now an issue with one test in the JSON deserialization: https://github.com/GuillaumeSmaha/swagger-core/blob/upgrade_fasterxml_jackson/modules/swagger-core/src/test/java/io/swagger/v3/core/deserialization/JsonDeserializationTest.java#L253

15:40:31.555 [main] ERROR i.s.v3.core.util.ReflectionUtils - Failed to resolve 'FakeType' into class
java.lang.ClassNotFoundException: FakeType
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	at io.swagger.v3.core.util.ReflectionUtils.loadClassByName(ReflectionUtils.java:49)
	at io.swagger.v3.core.util.ReflectionUtils.typeFromString(ReflectionUtils.java:31)
	at io.swagger.v3.core.util.reflection.ReflectionUtilsTest.typeFromStringTest(ReflectionUtilsTest.java:27)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:648)
	at org.testng.TestRunner.run(TestRunner.java:505)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
	at org.testng.TestNG.runSuites(TestNG.java:1049)
	at org.testng.TestNG.run(TestNG.java:1017)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:193)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:94)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
15:40:31.556 [main] ERROR i.s.v3.core.util.ReflectionUtils - Failed to resolve 'null' into class
java.lang.NullPointerException: null
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:315)
	at io.swagger.v3.core.util.ReflectionUtils.loadClassByName(ReflectionUtils.java:47)
	at io.swagger.v3.core.util.ReflectionUtils.typeFromString(ReflectionUtils.java:31)
	at io.swagger.v3.core.util.reflection.ReflectionUtilsTest.typeFromStringTest(ReflectionUtilsTest.java:28)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
	at org.testng.internal.Invoker.invokeMethod(Invoker.java:580)
	at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:716)
	at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:988)
	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
	at org.testng.TestRunner.privateRun(TestRunner.java:648)
	at org.testng.TestRunner.run(TestRunner.java:505)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
	at org.testng.SuiteRunner.run(SuiteRunner.java:364)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1208)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1137)
	at org.testng.TestNG.runSuites(TestNG.java:1049)
	at org.testng.TestNG.run(TestNG.java:1017)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:193)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:94)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
[ERROR] Tests run: 318, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.748 s <<< FAILURE! - in TestSuite
[ERROR] deserializeLongSchema(io.swagger.v3.core.deserialization.JsonDeserializationTest)  Time elapsed: 0.03 s  <<< FAILURE!
com.fasterxml.jackson.databind.JsonMappingException: 
Numeric value (3147483647) out of range of long (-9223372036854775808 - 9223372036854775807)
 at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: io.swagger.v3.oas.models.PathItem["get"]->io.swagger.v3.oas.models.Operation["parameters"]->java.util.ArrayList[0]->io.swagger.v3.oas.models.parameters.PathParameter["schema"]) (through reference chain: io.swagger.v3.oas.models.OpenAPI["paths"])
	at io.swagger.v3.core.deserialization.JsonDeserializationTest.deserializeLongSchema(JsonDeserializationTest.java:253)
Caused by: java.lang.IllegalArgumentException: 
Numeric value (3147483647) out of range of long (-9223372036854775808 - 9223372036854775807)
 at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: io.swagger.v3.oas.models.PathItem["get"]->io.swagger.v3.oas.models.Operation["parameters"]->java.util.ArrayList[0]->io.swagger.v3.oas.models.parameters.PathParameter["schema"])
	at io.swagger.v3.core.deserialization.JsonDeserializationTest.deserializeLongSchema(JsonDeserializationTest.java:253)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: 
Numeric value (3147483647) out of range of long (-9223372036854775808 - 9223372036854775807)
 at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: io.swagger.v3.oas.models.PathItem["get"]->io.swagger.v3.oas.models.Operation["parameters"]->java.util.ArrayList[0]->io.swagger.v3.oas.models.parameters.PathParameter["schema"])
	at io.swagger.v3.core.deserialization.JsonDeserializationTest.deserializeLongSchema(JsonDeserializationTest.java:253)
Caused by: com.fasterxml.jackson.core.exc.InputCoercionException: 
Numeric value (3147483647) out of range of long (-9223372036854775808 - 9223372036854775807)
 at [Source: UNKNOWN; line: -1, column: -1]
	at io.swagger.v3.core.deserialization.JsonDeserializationTest.deserializeLongSchema(JsonDeserializationTest.java:253)

I am checking fasterxml aand snakeyaml with their changelog: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.10 and https://bitbucket.org/asomov/snakeyaml/wiki/Changes but if you have any idea, feel free to share :)

@GuillaumeSmaha
Copy link
Contributor Author

GuillaumeSmaha commented Jul 20, 2019

It seems the big number thows now an exception, maybe related to FasterXML/jackson-core@c4dd84e
I disable it to see the other tests and now all key with number have double quotes around (200: becomes "200":) even with the parameter YAMLGenerator.Feature.MINIMIZE_QUOTES enabled ; is it normal ?

@cowtowncoder
Copy link
Member

My initial reaction is that the problem would be related to added Module Info in 2.10 (which 2.9 didn't have), and your running things on JDK 9+ or later? If so, stricter visibility checks may be imposed by JDK, without changes to either SnakeYAML or Jackson.

Now... those exception messages are highly confusing, but it could just be reporting wrong range (I think value is outside of int, not long range). I'd love a self-container (or just smaller) reproduction.
And yes, range checks may be stricter now, in sense that what may have been quietly truncated in the past is now consistently checked -- checks were added in places where they should have been (that is sense that checks existed in many places already, and should have been everywhere).
But from user perspective it may unexpected change, still.

As to Map keys (property names), I think there was a change but can not recall details. It's probably a bit complicated issue, wrt what exactly is attempted.

@cowtowncoder
Copy link
Member

@GuillaumeSmaha Actually, on quoting of Map keys, I'd be interested in knowing more about your special usage. The challenge here is to separate intents so that:

  1. If caller wants to write String key, then value like "100" absolutely should be quoted because otherwise YAML parser may decode it as number
  2. if caller wants to write integer/long key -- for which it should use method JsonGenerator.writeFieldId(...) (added in 2.8) -- value MUST NOT be quoted, to similarly retain typing

But to get all of this working, there is a bit of work to do. I think I will actually file a new issue to ensure that YAML generator at least implements writeFieldId(...) to avoid escaping values.
Trick however is to find a use case where this matters... so it'd be great if you had something that showcases the issue.

@GuillaumeSmaha
Copy link
Contributor Author

@cowtowncoder I just saw your commit 456f36f
Thank you, I keep looking to the other test of swagger-core.

@GuillaumeSmaha
Copy link
Contributor Author

Just to inform you. I have other error like that:

[ERROR] processModelWithPairProperties(io.swagger.v3.core.converting.ModelConverterTest)  Time elapsed: 0.01 s  <<< FAILURE!
java.lang.NoClassDefFoundError: org/yaml/snakeyaml/events/Event
	at io.swagger.v3.core.converting.ModelConverterTest.processModelWithPairProperties(ModelConverterTest.java:185)
Caused by: java.lang.ClassNotFoundException: org.yaml.snakeyaml.events.Event
	at io.swagger.v3.core.converting.ModelConverterTest.processModelWithPairProperties(ModelConverterTest.java:185)

[ERROR] testLocalTime(io.swagger.v3.core.resolving.Ticket2992Test)  Time elapsed: 0.013 s  <<< FAILURE!
java.lang.NoClassDefFoundError: org/yaml/snakeyaml/error/YAMLException
	at io.swagger.v3.core.resolving.Ticket2992Test.testLocalTime(Ticket2992Test.java:24)
Caused by: java.lang.ClassNotFoundException: org.yaml.snakeyaml.error.YAMLException
	at io.swagger.v3.core.resolving.Ticket2992Test.testLocalTime(Ticket2992Test.java:24)

I will look more in details the change done in 2.10 and see if it is related to this change.

@cowtowncoder
Copy link
Member

Interesting; that could be related to shading of SnakeYAML.

@GuillaumeSmaha
Copy link
Contributor Author

Yes I think also it can be related to SnakeYAML. I will dig in the diff between 1.23 and 1.24

@cowtowncoder
Copy link
Member

One additional note: I think this could be due to Jackson not catching and rethrowing some failures (done to insulate app calling Jackson from internal details of using snakeyaml), so it may be combination of issues. I should be able to add missing handling, if any, to YAML module quite easily, if stack trace can show the call path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants