Broken encoding in response body if application is running on OpenShift

[fedinskiy]

Describe the bug

Given:

1. Application uses quarkus-resteasy-reactive-jackson
2. The application is run on OpenShift
3. Application responds to HTTP request with response, which has HTTP status 200 and word "Slovník" in the body(the 6th letter is U+00ED, "LATIN SMALL LETTER I WITH ACUTE")

Expected behavior

The client should receive http response with "Slovník" in the body.

Actual behavior

The client receives http response with "Slovn?k" in the body.

How to Reproduce?

Reproducer:
https://github.com/fedinskiy/quarkus-test-suite/blob/openshift_fail/sql-db/hibernate-reactive/src/test/java/io/quarkus/ts/reactive/AbstractReactiveDatabaseIT.java#L37

Minimal reproducer:
https://github.com/fedinskiy/quarkus-reproducer/tree/openshift_fail
Scripts run.sh and stop.sh contain commands to start and stop the application on the OCP cluster respectively.
Run curl ${path to your app}/hello/broken to get the result.

Output of uname -a or ver

4.18.0-305.el8.x86_64

Output of java -version

Java version: 11.0.12, vendor: Oracle Corporatio

GraalVM version (if different from Java)

No response

Quarkus version or git rev

2c1be38

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.3 (ff8e977a158738155dc465c6a97ffaf31982d739)

Additional information

OpenShift 4.9

[quarkus-bot]

/cc @geoand, @iocanel

[geoand]

Do you have a cluster I could access to reproduce this?
Furthermore, what are the exact steps to reproduce it?

[fedinskiy]

@geoand the issue was found in this Jenkins run[1], so the answers would be:

1. OCP at https://api.ocp49.dynamic.quarkus:6443 is the best thing I can offer.
2. to run the Jenkins job again. Or, precisely, to run test OpenShiftPostgresql10IT#testI18N from io.quarkus.ts.qe:hibernate-reactive module, while being connected to an OCP cluster.

[1] https://main-jenkins-csb-quarkusqe.apps.ocp-c1.prod.psi.redhat.com/job/quarkus-main-rhel8-jdk11-openshift-new-ts-jvm/159/consoleFull

[geoand]

BTW, +100 for Thinking fast and slow in the tests :)

[geoand]

I test a sample RESTEasy Reactive application on Openshift 4.9 that simply does the following:

@Path("/hello")
public class TestResource {
    @GET
    public Multi<String> hello() {
        return Multi.createFrom().items("Γιώργος", "Slovník");
    }
}

and the response contains the proper characters.

So in short, I can't reproduce this.

[fedinskiy]

@geoand I wasn't able to reproduce the error on your example either. But this one worked:

@Path("/hello")
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
public class TestResource {

    @GET
    @Path("constant")
    public Uni<Response> hello() {
        return Uni.createFrom().item(Response.ok("Slovník").build());
    }
}

I presume, the gamechanger is javax.ws.rs.Produces annotation.

[geoand]

I still could not reproduce it

[stuartwdouglas]

Any chance you can capture the actual data that is being sent over the wire (e.g. via wireshark)?

[fedinskiy]

@stuartwdouglas sniffing VPN traffic is beyond my wireshark-fu capabilities, but I managed to capture request and response with curl's tracing. Results of --trace and --trace-ascii can be found here: https://gist.github.com/fedinskiy/567f6ccd14f83558d5b281268f2ca49e
The body looks like that 53 6c 6f 76 6e 3f 6b, so server output indeed contains question mark symbol('?', 3F in Unicode)

[gsmet]

@fedinskiy does your Quarkus app run with a non default locale/encoding?

[fedinskiy]

@gsmet I do not pass any parameters to the application, and locale on the openshift node looks like that:

sh-4.4$ locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

[gsmet]

I wouldn't be surprised if it could be related to using a non-UTF-8 locale. I have no idea what should be used as the default charset for content types if you're not defining any charset in it.

[fedinskiy]

Well, IETF considers[1] ISO-8859-1(yikes!) as a default, but recommends[2] to use UTF-8. To my understanding, this particular example should work with both, but the latter option looks like a better idea.

As a side note: since Quarkus is advertised as a Kubernetes-native framework[3], novice users(e.g. yours truly) expect that behaviour of their applications doesn't depend on the environment.

[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.7.1
[2] https://www.w3.org/International/questions/qa-choosing-encodings
[3] https://quarkus.io/kubernetes-native/

P.S. I am not sure, if anyone gets notifications about edits in the original post, so I will add it here: there is also a "minimal" reproducer: https://github.com/fedinskiy/quarkus-reproducer/tree/openshift_fail .
Scripts run.sh and stop.sh contain commands to start and stop the application on the OCP cluster respectively.
Run curl ${path to your app}/hello/broken to get the result.

[geoand]

I am not sure, if anyone gets notifications about edits in the original post

nope, GitHub does not notify on updates

[geoand]

The version of Quarkus you are using in the reproducer is very outdated. Can you try with 2.6.1.Final?

[fedinskiy]

Updated quarkus version in the reproducer. The result is the same.

[gsmet]

@geoand did you try locally with a POSIX locale?

[geoand]

I will give it a shot

[geoand]

So indeed changing the locale makes this. I think I know what is causing it

[geoand]

#22644 fixes the issue