TRIL tests on z/OS generates non-standard NaN formatting

[sarwat12]

While working on PR #7196 , we noticed that adding NaN value handling to the S390 float & double max/min evaluators seems to pass all the MaxMin.cpp tests on Linux on Z, but on z/OS, the tests with NaN operands keeps failing with the following exception:

28: [----------] 1444 tests from MaxMin/FloatMaxMin
28: unknown file: Failure
28: Unknown C++ exception thrown in the test body.
28: [  FAILED  ] MaxMin/FloatMaxMin.UsingConst/20, where GetParam() = ((0, NaNQ(1)), ("\x86\x94" "\x81\xA7", 50088187B0)) (0 ms)
28: unknown file: Failure
28: Unknown C++ exception thrown in the test body.
28: [  FAILED  ] MaxMin/FloatMaxMin.UsingConst/21, where GetParam() = ((0, NaNQ(1)), ("\x86\x94\x89\x95", 50088187A0)) (0 ms)
28: unknown file: Failure
28: Unknown C++ exception thrown in the test body.
28: [  FAILED  ] MaxMin/FloatMaxMin.UsingConst/22, where GetParam() = ((0, -NaNQ(1)), ("\x86\x94" "\x81\xA7", 50088187B0)) (1 ms)
28: unknown file: Failure
28: Unknown C++ exception thrown in the test body.
28: [  FAILED  ] MaxMin/FloatMaxMin.UsingConst/23, where GetParam() = ((0, -NaNQ(1)), ("\x86\x94\x89\x95", 50088187A0)) (0 ms)

The tests are failing most likely due to non-standard nan formatting where the TRIL generator raises an exception:

• std::snprintf("(dconst %f)", NAN) yields (dconst nan) on Linux/GCC, passing all the tests
• std::snprintf("(dconst %f)", NAN) yields (dconst NaNQ(1)) on zOS/XLC, where the tests fail.

This leads to the Float/Double MaxMin tests currently being disabled on z/OS. Opening up this issue to keep track of potential fixes for the correct NaN formatting on z/OS. @r30shah

[janvrany]

This is long known problem. I try to attack it in #5587 "Use custom function to format TRIL code" (not yet merged)

[r30shah]

Thanks @janvrany , @sarwat12 was working on handling NaN's correctly on Z. His PR (#7196) will enable the tril test on Linux on Z as the logic in the evaluator now simply return the operand which is NaN (So we would not generate any constant materialization).
For the z/OS as he said, formatting is the blocking, but the evaluator now have the support so please let us know if you have something for us to try on z/OS.

[janvrany]

Yes, once my PR #5587 gets in (waiting for review) there would have to be hopefully trivial change needed on z/OS to support NaNs - see https://github.com/eclipse/omr/pull/5587/files#diff-b2afab503b2a84d6f6fa84b75d153dd38e53b5b15b40798548af6f0777a67b5cR467-R469

I decided to do one step at a time, first get make drop-in replacement for std::snprintf() and after that address these details in another PRs.