[CI] GeoJdbcSqlSpecIT sometimes fails in static code

[droberts195]

There is an intermittent failure that is reported as being in org.elasticsearch.xpack.sql.qa.single_node.GeoJdbcSqlSpecIT classMethod.

An example is https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=oraclelinux-6/201/console

The exception is:

org.junit.AssumptionViolatedException: JTS inside H2 is using default local for toUpperCase() in string comparison making it fail to parse WKT on certain locales
	at com.carrotsearch.randomizedtesting.RandomizedTest.assumeTrue(RandomizedTest.java:725)
	at org.apache.lucene.util.LuceneTestCase.assumeTrue(LuceneTestCase.java:874)
	at org.elasticsearch.xpack.sql.qa.geo.GeoSqlSpecTestCase.lambda$static$0(GeoSqlSpecTestCase.java:34)
	at org.elasticsearch.xpack.sql.qa.jdbc.LocalH2.before(LocalH2.java:68)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:46)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:748)

I guess it means assumptions inside static initialisation code are not treated differently to other exceptions, and we need to think of an alternative way to mute the problem.

The assumption was recently added in #42200. An example of a build that suffered the underlying problem that caused the muting attempt is https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+intake/1431/

[elasticmachine]

Pinging @elastic/es-search

[henningandersen]

Same problem here:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=oraclelinux-7/207

[tvernum]

For SAML and Kerberos where we have similar issues with 3rd party libaries that are locale dependent we reset the locale to English if it's an unsupported one.
https://github.com/elastic/elasticsearch/blob/7.2/x-pack/plugin/security/src/test/java/org/elasticsearch/xpack/security/authc/saml/SamlTestCase.java#L38-L61

[matriv]

Two more failures here:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+intake/1533/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=sles-12/219/console

Reproduce

./gradlew :x-pack:plugin:sql:qa:single-node:integTestRunner --tests "org.elasticsearch.xpack.sql.qa.single_node.GeoJdbcSqlSpecIT" \
  -Dtests.seed=AEAA4DEF5937355A \
  -Dtests.security.manager=true \
  -Dtests.locale=en-US \
  -Dtests.timezone=UTC \
  -Dcompiler.java=12 \
  -Druntime.java=8

Example relevant log:

13:24:16 > Task :x-pack:plugin:sql:qa:single-node:integTestRunner
13:24:16 Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
13:24:35 
13:24:35 org.elasticsearch.xpack.sql.qa.single_node.GeoJdbcSqlSpecIT > classMethod FAILED
13:24:35     org.junit.AssumptionViolatedException: JTS inside H2 is using default local for toUpperCase() in string comparison making it fail to parse WKT on certain locales
13:24:35 
13:24:35 org.elasticsearch.xpack.sql.qa.single_node.GeoJdbcSqlSpecIT > classMethod FAILED
13:24:35     junit.framework.AssertionFailedError: Clean up static fields (in @AfterClass?) and null them, your test still has references to classes of which the sizes cannot be measured due to security restrictions or Java 9 module encapsulation:
13:24:35       - public static org.elasticsearch.xpack.sql.qa.jdbc.LocalH2 org.elasticsearch.xpack.sql.qa.geo.GeoSqlSpecTestCase.H2
13:24:35 
13:24:35         Caused by:
13:24:35         java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.nio.cs")
13:24:35 
13:24:35 
13:24:35 Suite: Test class org.elasticsearch.xpack.sql.qa.single_node.GeoJdbcSqlSpecIT
13:24:35   2> org.junit.AssumptionViolatedException: JTS inside H2 is using default local for toUpperCase() in string comparison making it fail to parse WKT on certain locales
13:24:35   2> junit.framework.AssertionFailedError: Clean up static fields (in @AfterClass?) and null them, your test still has references to classes of which the sizes cannot be measured due to security restrictions or Java 9 module encapsulation:
13:24:35       - public static org.elasticsearch.xpack.sql.qa.jdbc.LocalH2 org.elasticsearch.xpack.sql.qa.geo.GeoSqlSpecTestCase.H2
13:24:35 
13:24:35         Caused by:
13:24:35         java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.nio.cs")
13:24:35   2> REPRODUCE WITH: ./gradlew :x-pack:plugin:sql:qa:single-node:integTestRunner --tests "org.elasticsearch.xpack.sql.qa.single_node.GeoJdbcSqlSpecIT" -Dtests.seed=AEAA4DEF5937355A -Dtests.security.manager=true -Dtests.locale=en-US -Dtests.timezone=UTC -Dcompiler.java=12 -Druntime.java=8
13:24:35   2> NOTE: test params are: codec=Lucene80, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@38e8341), locale=tr, timezone=Africa/Lagos
13:24:35   2> NOTE: Linux 4.12.14-95.16-default amd64/Oracle Corporation 1.8.0_212 (64-bit)/cpus=16,threads=1,free=372552456,total=514850816
13:24:35   2> NOTE: All tests run in this JVM: [CliErrorsIT, CliExplainIT, CliFetchSizeIT, CliSelectIT, CliShowIT, GeoJdbcCsvSpecIT, GeoJdbcSqlSpecIT]
13:27:12 
13:27:12 1284 tests completed, 2 failed, 31 skipped
13:27:12 
13:27:12 Tests with failures:
13:27:12  - org.elasticsearch.xpack.sql.qa.single_node.GeoJdbcSqlSpecIT.classMethod

[imotov]

I am on it. Interesting, somehow it works when I run it on command line with this parameter, but fails when it runs in CI. I have an idea how to fix it though.