OsProbeTests.testOsStats and memory-related YAML test failures on Debian 8

[gwbrown]

Other associated failures from what appears to be the same root cause of misreported memory by the OS, all on Debian 8 runners:

• ClientYamlTestSuiteIT test {p0=cluster.stats/10_basic/cluster stats test}
• ClientYamlTestSuiteIT test {p0=cluster.stats/10_basic/get cluster stats returns cluster_uuid at the top level}
• DocsClientYamlTestSuiteIT test {yaml=reference/ml/anomaly-detection/apis/get-ml-info/line_41}

Build scan: https://gradle-enterprise.elastic.co/s/z3bger2wnj62m

Repro line: ./gradlew ':server:test' --tests "org.elasticsearch.monitor.os.OsProbeTests.testOsStats" -Dtests.seed=A4A541649A2C4A2C -Dtests.security.manager=true -Dtests.locale=es-PY -Dtests.timezone=Asia/Anadyr -Druntime.java=8

Reproduces locally?: No (on MacOS, don't have a local Debian box)

Applicable branches: 7.x, 7.11, 7.10, 6.8

Failure history: build stats, first recent failure at 2020-12-17T04:22:14.082Z on 7.11
Failure excerpt:

java.lang.AssertionError: 	
Expected: a value greater than <0L>	
     but: <0L> was equal to <0L>	
	at __randomizedtesting.SeedInfo.seed([A4A541649A2C4A2C:2FAF436319C6E61C]:0)	
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)	
	at org.junit.Assert.assertThat(Assert.java:956)	
	at org.junit.Assert.assertThat(Assert.java:923)	
	at org.elasticsearch.monitor.os.OsProbeTests.testOsStats(OsProbeTests.java:123)	
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)	
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)	
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)	
	at java.lang.reflect.Method.invoke(Method.java:498)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)	
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)	
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)	
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)	
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)	
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)	
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)	
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)	
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)	
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)	
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)	
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)	
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)	
	at java.lang.Thread.run(Thread.java:748)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[dimitris-athanasiou]

The failures in https://gradle-enterprise.elastic.co/s/2efkk2gimu5nm seem possibly related with memory misreported too.

[henningandersen]

Want to highlight that there is info in two other places on the cause of this:
#66885 (comment)
#67089 (comment)

I would be inclined to suggest that we add a check for 0 in OsProbe and then on debian 8 go and find the right size in /proc/meminfo.

[danhermann]

Thanks, @henningandersen. I'll look into that.

[ywangd]

These two:

• https://gradle-enterprise.elastic.co/s/sofhxlo2xg374 (7.x)
• https://gradle-enterprise.elastic.co/s/4poxdcostfxwc (7.11)

failed for different tests. But I think the underlying cause is the same, i.e. memory reporting issue on Debian OS. The error message is:

Warnings: [Values less than -1 bytes are deprecated and will not be supported in the next major version: [-28786565120b]]

The failed tests are (list them here so its sear:

org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=mixed_cluster/10_basic/Get index works}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=mixed_cluster/10_basic/Use the percolate query in mixed cluster}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=mixed_cluster/10_basic/Verify custom cluster metadata still exists during upgrade}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=mixed_cluster/10_basic/Verify nodes usage works}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=mixed_cluster/10_basic/Verify that we can still find things with the template}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=mixed_cluster/20_date_range/Insert more docs to java index}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=mixed_cluster/20_date_range/Insert more docs to joda index}

[probakowski]

There's more:

• https://gradle-enterprise.elastic.co/s/d6dh6txrsidf6
• https://gradle-enterprise.elastic.co/s/dyumuls6bn3la
• https://gradle-enterprise.elastic.co/s/rmjg2op2ugfym

[javanna]

heya @danhermann, I wonder if this failure from today is related, I would say so at first glance: https://gradle-enterprise.elastic.co/s/e2hir35m4taug/console-log?task=:server:test .

[danhermann]

@javanna, yes, thanks for the ping. I have opened #68742 to resolve that failure.