[CI] Failure in BWC on debian 8 tests with `_cat/nodes`. Error: Values less than -1 bytes are not supported #54415

benwtrent · 2020-03-30T12:29:25Z

Multiple BWC tests fail in similar ways.

Values less than -1 bytes are not supported: -44383772672b

This occurs in BWC tests with 7.7.

Some failures:

https://gradle-enterprise.elastic.co/s/rpa3bpzgm7bkg

https://gradle-enterprise.elastic.co/s/or7qqkbuuazto

Here is one of the many stack traces:

Full stack trace

org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:38624], URI [/_cat/nodes?h=master%2Cversion&error_trace=true], status line [HTTP/1.1 400 Bad Request]
    {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -46886875136b","stack_trace":"[Values less than -1 bytes are not supported: -46886875136b]; nested: IllegalArgumentException[Values less than -1 bytes are not supported: -46886875136b];
 org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)
 org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)
 org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)
 org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
 org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)
 org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)
 org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:49)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
 org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)
 org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)
 org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)
 org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
 org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
 org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)
 org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)
 org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)
 org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)
 org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
 io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
 io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
 io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
 io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
 io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
 io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
 io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
 io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
 io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
 io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
 io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -46886875136b
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:80)
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:75)
 org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:256)
 org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:316)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:116)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:113)
 org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
 org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
\t... 50 more
"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -46886875136b","stack_trace":"java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -46886875136b
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:80)
 org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:75)
 org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:256)
 org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:316)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:116)
 org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:113)
 org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
 org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)
 org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
 org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)
 org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)
 org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)
 org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)
 org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
 org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
 org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)
 org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)
 org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)
 org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)
 org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
 io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
 io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
 io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
 io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
 io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
 io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
 io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
 io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
 io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
 io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
 io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
 io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
 io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 java.base/java.lang.Thread.run(Thread.java:832)
"},"status":400}
        at __randomizedtesting.SeedInfo.seed([30DEEF650CBC9654:34BFE11467792139]:0)
        at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
        at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
        at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.lambda$waitForTemplates$0(XPackRestTestHelper.java:48)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:905)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:890)
        at org.elasticsearch.xpack.test.rest.XPackRestTestHelper.waitForTemplates(XPackRestTestHelper.java:44)

The text was updated successfully, but these errors were encountered:

elasticmachine · 2020-03-30T12:29:27Z

Pinging @elastic/es-core-features (:Core/Features/CAT APIs)

benwtrent · 2020-03-30T12:34:06Z

Here are some nifty server logs:
659.tar.gz

benwtrent · 2020-03-30T12:34:49Z

Related to PR: #54259

dakrone · 2020-03-30T15:23:18Z

@gwbrown I think we may need a way to be more lenient for BWC when receiving values from other nodes, as otherwise users may run into this problem with mixed version clusters, what do you think?

dakrone · 2020-03-30T16:04:38Z

Oh nevermind! sorry @gwbrown, I read this as relating to the TimeValue change but this is bytes, so ignore this!

gwbrown · 2020-03-30T16:41:03Z

Ha, I made the same mistake at first, then started wondering if I'd done some work on preventing negative byte values and just forgotten. Happy to help if you want assistance, of course!

ywangd · 2020-04-01T02:23:00Z

Another failure today:
https://gradle-enterprise.elastic.co/s/wthzhjyetha7c

Seems to be specific 7.x and debian08

https://gradle-enterprise.elastic.co/scans/tests?list.size=50&list.sortColumn=startTime&list.sortOrder=desc&search.buildToolType=gradle&search.buildToolType=maven&search.startTimeMax=1585699386158&search.startTimeMin=1585094586151&search.tags=CI&search.tzOffset=660&tests.container=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT&tests.sortField=FAILED&tests.test=test%20%7Bp0%3Dold_cluster/10_basic/Create%20a%20task%20result%20record%20in%20the%20old%20cluster%7D&tests.unstableOnly=true

williamrandolph · 2020-04-02T14:05:37Z

It also appears that the affected tests are UpgradeClusterClientYamlTestSuiteIT and MlConfigIndexMappingsFullClusterRestartIT#testMlConfigIndexMappingsAfterMigration. \

Now that we have an error trace on the response, there's a stack trace in the failing response for the ML test that we didn't have when this ticket was filed:

java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -46886875136b
	at org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:80)
	at org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:75)
	at org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:256)
	at org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:316)
	at org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:116)
	at org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:113)
	at org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
	at org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:70)
	at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43)
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:236)
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:213)
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:191)
	at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1129)
	at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:222)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)
	at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:214)
	at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:139)
	at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)
	at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)
	at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:832)

It appears that this error happens when OsStats.Mem has a larger value for free memory than for total memory:

        public ByteSizeValue getUsed() {
            return new ByteSizeValue(total - free);
        }

The first failure was on March 25th, not long after we cut the 7.7 branch. There have been consistent failures in the twice-daily bwc run for debian 8. Chart: gradle enterprise

mayya-sharipova · 2020-04-03T14:15:54Z

Recent failures for today on debian-8 and 7.x:

Log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+7.x+multijob-unix-compatibility/os=debian-8&&immutable/667/console
Build scan: https://gradle-enterprise.elastic.co/s/zgglcpzvad2m6

Failing tests:
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/10_basic/Create a task result record in the old cluster}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/10_basic/Create things in the cluster state that we'll validate are there after the upgrade}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/10_basic/Index percolator queries and use the percolate query in old cluster}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/20_date_range/Create index with java style index in 6}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/20_date_range/Create index with joda style index that is incompatible with java.time (>6.1)}
org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT test {p0=old_cluster/20_date_range/Create index with joda style index that is incompatible with java.time. (6.0)}
org.elasticsearch.xpack.restart.MlConfigIndexMappingsFullClusterRestartIT testMlConfigIndexMappingsAfterMigration

The same error message:
method [GET], host [http://127.0.0.1:44582], URI [/_cat/nodes?h=version%2Cmaster], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -48852353024b"}]

dimitris-athanasiou · 2020-04-06T13:54:59Z

Another failure: Jenkins, Build scan

romseygeek · 2020-04-14T13:49:40Z

And another: https://gradle-enterprise.elastic.co/s/oauuxgrtd6i5e/console-log?task=:x-pack:qa:full-cluster-restart:v7.7.0%23oldClusterTest

cbuescher · 2020-04-16T12:15:25Z

Another instance of the the MlConfigIndexMappingsFullClusterRestartIT.testMlConfigIndexMappingsAfterMigration failure on 7.x:
https://gradle-enterprise.elastic.co/s/i73lmau2ago5w

rg.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:59558], URI [/_cat/nodes?h=master%2Cversion&error_trace=true], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -43149541376b","stack_trace":"[Values less than -1 bytes are not supported: -43149541376b]; nested: IllegalArgumentException[Values less than -1 bytes are not supported: -43149541376b];
at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)
at org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)
at org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)
at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)\n\tat

tlrx · 2020-04-17T12:32:12Z

It happened again today on 7.x:

https://gradle-enterprise.elastic.co/s/negka76witrbo

This test is failing twice a day for some time now so I'm going to mute it (#55389).

I'm also assigning this issue to @jakelandis so that he can assign it to someone from the core/feature team.

…ngsAfterMigration (#55389) Relates #54415

ywangd · 2020-04-24T00:51:34Z

Failed again: https://gradle-enterprise.elastic.co/s/hsoc4hi5sf5p6 (7.x, debian)
Same issue as the one from my last shift.

test {p0=old_cluster/20_date_range/Create index with joda style index that is incompatible with java.time (>6.1)} 
test {p0=old_cluster/10_basic/Index percolator queries and use the percolate query in old cluster}
test {p0=old_cluster/20_date_range/Create index with joda style index that is incompatible with java.time. (6.0)}
test {p0=old_cluster/10_basic/Create a task result record in the old cluster}
test {p0=old_cluster/10_basic/Create things in the cluster state that we'll validate are there after the upgrade
test {p0=old_cluster/20_date_range/Create index with java style index in 6}

Error message is {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -40171786240b"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -40171786240b"},"status":400}

jkakavas · 2020-05-06T14:07:10Z

Still happening: https://gradle-enterprise.elastic.co/s/pzl2mgclmc2zq

andreidan · 2020-05-07T12:02:26Z

Still happening: https://gradle-enterprise.elastic.co/s/wyh6iqz76h54u

tvernum · 2020-05-08T03:28:06Z

Another one: https://gradle-enterprise.elastic.co/s/j6vigxs56aes2/

I'm going to try and mute more tests for this.

tvernum · 2020-05-08T06:22:23Z

I couldn't see a way to mute this easily, and I also tried to work out how this could happen, but didn't get anywhere.

Given that it's a problem on 7.x debian-8 only, I suspect this is actually an issue with the OS Management Bean on Java 8, and we should just work around it by detecting that free > total and return 0 or -1.

We could try and work out why, but I suspect that will just end up finding that it's a JDK bug we can't do anything about

danhermann · 2020-05-08T10:58:16Z

I spent some time on this yesterday and came to a similar conclusion regarding detecting free > total situations. The one possibility that I could come up with was that while @dakrone demonstrated in #42725 that negative values can be returned for either total or free memory, those occurrences may not coincide such that total is reported as negative but free is reported as positive thus resulting in a free > total situation. I've opened #56412 as a possible way to address it.

benwtrent · 2020-05-11T11:37:22Z

https://gradle-enterprise.elastic.co/s/cno7wedfdoqfg

https://gradle-enterprise.elastic.co/s/viy6zvnenjkn4

Same failure, but on 7.7.

Would be good to have the fix backported into 7.7 when possible.

DaveCTurner · 2020-05-14T12:16:40Z

More today:

https://gradle-enterprise.elastic.co/s/fnqczt5dpd7r6 (7.x, testing vs 7.7.1)
https://gradle-enterprise.elastic.co/s/qcsgtthijgysc (7.8, testing vs 7.7.1)
https://gradle-enterprise.elastic.co/s/efruirda6froi (7.x, testing vs 7.7.1)

danhermann · 2020-05-14T14:25:19Z

I will backport the fix for these into 7.7 and 7.8 shortly.

danhermann · 2020-05-18T13:58:25Z

Closing this as the fix has been backported to 7.7 and 7.8 in addition to 7.x and master.

IvanBoyko · 2020-07-09T16:43:36Z

In our case it seemed to be the swap, so switching it off with sudo swapoff -a immediately stoped the warnings.
However, we are ES 6.8.10 on Ubuntu 18.04 (didn't see this on ES 6.8.7)

benwtrent added >test-failure Triaged test failures from CI :Data Management/CAT APIs Text APIs behind /_cat labels Mar 30, 2020

williamrandolph changed the title ~~Failure in BWC tests with _cat/nodes. Error: Values less than -1 bytes are not supported~~ [CI] Failure in BWC on debian 8 tests with _cat/nodes. Error: Values less than -1 bytes are not supported Apr 2, 2020

tlrx mentioned this issue Apr 17, 2020

Mute MlConfigIndexMappingsFullClusterRestartIT.testMlConfigIndexMappingsAfterMigration #55389

Merged

tlrx added a commit that referenced this issue Apr 17, 2020

Mute MlConfigIndexMappingsFullClusterRestartIT.testMlConfigIndexMappi…

290361c

…ngsAfterMigration (#55389) Relates #54415

droberts195 mentioned this issue Apr 19, 2020

Negative used memory in QueryCacheStats causes IllegalArgumentException in _cat/nodes #55434

Closed

martijnvg mentioned this issue Apr 20, 2020

[CI] Failure in MixedClusterClientYamlTestSuiteIT p0=indices.data_stream/10_basic/Create data stream #54272

Closed

rjernst added the Team:Data Management Meta label for data/management team label May 4, 2020

danhermann mentioned this issue May 8, 2020

Report used memory as zero when total memory cannot be obtained #56435

Merged

danhermann added :Data Management/Stats Statistics tracking and retrieval APIs and removed :Data Management/CAT APIs Text APIs behind /_cat labels May 8, 2020

danhermann closed this as completed in #56435 May 8, 2020

benwtrent reopened this May 11, 2020

tvernum mentioned this issue May 14, 2020

IllegalArgumentException from node stats #56739

Closed

danhermann closed this as completed May 18, 2020

davidkyle mentioned this issue Sep 9, 2020

[ML] Unmute full cluster restart mapping test #62148

Closed

jkakavas mentioned this issue Aug 19, 2021

[CI] MixedClusterClientYamlTestSuiteIT test {p0=cluster.stats/10_basic/cluster stats test} failing #76717

Closed

droberts195 mentioned this issue Oct 15, 2021

Allow total memory to be overridden #78750

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Failure in BWC on debian 8 tests with `_cat/nodes`. Error: Values less than -1 bytes are not supported #54415

[CI] Failure in BWC on debian 8 tests with `_cat/nodes`. Error: Values less than -1 bytes are not supported #54415

benwtrent commented Mar 30, 2020 •

edited

Loading

elasticmachine commented Mar 30, 2020

benwtrent commented Mar 30, 2020

benwtrent commented Mar 30, 2020

dakrone commented Mar 30, 2020

dakrone commented Mar 30, 2020

gwbrown commented Mar 30, 2020

ywangd commented Apr 1, 2020

williamrandolph commented Apr 2, 2020

mayya-sharipova commented Apr 3, 2020

dimitris-athanasiou commented Apr 6, 2020

romseygeek commented Apr 14, 2020

cbuescher commented Apr 16, 2020 •

edited

Loading

tlrx commented Apr 17, 2020

ywangd commented Apr 24, 2020

jkakavas commented May 6, 2020

andreidan commented May 7, 2020

tvernum commented May 8, 2020

tvernum commented May 8, 2020

danhermann commented May 8, 2020

benwtrent commented May 11, 2020 •

edited

Loading

DaveCTurner commented May 14, 2020

danhermann commented May 14, 2020

danhermann commented May 18, 2020

IvanBoyko commented Jul 9, 2020

[CI] Failure in BWC on debian 8 tests with _cat/nodes. Error: Values less than -1 bytes are not supported #54415

[CI] Failure in BWC on debian 8 tests with _cat/nodes. Error: Values less than -1 bytes are not supported #54415

Comments

benwtrent commented Mar 30, 2020 • edited Loading

elasticmachine commented Mar 30, 2020

benwtrent commented Mar 30, 2020

benwtrent commented Mar 30, 2020

dakrone commented Mar 30, 2020

dakrone commented Mar 30, 2020

gwbrown commented Mar 30, 2020

ywangd commented Apr 1, 2020

williamrandolph commented Apr 2, 2020

mayya-sharipova commented Apr 3, 2020

dimitris-athanasiou commented Apr 6, 2020

romseygeek commented Apr 14, 2020

cbuescher commented Apr 16, 2020 • edited Loading

tlrx commented Apr 17, 2020

ywangd commented Apr 24, 2020

jkakavas commented May 6, 2020

andreidan commented May 7, 2020

tvernum commented May 8, 2020

tvernum commented May 8, 2020

danhermann commented May 8, 2020

benwtrent commented May 11, 2020 • edited Loading

DaveCTurner commented May 14, 2020

danhermann commented May 14, 2020

danhermann commented May 18, 2020

IvanBoyko commented Jul 9, 2020

[CI] Failure in BWC on debian 8 tests with `_cat/nodes`. Error: Values less than -1 bytes are not supported #54415

[CI] Failure in BWC on debian 8 tests with `_cat/nodes`. Error: Values less than -1 bytes are not supported #54415

benwtrent commented Mar 30, 2020 •

edited

Loading

cbuescher commented Apr 16, 2020 •

edited

Loading

benwtrent commented May 11, 2020 •

edited

Loading