Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Negative used memory causes IllegalArgumentException in _cat/nodes #42157

Closed
DaveCTurner opened this issue May 15, 2019 · 1 comment · Fixed by #42725
Closed

Negative used memory causes IllegalArgumentException in _cat/nodes #42157

DaveCTurner opened this issue May 15, 2019 · 1 comment · Fixed by #42725
Assignees
Labels
:Data Management/Stats Statistics tracking and retrieval APIs >test-failure Triaged test failures from CI

Comments

@DaveCTurner
Copy link
Contributor

The following build failed trying to report a negative number of bytes returned from OsStats.Mem#getUsed():

https://scans.gradle.com/s/hvu55m4ctmbd6/console-log?task=:qa:mixed-cluster:v7.2.0%23mixedClusterTestRunner#L675

This was in :qa:mixed-cluster:v7.2.0#mixedClusterTestRunner, and the stack trace looked as follows:

  1> [2019-05-15T12:28:35,981][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] [p0=cat.nodes/10_basic/Test cat nodes output] before test
  1> [2019-05-15T12:28:36,001][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] Stash dump on test failure [{
  1>   "stash" : {
  1>     "body" : {
  1>       "error" : {
  1>         "root_cause" : [
  1>           {
  1>             "type" : "illegal_argument_exception",
  1>             "reason" : "Values less than -1 bytes are not supported: -366415872b",
  1>             "stack_trace" : "[Values less than -1 bytes are not supported: -366415872b]; nested: IllegalArgumentException[Values less than -1 bytes are not supported: -366415872b];
  1>  at org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:639)
  1>  at org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:567)
  1>  at org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)
  1>  at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)
  1>  at org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)
  1>  at org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)
  1>  at org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:49)
  1>  at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)
  1>  at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
  1>  at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:238)
  1>  at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:215)
  1>  at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)
  1>  at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:193)
  1>  at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)
  1>  at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1101)
  1>  at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:224)
  1>  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  1>  at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)
  1>  at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:216)
  1>  at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:141)
  1>  at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:105)
  1>  at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:660)
  1>  at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
  1>  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
  1>  at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
  1>  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
  1>  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
  1>  at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
  1>  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
  1>  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
  1>  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
  1>  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
  1>  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
  1>  at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)
  1>  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)
  1>  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
  1>  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
  1>  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  1>  at java.base/java.lang.Thread.run(Thread.java:834)
  1> Caused by: java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -366415872b
  1>  at org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:72)
  1>  at org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:67)
  1>  at org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:252)
  1>  at org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:305)
  1>  at org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:105)
  1>  at org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:102)
  1>  at org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
  1>  at org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
  1>  ... 40 more
  1> "
  1>           }
  1>         ],
  1>         "type" : "illegal_argument_exception",
  1>         "reason" : "Values less than -1 bytes are not supported: -366415872b",
  1>         "stack_trace" : "java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -366415872b
  1>  at org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:72)
  1>  at org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:67)
  1>  at org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:252)
  1>  at org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:305)
  1>  at org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:105)
  1>  at org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:102)
  1>  at org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)
  1>  at org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)
  1>  at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)
  1>  at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)
  1>  at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:238)
  1>  at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:215)
  1>  at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)
  1>  at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:193)
  1>  at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)
  1>  at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1101)
  1>  at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:224)
  1>  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
  1>  at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)
  1>  at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:216)
  1>  at org.elasticsearch.transport.InboundHan  1> dler.messageReceived(InboundHandler.java:141)
  1>  at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:105)
  1>  at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:660)
  1>  at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
  1>  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
  1>  at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
  1>  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
  1>  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
  1>  at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
  1>  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
  1>  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
  1>  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
  1>  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)
  1>  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
  1>  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)
  1>  at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)
  1>  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)
  1>  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
  1>  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
  1>  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  1>  at java.base/java.lang.Thread.run(Thread.java:834)
  1> "
  1>       },
  1>       "status" : 400
  1>     }
  1>   }
  1> }]
  1> [2019-05-15T12:28:36,030][INFO ][o.e.b.MixedClusterClientYamlTestSuiteIT] [test] [p0=cat.nodes/10_basic/Test cat nodes output] after test
  2> REPRODUCE WITH: ./gradlew :qa:mixed-cluster:v7.2.0#mixedClusterTestRunner --tests "org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=cat.nodes/10_basic/Test cat nodes output}" -Dtests.seed=3246BE5B867D1D1C -Dtests.security.manager=true -Dtests.locale=ckb-IR -Dtests.timezone=America/Argentina/Mendoza -Dcompiler.java=12 -Druntime.java=11
  2> java.lang.AssertionError: Failure at [cat.nodes/10_basic:4]: expected [2xx] status code but api [cat.nodes] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -366415872b","stack_trace":"[Values less than -1 bytes are not supported: -366415872b]; nested: IllegalArgumentException[Values less than -1 bytes are not supported: -366415872b];\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:639)\n\tat org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:567)\n\tat org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)\n\tat org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:49)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:238)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:215)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:193)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1101)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:224)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:216)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:141)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:105)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:660)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\nCaused by: java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -366415872b\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:72)\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:67)\n\tat org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:252)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:305)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:105)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:102)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)\n\t... 40 more\n"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -366415872b","stack_trace":"java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -366415872b\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:72)\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:67)\n\tat org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:252)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:305)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:105)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:102)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:238)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:215)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:193)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1101)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:224)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:216)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:141)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:105)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:660)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"},"status":400}]
        Caused by:
        java.lang.AssertionError: expected [2xx] status code but api [cat.nodes] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -366415872b","stack_trace":"[Values less than -1 bytes are not supported: -366415872b]; nested: IllegalArgumentException[Values less than -1 bytes are not supported: -366415872b];\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:639)\n\tat org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:567)\n\tat org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)\n\tat org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:49)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:238)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:215)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:193)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1101)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:224)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:216)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:141)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:105)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:660)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\nCaused by: java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -366415872b\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:72)\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:67)\n\tat org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:252)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:305)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:105)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:102)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)\n\t... 40 more\n"}],"type":"illegal_argument_exception","reason":"Values less than -1 bytes are not supported: -366415872b","stack_trace":"java.lang.IllegalArgumentException: Values less than -1 bytes are not supported: -366415872b\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:72)\n\tat org.elasticsearch.common.unit.ByteSizeValue.<init>(ByteSizeValue.java:67)\n\tat org.elasticsearch.monitor.os.OsStats$Mem.getUsed(OsStats.java:252)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction.buildTable(RestNodesAction.java:305)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:105)\n\tat org.elasticsearch.rest.action.cat.RestNodesAction$1$1$1.buildResponse(RestNodesAction.java:102)\n\tat org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:37)\n\tat org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:47)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:68)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:64)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.finishHim(TransportNodesAction.java:238)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onOperation(TransportNodesAction.java:215)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$000(TransportNodesAction.java:142)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:193)\n\tat org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleResponse(TransportNodesAction.java:183)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1101)\n\tat org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:224)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:216)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:141)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:105)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:660)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1408)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:930)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:682)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:582)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:536)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:834)\n"},"status":400}]
@DaveCTurner DaveCTurner added :Data Management/Stats Statistics tracking and retrieval APIs >test-failure Triaged test failures from CI labels May 15, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

dakrone added a commit to dakrone/elasticsearch that referenced this issue May 30, 2019
We've had a situation where the MX bean reported negative values for the
free memory of the OS, in those rare cases we want to return a value of
0 rather than blowing up later down the pipeline.

In the event that there is a serialization or creation error with regard
to memory use, this adds asserts so the failure will occur as soon as
possible and give us a better location for investigation.

Resolves elastic#42157
dakrone added a commit that referenced this issue Jun 19, 2019
…42725)

* Return 0 for negative "free" and "total" memory reported by the OS

We've had a situation where the MX bean reported negative values for the
free memory of the OS, in those rare cases we want to return a value of
0 rather than blowing up later down the pipeline.

In the event that there is a serialization or creation error with regard
to memory use, this adds asserts so the failure will occur as soon as
possible and give us a better location for investigation.

Resolves #42157

* Fix test passing in invalid memory value

* Fix another test passing in invalid memory value

* Also change mem check in MachineLearning.machineMemoryFromStats

* Add background documentation for why we prevent negative return values

* Clarify comment a bit more
dakrone added a commit that referenced this issue Jun 19, 2019
…42725)

* Return 0 for negative "free" and "total" memory reported by the OS

We've had a situation where the MX bean reported negative values for the
free memory of the OS, in those rare cases we want to return a value of
0 rather than blowing up later down the pipeline.

In the event that there is a serialization or creation error with regard
to memory use, this adds asserts so the failure will occur as soon as
possible and give us a better location for investigation.

Resolves #42157

* Fix test passing in invalid memory value

* Fix another test passing in invalid memory value

* Also change mem check in MachineLearning.machineMemoryFromStats

* Add background documentation for why we prevent negative return values

* Clarify comment a bit more
dakrone added a commit that referenced this issue Jun 19, 2019
…42725)

* Return 0 for negative "free" and "total" memory reported by the OS

We've had a situation where the MX bean reported negative values for the
free memory of the OS, in those rare cases we want to return a value of
0 rather than blowing up later down the pipeline.

In the event that there is a serialization or creation error with regard
to memory use, this adds asserts so the failure will occur as soon as
possible and give us a better location for investigation.

Resolves #42157

* Fix test passing in invalid memory value

* Fix another test passing in invalid memory value

* Also change mem check in MachineLearning.machineMemoryFromStats

* Add background documentation for why we prevent negative return values

* Clarify comment a bit more
dakrone added a commit that referenced this issue Jun 19, 2019
…42725)

* Return 0 for negative "free" and "total" memory reported by the OS

We've had a situation where the MX bean reported negative values for the
free memory of the OS, in those rare cases we want to return a value of
0 rather than blowing up later down the pipeline.

In the event that there is a serialization or creation error with regard
to memory use, this adds asserts so the failure will occur as soon as
possible and give us a better location for investigation.

Resolves #42157

* Fix test passing in invalid memory value

* Fix another test passing in invalid memory value

* Also change mem check in MachineLearning.machineMemoryFromStats

* Add background documentation for why we prevent negative return values

* Clarify comment a bit more
dakrone added a commit that referenced this issue Jun 19, 2019
…42725)

* Return 0 for negative "free" and "total" memory reported by the OS

We've had a situation where the MX bean reported negative values for the
free memory of the OS, in those rare cases we want to return a value of
0 rather than blowing up later down the pipeline.

In the event that there is a serialization or creation error with regard
to memory use, this adds asserts so the failure will occur as soon as
possible and give us a better location for investigation.

Resolves #42157

* Fix test passing in invalid memory value

* Fix another test passing in invalid memory value

* Also change mem check in MachineLearning.machineMemoryFromStats

* Add background documentation for why we prevent negative return values

* Clarify comment a bit more
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Stats Statistics tracking and retrieval APIs >test-failure Triaged test failures from CI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants