Negative longs unsupported, use writeLong or writeZLong for negative numbers

[srikwit]

Elasticsearch version (bin/elasticsearch --version): 7.9.1

Plugins installed: [Custom]

JVM version (java -version):
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)

OS version (uname -a if on a Unix-like system): Linux 4.15.0-76-generic

Description of the problem including expected versus actual behavior:

Steps to reproduce:

I have been observing it with the cluster upgrade to 7.9.1

Provide logs (if relevant):

java.lang.IllegalStateException: Negative longs unsupported, use writeLong or writeZLong for negative numbers [-1000]
        at org.elasticsearch.common.io.stream.StreamOutput.writeVLong(StreamOutput.java:301) ~[elasticsearch-7.9.1.jar
:7.9.1]
        at org.elasticsearch.ingest.IngestStats$Stats.writeTo(IngestStats.java:197) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.ingest.IngestStats.writeTo(IngestStats.java:103) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.common.io.stream.StreamOutput.writeOptionalWriteable(StreamOutput.java:952) ~[elasticsear
ch-7.9.1.jar:7.9.1]
        at org.elasticsearch.action.admin.cluster.node.stats.NodeStats.writeTo(NodeStats.java:290) ~[elasticsearch-7.9
.1.jar:7.9.1]
        at org.elasticsearch.transport.OutboundMessage.writeMessage(OutboundMessage.java:87) ~[elasticsearch-7.9.1.jar
:7.9.1]
        at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:64) ~[elasticsearch-7.9.1.jar:7.
9.1]
        at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:159) ~[elasticsearch
-7.9.1.jar:7.9.1]
        at org.elasticsearch.transport.OutboundHandler$MessageSerializer.get(OutboundHandler.java:144) ~[elasticsearch
-7.9.1.jar:7.9.1]
        at org.elasticsearch.transport.OutboundHandler$SendContext.get(OutboundHandler.java:197) ~[elasticsearch-7.9.1.jar:7.9.1]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler$WriteOperation.buffer(Netty4MessageChannelHandler.java:213) ~[transport-netty4-client-7.9.1.jar:7.9.1]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.doFlush(Netty4MessageChannelHandler.java:147) ~[transport-netty4-client-7.9.1.jar:7.9.1]
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.flush(Netty4MessageChannelHandler.java:117) ~[transport-netty4-client-7.9.1.jar:7.9.1]

[cbuescher]

This looks like a duplicate of #52339, tagging the ingest team, please close if you think #52339 covers this case.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Ingest)

[danhermann]

This does look like a duplicate of #52339, but it appears that there may be multiple issues with the serialization of ingest stats as some bugs have already been fixed (#52543) but we are still seeing other errors such as this one. @srikwit, if you are able to share your ingest pipelines (or at least a list of the processors and whether they use conditionals), that would be helpful in tracking down the cause of this bug.

[srikwit]

@danhermann we use around 30 pipelines and looking at the previous bug reports, I am sharing one workflow which might be of relevance:

Pipeline 1:
Main:
    Script (script contains painless if)
    Script (script contains painless if)
    Script (script contains painless if)
    Remove (Unconditional)
    Script (Unconditional)
    Enrich
    Enrich
    Enrich
    Enrich
    Script (script contains painless if)
    Script (Unconditional)
    Remove (Unconditional)
    Enrich
    Enrich
    Enrich
    Enrich
    Enrich
    Enrich
    Script (script contains painless if)
    Drop (Conditional)
    Remove (Unconditional)
    Enrich
    Enrich
    GeoIP
    GeoIP
    GeoIP
    GeoIP
    Rename
    Rename
    Rename
    Rename
On failure:
    Set

Please let me know if this is helpful.

[danhermann]

@srikwit, thank you. That will probably help in narrowing down the possible sources of the bug. I'm going to close this issue as a duplicate of #52339 but leave a comment there about the additional information you've provided here.