Logstash 5.0.0-alpha5 - Can no longer connect to beats input #5756

ceeeekay · 2016-08-11T03:49:09Z

Upgrading from Logstash 5.0.0-alpha4 to Logstash 5.0.0-alpha5, filebeat clients can no longer connect to the beats input.

2016/08/11 00:06:24.596939 single.go:140: INFO Connecting error publishing events (retrying): read tcp 10.2.15.11:57029->10.2.15.21:10514: read: connection reset by peer

Example configs:

input {
  beats {
    port            => 10514
    ssl             => true
    ssl_certificate => "/etc/logstash/ssl/certs/dev-beats-ca.pem"
    ssl_key         => "/etc/logstash/ssl/private_keys/dev-beats-key.pem"
    type            => "beats"
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

filebeat:
  spool_size: 1
  registry_file: /var/lib/filebeat/registry
  config_dir: /etc/filebeat/conf.d/
output:
  logstash:
    enabled: true
    hosts: ["10.2.15.21:10514"]
    tls:
      disabled: false
      certificate_authorities: ["/etc/filebeat/dev-beats-ca.pem"]
      insecure: false

Prospectors are in the conf.d directory above, but aren't relevant here.

According to netstat the service is definitely listening on tcp/10514, and these configs were previously working with alpha4.

There is no mention of the error that I can see running Logstash with debug logging.

Fresh install of Logstash (dpkg purge, rm) does not change the behaviour.

The text was updated successfully, but these errors were encountered:

ceeeekay · 2016-08-11T04:46:02Z

Sorry, forgot the system stuff:

Ubuntu 14.04.4
logstash 1:5.0.0~alpha5-1
logstash-input-beats (3.1.0.beta4)
filebeat 5.0.0-alpha5 and 1.2.3

ph · 2016-08-11T13:18:06Z

@ceeeekay thanks for reporting this, beats input was completely rewritten from scratch in Java between the 2 alphas

So theses report are critical to us!

I have a few questions:

I presume this is a test environment, could you test without your certificates?
no errors on the Logstash side at all?

ceeeekay · 2016-08-11T23:55:30Z

Hi @ph

I can connect fine with SSL disabled.

I can also reproduce the fault elsewhere with SSL enabled. I've checked the alt names and expiry on the cert and they're all fine (and this was working prior to upgrading to alpha5).

I don't know if this is useful info but even trying to telnet to the beats port on an affected machine results in the connection being closed immediately, still with no errors logged in LS:

# telnet localhost 10514
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.

ph · 2016-08-16T12:35:26Z

@ceeeekay I did not get luck into reproducing this error but I want to get to the end of this.

it is possible to provide this:

The command line you used to generate the certificate
A certificate/private key to reproduce the issue

This would help me a lot.

Thanks

ceeeekay · 2016-08-17T00:06:54Z

Hi @ph

I've tried to go back to the beginning and create a new SSL key so I can give you the exact steps to reproduce, however I can't even get alpha5 to start with this new key, although it works fine with 2.3.4 (logstash-input-beats 2.2.9).

The error displayed by LS alpha5 is:
An unexpected error occurred! {:error=>#<NoMethodError: undefined method stop' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-3.1.0.beta4-java/lib/logstash/inputs/beats.rb:173:in stop'", "/usr/share/logstash/logstash-core/lib/logstash/inputs/base.rb:88:in do_stop'", "org/jruby/RubyArray.java:1613:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:366:in shutdown'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:252:in stop_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:261:in shutdown_pipelines'", "org/jruby/RubyHash.java:1342:in each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:261:in shutdown_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:123:in shutdown'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:237:in execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:157:in run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:66:in (root)'"], :level=>:fatal}`

We're using alt names as we're sending to multiple LS nodes.

Here's the ssl command used to generate the key & cert:
openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 -keyout dev-beats-key.pem -out dev-beats-ca.pem

And these changes at the bottom of openssl.cnf:

[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[alt_names]
DNS.1 = dev-ingress1
DNS.2 = dev-ingress2

Are you able to reproduce this latest error?

ceeeekay · 2016-08-17T01:40:50Z

Hi again @ph

I've spotted an error with the openssl config above and found it wasn't referring to the v3_req section, which was causing the startup error with alpha5 - although I'm not sure why that was upsetting it.

I now have a cert/key pair which Logstash is happy to start with, however I'm still getting the original connection reset issue with them.

Happy to provide these to you as they were only generated for this test.

Cheers.

ph · 2016-08-19T13:40:33Z

I am working on making theses errors more expliticit, expect a new version soon :)
Thanks for keeping the thread up to date.

ph · 2016-08-22T17:39:19Z

OK with have with this certificate a NullPointerException in the SSLSimpleBuilder class.

Exception in thread "main" java.lang.NullPointerException
    at org.logstash.netty.PrivateKeyConverter.generatePkcs8(PrivateKeyConverter.java:43)
    at org.logstash.netty.PrivateKeyConverter.convert(PrivateKeyConverter.java:39)
    at org.logstash.beats.Runner.main(Runner.java:39)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

ph · 2016-08-22T17:40:51Z


```
     * Returns the name of the primary encoding format of this key,
     * or null if this key does not support encoding.
```

The main problem is the PemReader cannot correctly read the provided files.

ph · 2016-08-22T20:08:00Z

Actually there is 3 errors in this scenario.

the ssl error are not correctly send back to the user.
The PKCS8 handling was only taking care of PemKeyPair and not PrivateKeyInfo
After fixing the 2 issues above we still get found no certificates in input stream, path is OK, c
content is there. (Even if the crt is currently expired it should find it)

ph self-assigned this Aug 11, 2016

ph added bug Pioneer Program bug in plugin v5.0.0-beta1 labels Aug 11, 2016

ceeeekay mentioned this issue Aug 12, 2016

3.1.0.beta4 - Misleading error when SSL key or cert is not readable logstash-plugins/logstash-input-beats#116

Closed

ph mentioned this issue Aug 24, 2016

NPE, Error in the log and make it work with Logstash 2.4 logstash-plugins/logstash-input-beats#119

Closed

suyograo added v5.0.0 and removed v5.0.0-beta1 labels Sep 13, 2016

ceeeekay mentioned this issue Oct 5, 2016

Logstash 5.0.0-beta1 - Beats input "failed to set certificate and key" #5998

Closed

suyograo closed this as completed Oct 10, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Logstash 5.0.0-alpha5 - Can no longer connect to beats input #5756

Logstash 5.0.0-alpha5 - Can no longer connect to beats input #5756

ceeeekay commented Aug 11, 2016 •

edited

Loading

ceeeekay commented Aug 11, 2016

ph commented Aug 11, 2016

ceeeekay commented Aug 11, 2016

ph commented Aug 16, 2016

ceeeekay commented Aug 17, 2016 •

edited

Loading

ceeeekay commented Aug 17, 2016 •

edited

Loading

ph commented Aug 19, 2016

ph commented Aug 22, 2016 •

edited

Loading

ph commented Aug 22, 2016 •

edited

Loading

ph commented Aug 22, 2016

Logstash 5.0.0-alpha5 - Can no longer connect to beats input #5756

Logstash 5.0.0-alpha5 - Can no longer connect to beats input #5756

Comments

ceeeekay commented Aug 11, 2016 • edited Loading

ceeeekay commented Aug 11, 2016

ph commented Aug 11, 2016

ceeeekay commented Aug 11, 2016

ph commented Aug 16, 2016

ceeeekay commented Aug 17, 2016 • edited Loading

ceeeekay commented Aug 17, 2016 • edited Loading

ph commented Aug 19, 2016

ph commented Aug 22, 2016 • edited Loading

ph commented Aug 22, 2016 • edited Loading

ph commented Aug 22, 2016

ceeeekay commented Aug 11, 2016 •

edited

Loading

ceeeekay commented Aug 17, 2016 •

edited

Loading

ceeeekay commented Aug 17, 2016 •

edited

Loading

ph commented Aug 22, 2016 •

edited

Loading

ph commented Aug 22, 2016 •

edited

Loading