journalbeat 6.7.0 doesn't work with AWS elasticsearch

[benny-medflyt]

journalbeat 6.5.4 was working fine. I upgraded to 6.7.0 and now it can't send the logs to elasticsearch. This is the error:

2019-04-02T17:52:45.542Z        ERROR   pipeline/output.go:100  Failed to connect to backoff(elasticsearch(http://vpc-xxxx-logs-yyyyyyyyyyyyyyyyyyy.us-east-1.es.amazonaws.com:80)): Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials
2019-04-02T17:52:45.542Z        INFO    pipeline/output.go:93   Attempting to reconnect to backoff(elasticsearch(http://vpc-xxxx-logs-yyyyyyyyyyyyyyyyyyy.us-east-1.es.amazonaws.com:80)) with 36 reconnect attempt(s)
2019-04-02T17:52:45.542Z        DEBUG   [elasticsearch] elasticsearch/client.go:715     ES Ping(url=http://vpc-xxxx-logs-yyyyyyyyyyyyyyyyyyy.us-east-1.es.amazonaws.com:80)
2019-04-02T17:52:45.542Z        INFO    [publish]       pipeline/retry.go:189   retryer: send unwait-signal to consumer
2019-04-02T17:52:45.542Z        INFO    [publish]       pipeline/retry.go:191     done
2019-04-02T17:52:45.542Z        INFO    [publish]       pipeline/retry.go:166   retryer: send wait signal to consumer
2019-04-02T17:52:45.542Z        INFO    [publish]       pipeline/retry.go:168     done
2019-04-02T17:52:45.546Z        DEBUG   [elasticsearch] elasticsearch/client.go:738     Ping status code: 200
2019-04-02T17:52:45.546Z        INFO    elasticsearch/client.go:739     Attempting to connect to Elasticsearch version 6.4.2
2019-04-02T17:52:45.546Z        DEBUG   [elasticsearch] elasticsearch/client.go:757     GET http://vpc-xxxx-logs-yyyyyyyyyyyyyyyyyyy.us-east-1.es.amazonaws.com:80/_xpack?human=false  <nil>

That error shows up over and over. The problem is it it trying to do a GET request to _xpack?human=false endpoint, which doesn't work:

[ec2-user@ip-172-31-xxx-xxx ~]$ curl --verbose http://vpc-xxxx-logs-yyyyyyyyyyyyyyyyyyy.us-east-1.es.amazonaws.com:80/_xpack?human=false
*   Trying 172.31.zzz.zzz...
* TCP_NODELAY set
* Connected to vpc-xxxx-logs-yyyyyyyyyyyyyyyyyyy.us-east-1.es.amazonaws.com (172.31.zzz.zzz) port 80 (#0)
> GET /_xpack?human=false HTTP/1.1
> Host: vpc-xxxx-logs-yyyyyyyyyyyyyyyyyyy.us-east-1.es.amazonaws.com
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< Date: Tue, 02 Apr 2019 17:56:27 GMT
< Content-Type: application/json
< Content-Length: 53
< Connection: keep-alive
< x-amzn-RequestId: a387e173-5570-11e9-805e-439d9255c66a
< Access-Control-Allow-Origin: *
< 
* Connection #0 to host vpc-xxxx-logs-yyyyyyyyyyyyyyyyyyy.us-east-1.es.amazonaws.com left intact
{"Message":"Your request: '/_xpack' is not allowed."}

This is the file I installed: https://artifacts.elastic.co/downloads/beats/journalbeat/journalbeat-6.7.0-x86_64.rpm

This is my config:

journalbeat.inputs:
- paths: []
  seek: cursor

output.elasticsearch:
  hosts: ["http://vpc-xxxx-logs-yyyyyyyyyyyyyyyyyyy.us-east-1.es.amazonaws.com:80"]

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

logging.level: debug

[andrewkroh]

There is a Journalbeat package without any X-Pack code (so it doesn't have the central management enroll sub-command) and it won't be trying any _xpack/ endpoints.

https://www.elastic.co/downloads/beats/journalbeat-oss
https://artifacts.elastic.co/downloads/beats/journalbeat/journalbeat-oss-6.7.0-x86_64.rpm

[benny-medflyt]

Thanks that fixes everything