Crash if logstash-* index does not exist

[andre-pt]

Kibana version: 4.5.1

OS version: Red Hat Enterprise Linux Server release 7.2 (Maipo)

Original install method (e.g. download page, yum, from source, etc.): download page

Description of the problem including expected versus actual behavior:
In my logstash config I have:

      elasticsearch {
        hosts => ["127.0.0.1:9200"]
        index => "myIndex-%{+YYYY.MM}"
        document_id => "%{id}"
      }

If I start Kibana, it crashes because it doesn't find a logstash-* index.

I would expect to be able to choose the index pattern I want afterwards.

Steps to reproduce:

1. I have just started a fresh elasticsearch with no data.
2. Started logstash, which created some indexes myIndex-*
3. Connected to Kibana using Chrome in Windows.

At this point Kibana gave me the following error:
Fatal Error
unknown error

Version: 4.5.1
Build: 9892
Error: unknown error
    at respond (http://127.0.0.1:10316/bundles/kibana.bundle.js?v=9892:76380:16)
    at checkRespForFailure (http://127.0.0.1:10316/bundles/kibana.bundle.js?v=9892:76341:8)
    at http://127.0.0.1:10316/bundles/kibana.bundle.js?v=9892:74959:8
    at processQueue (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9892:42360:29)
    at http://127.0.0.1:10316/bundles/commons.bundle.js?v=9892:42376:28
    at Scope.$eval (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9892:43604:29)
    at Scope.$digest (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9892:43415:32)
    at Scope.$apply (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9892:43712:25)
    at done (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9892:38161:48)
    at completeRequest (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9892:38359:8)

[Bargs]

The error your seeing looks like it's caused by an error response from Elasticsearch, or perhaps the Kibana backend. Could you recreate the fatal error with the Chrome devtools network tab open and see if there are any error responses? Also, do you see any errors in you Kibana or Elasticsearch logs?

[andre-pt]

Currently I'm not able to test it again in the Linux machine I reproduced the same time.

But using 4.5.2 (noticed now it was released a couple days ago) in windows did not error out.

I will try again in a couple of days

[andre-pt]

Running a clean ES, I have the following log entries as soon as I start kibana (server):

[2016-07-11 14:58:27,993][INFO ][cluster.metadata         ] [Cancer] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [config]
[2016-07-11 14:58:28,194][INFO ][cluster.routing.allocation] [Cancer] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).

Then I go to kibana and the following request returns 404, as expected because I don't have a logstash-* index:

Request URL: http://127.0.0.1:10316/elasticsearch/logstash-*/_mapping/field/*?_=1468242061451&ignore_unavailable=false&allow_no_indices=false&include_defaults=true
Request Method: GET
Status Code: 404 Not Found

At this point in the ES log I got:

[2016-07-11 14:59:03,575][INFO ][cluster.metadata         ] [Cancer] [.kibana] create_mapping [index-pattern]

The only way I found to make it work was to create a temporary logstash-* index.

[andre-pt]

I was able to reproduce also with 4.5.2:

Version: 4.5.2
Build: 9896
Error: unknown error
    at respond (http://127.0.0.1:10316/bundles/kibana.bundle.js?v=9896:76380:16)
    at checkRespForFailure (http://127.0.0.1:10316/bundles/kibana.bundle.js?v=9896:76341:8)
    at http://127.0.0.1:10316/bundles/kibana.bundle.js?v=9896:74959:8
    at processQueue (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9896:42385:29)
    at http://127.0.0.1:10316/bundles/commons.bundle.js?v=9896:42401:28
    at Scope.$eval (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9896:43629:29)
    at Scope.$digest (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9896:43440:32)
    at Scope.$apply (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9896:43737:25)
    at done (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9896:38186:48)
    at completeRequest (http://127.0.0.1:10316/bundles/commons.bundle.js?v=9896:38384:8)

Request URL: http://127.0.0.1:10316/elasticsearch/logstash-*/_mapping/field/*?_=1468242606029&ignore_unavailable=false&allow_no_indices=false&include_defaults=true
Request Method: GET
Status Code: 404 Not Found

Request Headers:

GET /elasticsearch/logstash-*/_mapping/field/*?_=1468242606029&ignore_unavailable=false&allow_no_indices=false&include_defaults=true HTTP/1.1
Host: 127.0.0.1:10316
Connection: keep-alive
Accept: application/json, text/plain, */*
kbn-version: 4.5.2
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36
Referer: http://127.0.0.1:10316/app/kibana
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-US,en;q=0.8
Cookie: _ga=GA1.1.819804584.1467361615

Response Headers:

HTTP/1.1 404 Not Found
Date: Mon, 11 Jul 2016 13:10:06 GMT
Server: Apache
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=3000000; includeSubDomains
Cache-Control: no-cache
Connection: keep-alive, close
Content-Length: 323
Content-Type: text/html
Es.Index: logstash-*
Es.Resource.Id: logstash-*
Es.Resource.Type: index_or_alias
Kbn-Name: kibana
Kbn-Version: 4.5.2

Note: I'm reaching kibana over a ssh tunnel, it might be useful info :)

[Bargs]

Hmmm, there's something else odd going on, the 404 isn't the cause of the fatal error (see my screenshot below of an initial load of a fresh Kibana 4.5.2 and ES 2.3.4).

What version of Elasticsearch are you using? Would it be possible to try accessing Kibana without the SSH tunnel?

[andre-pt]

I'm running elasticsearch-2.3.3 and no, SSH tunnel is the only way I can do it.

Currently I don't have access to other linux machine where I can replicate it locally.

[Bargs]

I just noticed the Server: Apache header in the response you included. Are you running an Apache proxy in front of Kibana? If so, what happens if you remove the proxy? Misconfigured proxies are often the source of strange issues like this.

[andre-pt]

that's a good hint!
The company where I'm working uses a proprietary system to create ssh tunnels, which is used to better manage which servers the employees have access.

Maybe they have a reverse proxy combined with the tunnel.

Thanks a lot for the help, I will close the issue.
André