Painless crashes with courier fetch warning on Kibana if user tries to index a new document.

[bhavyarm]

Kibana version: 5.2.0 & 5.1.1

Elasticsearch version: 5.2.0 & 5.1.1

Server OS version: darwin_x86_x64

Browser version: chrome latest

Browser OS version: OS X

Original install method (e.g. download page, yum, from source, etc.): BC2 for 5.2.0 pre-release

Description of the problem including expected versus actual behavior: If user creates a painless scripted field on an index and adds a new document into that index, painless script crashes with a courier fetch warning message on discover.

Steps to reproduce:

1. Created a painless scripted field into logstash: eg: doc['geo.dest.keyword'].value + ':' + doc['geo.src.keyword'].value
2. Index a new document into logstash: curl -XPOST -u elastic:changeme http://localhost:9200/logstash-2016.01.01/newType -d '
   {
   "title": "The Hobbit",
   "year": "1937",
   "@timestamp": "2016-01-01T11:36:07.963Z"
   }'
3. Click on discover - there is a courier fetch warning

Errors in browser console (if relevant):
None

Provide logs and/or server output (if relevant):
ES Log:

Caused by: java.lang.IllegalArgumentException: No field found for [geo.dest.keyword] in mapping with types []
at org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:80) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.painless.Executable$Script.execute(doc['geo.dest.keyword'].value + ':' + doc['geo.src.keyword'].value @ :5) ~[?:?]
at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:123) ~[?:?]
at org.elasticsearch.search.fetch.subphase.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:52) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:163) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:489) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:354) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:351) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$1(SecurityServerTransportInterceptor.java:224) ~[?:?]
at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:109) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$2(SecurityServerTransportInterceptor.java:263) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:56) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$null$2(ServerTransportFilter.java:164) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:127) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:121) ~[?:?]
at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:109) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$3(ServerTransportFilter.java:166) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:56) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:182) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$2(AuthenticationService.java:201) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:213) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:180) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:142) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:114) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:142) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:270) ~[?:?]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:610) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) ~[elasticsearch-5.2.0.jar:5.2.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.2.0.jar:5.2.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]

[Bargs]

Painless scripts need to be defensive if they're accessing fields that might not exist. You can check for the existence of a field by using doc.containsKey. So you might write your script like this:

if (doc.containsKey('geo.dest.keyword') && doc.containsKey('get.src.keyword') {
  return doc['geo.dest.keyword'].value + ':' + doc['geo.src.keyword'].value;  
}
else {
  return '';
}

[LeeDr]

@Bargs I think the courier fetch warning is straight from Elasticsearch? I understand your solution, but should Elasticsearch handle it better? Maybe we should create an issue in Elasticsearch repo?

[Bargs]

@LeeDr I already did awhile ago and the response was "working as intended" elastic/elasticsearch#22056

[Bargs]

Perhaps we could handle it more gracefully in Kibana, but I think that falls under #5088 so I think we can close this ticket.