Keep getting explicit index in bulk is not allowed

[yasin-amadmia-mck]

2016-05-12 12:44:22 +0100 [info]: Connection opened to Elasticsearch cluster => {:host=>"search-fluentd-aggregated-logs-xxxx.eu-west-1.es.amazonaws.com", :port=>443, :scheme=>"https", :aws_elasticsearch_service=>{:credentials=>#<Aws::Credentials access_key_id="xxxx">, :region=>"eu-west-1"}}
2016-05-12 12:44:22 +0100 [warn]: temporarily failed to flush the buffer. next_retry=2016-05-12 12:44:23 +0100 error_class="Elasticsearch::Transport::Transport::Errors::BadRequest" error="[400] {\"error\":\"ElasticsearchIllegalArgumentException[explicit index in bulk is not allowed]\",\"status\":400}" plugin_id="object:3fe50cdd5884"
  2016-05-12 12:44:22 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/base.rb:201:in `__raise_transport_error'
  2016-05-12 12:44:22 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/base.rb:312:in `perform_request'
  2016-05-12 12:44:22 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
  2016-05-12 12:44:22 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-transport-1.0.17/lib/elasticsearch/transport/client.rb:128:in `perform_request'
  2016-05-12 12:44:22 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/elasticsearch-api-1.0.17/lib/elasticsearch/api/actions/bulk.rb:88:in `bulk'
  2016-05-12 12:44:22 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.4.0/lib/fluent/plugin/out_elasticsearch.rb:236:in `send'
  2016-05-12 12:44:22 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-1.4.0/lib/fluent/plugin/out_elasticsearch.rb:229:in `write'
  2016-05-12 12:44:22 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/buffer.rb:345:in `write_chunk'
  2016-05-12 12:44:22 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/buffer.rb:324:in `pop'
  2016-05-12 12:44:22 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/output.rb:329:in `try_flush'
  2016-05-12 12:44:22 +0100 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.20/lib/fluent/output.rb:140:in `run'
2016-05-12 12:44:23 +0100 [warn]: temporarily failed to flush the buffer. next_retry=2016-05-12 12:44:25 +0100 error_class="Elasticsearch::Transport::Transport::Errors::BadRequest" error="[400] {\"error\":\"ElasticsearchIllegalArgumentException[explicit index in bulk is not allowed]\",\"status\":400}" plugin_id="object:3fe50cdd5884"
  2016-05-12 12:44:23 +0100 [warn]: suppressed same stacktrace
2016-05-12 12:44:25 +0100 [warn]: temporarily failed to flush the buffer. next_retry=2016-05-12 12:44:29 +0100 error_class="Elasticsearch::Transport::Transport::Errors::BadRequest" error="[400] {\"error\":\"ElasticsearchIllegalArgumentException[explicit index in bulk is not allowed]\",\"status\":400}" plugin_id="object:3fe50cdd5884"
  2016-05-12 12:44:25 +0100 [warn]: suppressed same stacktrace
2016-05-12 12:44:29 +0100 [warn]: temporarily failed to flush the buffer. next_retry=2016-05-12 12:44:37 +0100 error_class="Elasticsearch::Transport::Transport::Errors::BadRequest" error="[400] {\"error\":\"ElasticsearchIllegalArgumentException[explicit index in bulk is not allowed]\",\"status\":400}" plugin_id="object:3fe50cdd5884"
  2016-05-12 12:44:29 +0100 [warn]: suppressed same stacktrace

[aerickson]

@geek876 What was the solution? I'm seeing this now.

Update: I thought I had rest.action.multi.allow_explicit_index = true, but there's a terraform bug (hashicorp/terraform#3980) that wasn't setting it on my domain.

[yasin-amadmia-mck]

@aerickson same issue for me too. setting rest.action.multi.allow_explicit_index = true manually fixed it for me.