ccs - multiple remote clusters index pattern doesn't trigger a call _fields_for_wildcard #12966

nellicus · 2017-07-19T12:03:35Z

hello

I'm giving a go to ccs support in index pattern and not able to define a working index pattern.

note it works with a single definition londond:metricbeat-* which triggers a call

http://localhost:5601/api/index_patterns/_fields_for_wildcard?pattern=london%3Ametricbeat-*&meta_fields=%5B%22_source%22%2C%22_id%22%2C%22_type%22%2C%22_index%22%2C%22_score%22%5D

however it doesn't when using two remote clusters as the index pattern definition

london:metricbeat-*,newyork:metricbeat-*

kibana is connecting to a dedicated CCS node (which connects to the 2 clusters london/newyork holding the data) with settings

{
  "persistent": {
    "search": {
      "remote": {
        "london": {
          "seeds": [
            "127.0.0.1:9300"
          ]
        },
        "newyork": {
          "seeds": [
            "127.0.0.1:9301"
          ]
        }
      }
    }
  },
  "transient": {}
}

running _search from dev console in the above mentioned kibana instance works

GET london:metricbeat-*,newyork:metricbeat-*/_search
{
  "size": 0,
  "aggs": {
    "byTag": {
      "terms": {
        "field": "tags",
        "size": 10
      }
    }
  }
}
{
  "took": 12,
  "timed_out": false,
  "_shards": {
    "total": 10,
    "successful": 10,
    "failed": 0
  },
  "hits": {
    "total": 123956,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "byTag": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "london",
          "doc_count": 62626
        },
        {
          "key": "newyork",
          "doc_count": 61330
        }
      ]
    }
  }
}

I don't see kIbana calling the equivalent of

respons [11:46:38.726]  GET /api/index_patterns/_fields_for_wildcard?pattern=london%3Ametricbeat-*&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score 200 33ms - 9.0B

when using london:metricbeat-*,newyork:metricbeat-*

kibana logs in verbose mode when changing the index pattern definition from

london:metricbeat-* to london:metricbeat-*,newyork:metricbeat-*

show (not much happening when london:metricbeat-*,newyork:metricbeat-* is typed into index pattern value?)

 log   [11:56:52.448] [debug][monitoring-ui] Received Monitoring event data
  log   [11:56:52.489] [debug][esqueue][reporting][worker] j5axql3r0lwg3a73a9cn4gv0 - querying for outstanding jobs
  log   [11:56:52.491] [debug][esqueue][reporting][worker] j5axql3r0lwg3a73a9cn4gv0 - 0 outstanding jobs returned
respons [11:56:52.629]  GET /api/index_patterns/_fields_for_wildcard?pattern=london%3Ametricbeat-*&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score 200 46ms - 9.0B
respons [11:56:53.245]  GET /api/reporting/jobs/list_completed_since?since=2017-05-02T09%3A57%3A42.059Z 200 6ms - 9.0B
  log   [11:56:54.202] [debug][plugin] Checking Elasticsearch version
  log   [11:56:54.209] [debug][plugin] Checking Elasticsearch version
  log   [11:56:55.491] [debug][esqueue][reporting][worker] j5axql3r0lwg3a73a9cn4gv0 - querying for outstanding jobs
  log   [11:56:55.492] [debug][esqueue][reporting][worker] j5axql3r0lwg3a73a9cn4gv0 - 0 outstanding jobs returned
respons [11:56:56.271]  POST /api/console/proxy?path=_aliases&method=GET 200 21ms - 9.0B
respons [11:56:56.274]  POST /api/console/proxy?path=_mapping&method=GET 200 26ms - 9.0B
  log   [11:56:56.706] [debug][plugin] Checking Elasticsearch version
  log   [11:56:56.717] [debug][plugin] Checking Elasticsearch version
  ops   [11:56:57.448]  memory: 95.6MB uptime: 0:17:41 load: [1.07 1.16 1.21] delay: 0.429
  log   [11:56:57.450] [debug][monitoring-ui] Received Monitoring event data
  log   [11:56:58.492] [debug][esqueue][reporting][worker] j5axql3r0lwg3a73a9cn4gv0 - querying for outstanding jobs
  log   [11:56:58.493] [debug][esqueue][reporting][worker] j5axql3r0lwg3a73a9cn4gv0 - 0 outstanding jobs returned
  log   [11:56:59.205] [debug][monitoring-ui] Sending Monitoring payload to Elasticsearch
  log   [11:56:59.209] [debug][plugin] Checking Elasticsearch version
  log   [11:56:59.227] [debug][plugin] Checking Elasticsearch version
respons [11:57:00.175]  GET /api/reporting/jobs/list_completed_since?since=2017-05-02T09%3A57%3A42.059Z 200 13ms - 9.0B
respons [11:57:01.051]  GET /api/index_patterns/_fields_for_wildcard?pattern=london%3Ametricbeat-*&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score 200 53ms - 9.0B
  log   [11:57:01.494] [debug][esqueue][reporting][worker] j5axql3r0lwg3a73a9cn4gv0 - querying for outstanding jobs
  log   [11:57:01.497] [debug][esqueue][reporting][worker] j5axql3r0lwg3a73a9cn4gv0 - 0 outstanding jobs returned
  log   [11:57:01.713] [debug][plugin] Checking Elasticsearch version
  log   [11:57:01.751] [debug][plugin] Checking Elasticsearch version
  ops   [11:57:02.448]  memory: 95.5MB uptime: 0:17:46 load: [0.98 1.14 1.21] delay: 0.151
  log   [11:57:02.449] [debug][monitoring-ui] Received Monitoring event data
respons [11:57:03.269]  GET /api/reporting/jobs/list_completed_since?since=2017-05-02T09%3A57%3A42.059Z 200 9ms - 9.0B
  log   [11:57:04.221] [debug][plugin] Checking Elasticsearch version
  log   [11:57:04.286] [debug][plugin] Checking Elasticsearch version
  log   [11:57:04.494] [debug][esqueue][reporting][worker] j5axql3r0lwg3a73a9cn4gv0 - querying for outstanding jobs
  log   [11:57:04.495] [debug][esqueue][reporting][worker] j5axql3r0lwg3a73a9cn4gv0 - 0 outstanding jobs returned
respons [11:57:04.952]  GET /api/index_patterns/_fields_for_wildcard?pattern=london%3Ametricbeat-*&meta_fields=_source&meta_fields=_id&meta_fields=_type&meta_fields=_index&meta_fields=_score 200 19ms - 9.0B
  log   [11:57:06.732] [debug][plugin] Checking Elasticsearch version
  log   [11:57:06.824] [debug][plugin] Checking Elasticsearch version

yml configs

 $ find . -type f -name "*.yml" |  egrep '(elasticsearch|kibana)\.yml' | xargs egrep '^[^#]+'
./ccs/kibana-5.5.0-linux-x86_64/config/kibana.yml:elasticsearch.username: "elastic"
./ccs/kibana-5.5.0-linux-x86_64/config/kibana.yml:elasticsearch.password: "changeme"
./ccs/kibana-5.5.0-linux-x86_64/config/kibana.yml:elasticsearch.url: "http://localhost:9202"
./ccs/elasticsearch-5.5.0/config/elasticsearch.yml:http.port: 9202
./ccs/elasticsearch-5.5.0/config/elasticsearch.yml:transport.tcp.port: 9302
./ccs/elasticsearch-5.5.0/config/elasticsearch.yml:cluster.name: "dccs"
./london/elasticsearch-5.5.0/config/elasticsearch.yml:http.port: 9200
./london/elasticsearch-5.5.0/config/elasticsearch.yml:transport.tcp.port: 9300
./london/elasticsearch-5.5.0/config/elasticsearch.yml:cluster.name: "london"
./newyork/elasticsearch-5.5.0/config/elasticsearch.yml:http.port: 9201
./newyork/elasticsearch-5.5.0/config/elasticsearch.yml:transport.tcp.port: 9301
./newyork/elasticsearch-5.5.0/config/elasticsearch.yml:cluster.name: "newyork"

am I missing something?

The text was updated successfully, but these errors were encountered:

epixa · 2017-07-19T12:12:49Z

Does london,newyork:metricbeat-* work? I can't remember if I've tried that or not, but it doesn't seem right that you could specify different indices based on the cluster name, so this syntax seems more inline with how I think about CCS. Whatever the appropriate syntax is though, we should support it.

Edit: Whoops, I just re-read and saw your mention about the syntax working via console. We should support it, then.

nellicus · 2017-07-19T12:23:29Z

Does london,newyork:metricbeat-* work?

nope :(

newyork:metricbeat-* does

nellicus · 2017-07-19T12:41:30Z

repro tar.gz
sorry forgot to wipeout the data folders (all metricbeat data)

nellicus · 2017-07-19T13:27:06Z

current workaround , use
*:metricbeat-*

thanks @LeeDr

alexfrancoeur · 2017-07-20T14:48:44Z

I'm also able to reproduce locally on 5.5

Seems to be an issue with index pattern management.

@epixa I had thought that london,newyork:metricbeat-* syntax would work as well but must have misread the initial issue. The syntax needs to be as described earlier - cluster1:secrepo*,cluster2:secrepo*

nellicus · 2017-07-20T15:27:00Z

@alexfrancoeur my repro was on 5.5 too

btw may I ask what behavior the notation london,newyork:metricbeat-* would cause?

do we imply (:*) for the indices of london cluster there?

alexfrancoeur · 2017-07-20T17:50:21Z

The expectation around london,newyork:metricbeat-* was that your index pattern would grab metricbeat-* indices from london and newyork. It's the equivalent of london:metricbeat-*,newyork:metricbeat-* but a misunderstanding around syntax.

skearns64 · 2017-07-21T03:35:25Z

@nellicus - I hate making suggestions without being able to test them first, but I does *:metricbeat-* work in this case?

Even if that works, we should definitely still address this issue - we should be able to specify a subset of all configured CCS clusters in the index pattern - but I expect the most common case will be searching all configured CCS clusters, and I bet that's the main path we tested during development.

nellicus · 2017-07-21T10:07:26Z

@skearns64 yes it does! (#12966 (comment))

epixa added bug Fixes for quality problems that affect the customer experience :Discovery Feature:Cross Cluster Search labels Jul 19, 2017

alexfrancoeur added the :Management label Jul 20, 2017

alexfrancoeur added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Jul 31, 2017

chrisronline mentioned this issue Feb 5, 2018

[Management] Handle commas as separations in the query for CCS purposes #16535

Merged

chrisronline closed this as completed in #16535 Feb 8, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ccs - multiple remote clusters index pattern doesn't trigger a call _fields_for_wildcard #12966

ccs - multiple remote clusters index pattern doesn't trigger a call _fields_for_wildcard #12966

nellicus commented Jul 19, 2017 •

edited

Loading

epixa commented Jul 19, 2017 •

edited

Loading

nellicus commented Jul 19, 2017

nellicus commented Jul 19, 2017 •

edited

Loading

nellicus commented Jul 19, 2017

alexfrancoeur commented Jul 20, 2017 •

edited

Loading

nellicus commented Jul 20, 2017 •

edited

Loading

alexfrancoeur commented Jul 20, 2017 •

edited

Loading

skearns64 commented Jul 21, 2017 •

edited

Loading

nellicus commented Jul 21, 2017

ccs - multiple remote clusters index pattern doesn't trigger a call _fields_for_wildcard #12966

ccs - multiple remote clusters index pattern doesn't trigger a call _fields_for_wildcard #12966

Comments

nellicus commented Jul 19, 2017 • edited Loading

epixa commented Jul 19, 2017 • edited Loading

nellicus commented Jul 19, 2017

nellicus commented Jul 19, 2017 • edited Loading

nellicus commented Jul 19, 2017

alexfrancoeur commented Jul 20, 2017 • edited Loading

nellicus commented Jul 20, 2017 • edited Loading

alexfrancoeur commented Jul 20, 2017 • edited Loading

skearns64 commented Jul 21, 2017 • edited Loading

nellicus commented Jul 21, 2017

nellicus commented Jul 19, 2017 •

edited

Loading

epixa commented Jul 19, 2017 •

edited

Loading

nellicus commented Jul 19, 2017 •

edited

Loading

alexfrancoeur commented Jul 20, 2017 •

edited

Loading

nellicus commented Jul 20, 2017 •

edited

Loading

alexfrancoeur commented Jul 20, 2017 •

edited

Loading

skearns64 commented Jul 21, 2017 •

edited

Loading