Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Monitoring] Logstash/Beats/APM Server Standalone Cluster monitoring displays a blank page if accessed through CCS #96189

Open
hendry-lim opened this issue Apr 3, 2021 · 2 comments
Labels
bug Fixes for quality problems that affect the customer experience Team:Monitoring Stack Monitoring team triage_needed

Comments

@hendry-lim
Copy link

hendry-lim commented Apr 3, 2021

Kibana version: 7.12.0 7.12.1 7.13.x 7.14.0

Elasticsearch version: 7.12.0 7.12.1 7.13.x 7.14.0

Browser version: Firefox 87.0 88.0

Describe the bug:
No Logstash nodes are displayed on Stack Monitoring on Overview, Nodes, and Pipelines if Logstash is under a Standalone Cluster and accessed through CCS.

Steps to reproduce:

  1. Set up and run Logstash to output to non-Elasticsearch output
  2. Set up Logstash monitoring using Metricbeat
  3. Set up CCS from a different cluster to access Logstash monitoring data
  4. Go to Stack Monitoring --> Standalone Cluster
    image
  5. Click on either Overview, Nodes, or Pipelines
  6. Observe a blank list/charts are displayed
    image
    image

It's working fine Logstash that sends data directly to an Elasticsearch cluster.

Expected behavior:
Charts/list should be populated with relevant Logstash instances and metrics.

Relates to #93617

Tried with 7.13.0-SNAPSHOT
Direct access to the cluster that stores the monitoring data
image

Access to the monitoring data via CCS
image

No error for Beats, but observed the following errors for APM Server

Stack Monitoring standalone cluster error
POST http://localhost:5601/api/monitoring/v1/clusters/__standalone_cluster__/apm/instances
TypeError: data is undefined
    ApmServerInstances http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    ds http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    xa http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    fl http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    ml http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    ol http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    Gr http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    unstable_runWithPriority http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:398
    qr http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    Gr http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    Vr http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    el http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    Wl http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    Gl http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    render http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390
    renderReact http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    renderComponent http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    controller http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    $digest http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $apply http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    updateData http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    promise callback*base_controller_MonitoringViewBaseController/this.updateData http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    base_controller_MonitoringViewBaseController http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    $eval http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    applyAsync http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    C http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $eval http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $apply http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    setTimeout handler*cn/o.defer http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $applyAsync http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    titleProvider http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    base_controller_MonitoringViewBaseController http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    base_eui_table_controller_MonitoringViewBaseEuiTableController http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    controller http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    instantiate http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    get http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    link http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    he http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    Pe http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    se http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    le http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    oe http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    ge http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    _ http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    update http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    $broadcast http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    commitRoute http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    u http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    u http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $digest http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    evalAsync http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    setTimeout handler*cn/o.defer http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $evalAsync http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    get http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    u http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    e http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    h http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    resolve http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    C http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    a http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    y http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    C http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    onload http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    get http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    y http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    y http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    u http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    u http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $digest http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    get http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    x http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    T http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    x http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    S http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    dispatch http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:353
    handle http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:353
kbn-ui-shared-deps.js:390:77072
@hendry-lim hendry-lim added the bug Fixes for quality problems that affect the customer experience label Apr 3, 2021
@botelastic botelastic bot added the needs-team Issues missing a team label label Apr 3, 2021
@hendry-lim hendry-lim changed the title [Monitoring] Logstash Standalone Cluster monitoring displays a blank page if accessed through CCS [Monitoring] Logstash/Beats/APM Server Standalone Cluster monitoring displays a blank page if accessed through CCS Apr 7, 2021
@mikecote mikecote added Team:Monitoring Stack Monitoring team triage_needed and removed needs-team Issues missing a team label labels Apr 8, 2021
@elasticmachine
Copy link
Contributor

Pinging @elastic/stack-monitoring (Team:Monitoring)

@hendry-lim
Copy link
Author

hendry-lim commented Aug 21, 2021

Issue persists in 7.15.0-SNAPSHOT.

Kibana error log when viewing APM server instances under Standalone Cluster
{
  "type": "response",
  "@timestamp": "2021-08-21T14:22:18+00:00",
  "tags": [],
  "pid": 1215,
  "method": "post",
  "statusCode": 200,
  "req": {
    "url": "/api/monitoring/v1/clusters/__standalone_cluster__",
    "method": "post",
    "headers": {
      "host": "localhost:5601",
      "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0",
      "accept": "application/json, text/plain, */*",
      "accept-language": "en-US,en;q=0.5",
      "accept-encoding": "gzip, deflate",
      "referer": "http://localhost:5601/app/monitoring",
      "kbn-system-request": "true",
      "content-type": "application/json;charset=utf-8",
      "kbn-version": "7.15.0-SNAPSHOT",
      "content-length": "101",
      "origin": "http://localhost:5601",
      "dnt": "1",
      "connection": "keep-alive",
      "sec-fetch-dest": "empty",
      "sec-fetch-mode": "cors",
      "sec-fetch-site": "same-origin"
    },
    "remoteAddress": "172.20.0.1",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0",
    "referer": "http://localhost:5601/app/monitoring"
  },
  "res": {
    "statusCode": 200,
    "responseTime": 88,
    "contentLength": 406
  },
  "message": "POST /api/monitoring/v1/clusters/__standalone_cluster__ 200 88ms - 406.0B"
}{
  "type": "log",
  "@timestamp": "2021-08-21T14:22:18+00:00",
  "tags": ["error", "plugins", "monitoring", "monitoring"],
  "pid": 1215,
  "message": "TypeError: Cannot destructure property 'beats' of '(intermediate value)(intermediate value)(intermediate value)' as it is undefined.\n    at handleResponse (/usr/share/kibana/x-pack/plugins/monitoring/server/lib/apm/get_apms.js:46:5)\n    at getApms (/usr/share/kibana/x-pack/plugins/monitoring/server/lib/apm/get_apms.js:181:10)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at async Promise.all (index 1)\n    at Object.handler (/usr/share/kibana/x-pack/plugins/monitoring/server/routes/api/v1/apm/instances.js:51:31)\n    at handler (/usr/share/kibana/x-pack/plugins/monitoring/server/plugin.js:411:28)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"
}{
  "type": "log",
  "@timestamp": "2021-08-21T14:22:18+00:00",
  "tags": ["error", "http"],
  "pid": 1215,
  "message": "Error: Cannot wrap an error\n    at new module.exports (/usr/share/kibana/node_modules/@hapi/hoek/lib/error.js:23:19)\n    at Object.module.exports [as assert] (/usr/share/kibana/node_modules/@hapi/hoek/lib/assert.js:20:11)\n    at Toolkit.response (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:191:18)\n    at HapiResponseAdapter.toSuccess (/usr/share/kibana/src/core/server/http/router/response_adapter.js:97:43)\n    at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n    at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"
}{
  "type": "error",
  "@timestamp": "2021-08-21T14:22:18+00:00",
  "tags": [],
  "pid": 1215,
  "level": "error",
  "error": {
    "message": "Internal Server Error",
    "name": "Error",
    "stack": "Error: Internal Server Error\n    at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:61:19)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:177:34)\n    at runMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:95:5)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"
  },
  "url": "http://localhost:5601/api/monitoring/v1/clusters/__standalone_cluster__/apm/instances",
  "message": "Internal Server Error"
}{
  "type": "response",
  "@timestamp": "2021-08-21T14:22:18+00:00",
  "tags": [],
  "pid": 1215,
  "method": "post",
  "statusCode": 500,
  "req": {
    "url": "/api/monitoring/v1/clusters/__standalone_cluster__/apm/instances",
    "method": "post",
    "headers": {
      "host": "localhost:5601",
      "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0",
      "accept": "application/json, text/plain, */*",
      "accept-language": "en-US,en;q=0.5",
      "accept-encoding": "gzip, deflate",
      "referer": "http://localhost:5601/app/monitoring",
      "content-type": "application/json;charset=utf-8",
      "kbn-version": "7.15.0-SNAPSHOT",
      "content-length": "81",
      "origin": "http://localhost:5601",
      "dnt": "1",
      "connection": "keep-alive",
      "sec-fetch-dest": "empty",
      "sec-fetch-mode": "cors",
      "sec-fetch-site": "same-origin"
    },
    "remoteAddress": "172.20.0.1",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0",
    "referer": "http://localhost:5601/app/monitoring"
  },
  "res": {
    "statusCode": 500,
    "responseTime": 57,
    "contentLength": 97
  },
  "message": "POST /api/monitoring/v1/clusters/__standalone_cluster__/apm/instances 500 57ms - 97.0B"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Team:Monitoring Stack Monitoring team triage_needed
Projects
None yet
Development

No branches or pull requests

3 participants