Kibana crash: TypeError: request.abort is not a function

[tsullivan]

Kibana version: 8.0.0/master
Original install method (e.g. download page, yum, from source, etc.): source

Describe the bug: After enabling APM integration for Kibana, the Kibana server crashes if there is a logged error.

Steps to reproduce:

1. Enable APM in Kibana (See PR )
2. Start Kibana
3. Run a PDF Report. The server will expectedly log an error found in the headless browser: Refused to execute inline script because it violates the following Content Security Policy directive:
4. Kibana crashes

Expected behavior:

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
These logs are from running Kibana with node scripts/kibana but running with yarn start reproduces the crash as well.

  log   [16:22:34.276] [info][listening] Server running at http://localhost:5601
  log   [16:22:34.492] [info][server][Kibana][http] http server running at http://localhost:5601
  log   [16:22:37.136] [warning][esqueue][queue-worker][reporting] k7l3c5qf1xr4cbae0c1hmx7k - Failing job k7l37vz31xh7cbae0cdk8n9e
  log   [16:22:37.159] [info][esqueue][queue-worker][reporting] k7l3c5qf1xr4cbae0c1hmx7k - Job marked as failed: /.reporting-2020.03.08/k7l37vz31xh7cbae0cdk8n9e
  log   [16:25:21.347] [info][queue-job][reporting] Successfully queued job: k7l3fqpw1xr4cbae0c478xfs
  log   [16:25:22.799] [info][esqueue][queue-worker][reporting] k7l3c5qf1xr4cbae0c1hmx7k - Job marked as claimed: /.reporting-2020.03.08/k7l3fqpw1xr4cbae0c478xfs
  log   [16:25:22.800] [info][esqueue][queue-worker][reporting] k7l3c5qf1xr4cbae0c1hmx7k - Starting job
  log   [16:25:22.845] [info][browser-driver][execute][k7l3fqpw1xr4cbae0c478xfs][printable_pdf][reporting] Creating browser page driver
  log   [16:25:23.147] [info][execute][k7l3fqpw1xr4cbae0c478xfs][printable_pdf][reporting] opening url https://spicy.local:443/kbn/app/kibana#/dashboard/722b74f0-b882-11e8-a6d9-e546fe2bba5f?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!f%2Cvalue%3A900000)%2Ctime%3A(from%3A'2020-03-07T07%3A00%3A00.000Z'%2Cto%3A'2020-03-09T07%3A00%3A00.000Z'))&_a=(description%3A'Analyze%20mock%20eCommerce%20orders%20and%20revenue'%2Cfilters%3A!()%2CfullScreenMode%3A!f%2Coptions%3A(hidePanelTitles%3A!f%2CuseMargins%3A!t)%2Cpanels%3A!((embeddableConfig%3A(vis%3A(colors%3A('Men!'s%20Accessories'%3A%2382B5D8%2C'Men!'s%20Clothing'%3A%23F9BA8F%2C'Men!'s%20Shoes'%3A%23F29191%2C'Women!'s%20Accessories'%3A%23F4D598%2C'Women!'s%20Clothing'%3A%2370DBED%2C'Women!'s%20Shoes'%3A%23B7DBAB)))%2CgridData%3A(h%3A10%2Ci%3A'1'%2Cw%3A36%2Cx%3A12%2Cy%3A18)%2Cid%3A'37cc8650-b882-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'1'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A(vis%3A(colors%3A(FEMALE%3A%236ED0E0%2CMALE%3A%23447EBC)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A11%2Ci%3A'2'%2Cw%3A12%2Cx%3A12%2Cy%3A7)%2Cid%3Aed8436b0-b88b-11e8-a6d9-e546fe2bba5f%2CpanelIndex%3A'2'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A7%2Ci%3A'3'%2Cw%3A18%2Cx%3A0%2Cy%3A0)%2Cid%3A'09ffee60-b88c-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'3'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A7%2Ci%3A'4'%2Cw%3A30%2Cx%3A18%2Cy%3A0)%2Cid%3A'1c389590-b88d-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'4'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A11%2Ci%3A'5'%2Cw%3A48%2Cx%3A0%2Cy%3A28)%2Cid%3A'45e07720-b890-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'5'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A10%2Ci%3A'6'%2Cw%3A12%2Cx%3A0%2Cy%3A18)%2Cid%3A'10f1a240-b891-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'6'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A11%2Ci%3A'7'%2Cw%3A12%2Cx%3A0%2Cy%3A7)%2Cid%3Ab80e6540-b891-11e8-a6d9-e546fe2bba5f%2CpanelIndex%3A'7'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A(vis%3A(colors%3A('0%20-%2050'%3A%23E24D42%2C'50%20-%2075'%3A%23EAB839%2C'75%20-%20100'%3A%237EB26D)%2CdefaultColors%3A('0%20-%2050'%3A'rgb(165%2C0%2C38)'%2C'50%20-%2075'%3A'rgb(255%2C255%2C190)'%2C'75%20-%20100'%3A'rgb(0%2C104%2C55)')%2ClegendOpen%3A!f))%2CgridData%3A(h%3A11%2Ci%3A'8'%2Cw%3A12%2Cx%3A24%2Cy%3A7)%2Cid%3A'4b3ec120-b892-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'8'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A(vis%3A(colors%3A('0%20-%202'%3A%23E24D42%2C'2%20-%203'%3A%23F2C96D%2C'3%20-%204'%3A%239AC48A)%2CdefaultColors%3A('0%20-%202'%3A'rgb(165%2C0%2C38)'%2C'2%20-%203'%3A'rgb(255%2C255%2C190)'%2C'3%20-%204'%3A'rgb(0%2C104%2C55)')%2ClegendOpen%3A!f))%2CgridData%3A(h%3A11%2Ci%3A'9'%2Cw%3A12%2Cx%3A36%2Cy%3A7)%2Cid%3A'9ca7aa90-b892-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'9'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A18%2Ci%3A'10'%2Cw%3A48%2Cx%3A0%2Cy%3A54)%2Cid%3A'3ba638e0-b894-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'10'%2Ctype%3Asearch%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A(hiddenLayers%3A!()%2CisLayerTOCOpen%3A!f%2CmapCenter%3A(lat%3A45.88578%2Clon%3A-15.07605%2Czoom%3A2.11)%2CopenTOCDetails%3A!())%2CgridData%3A(h%3A15%2Ci%3A'11'%2Cw%3A24%2Cx%3A0%2Cy%3A39)%2Cid%3A'2c9c1f60-1909-11e9-919b-ffe5949a18d2'%2CpanelIndex%3A'11'%2Ctype%3Amap%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A15%2Ci%3A'12'%2Cw%3A24%2Cx%3A24%2Cy%3A39)%2Cid%3Ab72dd430-bb4d-11e8-9c84-77068524bcab%2CpanelIndex%3A'12'%2Ctype%3Avisualization%2Cversion%3A'8.0.0'))%2Cquery%3A(language%3Akuery%2Cquery%3A'')%2CtimeRestore%3A!t%2Ctitle%3A'%5BeCommerce%5D%20Revenue%20Dashboard'%2CviewMode%3Aview)&forceNow=2020-03-09T23%3A25%3A21.283Z
  log   [16:25:23.655] [error][browser-driver][execute][headless-browser-console][k7l3fqpw1xr4cbae0c478xfs][printable_pdf][reporting] Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.

  log   [16:25:33.438] [info][execute][k7l3fqpw1xr4cbae0c478xfs][printable_pdf][reporting] handled 107 page requests
TypeError: request.abort is not a function
    at AbortSignal.options.signal.addEventListener (/Users/tsullivan/elastic/kibana/src/core/server/elasticsearch/cluster_client.ts:67:19)
    at Timeout._onTimeout (/Users/tsullivan/elastic/kibana/node_modules/abortcontroller-polyfill/dist/cjs-ponyfill.js:153:27)
    at ontimeout (timers.js:436:11)
    at tryOnTimeout (timers.js:300:5)
    at listOnTimeout (timers.js:263:5)
    at Timer.processTimers (timers.js:223:10)
  log   [16:25:34.927] [warning][browser-driver][execute][k7l3fqpw1xr4cbae0c478xfs][printable_pdf][reporting] Request to [https://spicy.local/kbn/internal/search/es] failed! [net::ERR_ABORTED]. This error will be ignored.
Sending error to Elastic APM { id: 'b21f3fc287bdfcecfd07858adaa55b96' }

Any additional context:
It appears the line of code trigging the TypeError is https://github.com/elastic/kibana/blob/56041f0/src/core/server/elasticsearch/cluster_client.ts#L67

[tsullivan]

Hm, I haven't been able to reproduce this in the last couple of days. It was possibly related to 403 errors between apm-server and ES.

[tsullivan]

Sorry, this does look like there's a problem because I am still seeing this. I earlier switched to Reporting from Canvas, and wasn't getting this crash. But switching back to Dashboard reporting is giving me the crash.

[tsullivan]

I've added logging before the request.abort statement and here's what we get:

code change:

          if (typeof request.abort !== 'function') {
            throw new Error(`incorrect type for request: ${request}`);
          }
          request.abort();
          reject(new Error('Request was aborted'));

the result:

Error: incorrect type for request: [object Promise]

[mshustov]

elasticsearch library client returns a promise with additional abort method. https://github.com/elastic/elasticsearch-js-legacy/blob/16.x/src/lib/transport.js#L209
Does APM library patch native Promise object? @vigneshshanmugam

[vigneshshanmugam]

@restrry Node.js agent patches the elasticsearch client request and returns a different promise than the one elasticsearch request returns which seems to be the source of the problem AFAIK.

More details - elastic/apm-agent-nodejs#1565

Issue seems to be a duplicate of #52617

[mshustov]

@vigneshshanmugam thanks for the clarification! Any plans to fix it soon? I'm not sure that we want to merge the workaround from #59967, as it changes ES data fetching flow. Thus it's hard to predict what consequences we might face later.

[vigneshshanmugam]

@restrry Totally understand, I am not sure about the timeline when we will fix it, but I can have a look at it tomorrow and provide more details here.

[tsullivan]

@vigneshshanmugam do you think it's possible we could get a fix before 7.7?

[vigneshshanmugam]

@tsullivan I am not so sure on the timeframe of the issue as there is a open PR in the Node.js agent to address this elastic/apm-agent-nodejs#1566

[tsullivan]

Thanks for the follow-up @vigneshshanmugam !

I noticed there are no version labels on the PR in the apm-agent repo. Do you know what version of Kibana will have that fix?

cc @bmcconaghy @pmuellr