[Ingest Manager] POST /setup crashes Kibana with Unhandled Promise rejection

[jfsiii]

Steps to reproduce

0. Start Kibana with the registry unavailable. e.g. use xpack.ingestManager.registryUrl flag to set it to an "bad" address or something like browser or networks tools to block traffic to https://epr.elastic.co/
   • custom registry is Gold+ so start ES in one terminal w/ yarn es snapshot --license=trial -E xpack.security.authc.api_key.enabled=true
   • Kibana in another with yarn start --no-base-path --xpack.ingestManager.enabled=true --xpack.ingestManager.registryUrl=http://localhost:9999
1. Visit http://localhost:5601/app/ingestManager in the browser or run curl -X POST 'http://localhost:5601/api/ingest_manager/setup' -H 'kbn-xsrf: <string>' --user elastic:changeme
2. Observe Kibana crash with

Unhandled Promise rejection detected:

{ RegistryError: Error connecting to package registry at http://localhost:9999/search?package=endpoint&internal=true&experimental=true: request to http://localhost:9999/search?package=endpoint&internal=true&experimental=true failed, reason: connect ECONNREFUSED 127.0.0.1:9999
    at getResponse (/Users/jfsiii/work/kibana/x-pack/plugins/ingest_manager/server/services/epm/registry/requests.ts:22:11)
    at process._tickCallback (internal/process/next_tick.js:68:7) name: 'RegistryError' }

Terminating process...
 server crashed  with status code 1

Notes

Not all Ingest UI routes/views were crashing. Other EPM API requests seem to work as expected before opening UI

> curl  http://localhost:5601/api/ingest_manager/epm/categories --user elastic:changeme
{"statusCode":500,"error":"Internal Server Error","message":"Error connecting to package registry at http://localhost:9999/categories: request to http://localhost:9999/categories failed, reason: connect ECONNREFUSED 127.0.0.1:9999"}

>  curl  http://localhost:5601/api/ingest_manager/epm/packages/nginx-0.1.2 --user elastic:changeme
{"statusCode":500,"error":"Internal Server Error","message":"Error connecting to package registry at http://localhost:9999/package/nginx/0.1.2: request to http://localhost:9999/package/nginx/0.1.2 failed, reason: connect ECONNREFUSED 127.0.0.1:9999"}

> curl  http://localhost:5601/api/ingest_manager/epm/packages --user elastic:changeme
{"statusCode":500,"error":"Internal Server Error","message":"Error connecting to package registry at http://localhost:9999/search: request to http://localhost:9999/search failed, reason: connect ECONNREFUSED 127.0.0.1:9999"}

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[jfsiii]

Looking at the network tab there was a 500 POST /setup. I see where the failure is happening and am working on a branch now.

[jfsiii]

My branch is at https://github.com/jfsiii/kibana/tree/73997-crash-on-setup-without-registry

WIP. Just logging to help anyone else who's interested. Not sure why the await setupIngestManager doesn't fail.

https://github.com/jfsiii/kibana/blob/358edb205f0dc96331869ff17f3bdcc2a760c2b6/x-pack/plugins/ingest_manager/server/routes/setup/handlers.ts#L89

[ruflin]

@jfsiii Thanks for investigating. We should also try to get this into 7.9.0 or 7.9.1 to make sure in case a 7.9 registry is not available one day, KB does not crash.

[jfsiii]

I found a fix jfsiii@39f6320 and will now clean up the console.logs and add test(s). I might close this and create a new PR.

a few cURL calls to POST /setup which fail w/500 502 as expected

± ~/work/kibana [73997-crash-on-setup-without-registry @ 39f6320] ✓ ‣ curl -X POST 'http://localhost:5601/api/ingest_manager/setup' -H 'kbn-xsrf: <string>' --user elastic:changeme
{"statusCode":502,"error":"Bad Gateway","message":"Error connecting to package registry at http://localhost:9999/search?package=system&internal=true&experimental=true: request to http://localhost:9999/search?package=system&internal=true&experimental=true failed, reason: connect ECONNREFUSED 127.0.0.1:9999"}^[[A
± ~/work/kibana [73997-crash-on-setup-without-registry @ 39f6320] ✓ ‣ curl -X POST 'http://localhost:5601/api/ingest_manager/setup' -H 'kbn-xsrf: <string>' --user elastic:changeme
{"statusCode":502,"error":"Bad Gateway","message":"Error connecting to package registry at http://localhost:9999/search?package=system&internal=true&experimental=true: request to http://localhost:9999/search?package=system&internal=true&experimental=true failed, reason: connect ECONNREFUSED 127.0.0.1:9999"}
± ~/work/kibana [73997-crash-on-setup-without-registry @ 39f6320] ✓ ‣ curl -X POST 'http://localhost:5601/api/ingest_manager/setup' -H 'kbn-xsrf: <string>' --user elastic:changeme
{"statusCode":502,"error":"Bad Gateway","message":"Error connecting to package registry at http://localhost:9999/search?package=system&internal=true&experimental=true: request to http://localhost:9999/search?package=system&internal=true&experimental=true failed, reason: connect ECONNREFUSED 127.0.0.1:9999"}
± ~/work/kibana [73997-crash-on-setup-without-registry @ 39f6320] ✓ ‣

the logging from those calls

handler try await setupIngestManager
setupIngestManager
handler catch { RegistryError: Error connecting to package registry at http://localhost:9999/search?package=system&internal=true&experimental=true: request to http://localhost:9999/search?package=system&internal=true&experimental=true failed, reason: connect ECONNREFUSED 127.0.0.1:9999
    at getResponse (/Users/jfsiii/work/kibana/x-pack/plugins/ingest_manager/server/services/epm/registry/requests.ts:26:11)
    at process._tickCallback (internal/process/next_tick.js:68:7) name: 'RegistryError' }
server    log   [15:06:22.313] [error][ingestManager][plugins] Error connecting to package registry at http://localhost:9999/search?package=system&internal=true&experimental=true: request to http://localhost:9999/search?package=system&internal=true&experimental=true failed, reason: connect ECONNREFUSED 127.0.0.1:9999
exit A
handler try await setupIngestManager
setupIngestManager
handler catch { RegistryError: Error connecting to package registry at http://localhost:9999/search?package=system&internal=true&experimental=true: request to http://localhost:9999/search?package=system&internal=true&experimental=true failed, reason: connect ECONNREFUSED 127.0.0.1:9999
    at getResponse (/Users/jfsiii/work/kibana/x-pack/plugins/ingest_manager/server/services/epm/registry/requests.ts:26:11)
    at process._tickCallback (internal/process/next_tick.js:68:7) name: 'RegistryError' }
server    log   [15:06:23.531] [error][ingestManager][plugins] Error connecting to package registry at http://localhost:9999/search?package=system&internal=true&experimental=true: request to http://localhost:9999/search?package=system&internal=true&experimental=true failed, reason: connect ECONNREFUSED 127.0.0.1:9999
exit A
handler try await setupIngestManager
setupIngestManager
handler catch { RegistryError: Error connecting to package registry at http://localhost:9999/search?package=system&internal=true&experimental=true: request to http://localhost:9999/search?package=system&internal=true&experimental=true failed, reason: connect ECONNREFUSED 127.0.0.1:9999
    at getResponse (/Users/jfsiii/work/kibana/x-pack/plugins/ingest_manager/server/services/epm/registry/requests.ts:26:11)
    at process._tickCallback (internal/process/next_tick.js:68:7) name: 'RegistryError' }
server    log   [15:06:24.594] [error][ingestManager][plugins] Error connecting to package registry at http://localhost:9999/search?package=system&internal=true&experimental=true: request to http://localhost:9999/search?package=system&internal=true&experimental=true failed, reason: connect ECONNREFUSED 127.0.0.1:9999
exit A